← All projects

Engineering field note / Documented build

A pipeline can run successfully and still publish the wrong data.

With chile-hub, the question was never how many datasets we could publish. It was how to stop plausible-looking files from introducing errors nobody noticed.

Carlos Ortega González · Built and maintains chile-hub Prepared October 2026

Public sourceExecutable contractsDocumented limitations

01 / The failure mode

The disappearing zero

In Chile, Iquique's territorial identifier is 01101. Coerce it to an integer and it becomes 1101. Both look legitimate in a spreadsheet, but they are no longer the same join key.

Correct identifier01101
After integer coercion1101
raw = "01101"
wrong = str(int(raw))  # "1101"
assert raw != wrong

A minimal reproduction of a failure mode, not a production incident report.

The result: a join can drop valid rows without throwing an error. The final report looks credible but is incomplete.

02 / Architecture decision

Validation decides what gets published

The workflow separates extraction, building, verification and publication. Data contracts, provenance and release rules are operational controls, not decorative README claims.

  1. 01

    Extract

    Acquire source data and preserve provenance.

  2. 02

    Build

    Normalize formats and keys without losing information.

  3. 03

    Verify

    Check schema, coverage and dataset-level invariants.

  4. 04

    Publish

    Release only datasets that meet the applicable policy.

03 / Inspectable decisions

Three choices that make the system auditable

Fixed-width strings for identifiers

CUT codes remain strings so leading zeros survive extraction, transformation and joins.

Read the identifier ADR ↗

Executable data contracts

Each dataset declares verifiable constraints. An upstream format change must not quietly become trusted output.

Inspect the contracts ↗

Publication maturity lanes

Fragile sources and unresolved licensing do not automatically qualify as stable, publishable data.

Read inclusion criteria ↗

04 / What did not magically disappear

Reliability sometimes means saying “not yet”

External sources change, become unavailable and sometimes block automation. chile-hub documents those constraints and distinguishes stable datasets from candidates. A declared snapshot is not the same as newly fetched data.

This case documents inspectable design choices and implementation, not an independent audit or a guarantee that errors are impossible. PyPI downloads are not presented as unique users or installations.

05 / Inspect the work

Don't take this page on faith

Every engineering claim is linked to public source material. Follow the evidence and decide whether the design holds up.

01Source and historyOpen, maintained repository02Release safetyPipeline verification implementation03Engineering recordOriginal technical case study and tradeoffs04Persistent citationZenodo project DOI record

06 / Beyond open data

What does this mean for your automation?

Recurring reports, permit collectors and scheduled folder syncs need similar discipline: validated inputs, deterministic outputs, safe reruns and visible failures. The systems differ; the engineering principles travel.

Explore Python automation services →