Public figures do not make proof
French open data is plentiful and free, which is what makes it misleading. What data journalism teaches an engineer, and the step almost nobody takes.
Data journalism and data engineering handle the same files and do not answer to the same judge. In a company a table is signed off by whoever commissioned it. In a newspaper it is attacked by the reader who disagrees. The second test is the harsher one, and that is why it is worth looking at from inside the first.
In France most of it is open and documented: the interministerial portal, the land registry, the address base, the geoportal, and datasets published directly by operators such as the railways or the electricity distribution network. The difficulty is not access, it is joining: two public files almost never share a granularity or a cut-off date, and that is where the errors get made that nobody sees afterwards.
- data.gouv.fr
- cadastre.data.gouv.fr
- adresse.data.gouv.fr
- geoportail.gouv.fr
- data.sncf.com
- Banque de France · Webstat
When the figure is not published you have to read it where it is displayed. That is a confession rather than a technique: extraction from a page reflects today’s layout and breaks at the first change. It is defensible for a one-off investigation, never for a system meant to hold, and confusing the two is the surest way to ship a pipeline that will fall over on a Tuesday.
The tooling I kept from that period is the tooling that stays small: a dataset published as a database you can query without a server, or a notebook that runs in the browser. In both cases the piece and the data travel together, and the reader can redo the calculation. That is exactly what a corporate pipeline almost never allows, and one of the few things it would gain by copying.
In a research paper a figure illustrates a result established elsewhere. In a news piece the figure is the result. That changes everything: the choice of scale, of origin, of palette becomes an editorial choice, and a continuous palette applied to categories lies without being heard. The tools that produce these figures are excellent; the responsibility remains undivided.
My 2020 notes were half a directory of links, and I am not republishing them as such: a directory’s freshness is what expires soonest. What has not aged is the order of operations: find, join, verify, publish. The one step I see systematically skipped in companies is the third, and it is the only one that separates a table from proof.