Every applied-research program that trains models on an operator's confidential wells has the same throat at its narrowest point: how many wells actually show up, and how fast. We have written before about that bottleneck directly, where classification error tracked the arrival of labelled wells far more closely than any architecture change (The Data Bottleneck Is the Real Bottleneck in Subsurface AI). This piece is about the other half of the response, the part that never appeared in a single conference deck or journal submission: while the confidential wells trickled in, we were quietly harvesting a national public archive to keep the pipeline moving.
The open-data pillar most readers of our work already know is FORCE 2020, the Norwegian Continental Shelf competition set we used to rehearse well-to-well correlation on 118 public wells before touching client logs (Well-to-Well at Field Scale). FORCE gave us curated, labelled, benchmark-grade logs. What it did not give us was practice at the messy front of the pipeline: fetching raw records from a live repository, one well at a time, across the full spread of record types a real subsurface database holds. For that we went to a national archive and wrote a crawler.
Why a national repository, and not just a competition set
A curated competition dataset is a finished product. Someone has already decided which wells belong, cleaned the curves, aligned the depths, and handed you a tidy archive. That is what makes it useful for benchmarking and a poor rehearsal for the operational reality of client data, which arrives raw, uneven, and in fragments.
A national well archive sits at the other end. Public repositories such as the Dutch NLOG service publish decades of onshore and offshore subsurface records through a live mapviewer API: per-well details, core data, deviation surveys, lithostratigraphy, log documents, production history. It is a working database, with the same shape and the same irregularities you will meet when a client finally opens their own vault. Learning to pull from it is learning to pull from theirs.
That is the first thing a national repository buys you: the pipeline rehearsal. The second is a pool of pretraining candidates, real logs with real geological structure a model can see before it ever meets a confidential well. The third is correlation practice on genuine stratigraphy rather than a synthetic proxy. None of the three requires an NDA, and all three are spent before the scarce, protected data is.
Eight endpoint families, one resumable loop
The crawler is small on purpose: 54 lines of Python against the archive's REST API. It reads a pre-fetched list of wells, each identified by a stable borehole id, and for every well it walks eight endpoint families in turn: basic details, core runs, deviation surveys, documents, lithostratigraphy interpretations, log documents, production figures, and sample measurements. Each family is a separate JSON call. Eight calls per well, one well after another, saved to disk as it goes.
Two design choices carry the whole thing. The first is that the loop is keyed on the borehole id, not on a row counter or a URL guess, so a well is either fully requested or skipped as a unit. The second is that it is resumable: before it fetches a well it checks whether that well is already on disk, and if it is, it moves on. Combined, those two turn an unreliable, hours-long harvest against a public server into something you can start, interrupt, and restart without penalty. Nothing is re-downloaded. The frontier just picks up where it stopped.
The instrument above is the argument in one frame. The eight endpoint families fan into a single crawl loop on the left; the sweep across the well list runs on the right. The orange checkpoint is the piece that matters. With resume on, everything left of it is already saved and gets skipped on the next run, which is what lets an unattended crawl against a rate-limited government API actually finish. A companion downloader, 29 lines, does the same job for the bulky log-document files, pulling each from the archive's logdocument/{id} endpoint into a per-well folder, and it follows the identical rule: on restart, skip what is already there.
One robustness detail carries more weight than its size. A public archive answers a request for a record type a given well does not have with an empty success, not an error. The crawler treats a 200 as content to save and a 204 as a valid "no record here for this well," and both advance the loop. Handle only the 200 and let the 204 fall through as an exception, and the crawl dies on the first well that lacks, say, a production figure, which is most of them. That empty-but-valid response is the single most common way a naive government-API scraper stalls overnight.
The etiquette of mining a government geo-API
Public does not mean unlimited, and a national repository is shared infrastructure. A few rules kept our crawl welcome.
Key on the archive's own stable id. Borehole ids do not change when a well is renamed or a record revised, so a run keyed on them is idempotent: re-running never duplicates and never re-fetches. Guessing sequential ids or scraping the HTML mapviewer instead of the JSON API is how you both miss records and hammer the server.
Make the crawl resumable from the first line, not as a later patch. The skip-if-saved check is four lines and it changes the operational character of the whole job. Without it, any interruption, a dropped connection or a nightly maintenance window, costs you the entire run. With it, the cost of an interruption is one well.
Fetch per well and per endpoint, in series, rather than firing parallel requests at a public host. The point is not raw speed; it is a harvest that completes without getting throttled. Eight ordered calls per well, moving on cleanly when a record is absent, reads to the server like a patient client rather than a flood.
Respect the licence and keep the provenance. Open for rehearsal and pretraining is not the same as open for redistribution, and the two data pillars must never touch. The confidential client wells lived only where the NDA required, and the open harvest lived separately, so that nothing from one crossed into the other.
What the open pillar actually changed
None of this replaced the client's wells. The models that mattered were trained and validated on the operator's confidential logs, because that is the reservoir the operator was paying us to understand. What the national archive changed was the order of spending. By the time a protected well arrived, the crawl-and-parse pipeline had already been exercised against a live repository, the record types were already understood, and the correlation machinery had already run on real stratigraphy. The scarce, NDA-bound data was spent last, on the questions only it could answer, instead of first, on questions a free national archive could rehearse.
That is the case for treating open subsurface data as the antidote to scarcity. Not because the public logs are as good as the client's, but because they let you get your mistakes out of the way before the meter starts.
Limitations
The crawler was built for one national archive's API and its specific endpoint layout; the eight-family structure and the id scheme are that repository's, and a different national service would need its own mapping even though the resumable, per-well pattern carries over. The well-list length shown in the instrument is illustrative, chosen to make the frontier sweep legible; the scripts read whatever the pre-fetched list holds. Open national data narrows the proprietary-data bottleneck for pipeline rehearsal, pretraining, and correlation practice, but it does not close it: the model that ships still has to be trained and validated on the client's own reservoir, and the accuracy that decision-makers act on comes from those confidential wells, not the public ones.
References
[1] NLOG, the Dutch Oil and Gas Portal (national subsurface data repository), TNO Geological Survey of the Netherlands. Public well and log records via the mapviewer REST API. https://www.nlog.nl/
[2] Bormann, P., Aursand, P., Dilib, F., Dischington, P., and Manral, S. FORCE 2020 Well Log and Lithofacies Dataset for Machine Learning Competition. Zenodo (2020). Record 4351156. https://doi.org/10.5281/zenodo.4351156
