“The model is the easy part. Closing the gap between a notebook that works and a system that holds up at production cadence — that is where ML projects in oil & gas go to die.
”
The oil and gas industry generated approximately $3.3 trillion in annual revenue in 2019, making it one of the largest enterprises in the world. Across upstream, midstream, and downstream operations, the industry produces enormous quantities of data continuously — and depends heavily on sophisticated technology to extract operational insight, prevent equipment malfunctioning, and improve efficiency.
Annual oil & gas revenue worldwide (2019)
Operational data generated per major operator
Of enterprise ML projects never reach production — McKinsey, Gartner, IDC all converge here
In recent years, oil and gas has positioned itself as a technology trend-setter — automating significant portions of the value chain and increasingly leaning on artificial intelligence (AI) to make decisions that were once the sole domain of senior engineers.
But there's a gap between building an ML model and running an ML model in production. Closing that gap is what MLOpsStandardisation and streamlining of the machine-learning lifecycle. Borrows DevOps's automation patterns and adapts them for the data-and-model layer (where unit tests don't transfer cleanly). is for. And in oil and gas more than most industries, that gap is where ML projects go to die.
What the big players are doing
Royal Dutch Shell continuously looks for new ways to keep up with growing demand for dwindling resources. Their software development organisation runs on:
- Cloud collaboration — multi-team development across geographies
- Software-as-a-service — internal SaaS replacing legacy desktop tools
- Faster-moving environments — release cadences measured in days, not quarters
- DevOps — for the software side; the ML side is following
The question isn't whether oil and gas will adopt these patterns. It's how quickly the rest of the industry follows the leaders.
The answer to "what next" — once you've gotten DevOps reasonably solid — is a robust and scalable MLOps pipeline.
What MLOps actually is
MLOps is the standardisation and streamlining of machine-learning lifecycle management. It automates the lifecycle from data preparation through model building, production deployment, monitoring, and retraining.
Three reasons managing ML at scale is harder than managing software at scale:
a) Data is constantly changing, and business requirements shift on top of that. A model trained on Q1 data can underperform by Q3 — but only if you measure it.
b) Results need to be continually relayed back to the business, so model behaviour on production data stays aligned with the business outcomes the model was supposed to drive. The model's accuracy ≠ the business value the model produces.
c) The ML lifecycle crosses three teams: business, data science, and IT. None of them use the same tools or share a baseline vocabulary. The handoff between them is where things break.
MLOps borrows heavily from DevOps — both centre around robust automation and trust between teams. The mechanics differ (you don't unit-test a model the same way you unit-test a function), but the cultural pattern transfers cleanly.
The closed-loop pipeline
A working MLOps pipeline for an oil and gas application — say, seismic-section interpretation — is a closed loop, not a linear hand-off. Six stages, with retraining feeding back to collection.
- 1
Data collection
Raw SEG-Y, LAS, sensor telemetry. ETL pipelines convert to numpy / parquet / arrow.
- 2
Data aggregation
Provenance tracking, access control, schema enforcement. The audit-trail layer.
- 3
Model training
Reproducibility, hyperparameter tracking, distributed compute orchestration.
- 4
Deployment
REST API for online, batch for archives, drift monitoring, canary rollouts.
- 5
Monitoring
Production drift detection. Business KPI tracking. Alerts when accuracy slips.
- 6
Retraining
Automatic loop back to training when drift breaches threshold. Closed loop.
A look at each stage in turn — with the specifics that matter in oil and gas.
1. Data collection
Seismic sections are noisy, semi-structured datasets of seismic events collected over a specific area. The raw time-section data lives in SEG-Y formatThe 1970s-era industry-standard file format for seismic survey data. Arcane by modern standards; every upstream ML pipeline starts with a SEG-Y → numpy converter. — the industry standard since the 1970s, and one of the more arcane file formats in any technical domain.
Detailed ETL pipelines are needed to convert SEG-Y to numpy arrays (or any modern in-memory ML format). A reasonable AWS-centric reference for ETL on geospatial data: chasing earthquakes — Redshift ETL on unstructured data.
2. Data aggregation
This step combines data engineering with data science. Goals: data quality control, security, and integrity. In a regulated industry, this layer carries the audit trail — without it, the rest of the stack is non-compliant.
Specifics that matter in oil and gas:
- Provenance tracking — every datapoint tagged with survey ID, vintage, processing parameters
- Access control — joint-venture data has stricter sharing rules than wholly-owned-asset data
- Schema enforcement — operators have learned the hard way that a silent column-rename between two surveys breaks every downstream model
3. Model training
Deep learning is increasingly the framework of choice — neural networks deal natively with the spatial structure that seismic, log, and core data all have. Training itself is the easy part. The hard parts:
- Reproducibility — given the same data and the same code, can you regenerate the same model? In practice, GPU non-determinism, library version drift, and random seeds all conspire to break this.
- Hyperparameter management — tracking which values were tried, which won, and why.
- Compute orchestration — distributed training, on-prem GPUs, cloud bursts when on-prem is full. All this needs to be invisible to the data scientist.
4. Deployment
Once the model is trained, you evaluate the results — and if they pass the bar, push to production. This is where most oil and gas ML projects historically stalled.
Production deployment for ML in oil and gas typically means:
- REST API for online inference (interactive interpretation tools, real-time-ish dashboards)
- Batch scoring for offline inference (reprocessing historical archives)
- Drift monitoringProduction data slowly diverging from training data — formation conditions change, instruments age, processing pipelines evolve. A model trained on Q1 data underperforms by Q3, but only if you measure it. — the data your model sees in production won't match the data it was trained on; you need to know when the gap becomes problematic
- Canary deploymentsRoll a new model to 1–5% of traffic first, compare against the production baseline, only ramp up if metrics hold. Protects against silent regressions during ship. + A/B harnesses — so you can ship model updates without blowing up the existing pipeline
What investing in MLOps actually buys you
Three concrete outcomes:
- Time-to-production drops from quarters to weeks. A new model goes from "the data scientist has a notebook that works" to "running on the operator's well files" in days, not months.
- Cross-survey reuse becomes possible. A model trained on field A can be deployed to field B without rebuilding the whole pipeline. Capability compounds.
- Compliance comes for free. When the audit trail is built into the pipeline rather than retrofitted, regulatory submissions stop being projects in their own right.
Glossary
- Canary deployment
- Roll a new model to 1–5% of traffic first, compare against the production baseline, only ramp up if metrics hold. Protects against silent regressions during ship.
- Drift
- Production data slowly diverging from training data — formation conditions change, instruments age, processing pipelines evolve. A model trained on Q1 data underperforms by Q3, but only if you measure it.
- MLOps
- Standardisation and streamlining of the machine-learning lifecycle. Borrows DevOps's automation patterns and adapts them for the data-and-model layer (where unit tests don't transfer cleanly).
- SEG-Y
- The 1970s-era industry-standard file format for seismic survey data. Arcane by modern standards; every upstream ML pipeline starts with a SEG-Y → numpy converter.
Where to start
If you're an operator looking at this and wondering where to start, the answer is boring: don't start with the model. Start with the data pipeline. A trustworthy data layer is what unlocks the rest. Without it, every model you ship is a one-off.
The model is the easy part — once your data is in shape.
