Halfway through a three-year AI programme for a major operator in Oman, the compute bill stopped behaving. We had approved a mid-six-figure budget the previous December for a fractured-carbonate image-log project, work that ran on a private datacenter of experimentation servers, a data-processing box, an older stack of cards, and an NVIDIA A100. Then two things happened at once. Russia invaded Ukraine, and the price of the gas that powered the racks went vertical. And the supervised model phase, the one that trained the fracture and bedding detectors, turned out to need more than twice the compute we had scoped. In October 2022 we had to go back to the client and ask for more money on a programme that was already funded and already running.
That is the moment most engagements quietly rot. A vendor comes back asking for a top-up, the number is round, the justification is a paragraph, and the client's trust in the original estimate collapses along with the schedule. We did the opposite, and the reason it worked is the whole point of this piece: we did not ask for a number. We handed over a ledger.
The two things that broke at once
The original approved budget was 533,000 USD, signed the previous December for a 20-month scope. The additional request came to 104,300 USD, and the first discipline was that it split cleanly into two named buckets rather than one blended ask. Bucket one, 53,300 USD, covered extra AI infrastructure: the project had split into parallel supervised and unsupervised tracks (the unsupervised branch won for vugs and beddings, the supervised branch for fractures), which doubled the compute paths, and Phase 3 added an 80-plus-well well-to-well correlation workload we had to provision for. Bucket two, 51,000 USD, was an energy and utility contingency, roughly OMR 20,000 across twelve months, and it existed entirely because of the war.
The energy line was not a hedge. Gas prices were up 400% against pre-war levels, 600% since the project kicked off the previous December, and 1300% since September 2021, tracked against the traded European gas benchmark. Electricity was up 394%. DGX-class hosting on the open market had moved from 12,000 to 15,000 USD a month before the war to north of 20,000 USD a month after it. Every one of those figures went into the request as a sourced number with a baseline attached, because a contingency that says "energy is expensive now" is a plea, and a contingency that says "gas is up 400% versus pre-war on this named index" is an audit line.
The ledger, not the ask
The compute bucket is where the itemisation did the real work. Instead of asserting that the model phase had run over, we tabled the budgeted-against-actual GPU hours for every phase.
Phase 1 had been budgeted at 1200 hours and roughly 20,000 USD, and it landed on both. Phase 2 had also been budgeted at 1200 hours, and it actually consumed 2600 hours, 2.17 times the plan, for about 44,700 USD against the 20,000 budgeted, an overrun of 24,700 USD. Phase 3 had been scoped at 5800 hours and 100,000 USD, and the forecast was 7500 hours and 128,600 USD, 1.29 times over, an overrun of 28,600 USD. Those two overruns, 24,700 and 28,600, summed to 53,300 USD, and that was not a coincidence chosen after the fact. That sum was the infrastructure bucket. The client could read the top-line ask, follow the line down to the phase, and see the hours that produced it.
This is a different negotiating posture than the usual overrun conversation. When the number is defended by a table, the reviewer stops arguing about the number and starts checking the table, which is a far healthier conversation to be in. The question shifts from "why should we trust your estimate" to "is 2600 hours a fair count of what Phase 2 actually ran," and that second question has a factual answer we could produce from the run logs. A single ML run took six to eighteen hours; Phase 3 was going to push concurrent runs from a handful to somewhere near sixty to ninety at once. None of that is arguable in the abstract. All of it is checkable line by line.
The moat we had already built
The itemised overrun explained why the bill grew. It did not, on its own, explain why the client should believe the bill would not keep growing without limit as the war dragged on. That reassurance came from a piece of infrastructure planning that predated the crisis by two years.
We had bought GPU capacity before the pandemic and signed a multi-year hardware partnership in 2020, and we ran the workload in a private datacenter on a rate that was fixed before any of this happened. The rate card told the story: the industry baseline was 30 USD per GPU-hour, our standard managed rate was 24.60 USD per GPU-hour, and the rate this engagement actually paid was 17.22 USD per GPU-hour, a 30% proposal-phase discount locked in 2020. When the open market for DGX hosting jumped past 20,000 USD a month, our per-hour rate did not move, because it had been negotiated before the spike. The change request could therefore promise something a spot-priced vendor could not: the overrun was bounded, because the unit price was fixed while everyone else's floated up.
The A100 itself did the rest of the bounding. Rather than buy more cards to meet the parallel-run demand, we partitioned the one A100 into as many as 28 Multi-Instance GPU slices, so a single physical card presented as many isolated instances, and pushed sustained utilisation to 80-90%. That is the difference between "we need more hardware" and "we need to use the hardware we already priced more densely." The change request asked for contingency, not a capital expansion, and the MIG plan was the evidence.
Why a change request can build trust
The instinct when a project runs over is to absorb the overrun quietly or to bury it in a vague top-up, on the theory that visibility is dangerous. On this engagement the opposite held. The transparent request, with its itemised compute-hour table, its two named buckets, its sourced energy baselines, and its fixed-rate moat, was more trust-building than silence would have been. It showed the client that we knew, to the hour, where their money had gone and where it was going, and that we had built the cost defence into the infrastructure before the crisis rather than scrambling for it after.
The general lesson transfers past this reservoir and past this war. When you have to ask a client for more money mid-programme, the number is almost never the problem. The problem is whether the number is legible. An overrun defended by a table invites scrutiny and survives it. An overrun defended by a paragraph invites suspicion and rarely does. The 104,300 USD we asked for was large, but every dollar of it pointed at a line the client could check, and that is why the conversation ended in a signature rather than a standoff.
Limitations
The figures here are from a single confidential engagement in a fractured carbonate reservoir in Oman, and they reflect one operator's cost structure and one moment in the 2022 energy market; the percentage moves are tied to specific baselines (pre-war, project start, and September 2021) and should not be read as a general forecast. The compute-hour counts and dollar figures come from the engagement's own additional-budget request and its cost-comparison ledger, so they are internal accounting rather than an independently audited benchmark. The GPU-hour rate card reflects hardware and a vendor deal negotiated in 2020 under conditions specific to that period, and the concurrent-run and utilisation figures are the plan we scoped for Phase 3, not a post-hoc measurement of the whole programme.
