Skip to main content
Reading viewAll insights →
BLOG6 min read

Self-Supervised Pretraining on Teleseismic Waveforms

Self-Supervised Pretraining on Teleseismic Waveforms
Tannistha Maitiby Tannistha MaitiHead of Engineering · 16 Nov 2026
Share

Seismology is bottlenecked by labels and drowning in waveforms, which is the setting masked autoencoding was built for. The part worth being careful about is how you pick the pretraining run, because reconstruction error and downstream transfer rank the same six candidates in opposite orders, and the metric that is easy to compute selects the weakest encoder on the board.

Supervised deep learning in seismology is bottlenecked by labels. Phase picks, event classifications and inverted structure are scarce and expensive; raw waveforms are abundant to the point of embarrassment, with modern networks recording petabytes of traces whose overwhelming majority carry no label at all.

Self-supervised learning turns that asymmetry into a training signal by inventing a pretext task whose answer is already inside the data. Masked-language modelling predicts hidden tokens [2]; the masked autoencoder reconstructs hidden image patches [1]. Both learn representations that transfer to labelled tasks with a fraction of the labels, and both port to teleseismic wavefields with some care.

The pretext task, and the adaptations that matter

A seismogram is divided into non-overlapping time patches. A large random fraction is masked, a Transformer encoder [6] ingests only the visible patches, a lightweight decoder reconstructs the full trace, and the loss is computed only on the masked samples [1]. Because the encoder never sees the masked positions, it has to infer them from wavefield structure.

Three adaptations do the work of making this seismological rather than generic. Masking whole time spans rather than scattered samples prevents the model from solving the task by trivial interpolation. Multi-component inputs let it learn the polarisation relationships that carry converted-phase information. And normalisation has to preserve relative amplitude, because amplitude ratios are structure, not scale.

The pretrained encoder is then fine-tuned, or linearly probed, with a small task head. Scarce labels are spent on adaptation rather than on learning wavefield structure from nothing, which is the whole point of the exercise.

The two metrics disagree, and one of them is the easy one

Cost of selecting on the pretext loss
The loss rewards the slender end of the hull. Transfer stands tallest at the other.
9.9accuracy points given up at 500 labels
A hull drawn by its plan and its elevation. Six frames run from the 15 percent mask ratio at the bow to 90 percent at the stern. The width of each frame is its reconstruction error, from 0.08 at the bow to 0.643 at the stern, so the bow is the slender end. The height of each frame is its downstream accuracy at 500 labels, and the tallest is the 75 percent frame at 72.4%. The waterline stands at 59%, the accuracy of a random initialisation on the same labels, and the part of the hull above it is the pretraining dividend of 13.4 points. Selecting on the pretext loss picks the 15 percent bow, which stands 9.9 accuracy points lower than the 75 percent crown. The floor shows the plan and the back wall shows the sheer line for all five label budgets.
Inspected frame
75% masking
Reconstruction error
0.53
Downstream accuracy
72.4%
Above random init
13.4 pts
Look at the hull fromthe plan or the sideor drag the figure to turn it
below the waterline: what the labels alone buyabove it: the pretraining dividendthe run the pretext loss picksthe run that transfers bestplan on the floor: width is reconstruction error
Height is accuracy measured up from the keel at zero; the rules on the wall stand at 20, 40, 60 and 80 percent. The skin between two frames is straight because the model has six runs, not a continuum. A demonstration, not a pretraining run.

At 500 labels this encoder reaches 72.4% against 59% from a random initialisation. Reconstruction error is lowest at 15% masking and transfer is best at 75%, so selecting the run on the pretext loss gives up 9.9 accuracy points. The gap between the two metrics is not noise: they rank the same six runs in opposite orders.

The six pretraining runs, ranked by each metric, at 500 downstream labels.
mask ratioreconstruction errordownstream accuracyselected by
15%0.0862.5%the pretext loss
30%0.19366.8%
45%0.30569.9%
60%0.41871.8%
75% (shown)0.5372.4%transfer
90%0.64371.8%
Downstream accuracy for a masked-waveform encoder, over the mask ratio it was pretrained at and the labels available afterwards, drawn as a hull. Six frames run from 15 percent masking at the bow to 90 percent at the stern. A frame's width on the plan is its reconstruction error, which rises monotonically toward the stern, so a team selecting its run on the pretext loss takes the slender bow, which is the lowest frame at every label budget. A frame's height is its downstream accuracy, and it peaks at 75 percent masking at every label budget here. Choosing on the pretext loss instead gives up 21.5 accuracy points at 50 labels and 2.7 at 5,000. The waterline is what a random initialisation reaches on the same labels, so the freeboard above it is the pretraining dividend. A demonstration of the behaviour our ES-3409 note describes, not a pretraining run.

Reconstruction error rises monotonically with the mask ratio. That is not a surprise: hiding more of the trace makes the pretext task harder, always, and no amount of representation quality reverses it.

Downstream transfer does something else. At a low mask ratio the task is solvable by local smoothness, so the encoder learns interpolation, which is not a representation of anything seismological. Push the ratio into the hard regime and the task can only be solved by learning where phases arrive, how coda decays, how P and S wavefields differ. That is the representation that transfers, and it peaks around 75%, mirroring the high mask ratios that make vision masked autoencoders effective [1].

Put those two facts together and the consequence is sharp. A team that selects its pretraining run on the pretext loss picks 15% masking, which is the narrowest frame of the hull and the lowest one at every label budget. Selecting on transfer picks 75% at every label budget on the board. The metrics do not merely differ in value; they rank the same six runs in opposite orders, and the cost of choosing on the easy one is 21.5 accuracy points at 50 labels, still 9.9 at 500, and 2.7 at 5,000.

This is worth distinguishing from an argument we published on 9 September, which was about a vendor's accuracy scalar being a marginal statistic over an estimator's output. That one is about reading a number correctly. This one is about model selection: the pretext loss is not a badly-read proxy for downstream utility, it is not a proxy for it at all, and a pipeline that early-stops or checkpoint-selects on reconstruction is optimising a quantity nobody downstream will ever consume.

The second thing the block shows is quieter. Slide the label budget back and the whole hull rises while the waterline closes on the crown and the pretraining dividend narrows: 29 points of advantage at 50 labels, under 4 at 5,000. Pretraining is worth most exactly where labels are scarce, which is the regime that motivated it, and its case gets weaker as a labelled catalogue grows.

What this does not fix

Five limitations deserve naming before anyone budgets a pretraining run.

Pretraining data bias propagates. An encoder trained on one network's instruments and noise may transfer poorly to another, and domain shift here is not hypothetical. Reconstruction quality is not the goal and correlates only loosely with downstream utility, which is the whole subject above. The pretext task rewards learning any predictable structure, and instrument and site artefacts are extremely predictable, so a model can spend capacity on the recording system rather than on the earth.

Synthetic pretraining transfers to real data only as well as the synthetics are realistic. And the compute is substantial, justified only when several downstream heads reuse the same encoder, which is the real economic test: one pretraining run, many task heads, or it does not pay.

None of these is disqualifying. They are the conditions under which the method is worth running, and they are all observable before the budget is spent.

Key takeaways

  1. Waveforms are abundant and labels are scarce, which is the asymmetry masked autoencoding converts into a training signal.
  2. Span masking, multi-component inputs and amplitude-preserving normalisation are what make the generic recipe seismological.
  3. Reconstruction error rises monotonically with the mask ratio while transfer peaks in the hard regime near 75%, so the two metrics rank the same runs in opposite orders.
  4. Selecting a pretraining run on the pretext loss picks the weakest encoder on the board: 21.5 accuracy points given up at 50 labels, 2.7 at 5,000.
  5. The pretraining dividend narrows as labels accumulate, so the method is worth most in exactly the low-label regime that motivates it.
  6. The pretext task rewards any predictable structure, including instrument and site artefacts, so evaluate on transfer and never on reconstruction.

References

  1. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. arXiv:2111.06377
  2. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arXiv:1810.04805
  3. T. Chen, S. Kornblith, M. Norouzi, G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. arXiv:2002.05709
  4. W. Zhu, G. C. Beroza. PhaseNet: a deep-neural-network-based seismic arrival-time picking method. Geophysical Journal International, 2019. arXiv:1803.03211
  5. S. M. Mousavi, W. L. Ellsworth, W. Zhu, L. Y. Chuang, G. C. Beroza. Earthquake transformer. Nature Communications, 2020. doi:10.1038/s41467-020-17591-w
  6. A. Vaswani, N. Shazeer, N. Parmar, et al. Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762
  7. The companion argument about reading an accuracy scalar correctly: The error that cancels.
Tannistha Maiti
Tannistha Maiti

Head of Engineering

More from EarthScan

Related research

All insights →
The Synthetic-Data Flywheel Is the Moat
Insight

The Synthetic-Data Flywheel Is the Moat

32 Synthetic Models Beat 1,000 Stations
Insight

32 Synthetic Models Beat 1,000 Stations

Shear-Wave Splitting: Labels You Can Trust
Insight

Shear-Wave Splitting: Labels You Can Trust

Stay ahead

EarthScan insights, in your inbox.

Field-tested research on subsurface and energy-transition AI. About twice a month. No noise.

We use your email only for this newsletter. Unsubscribe anytime Privacy.