Supervised deep learning in seismology is bottlenecked by labels. Phase picks, event classifications and inverted structure are scarce and expensive; raw waveforms are abundant to the point of embarrassment, with modern networks recording petabytes of traces whose overwhelming majority carry no label at all.
Self-supervised learning turns that asymmetry into a training signal by inventing a pretext task whose answer is already inside the data. Masked-language modelling predicts hidden tokens [2]; the masked autoencoder reconstructs hidden image patches [1]. Both learn representations that transfer to labelled tasks with a fraction of the labels, and both port to teleseismic wavefields with some care.
The pretext task, and the adaptations that matter
A seismogram is divided into non-overlapping time patches. A large random fraction is masked, a Transformer encoder [6] ingests only the visible patches, a lightweight decoder reconstructs the full trace, and the loss is computed only on the masked samples [1]. Because the encoder never sees the masked positions, it has to infer them from wavefield structure.
Three adaptations do the work of making this seismological rather than generic. Masking whole time spans rather than scattered samples prevents the model from solving the task by trivial interpolation. Multi-component inputs let it learn the polarisation relationships that carry converted-phase information. And normalisation has to preserve relative amplitude, because amplitude ratios are structure, not scale.
The pretrained encoder is then fine-tuned, or linearly probed, with a small task head. Scarce labels are spent on adaptation rather than on learning wavefield structure from nothing, which is the whole point of the exercise.
The two metrics disagree, and one of them is the easy one
- Inspected frame
- 75% masking
- Reconstruction error
- 0.53
- Downstream accuracy
- 72.4%
- Above random init
- 13.4 pts
At 500 labels this encoder reaches 72.4% against 59% from a random initialisation. Reconstruction error is lowest at 15% masking and transfer is best at 75%, so selecting the run on the pretext loss gives up 9.9 accuracy points. The gap between the two metrics is not noise: they rank the same six runs in opposite orders.
| mask ratio | reconstruction error | downstream accuracy | selected by |
|---|---|---|---|
| 15% | 0.08 | 62.5% | the pretext loss |
| 30% | 0.193 | 66.8% | |
| 45% | 0.305 | 69.9% | |
| 60% | 0.418 | 71.8% | |
| 75% (shown) | 0.53 | 72.4% | transfer |
| 90% | 0.643 | 71.8% |
Reconstruction error rises monotonically with the mask ratio. That is not a surprise: hiding more of the trace makes the pretext task harder, always, and no amount of representation quality reverses it.
Downstream transfer does something else. At a low mask ratio the task is solvable by local smoothness, so the encoder learns interpolation, which is not a representation of anything seismological. Push the ratio into the hard regime and the task can only be solved by learning where phases arrive, how coda decays, how P and S wavefields differ. That is the representation that transfers, and it peaks around 75%, mirroring the high mask ratios that make vision masked autoencoders effective [1].
Put those two facts together and the consequence is sharp. A team that selects its pretraining run on the pretext loss picks 15% masking, which is the narrowest frame of the hull and the lowest one at every label budget. Selecting on transfer picks 75% at every label budget on the board. The metrics do not merely differ in value; they rank the same six runs in opposite orders, and the cost of choosing on the easy one is 21.5 accuracy points at 50 labels, still 9.9 at 500, and 2.7 at 5,000.
This is worth distinguishing from an argument we published on 9 September, which was about a vendor's accuracy scalar being a marginal statistic over an estimator's output. That one is about reading a number correctly. This one is about model selection: the pretext loss is not a badly-read proxy for downstream utility, it is not a proxy for it at all, and a pipeline that early-stops or checkpoint-selects on reconstruction is optimising a quantity nobody downstream will ever consume.
The second thing the block shows is quieter. Slide the label budget back and the whole hull rises while the waterline closes on the crown and the pretraining dividend narrows: 29 points of advantage at 50 labels, under 4 at 5,000. Pretraining is worth most exactly where labels are scarce, which is the regime that motivated it, and its case gets weaker as a labelled catalogue grows.
What this does not fix
Five limitations deserve naming before anyone budgets a pretraining run.
Pretraining data bias propagates. An encoder trained on one network's instruments and noise may transfer poorly to another, and domain shift here is not hypothetical. Reconstruction quality is not the goal and correlates only loosely with downstream utility, which is the whole subject above. The pretext task rewards learning any predictable structure, and instrument and site artefacts are extremely predictable, so a model can spend capacity on the recording system rather than on the earth.
Synthetic pretraining transfers to real data only as well as the synthetics are realistic. And the compute is substantial, justified only when several downstream heads reuse the same encoder, which is the real economic test: one pretraining run, many task heads, or it does not pay.
None of these is disqualifying. They are the conditions under which the method is worth running, and they are all observable before the budget is spent.
Key takeaways
- Waveforms are abundant and labels are scarce, which is the asymmetry masked autoencoding converts into a training signal.
- Span masking, multi-component inputs and amplitude-preserving normalisation are what make the generic recipe seismological.
- Reconstruction error rises monotonically with the mask ratio while transfer peaks in the hard regime near 75%, so the two metrics rank the same runs in opposite orders.
- Selecting a pretraining run on the pretext loss picks the weakest encoder on the board: 21.5 accuracy points given up at 50 labels, 2.7 at 5,000.
- The pretraining dividend narrows as labels accumulate, so the method is worth most in exactly the low-label regime that motivates it.
- The pretext task rewards any predictable structure, including instrument and site artefacts, so evaluate on transfer and never on reconstruction.
References
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. arXiv:2111.06377
- J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arXiv:1810.04805
- T. Chen, S. Kornblith, M. Norouzi, G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. arXiv:2002.05709
- W. Zhu, G. C. Beroza. PhaseNet: a deep-neural-network-based seismic arrival-time picking method. Geophysical Journal International, 2019. arXiv:1803.03211
- S. M. Mousavi, W. L. Ellsworth, W. Zhu, L. Y. Chuang, G. C. Beroza. Earthquake transformer. Nature Communications, 2020. doi:10.1038/s41467-020-17591-w
- A. Vaswani, N. Shazeer, N. Parmar, et al. Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762
- The companion argument about reading an accuracy scalar correctly: The error that cancels.




