Insights into Pre-training via Simpler Synthetic Tasks

Pre-training a T5 on a synthetic dataset substantially outperforms over training from scratch and achieves up to 67% of the benefits of natural pre-training.

Using synthetic pre-training tasks can achieve significant gains in downstream tasks and can be simplified while still retaining much of its gains.