Efficient Training of Language Models to Fill in the Middle

Shows that autoregressive LMs can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end.

Training autoregressive language models with a large fraction of data transformed using fill-in-the-middle (FIM) data augmentation does not harm the original generative capability. FIM is a simple and efficient method that can be used to improve language models.