CoLT5: Faster Long-Range Transformers with Conditional Computation

Proposes CoLT5, a long-input Transformer model that employs conditional computation to devote more resources to important tokens in both feedforward and attention layers, achieving stronger performance than LongT5 with much faster training and inference.

Use CoLT5 to process long documents with better performance and faster training and inference, especially in natural language processing tasks. CoLT5 can also make use of extremely long inputs with strong gains up to 64k input length.