Super-NaturalInstructions: A Benchmark for Generalization to Unseen Instructions in Task-Oriented Language Understanding

This paper introduces a benchmark of 1,616 diverse NLP tasks and expert-written instructions to test models' ability to generalize to unseen instructions. It also introduces Tk-Instruct, a transformer model trained to follow a variety of in-context instructions, which outperforms existing instruction-following models on the benchmark.

This benchmark and model can help improve task-oriented language understanding for businesses that rely on NLP models for tasks such as classification, extraction, infilling, sequence tagging, text rewriting, and text composition. By focusing on diversity rather than volume in fine-tuning models, businesses may achieve better cross-task generalization.