Inverse Scaling can become U-shaped
This paper examined inverse scaling tasks evaluated on models of up to 540B parameters, and found that only four out of the eleven tasks remain inverse scaling, while six out of the eleven tasks exhibit what they call "U-shaped scaling" - performance decreases up to a certain model size, and then increases again up to the largest model evaluated.
This research suggests that scaling up language models can improve performance by unlocking emergent abilities, and that the inverse scaling trend observed in previous studies may not hold for larger models. This can inform businesses in industries that rely on language models and NLP, such as chatbots, virtual assistants, and language translations, to invest in larger models to unlock better performance.
Crosslingual Generalization through Multitask Finetuning
This paper applies multitask prompted finetuning (MTF) to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. They found that finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus, and that training on machine-translated prompts can lead to better performance on human-written prompts in the respective languages.
This research can benefit businesses by suggesting that large multilingual language models can be finetuned on English tasks with English prompts to improve generalization to non-English languages. This can be useful for businesses that operate globally and need language models that can understand different languages. Additionally, training on machine-translated prompts can also lead to better performance on human-written prompts, which can save businesses time and resources in creating human-written prompts for each language they operate in.