LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Develops a procedure for Int8 matmul for feed-forward and attention projection layers in transformers, which halves the memory for inference while retaining fp32 performance.

Enables businesses to use large language models without requiring significant GPU memory for inference, making such models more accessible and improving performance.