The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

• Authors: Shuming Ma, et al.
• Published: February 27, 2024
• Research Area: Machine Learning, Model Compression

By unveiling BitNet b1.58, we witness a revolutionary transition towards 1-bit Large Language Models (LLMs) where each parameter uses only ternary values {-1, 0, 1}. This ground-breaking formation aligns closely with full-precision Transformer LLMs in perplexity and end-task performance, unveiling a new cornerstone in the cost-effective AI domain.

• Establishes a novel 1-bit LLM, mirroring full-precision models in efficacy.
• Charts a pathway for scalable and cost-efficient AI modeling.
• Paves the way for custom hardware design optimized for 1-bit LLMs.

BitNet b1.58’s balancing act between top-tier performance and economic efficiency heralds a new era for machine learning models, making AI more accessible and sustainable. This work spotlights the potential to reshape future research and development, with implications stretching across various AI and machine learning applications.