The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

1-bit Large Language Models are revolutionizing the AI realm, as demonstrated by the recent research on BitNet. The paper presents BitNet b1.58, an innovative model in which every parameter is ternary {-1, 0, 1}, yet matches the performance of its full-precision counterparts. This paradigm shift is not only a triumph in terms of computing efficiency but also sets a new benchmark for high-performing, cost-effective future LLMs.

Key points from the study include:

• BitNet b1.58 successfully achieves the same level of perplexity and end-task performance as full-precision Transformer LLMs.
• It offers substantial benefits in latency, memory, throughput, and energy consumption.
• The paper delineates a new scaling law and a trailblazing recipe for future LLMs.

The introduction of 1.58-bit LLMs like BitNet b1.58 is a game-changer, heralding a new computational ethos and paving the way for specialized hardware optimized for cost-effective LLMs. The environmental and economic impact of this advancement could be vast and enduring.