AttentionStore is a new attention mechanism that presents a cost-effective solution for serving large language models (LLMs) engaged in multi-turn conversations. The system’s key innovation lies in the ability to reuse key-value (KV) caches, thereby significantly reducing serving costs attributed to historical token computations.

• Introduces hierarchical KV caching for all requests using memory/storage mediums.
• Implements layer-wise pre-loading and asynchronous saving to minimize access overheads during GPU computations.
• Features scheduler-aware fetching/eviction to optimize cache placement based on job scheduler hints.
• Employs a positional encoding decoupling strategy to maintain KV cache validity despite context window overflow.
• Demonstrates improvements such as an 88% reduction in time to the first token (TTFT) and an 8.2x increase in prompt prefilling throughput.

AttentionStore’s approach to efficient LLM serving has the potential to revolutionize the industry, particularly in the context of cloud-based applications and conversational AI platforms. The ability to parlay saved computations into cost reductions while maintaining high throughputs suggests promising applications in both enterprise and consumer environments. As LLMs continue to scale, AttentionStore provides a viable path toward sustainable and efficient AI communication systems.