Speeding Up LLMs with Speculative Decoding investigates a novel draft verification method that identifies the optimal moment and tokens for speculative decoding in LLM inference.

• Proposed algorithm based on optimal transport problem for block-level decoding.
• Empirical evaluation across various tasks and datasets.
• Substantial wall-clock speedup gains observed over token-level verification.

This approach to speculative decoding exemplifies how strategic algorithmic refinements can significantly improve the performance and efficiency of LLMs in practical applications.