Minisymposium Presentation

As Large Language Models (LLMs) grow in size and capability, their high computational cost poses a major challenge for real-time applications, making efficient inference a critical research problem. Speculative Decoding (SD) has emerged as a promising technique to accelerate LLM inference by leveraging a smaller draft model to generate candidate tokens, which are then verified in parallel by a larger target model to ensure statistical consistency. However, the need for frequent verification calls to the target LLM limits the potential speedup of SD. We propose SPRINTER, which utilizes a low-complexity verifier trained to predict if tokens generated by the draft model would be accepted by the target LLM. By performing approximate sequential verification, SPRINTER eliminates the need for constant verification by the target LLM and is only invoked when a token is deemed unacceptable. This significantly reduces the number of calls to the larger model, enabling further acceleration. We present a theoretical analysis of SPRINTER, examining the statistical properties of the generated tokens and the expected reduction in latency as a function of the verifier. Our evaluations on multiple datasets and model pairs demonstrate that approximate verification can maintain high-quality generation while achieving even greater speedups.