Latency

Speculative decoding is one of the most important tricks behind fast LLM serving. The basic idea is simple: Use a small fast model to guess several future tokens, then use the original large model to verify those guesses in one pass. If the guesses are right, the large model moves forward by multiple tokens instead of one. If a guess is wrong, the large model corrects it and the system continues. ...

When I evaluate an LLM system, one of the first latency metrics I look at is TTFT, or time to first token. This metric answers a simple question: After a user sends a request, how long does it take before the first output token appears? That sounds narrow, but it matters a lot. Users usually forgive a response that streams steadily after it starts. What feels bad is the dead time before anything appears on screen. ...