Why streaming beats waiting

A chat that blocks until the full reply is ready feels broken, even when the model is fast. The user sent a message. They are looking at a spinner. Meanwhile the model may already have the first sentence. You are hiding progress that already exists.

Streaming sends each token to the browser as the model produces it — the same pattern every consumer chat product uses. Total time can be similar. Perceived time is not. A 4-second wait with words at 400ms feels faster than a 3-second wait with a spinner, because the first number is “the product is working” and the second is “is it stuck?”

This only works because of Day 1: the model cannot skip to the end of the paragraph. Token two does not exist until token one is chosen. You might as well show them in that order.

On the server you call streamText instead of generateText. The AI SDK opens a stream from the provider (or Gateway), and you turn that into an HTTP response the client hook knows how to read.

On the client, useChat from @ai-sdk/react holds the messages array and appends tokens as they arrive. You do not parse Server-Sent Events yourself. You do not put the API key in this file.

The messages you send are the same array as yesterday. Streaming is delivery, not a different mental model. Tool calls, later, ride the same stream as extra parts on the assistant message.

Two clocks matter. Time-to-first-token (TTFT) is how long until the first word paints. That is the clock users feel. Time-to-last-token is how long until the model stops. That is closer to what you pay for, plus a bit of network.

A bigger model is often slower to first token and better at hard instructions. A smaller model is snappier and enough for short copy. Streaming does not make a slow model cheap. It makes a slow model honest: the user sees work happening.

If the first token takes two seconds, look at cold starts, the route’s region, and whether you stuffed a huge system prompt. Streaming will not hide a 2,000-token preamble.

Scripts, cron jobs, and tests should wait. A stream is harder to assert on. Classification and “extract this object” should wait (or use structured output) so you never render a half-field.

If the answer is one word, streaming buys you nothing. If the answer is a page, it is the difference between a product and a spinner.

You also cannot easily “undo” a streamed sentence the user already read. If you must validate the whole answer before showing it (moderation, a schema), wait, then render once. Streaming is a contract with the user: what they see is the answer, as it is born.