Cost, latency & streaming

Token budgets and caching

Every LLM call costs money proportional to tokens in and out, and every extra token of prompt (a long system message, a big chunk of booking history) is a recurring cost paid on every single call. Trim prompts to what the task actually needs. If the exact same prompt is likely to repeat — a canned FAQ answer, a common reschedule pattern — cache the response keyed on the prompt, the same instinct as caching a database query from module 34, so you pay for the generation once instead of every time.

Route cheap-model-first: not every question needs your most capable (and most expensive) model. A simple FAQ lookup can go to a small, fast, cheap model; only escalate to a bigger model when the small one is unsure or the task is genuinely harder. Most support traffic is simple.