Architecting a production-grade AI SaaS with Next.js and FastAPI

Why split Next.js and FastAPI at all?

Next.js is excellent for the product surface — routing, auth, server components, the dashboard. But heavy AI work (long generations, batched embeddings, multimodal pre/post-processing) belongs in a service you can scale and deploy independently, and Python is where the AI ecosystem lives. FastAPI gives you typed request/response with Pydantic and first-class async streaming. So Next.js owns the experience; FastAPI owns the model work; they talk over a thin, typed contract.

Never block the UI: stream or queue

There are two kinds of AI work and each has a pattern. Interactive generations (a chat reply) should stream token-by-token so the user sees progress immediately. Long-running jobs (a 2-minute report, a batch of embeddings) should go on a queue and notify when done — blocking a request for two minutes is how you get timeouts and rage-quits.

# FastAPI: stream tokens as they arrive
@app.post('/generate')
async def generate(req: PromptIn):
    async def tokens():
        async for chunk in llm.stream(req.prompt):
            yield f'data: {chunk}\n\n'   # SSE frame
        yield 'data: [DONE]\n\n'
    return StreamingResponse(tokens(), media_type='text/event-stream')

On the Next.js side, the client reads that stream and appends tokens as they arrive — a few lines of typed code, and the perceived latency drops from 'is it broken?' to 'it's typing'.

const res = await fetch('/api/generate', { method: 'POST', body });
const reader = res.body!.getReader();
const dec = new TextDecoder();
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  for (const line of dec.decode(value).split('\n\n')) {
    if (line.startsWith('data: ') && !line.includes('[DONE]'))
      append(line.slice(6));   // render token
  }
}

Long tasks: a queue, not a prayer

For jobs that outlive a request, enqueue them (Redis/BullMQ on the Node side, or a task queue on the Python side) and return a job id immediately. The client polls or subscribes for completion. This is the same discipline as any bursty, expensive workload: backpressure instead of a crash, retries instead of lost work.

Cost is an architecture decision, not an invoice surprise

• Right-size the model per task — the cheapest model that passes your evals, not the biggest.
• Cache aggressively: identical inputs should never re-spend (hash the prompt/context).
• Cap context and set per-feature token budgets with alerting.
• Ground answers with retrieval so you can use a smaller model and still be correct.

Key takeaways

• Next.js for the product, FastAPI for the model work, a thin typed contract between them.
• Stream interactive generations over SSE; queue long jobs and notify.
• Treat cost as a design constraint — right-size, cache, cap, and ground.
• The production difficulty is streaming, queuing and cost, not the LLM call itself.