APIs for AI products
REST and GraphQL APIs with streaming, versioning, typed contracts and errors that tell the client what to do next.
Discipline08 · Backend Engineering
In shortUpdated 30 Oct 2026 · Labs 24 FDE Squad
AI backend development at Labs 24, in short
AI backend development is building the server side of AI products: the APIs, queues, workflows, data models and caches that call models and tools reliably. Model calls are slow, can fail and cost money per request, so AI backends need streaming, durable execution, idempotent writes and fallbacks that ordinary web backends often skip. Labs 24 builds them with forward deployed engineers.
- APIs
- REST and GraphQL with streaming responses, versioning and clear error contracts
- Long tasks
- Queues and durable workflows that resume after crashes instead of starting over
- Writes
- Idempotency keys so a retried request never creates a second record
- Resilience
- Timeouts, retries, provider fallbacks, rate limits and backpressure
- Stack
- Python, TypeScript, Postgres on Neon, Redis, Temporal, queues, Docker
- Pricing
- Set after an initial embed and review of the current system
01 / What it means
A model call is the slowest, flakiest thing in your stack.
A database answers in milliseconds and costs nothing per query. A model call can take ten seconds, fail one time in a hundred and cost money every time. Backends built for ordinary web apps break in new ways when you put AI behind them. Ours are designed around that.
Designed for the failure cases firstAI behind an ordinary backend
- Request waits 40 seconds, then times out
- Server restarts and half-done work is lost
- User clicks twice, two records are created
- One busy customer slows everyone down
AI backend built forward deployed
- Answers stream as they are generated
- Long tasks resume from the last finished step
- Idempotency keys on every write
- Rate limits and queues per customer
- Fallback to a second model provider
02 / Capabilities
AI backend development capabilities.
Eight parts of backend engineering for AI products we own end to end, from the API contract to the load test.
Async jobs and queues
Work that takes longer than a page load goes on a queue, gets a job id, and reports progress back to the user.
Durable workflows
Multi-step AI tasks checkpointed after every step, so a crash or deploy resumes the work instead of repeating it.
Data models and databases
Postgres schemas for conversations, jobs, documents and audit trails, with migrations and multi-tenant separation.
Caching and performance
Response and prompt caching, connection pooling and parallel calls, tuned against a latency budget per endpoint.
Auth, tenancy and limits
SSO, API keys, per-tenant data separation, and rate limits so one heavy customer can't slow down the rest.
Resilience
Timeouts, retries with backoff, circuit breakers, provider fallbacks and idempotent writes on every path that changes data.
Testing and load
Contract tests, failure-injection tests and load tests at peak traffic before any AI feature goes live.
03 / Failure drill
What happens when things break.
Every AI backend meets these five failures sooner or later. Pick one to see where it hits the request path, and what the user sees with and without a backend built for it.
API gatewayAuth, limitsQueueJobs bufferedWorkflowSteps checkpointedModel callWith fallbackWriteIdempotentStream backAs it arrives
Without it
The request hangs for a minute, the user retries, and you pay for both calls.
With our backend
Timeout after a few seconds, one retry, then fallback to a second provider. The user sees text starting within a second.
Without it
Half-finished work: the invoice was drafted but never posted, and nobody knows.
With our backend
The workflow resumes on another server from the last completed step. Nothing is repeated, nothing is lost.
Without it
Two credit notes are created and finance spends an afternoon finding the duplicate.
With our backend
An idempotency key returns the first result for the second request. One write, always.
Without it
Everyone times out, the database falls over and the provider starts rejecting calls.
With our backend
Per-customer rate limits, a queue that absorbs the spike and workers that scale out. Slower for a minute, never down.
Without it
A spinner for forty seconds, the user closes the tab and the result is thrown away.
With our backend
A job id comes back at once, progress streams to the screen, and the result is waiting when they return.
04 / Patterns and latency
Backend patterns for LLM applications.
The patterns we use on almost every AI backend, and where the time goes in a typical answer.
| Pattern | Problem it solves | Where we use it |
|---|---|---|
| Streaming responses | Users stare at a spinner while the model writes | Chat, drafting, summaries |
| Async jobs with status | Tasks longer than a page load time out | Document processing, reports |
| Durable workflows | Crashes and deploys lose multi-step work | Agents, approvals, batch runs |
| Idempotency keys | Retries create duplicate records or payments | Every write to a system of record |
| Circuit breakers and fallbacks | One provider outage takes the product down | Every model and third-party call |
| Rate limits and backpressure | One customer's burst slows everyone | Multi-tenant APIs |
| Append-only event log | No record of what the AI did and why | Audit trails, replay, debugging |
Where the time goes in a 2.4-second answer
- Auth and routing 40 ms
- Retrieval 220 ms
- Model, first token 650 ms
- Tool call 380 ms
- Generate and stream 1,110 ms
Example budget. With streaming, the user starts reading after about 0.9 seconds, not 2.4.
05 / How it ships
Contract first, failures second, load last.
We agree the API, build for the failure cases, then prove it under peak load. A demo every Friday, on your infrastructure.
Every failure path tested before go-liveStep 01 · Contract
API and data model
Endpoints, payloads, error codes and the data model agreed with the teams that will call them.
- API contract
- Data model
- Latency budgets
Step 02 · Build
Happy path, streaming
The main flow running end to end with streaming, queues and durable workflows in place.
- Core endpoints
- Workflows and queues
- Friday demo
Step 03 · Break
Failure paths
We kill servers, cut providers off and send duplicates, then fix whatever does not recover cleanly.
- Failure-injection tests
- Idempotency verified
- Fallbacks working
Step 04 · Load
Peak traffic, then live
Load test at several times expected peak, tune, then release behind monitoring.
- Load test report
- Dashboards and alerts
- Runbook and handover
06 / Stack
AI backend stack.
Boring, proven tools wherever possible, so your team can run and extend the backend after handover.
Languages and APIs
PythonTypeScriptRESTGraphQLServer-sent eventsWebSockets
Data and jobs
PostgresNeonRedisTemporalQueues
Run and test
DockerCI/CDLoad testingLangfuseAlerting
Hosting, scaling and on-call for the backend are covered in Production & Cloud Engineering.
07 / In production
AI backends in production.
Under NDA, so clients stay unnamed. Every backend runs on client infrastructure with full IP transfer.
Multi-app coupon campaign engine
One backend serving several consumer apps, with per-app rate limits and queues for launch-day spikes.
Multi-tenant API
Digital banking offer engine
Backend packaged as a licensed module, with versioned APIs the bank's own apps call.
Versioned APIs
Regional logistics dispatch
Durable workflows track each job from assignment to proof of delivery, even when drivers reply hours later.
Durable workflows
08 / The squad around it
The backend holds the squad's work together.
Agents, models, retrieval and integrations all run through the backend. These are the disciplines it serves most.
01
Agent Engineering
Long-running agents on durable workflows.
02LLM Engineering
Model calls with routing and fallbacks.
04Evals & Reliability
Traces on every request path.
05Production & Cloud
Hosting, scaling and on-call.
06Integration
Connectors the backend calls.
09Software Engineering
The product and screens on top.
09 / FAQ
AI backend development FAQ.
What is AI backend development?
AI backend development is building the server side of AI products: the APIs, queues, workflows, data models and caches that call language models and tools reliably. Because model calls are slow, can fail and cost money per request, AI backends need streaming, durable execution, idempotent writes and provider fallbacks.
How is an AI backend different from a normal web backend?
A normal backend mostly talks to a database that answers in milliseconds. An AI backend also calls models that can take seconds, fail intermittently and cost money per call, and runs multi-step tasks. That calls for streaming, async jobs, durable workflows, retries with fallbacks and strict idempotency.
What is durable execution and why do AI agents need it?
Durable execution records the progress of a multi-step task after every step, so if a server crashes or redeploys the task resumes from where it stopped instead of repeating or losing work. AI agents that call several tools over minutes or hours need it. Labs 24 typically uses Temporal for this.
How do you stream LLM responses to users?
The backend forwards tokens to the client as the model generates them, using server-sent events or WebSockets, so users start reading in under a second. Labs 24 keeps streaming working through retries and fallbacks, and stores the final answer once complete.
How do you stop duplicate actions when requests are retried?
Every request that changes data carries an idempotency key. If the same request arrives twice, because of a double click, a network retry or a resumed workflow, the backend returns the first result instead of writing again.
How do you handle an LLM provider outage?
With timeouts, limited retries with backoff, circuit breakers and automatic fallback to a second provider or a self-hosted model, so users see slower answers rather than errors.
Which languages and databases do you use for AI backends?
Usually Python or TypeScript, Postgres (often on Neon) with Redis for caching and queues, and Temporal for durable workflows. Labs 24 fits the client's existing stack where it can, so their team can maintain it.
What are the steps in building an AI backend?
Labs 24 agrees the API contract and data model, builds the main flow with streaming and durable workflows, deliberately breaks it to test the failure paths, then load-tests at peak and releases behind monitoring.
How much does AI backend development cost?
Labs 24 prices after an initial embed and review of the current system, once the endpoints, traffic and integrations are known. Code and infrastructure are handed over with full IP.