RAG is not enough: what production AI actually needs

A retrieval-augmented assistant can be built in a weekend. Chunk the documents, embed them, retrieve the top passages, hand them to a model with a reasonable prompt. It will demo beautifully. It will also fail in production in ways the demo cannot reveal.

Where the weekend version breaks

• Retrieval returns plausible but wrong passages, and the model dutifully synthesises them into a confident answer.
• Nobody notices quality degrading, because there is no continuous measurement — only the impression of the person who last tried it.
• The knowledge base changes and answers silently drift, with no regression test to catch it.
• A user asks something outside scope and receives an invented answer instead of a refusal.
• An answer turns out to be wrong six months later and nobody can reconstruct why the system produced it.

The four layers that make it production-grade

A golden dataset, written by the business

Two to three hundred question-and-answer pairs, authored by the people who currently answer these questions, with the correct answer and the source it should cite. This is the single highest-value artefact in an enterprise AI programme and it is almost always the one clients want to skip.

Without it, "is the assistant good?" is a matter of opinion. With it, it is a number that runs in continuous integration on every prompt change, every model upgrade and every content update.

Calibrated refusal

A production assistant must be able to say it does not know. That requires a retrieval confidence signal, a threshold calibrated against the golden dataset, and a handover path that captures the question so the gap can be closed.

Typed tool contracts

The moment an agent does something rather than merely says something, free-form tool invocation becomes unacceptable. Tools need typed schemas, permission scoping per user, idempotency keys and a complete audit record of every call and its result.

const raiseServiceTicket = defineTool({
  name: 'raise_service_ticket',
  description: 'Create a support ticket for the authenticated customer.',
  parameters: z.object({
    category: z.enum(['billing', 'technical', 'complaint']),
    summary: z.string().max(200),
    severity: z.enum(['low', 'medium', 'high']),
  }),
  // Scoped to the caller: an agent can never act for another customer.
  execute: async (input, { customerId, idempotencyKey }) =>
    ticketing.create({ ...input, customerId, idempotencyKey }),
});

Decision records

Every interaction should persist the retrieved passages, the prompt version, the model and its version, the tools called and the final output. Storage is inexpensive. Being unable to explain a decision to a regulator eighteen months later is not.

The honest cost

In our engagements, the retrieval and generation logic is roughly fifteen per cent of the effort. Evaluation harnesses, guardrails, tool contracts, observability and the governance pack account for the rest. Budgeting for the fifteen per cent is the single most common reason enterprise AI pilots never reach production.