The AI Feature Launch Playbook: From Idea to Production in Six Steps

01Who this playbook is for

Product leaders, CTOs and operations heads who have an AI idea that worked in a demo and now need it to work every day, for real users, without surprises. It assumes the feature uses a large language model from a provider such as OpenAI, Anthropic or Google, possibly with your own documents or data. The steps apply whether you build in-house or with a partner.

02Why a playbook is needed

Demos are easy. Production is not. In July 2024 Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. Each of those four causes maps to a step below.

03The six steps at a glance

04Step 1: Give the feature one job

Write the job in one sentence a customer would recognise: "summarise a support ticket so the agent can reply faster" or "answer policy questions from our own documents, with sources".

Then add:

• The user and the moment. Who uses it, inside which screen or workflow, and what they do next with the answer.
• Three good examples and three bad ones. Real inputs, with the answer you would accept and the answer you would not.
• What the feature must never do. Give financial or legal advice, change a record without approval, reveal one customer's data to another.
• Why AI at all. Google's People + AI Guidebook notes that a rule or heuristic often works as well as, or better than, an AI version. If a rule can do the job, use the rule.

A feature with one clear job can be tested. A feature that "helps with everything" cannot.

What a good brief looks like

05Step 2: Prepare the data before the model

Most AI features fail on data, not on the model. Before building, answer:

• Where does the information the feature needs live today, and who owns it?
• Who is allowed to see which parts of it, and how will the feature respect that?
• How current does it need to be, and how will it stay current?
• Is any of it personal or sensitive data, and does your privacy policy and contract allow this use?

Connect the sources, clean the obvious problems and make sure the feature can only see what the signed-in user is already allowed to see. For features that answer from documents, see our explainer on retrieval-augmented generation: permissions must be enforced at retrieval time, not in the prompt.

Choosing a model comes after this

Pick the model last, against your evaluation set. Consider quality on your examples, cost per request at expected volume, response time, the provider's data terms, and where the data is processed. Business versions of the major providers' APIs offer terms that exclude training on your inputs; check the current terms for the plan you use.

06Step 3: Agree the quality bar

Collect 50 to 200 real examples of the task with the answer you would accept. This is your evaluation set. Anthropic's guidance on building evaluations gives three useful rules: make evaluations task-specific and include edge cases, automate grading where you can, and prefer more test cases with automated grading over a handful graded by hand.

Decide in writing:

Include edge cases on purpose: empty or irrelevant input, very long input, hostile input, and questions experts would disagree on. Track the share of answers that meet the bar on every version of the prompt, model and data.

Pick the simplest pattern that does the job

Move down the table only when the row above cannot do the job. Each step down adds capability and adds things that can go wrong.

07Step 4: Build with guardrails

The OWASP Top 10 for LLM Applications 2025 is a practical checklist of what goes wrong. The ones that matter most for a first feature:

Also:

• Log every input and output with the sources used, so problems can be traced.
• Give users a way to flag a bad answer in one click, and route flags to the owner.
• Prefer workflows to agents at first. Anthropic's guidance on building effective agents recommends the simplest solution that works, adding autonomy only when needed. A fixed sequence of steps is easier to test than a model that chooses its own.

For a wider governance frame, the NIST AI Risk Management Framework organises the work into four functions: govern, map, measure and manage. Its Generative AI Profile (NIST AI 600-1) lists risks specific to generative systems.

08Step 5: Launch small

Release to a small, willing group with human review switched on. In the first weeks, watch four numbers:

• Quality: the share of answers that meet the bar, from your evaluation set and from user flags.
• Override rate: how often people change or discard the output.
• Time saved: measured on the real task, not estimated.
• Cost per useful answer: model and infrastructure cost divided by answers that were used.

Widen access in steps, for example one team, then one department, then everyone, and only while quality holds. Keep human review for any output that affects a customer, money or a record.

A rollout plan you can copy

09Step 6: Keep measuring

Models, prompts and data change, and each change can move quality in either direction. Re-run the evaluation set on every change, review flagged answers weekly and keep a short log of what changed and why. Add new failure examples to the evaluation set as you find them, so the same mistake cannot return unnoticed.

Name one owner for the feature in production: a person who reads the flags, approves changes and can switch the feature off. Features without an owner drift.

10Design the experience for imperfect answers

Every AI feature is sometimes wrong. The interface decides whether that is a minor correction or a lost customer:

• Set expectations. Say what the feature does and what it does not do, in one line, where people use it.
• Show the evidence. Link each answer to the sources or records it used, so people can check it in seconds.
• Make correction cheap. Editing a draft should be faster than writing from scratch, and rejecting it should take one click.
• Allow "I don't know". A feature that admits a gap is trusted more than one that guesses.
• Keep people in charge of actions. Suggest, then let a person confirm anything that changes a record or reaches a customer.

11Who does what

An AI feature touches product, data, engineering and risk. Name the people before step 1, even if one person holds two roles.

12Cost and speed, planned up front

Model usage is billed by the amount of text in and out, so cost grows with prompt length, retrieved context and volume. Estimate it before launch:

1. Take ten typical requests from your evaluation set and measure the input and output size of each.
2. Multiply by the provider's current price for the model you plan to use, then by expected daily volume.
3. Add retrieval, hosting and logging costs.
4. Set a budget per request and per user, and an alert when daily spend passes a threshold.

Ways to bring cost and response time down without hurting quality: send only the passages the task needs, cache answers to repeated questions, use a smaller model for simple steps such as classification and a larger one only where it earns its cost, and stream long answers so users see progress. Re-check quality on the evaluation set after each of these changes.

13Privacy, transparency and records

• Personal data: if the feature processes personal data, record the purpose and lawful basis, minimize what is sent to the model, and set a retention period for logs. Under UK and EU GDPR, a data protection impact assessment is often expected for new uses of personal data at scale.
• Telling users: say clearly when content is AI-generated or when people are talking to an AI system. The EU AI Act (Regulation (EU) 2024/1689) includes transparency obligations of this kind, applied in stages since it entered into force on 1 August 2024.
• Keep a record: the brief, the data map, the evaluation results at launch and the change log together form the documentation a reviewer, auditor or enterprise customer will ask for.

14Templates you can copy

Evaluation set

Change log entry

15When to stop

Decide the stop conditions before launch, so stopping is a decision rather than an embarrassment:

• The pass rate cannot reach the agreed bar after a fixed number of improvement cycles.
• The override rate stays so high that people are rewriting most of the output.
• Cost per useful answer is higher than the value of the time saved.
• A "never" failure reaches a customer and the cause cannot be fixed with a control.

Stopping a feature that does not meet its bar is a good outcome. It frees budget for the one that will.

16A worked example: support reply drafts

1. Job: draft replies to billing questions from the help centre and the customer's last three invoices.
2. Data: help centre articles (public), invoices (per customer, permission-checked at retrieval).
3. Quality bar: 150 past tickets graded by two senior agents; launch at 90 percent, zero promises of refunds.
4. Guardrails: read-only, no prices outside the invoices, sources shown, one-click flag.
5. Launch: six agents for two weeks, every draft reviewed; then one team; then all agents.
6. Measure: override rate, handling time per ticket, flags per week, cost per sent reply; evaluation re-run on every prompt change.

17Common mistakes

• Choosing the model before writing the job and the evaluation set.
• Testing on the five examples that made the demo look good.
• Putting permission rules in the prompt instead of in the data layer.
• Letting the feature take actions in the first release.
• Launching to everyone at once, so the first visible mistake ends the project.
• No budget for running costs after launch.

18Checklist before launch

• One job, written down, with good, bad and "never" examples
• Data connected, current and permissioned per user
• Evaluation set with an agreed pass rate, passing
• Guardrails mapped to the OWASP risks above
• Logging, flagging and cost alerts live
• A small launch group and a rollout plan
• A named owner who can switch it off

19Terms used in this playbook

• Evaluation set. A collection of real examples with accepted answers, used to measure quality before launch and after every change.
• Pass rate. The share of the evaluation set the feature must handle well before it can go live.
• Guardrail. A check that runs around the model, such as filtering what data it can see or blocking certain outputs.
• Grounding. Making the model answer from supplied sources rather than from general knowledge.
• Prompt injection. Text, often hidden in a document or message, that tries to change the system's instructions.
• Feature flag. A switch that turns a feature on for chosen users without a new release.

20How we use this at CodeWorx

This is the sequence we follow when we add AI to a client's product or workflow. Our experience is that the evaluation set does more for quality than any choice of model, and that the first release should read and suggest before it is ever allowed to act. See applied AI and intelligent systems for how we run it, and why AI pilots stall for the failure patterns this playbook is designed to avoid.