Categories
AI Product / Blog

Building a Real Product With the Claude API: What I Actually Learned

I want to write the post I wish had existed before I started.

Most writing about building with Claude API or any LLM API falls into one of two categories: breathless hype about how AI will change everything, or step-by-step tutorials that stop right where the real problems begin. Neither of those helped me when I was three weeks into an integration that wasn’t working the way I expected.

This is the honest account. The product, the decisions, the things that broke, and what I would do differently.

The Problem

The client was a mid-sized media company — a WordPress-based publication with a small editorial team producing several long-form articles a week. The bottleneck wasn’t writing. It was briefing.

Before any article was written, an editor had to produce a brief: the angle, the key questions to answer, the competing coverage to be aware of, the suggested sources. For a team producing 15–20 pieces a week, that briefing work was consuming two to three hours a day of a senior editor’s time. Time they would rather have spent editing the actual work.

They had tried to solve this before. Templates. A shared Google Doc with a prompt they’d paste into ChatGPT. A Notion database that nobody updated. None of it stuck because none of it lived where the editors actually worked — inside the WordPress backend where articles were created.

The ask was straightforward: build something inside WordPress that would help editors generate first-draft briefs based on a working title, without making them context-switch to another tool.

Why Claude API

I chose Claude for three reasons, none of which were loyalty to a brand.

First, the output quality on structured, context-rich editorial tasks was consistently better in my testing. I ran the same brief-generation prompt through three APIs. Claude’s output required less cleanup and followed the format I specified more reliably.

Second, the context window. Editorial briefs benefit from being able to include the article title, a category taxonomy, related articles already published, and sometimes a rough notes dump from the commissioning editor. That’s a lot of input. I needed room to work.

Third, I wanted streaming support. I already suspected the latency problem was going to be my biggest UX challenge, and I needed an API that would let me stream tokens to the frontend. Claude’s API does this cleanly.

The Design Decisions That Mattered

Decision 1: Where in the workflow to live

The first prototype surfaced the feature as a standalone panel in the WordPress sidebar — a “Generate Brief” button that appeared on every new post. This was wrong.

Editors told me they didn’t want it at post creation. They wanted it earlier — during the commissioning stage, before an article was even assigned. The feature was solving the right problem in the wrong place in the workflow.

I moved it upstream. The revised version lived on the editorial planning board (a custom post type used for commissioning), not on the article post itself. Usage tripled after that change.

The lesson has nothing to do with AI: build the feature where the workflow actually lives, not where it’s technically convenient to add it.

Decision 2: Streaming vs. waiting

The brief generation call takes between three and six seconds depending on the complexity of the input. I tested three approaches to making that wait bearable.

Option A was a spinner with a “Generating your brief…” message. Users clicked the button again. Every time. They assumed it had frozen.

Option B was a skeleton loading state — placeholder lines that gave a sense of the structure being built. Better, but users still checked their email during the wait and lost context when it finished.

Option C was streaming. Text appeared token by token, typewriter-style, from the moment the first tokens arrived from the API. Editors watched it build. They stayed engaged. A few told me they found themselves reading as it generated and already had notes forming before it finished.

Streaming won decisively and it wasn’t close. If you are building with Claude API and your response time is over two seconds, streaming is not optional. It changes the product.

Decision 3: Prompt architecture

The first prompt I shipped was entirely engineered by me. It was structured, detailed, and almost completely useless to the people using it.

The briefs it produced were too long, too generic, and structured in a way that matched how I thought about editorial briefs, not how the editors did. I had written a good prompt for myself.

I fixed this by sitting down with the senior editor for ninety minutes and asking her to walk me through how she actually wrote a brief when she did it manually. Not the ideal version — the real version, with all the shortcuts and priority calls.

That conversation changed the prompt entirely. The structure shortened. The tone changed. References to competing coverage moved from a late-document appendix (where I had put them) to the second element (where she always put them). The “suggested questions” section that I thought was the most valuable part got cut — she never used it when briefing writers.

The best prompt engineering I did on this project happened in a conversation that had nothing to do with prompts.

What Didn’t Work

The “just regenerate” problem. Because regeneration was one button click, editors quickly developed a habit of clicking it multiple times and picking the version they liked best. This sounds fine until you realise that each regeneration is a different interpretation of the same input — and the act of choosing between them became a task of its own. Briefs that had previously taken one generation started requiring three or four.

I added a light friction: a “refine” mode where editors could note what they wanted to change before regenerating, rather than blank regenerating. This produced better briefs, reduced regeneration volume by about 60%, and — unexpectedly — produced briefs that editors felt more ownership over because their judgment was part of them.

Context window mismanagement (early versions). In the first builds I was including everything: article title, category, recent articles in the same category, SEO keywords, any commissioning notes. The outputs were comprehensive and wandered. I was giving the model so much that it tried to address all of it.

I progressively restricted the input until I found the minimum context that produced a useful output. Ironically, less context produced more focused briefs. The model needs direction, not information dumps.

Error state design. API calls fail. Rate limits happen. Network blips happen. My first error handling was a red toast notification that disappeared in three seconds. Editors didn’t see it, clicked the button again, got another failure, and assumed the feature was broken.

Error states need to be persistent, clear, and tell the user what to do next — not just that something went wrong.

Five Things I Would Tell Myself Before Starting

1. Solve the UX for latency before you write a single prompt. Decide whether you’re streaming or not, and design the loading state first. Everything else follows from that decision. A fast, bad-output feature has a path to good. A slow, bad-output feature gets abandoned.

2. Your first prompt is for you, not for the user. Don’t ship until you’ve watched a real user interact with the output and told you what’s wrong with it. The gap between what you think is useful and what is actually useful will surprise you every time.

3. The integration is the easy part. The Claude API is well-documented, straightforward to authenticate, and reasonably predictable. The hard work is product thinking: where does this live, what does it replace, what friction does it add, what friction does it remove.

4. Build for the unhappy path from the start. Errors, timeouts, rate limits, and malformed outputs are not edge cases. They’re regular occurrences at scale. Design your error states with the same care you design your success states.

5. Less context is often more. Give the model what it needs to do the specific job, not everything you think might be relevant. Specificity of input produces specificity of output. Information dumps produce comprehensive wandering.

What It Looked Like at the End

The feature shipped after about six weeks of build and iteration. The senior editor who had been spending two to three hours a day on briefs now spends about forty minutes. The briefs are not perfect — she edits them every time — but they’re good enough to be a starting point rather than a blank page.

That’s a meaningful outcome. Not because of AI. Because someone understood a workflow problem well enough to build something that actually fit inside it.

Building with Claude API is accessible. The documentation is good, the API is clean, and the capability is genuinely impressive. What is not simple is building something useful with it — that requires product thinking that no API can substitute for.

If This Was Useful

I write occasionally about building AI products, consulting, and the decisions that don’t make it into tutorials. If that interests you, the newsletter is the best way to follow along — sign up below.

If you’re working on an AI integration and could use a thinking partner or a second set of eyes on the product decisions: that’s something I do.


Categories
AI Product / Blog

The Practitioner’s Guide to Building AI Products

Most AI products fail before they ship. Not because the model is wrong, not because the infrastructure is too expensive — but because the team skipped the hard thinking and went straight to building.

I’ve been building with AI since before it was the thing everyone was building with. I’ve seen the pattern repeat: smart people, real budget, genuine ambition — and a product that never quite works in the real world. This is my attempt to give you the map I wish I’d had earlier.

Why Most AI Products Fail

Here’s the uncomfortable truth: most teams treat AI as a feature before they understand the problem it’s solving.

They see what GPT-4 can do in a demo, convince themselves it applies to their use case, and start engineering before anyone has asked the hard questions. The result is an AI-shaped layer on top of a problem that either wasn’t solved by the non-AI version either, or didn’t need AI at all.

The failure modes cluster around three things:

1. Optimising for wow over utility. The demo is great. The model responds coherently, the UI is slick, the investor call goes well. Then real users arrive and the cracks show. They ask questions the model wasn’t tuned for. Edge cases multiply. The product works 80% of the time, which in most contexts is worse than a simpler tool that works 95% of the time.

2. Treating latency and reliability as engineering problems only. Your PM instinct might say “we’ll fix performance later.” In AI products, that instinct is wrong. Users form their trust relationship with an AI product in the first five interactions. A slow, inconsistent response in session one can kill retention permanently — regardless of how much better it gets in week three.

3. Ignoring the human in the loop. The best AI products I’ve worked on all had a clear model for when the AI should act, when it should ask, and when it should get out of the way entirely. Teams that never define this end up with products that either do too much autonomously (and get things wrong in ways that erode trust) or interrupt users constantly (and become annoying faster than any competitor ever could).

The Right Starting Questions

Before you write a line of code or pick a model, answer these four questions honestly:

What is the specific human task this replaces, augments, or accelerates? Not a category. A task. “AI for customer support” is not an answer. “Drafting the first response to a refund request so a human agent can review and send in under 30 seconds” is an answer. Specificity here is everything.

Where does the AI need to be right, and where can it be approximately right? AI products are probabilistic by nature. That’s fine for some use cases and catastrophic for others. A product that helps a PM draft a PRD can tolerate hallucinations that a human will catch. A product that flags compliance risks in legal contracts cannot. Know your error tolerance before you commit to a model architecture.

What does failure look like to the user? Not what does a failed API call look like — what does the user experience when the AI gets it wrong? Do they know it’s wrong? Can they correct it? Does a wrong output cascade into something worse? Map the failure experience before you design the success experience.

What does the non-AI version of this look like? If you can’t describe the manual or rule-based version, you don’t understand the problem well enough to build the AI version. This isn’t a rhetorical question — the non-AI version is often your fallback, your eval baseline, and your user’s mental model all at once.

A Practical Framework for Building AI Products

This is what I use. It isn’t academic — it’s what I’ve iterated to after shipping things that worked and things that didn’t.

Step 1: Define the unit of value

AI product management best practices usually start with user stories. I start differently: I start with the unit of value the AI produces.

For a writing assistant, is the unit a complete draft? A single sentence? A structural suggestion? For a customer service bot, is it a resolved ticket? A deflected contact? A satisfied user?

The unit of value determines your eval criteria, your success metric, and your roadmap priorities. Everything downstream from here is easier when this is sharp.

Step 2: Build the eval before the product

This is the move most teams skip, and it’s the one that saves the most time.

Before you build the product, build the thing that tells you whether the product is working. This means: a set of real (or realistic) inputs, the outputs you expect, and a scoring mechanism. It doesn’t have to be automated on day one. A spreadsheet with 50 test cases and a human grader is infinitely better than shipping and waiting for support tickets.

When the eval exists before the product, every model change, prompt change, and architecture decision has a ground truth to test against. Without it, you’re flying blind and calling it iteration.

Step 3: Design the trust boundary explicitly

Every AI product has a trust boundary: the point at which the AI’s output becomes consequential enough that a human should review it before it has effect.

Design this on purpose. Don’t let it emerge from your implementation. Some products should have the trust boundary at “never” (AI drafts, human always reviews). Some can push it to “usually” (AI acts, human reviews on request). Very few should be at “always” (AI acts autonomously, no review path).

The right answer depends on your domain, your error tolerance (see starting question #2), and your users’ existing trust in automation. The mistake is never deciding.

Step 4: Ship the smallest thing that tests the real constraint

The real constraint is almost never the model. It’s the data pipeline, the user workflow, the latency requirement, the regulatory environment, or the cost at scale.

Ship the smallest version that puts the real constraint under load. Not the real constraint as you imagine it — as it actually exists in your production environment, with your real users, doing your real task.

I’ve seen teams spend three months fine-tuning a model when the blocking issue turned out to be that users didn’t trust AI output unless it showed its reasoning — a prompt engineering fix that took two days. Build to find the real constraint fast.

Step 5: Instrument everything from day one

AI products have a data flywheel that traditional software products don’t. Every user interaction is a potential training signal, an eval case, or a trust-calibration datapoint.

But only if you collect it.

From the first production user: log inputs, log outputs, log corrections, log abandonment. Don’t wait until you have scale to care about data quality. The decisions you make about data collection in the first month determine what you can do in month six.

Common Traps (That I’ve Fallen Into Myself)

The context window trap. When the model has a large context window, it’s tempting to throw everything in and let the model figure it out. This works in demos. In production, it increases latency, increases cost, and often decreases output quality because the signal-to-noise ratio drops. Be ruthless about what goes into context.

The prompt engineering treadmill. Prompts are not a stable foundation. A prompt that works perfectly today can degrade with a model update, a change in user behaviour, or even drift in the distribution of inputs over time. Treat your prompts like code: version them, test them, review changes. If your product’s quality lives entirely in an unmarked string in your codebase, you’re one deployment away from a regression you can’t explain.

The cost model surprise. Token costs are predictable. Usage patterns are not. A feature that costs $0.02 per interaction sounds cheap until your power users hit it 300 times a day. Model your cost per user segment, not just per call — and build cost controls before you need them.

Confusing technical capability with product-market fit. The model can do it. That doesn’t mean users want it, trust it, or will pay for it. These are different questions. Capability is a necessary condition for an AI product, not a sufficient one. Keep testing the PMF assumption even when the demo is impressive.

Skipping the edge case conversation. Edge cases in AI products aren’t just technical edge cases — they’re the inputs your users will actually produce that your happy-path design didn’t anticipate. Run a structured edge case session with real users early. What’s the weirdest thing they’d try? What happens? Is it acceptable?

The Short Version

Building AI products well is a practice, not a formula. The teams that get it right share a few habits: they define specificity before they define architecture, they build evaluation infrastructure before they build product, and they treat trust as a design constraint — not an afterthought.

The teams that struggle are usually moving too fast to stop and answer the hard questions. Speed matters in product. But in AI product development, the questions you skip at the start have a way of becoming the crises you manage at the end.

Work With Me

If you’re a founder or PM building your first AI product and want a practitioner in your corner — for strategy, architecture review, or hands-on build support — let’s talk.

And if you’d rather start with the writing: I publish Ship It, a newsletter for people building with AI. Short, opinionated, no filler. Sign up here.