20 real calls from the build: what was actually true, what the options were, what I picked, and why. Not a project retro written after the fact; each of these is documented the same week it happened.
Decision 1
The app had a documented set of design tokens: a type scale, a spacing scale, colour roles, the lot. An audit measured how much of it the code actually used. Type was at 13%. Spacing was at 6%.
Treat the low numbers as a documentation problem and describe what the app really did, or treat the system as the target and move the app onto it.
Moved the app. Every font size snapped to the nearest step on the scale and every spacing value to the nearest step on a new one, which changed density visibly on every screen.
Six font sizes sat inside a six-pixel band, and a one-pixel difference cannot rank anything at reading distance. That is the mechanical reason the screens read as cluttered: size was doing no work, so weight and colour were doing all of it and everything competed at once. A scale that nothing consumes is not a system, it is a document. The visible change in density was the point rather than a side effect to minimise.
Decision 2
Searching the stylesheets for problems kept coming back clean while the pages still had them.
Keep grepping for the values and class names you expect to find, or measure the rendered page for the defect's shape.
Walked every element in the browser comparing computed values, and looked for the shape of each defect rather than its name.
Every real find came from the second method. A pill written as half its own height never matches a search for the pill value. A page rendering completely empty on mobile has no wrong value anywhere in it. A banner overhanging its container by 24 pixels on each side looked correct because the overhang was symmetric and the page hid its own scrollbar. Counting occurrences in source also misled in both directions: it undercounted the radius problem and overcounted a different one by a factor of six. The stylesheet tells you what was written. Only the rendered page tells you what a person sees.
Decision 3
“Make it feel less generated” is not something anyone can act on, and the pages kept drifting back toward the same look.
Keep relying on taste and catch it in review, or write a checkable list of the specific shapes to avoid.
A named anti-pattern list on the style guide: a pastel fill with a coloured left rail, everything at one radius with one shadow, emoji standing in for hierarchy, a decorative icon doing a heading's job, arrows trailing off buttons, everything centred.
Every item on that list was reached for at some point while building this and then removed, so it is a record rather than a theory. The value is that it turns a matter of taste into a pass-or-fail check anyone can run, including me six months from now. It caught real things immediately: fourteen emoji feature cards, three callouts wearing the exact shape the list forbids, and one of them was on the style guide's own page.
Decision 4
The original plan let you branch from any past version. It was a full version tree, like a Git history you could fork sideways.
Build the branch-and-merge UI the plan described, or restrict the product to a single linear history.
Restore-only: reverting to an older version creates a new version at the head of one line. No branches, no branch picker.
The rest of the product already runs on a simpler model. Every AI re-run forks forward, never sideways. A branch tree solves a problem nobody was asking about yet, and it's real UI and data-model complexity for a feature that would sit unused in v1. I cut it, wrote down why, and moved on.
Decision 5
The whole point of phase-gated editing is that a human's manual edits can never be silently overwritten by an AI re-run.
Trust the application code to always insert a new version instead of updating an existing one, or enforce it at the database layer.
A database-level rule blocks any update to a version's content once it's finalized or already has a version forked from it. That holds regardless of what the application code tries to do.
A promise that only lives in application logic breaks the first time someone writes a one-off script, a migration, or a bug. If a trust claim is core to the product, it has to be structurally true, not just usually true.
Decision 6
The product's entire value proposition is that nothing in the generated document is invented. Every requirement traces back to a real source.
Prompt the model to only use supported information and trust it, or add a mechanical check on top.
Every extracted requirement carries the exact source quote it came from. Before anything reaches the document, that quote is checked against the actual source text, and content that doesn't match gets dropped.
Telling a language model to “only use what's actually there” is an instruction, not a guarantee. Models drift toward sounding complete even when the underlying material is thin. If “nothing is invented” is the core promise, the check that makes it true can't be optional.
Decision 7
The plan assumed you'd tune two knobs to balance cost against quality across the pipeline's stages: which model to use, and a temperature setting.
Force the API to accept parameters it doesn't support, switch providers, or find an equivalent lever within what's actually available.
Kept a single model throughout and expressed the same cheap-vs-careful distinction through the model's own reasoning-depth setting instead.
The plan was written before the specific tool was locked in. When reality and the plan disagreed, the plan was the thing that moved, not the API.
Decision 8
A public “Sign in with Google” button sits directly in front of a paid AI API key. Anyone who gets in can spend real money.
A single allowlist and call it done, or assume an approved account can still be a risk on its own.
Three independent checks, not one gate: an email allowlist, a hard lifetime run limit per account, and a rate limit on both the account and the network it's coming from.
Being on the allowlist answers “should this person be here at all,” not “could this specific session be misused right now.” Those are different questions, and only checking the first one leaves the actual cost exposure unprotected.
Decision 9
After a visual pass, the app's header and its content area were fighting each other for attention. Too much of the same color everywhere, and nothing read as a distinct region.
First attempt: lighten the header so it stood apart from the page. Looked at it live and it felt backwards. The header lost the identity that made the app recognizable, while the document underneath still needed to be the most legible thing on screen.
Reversed course: kept the header's original color as the app's identity, and pushed the reading area (the page canvas and every content card) toward white instead.
The first version wasn't wrong on paper, it was wrong in the browser. Shipping a decision, actually looking at it, and being willing to undo it in the same sitting is a more honest signal than getting it right the first time and never questioning it again.
Decision 10
The evaluate step scored the generated document once and stopped. If it found a weak section, the score just said so. Nothing acted on that finding.
Leave it as a single pass and let a person decide what to fix, or give the critic tools to fix specific gaps itself and let it decide whether to use them.
The critic gets two tools: rewrite one named section, or search the document's own already-verified source material for evidence it might have missed. It chooses on its own whether either is worth doing, capped at a few tries with a hard stop if two searches in a row turn up nothing useful.
The bar for this one was higher than "can the model call a tool." The whole product's value proposition rests on every claim tracing back to real source material, so a self-revising loop had to inherit that same discipline, not just add a demo-able feature. Every tool call has to name the specific gap it's closing, and a rewritten section only ever replaces the exact field the model was asked to fix. The code enforces that by construction; it isn't just an instruction the model is trusted to follow. The result: on a genuinely thin draft, the loop found and fixed real gaps across two revisions and improved the score, without inventing a single number it didn't have evidence for.
Decision 11
The Research Agent could replace the old "chunk every file, extract every chunk" step with a real per-document loop, but that loop reads one document at a time and decides whether to keep going. That trades away the old step's concurrency-5 parallel extraction. On a typical 5-10 document run, that's measurably slower for a similar result.
A genuinely agentic per-document loop that makes a real judgment call on each document (matches the bar the other two agentic stages already set, but slower), or a faster layer that triages documents in batches and checks in periodically. That's quicker, but not an actual tool-use loop making its own decisions.
Chose the slower, genuinely agentic version.
This product already has a standard for what counts as an agent. That's why the Roadmap Signal stage stays a single plain call instead of being dressed up as a fourth agent for no real reason. The faster alternative would have failed that same bar: no real autonomous decision, just a deterministic system with better manners. Speed lost to being honest about what's actually a judgment call and what's just formatting one to look more impressive.
Decision 12
The contradiction-resolution agent needed a shape: investigate a conflicting pair of requirements, then decide whether one clearly wins or the pair should be flagged for a human. Two real designs came out of the same review, not one obvious answer.
A per-candidate loop, where each contradiction gets its own investigation and its own resolve-or-flag decision, or one shared turn that reasons about every candidate in the run together, which costs less per run.
Kept the per-candidate loop, but tightened its tool-call budget (2 calls instead of 3) to bring the cost closer to what the cheaper option would have spent.
Per-candidate is what makes the loop's strongest guardrail actually enforceable: a resolution's claimed evidence has to be backed by a real tool call made for that specific candidate, not just some call made somewhere during the run. Collapsing to one shared turn would have made that check impossible to verify. That trades a real fabrication guardrail for a cost saving, and it isn't a trade this product's core promise can afford.
Decision 13
An agent that calls tools and decides what to do next is usually built on top of a framework, something like LangChain or LangGraph, that handles the tool-execution loop for you.
Reach for one of those frameworks, or write the loop directly against the model provider's own tool-use API.
Wrote it directly against the API: a plain loop that inspects the model's response, decides whether a tool was called, executes it, and merges the result back in.
The guardrails that actually matter here are exactly the kind of thing a framework abstracts into its own internals: checking a tool's target against a real allowlist before trusting it, merging back only the one field the model was asked to change, and capping how many times it can loop. I wanted those checks visible in code I could point to, not hidden inside someone else's execution layer. For a piece meant to show how the agent actually works, that visibility is the point, not a limitation.
Decision 14
Each turn of the agent loop makes its own call to the model. The loop already checks an overall time budget before starting a new turn, but that check only runs between turns, not while one is in flight.
Add a hard timeout on every individual call now, or rely on the existing between-turn check and accept that one unusually slow call could run long.
Left it as a documented gap rather than a silent one. The between-turn check is still the only defense today.
The pipeline as a whole already has a hard ceiling: the hosting platform ends a request past its time limit regardless. So the real failure mode here is "this run takes longer than it should," not "the app hangs forever." Between building that second, narrower safety net and spending the time on guardrails that prevent bad output instead, I chose the guardrails, and wrote down exactly what's still open rather than let it go unmentioned. A gap you can name is a smaller risk than one nobody wrote down.
Decision 15
A later pass added a PM / Lead PM / Approver role switcher that changes what's usable across the whole app. This is still a single-user, allowlist-gated portfolio piece with no real organization or member table anywhere in the schema.
Build a real roles table and server-enforced permission checks now, even though there's no second real user to protect anything from, or simulate the interaction entirely client-side and say so plainly.
A role picker stored in the browser, changing which buttons render and which review-status options are selectable. No server check backs any of it.
Building real enforcement for an app with exactly one real user would be theater in the other direction: complexity that protects nothing, just to look thorough. The honest move was to build the interaction fully (a real approval-gate workflow that genuinely disables the right controls) and say directly, on the app's own roadmap page, exactly what a real deployment would still need. A demo that quietly implies it's secure is worse than one that states what it is.
Decision 16
"Org-controlled PRD templates" could mean two different builds: a fixed set of presets that reorder or hide existing sections, or a true section editor where an org invents entirely new fields.
Rewrite the generation schema and every downstream consumer, the prompt, the critic, the regenerate-section tool, both canvas components, around a dynamic section list, or keep the schema exactly as it is and only vary what's rendered.
Three fixed presets, each an ordered subset of the same 12 sections the AI always generates. The pipeline and its output schema never changed.
The AI still writes all 12 sections on every run regardless of which template is picked. That's a real constraint I chose to keep, not a corner cut: rewriting the generation schema this late in a multi-phase initiative was a lot of risk for a capability nobody had actually asked for yet. A real section editor is the obvious next step if an org needs one. Reordering existing sections proves the concept at a fraction of the cost.
Decision 17
The only check for whether there was enough information to work with used to run after a full draft already existed. By the time a user found out the research was thin, they were already looking at a weak draft.
Build a genuinely new agent for this earlier check, or reuse the Critique Agent's exact judgment schema at a point before a draft exists, run against the synthesized understanding instead of a finished document.
Reused the existing schema. It runs a second time, earlier, and if it judges the research too thin to draft from, it blocks with specific questions instead of a guess, with a Generate anyway override for anyone who wants to proceed regardless.
A second agent making the same underlying judgment is exactly the kind of duplication that already burned this project once: the quality score used to drift from its own section scores because two separate code paths computed it slightly differently. Reusing the schema instead of building a new one closes off a repeat of that same bug class, not just tidiness. The sufficiency bar itself also got tightened mid-build, after a real test run passed a source set that was too thin to draft from responsibly. That got caught by running a real, deliberately weak input through the check, not by reasoning about it in advance.
Decision 18
A persistent knowledge base now survives across runs and versions: extracted goals, requirements, personas, and risks a person can later edit, merge, or archive. A finalized version's content, though, is a database-enforced promise that it never changes. Those two facts collide the moment a version needs to reference knowledge that might later move.
Let a version reference the knowledge base live, simpler to build, but it breaks the never-changes guarantee the instant someone edits a knowledge row a version already pointed to. Or snapshot the knowledge state into the version at the moment it's generated.
Always snapshot. A version gets its own frozen copy of the knowledge it drew on, never a live reference to a row that could move out from under it.
The real tension here is a knowledge base meant to get better over time, sitting right next to a document promise that must never lie about what it was actually based on. A frozen copy is what lets both be true at once, instead of one guarantee losing to the other. Editing or merging a knowledge item later never rewrites what a past version says it was built on.
Decision 19
The draft's quality score used to be a bare number the AI stated about its own work, with no mechanical check behind it. That already got fixed once: it's computed from real per-section scores instead of trusted as a claim. This time, three new numbers needed the same treatment, not just the one score.
Let the model self-report a rough grounding percentage for these too, cheaper to build. Or compute each one mechanically from data already stored and already true: what actually got cited, what actually got resolved.
Mechanical every time. Evidence coverage, citation accuracy, and contradictions resolved are all computed from stored data, none of them ever asked of the model, same as the original quality score.
This created a real problem in the product itself, not just the pipeline. A bare percentage sitting next to an AI-judged score reads like a second, competing opinion about quality unless it's obviously framed as a different kind of thing. That's why the panel has a section literally titled Counted, not judged, with a sentence that says so directly instead of trusting a reader to work it out. A count and a judgment call can look identical on a page if the difference isn't labeled.
Decision 20
Wanted real tracing (every Claude call, grouped into one trace per pipeline run) without changing how any pipeline stage is written. Two real infrastructure facts stood in the way. A serverless function can be torn down right after it sends its response, and the tracing library uploads its data in the background, not instantly. Separately, this app has a real Edge runtime code path (the routing middleware) with none of the Node.js internals the tracing library needs.
For the first problem: trust the library's own default timing, or explicitly wait for it to finish uploading before letting the function exit. For the second: import the tracing library normally and hope the build tooling keeps it out of the Edge bundle on its own, or only load it inside a code branch guaranteed to never run on the Edge runtime.
Explicitly wait for the flush every time, right alongside the existing cleanup step. Load the tracing library dynamically, inside a Node-only branch, and confirm it by checking the actual compiled Edge output for zero references to it, not just assuming the bundler handled it.
Both of these are the kind of bug that looks completely fine in local development and either never breaks visibly, or doesn't break until it's live. Skipping the explicit flush would mean traces silently disappearing some fraction of the time, with no error to point to. The verification step matters as much as the fix: grepping the built middleware file for OpenTelemetry and Langfuse references and finding zero is what actually confirms the Edge bundle stayed clean, not the assumption that it would.