# How do I run my products with AI agents writing the code? > A case study from three days of building AI Agents Live 2.0, with a control plane in plain files, one gate for every change, a model measured against past human decisions and a hard cap on spend - Author: Ash Gaikwad (https://ashgaikwad.com) - Published: 2026-11-09 - Topics: case study, ai agent, claude code, verification, llm cost - Canonical: https://ashgaikwad.com/blogs/how-i-run-my-products-with-ai-agents I run my products with AI agents writing the code, and I keep control with four things that sit around the agents, not inside them. A folder of plain files tells every agent what we are doing and where the last one stopped. One command gates every change. A model's judgment is measured against decisions people already made before it may act alone. Every model call reserves money before it runs, under a hard daily cap. My own job moved from writing code to deciding what gets built and checking that it works. I am Ash, a Staff engineer in Toronto with ten years at Silicon Valley companies. This is a case study of one release, AI Agents Live 2.0, which agents built from 6 to 8 November 2026 in 111 commits while I made the decisions. I wrote it for CTOs and engineering leads who want to know what this looks like in practice, including the parts that went wrong. As I write this, the release is built and pushed but not deployed. The last steps before the deploy are mine (keys, provider caps, a payment setting and one manual test), and three of the bugs below are still open. ![How a change moves through my setup. I decide. The decision goes into plain files every agent reads first, with a handoff for each piece of work. An agent builds in the product's own repository, where its hooks and rules load. Every change passes one gate, the full suite plus a second agent's review and a browser walk. I run one manual test, then deploy. Every correction goes back into the files as a rule or a test.](/images/blogs/agent-run-product-loop.svg) ## What the agents built AI Agents Live started as a directory of AI agents. Version 2.0 turns it into a tool that helps an agent's maker get found and recommended by AI assistants, which people call generative engine optimization (GEO). The release had ten steps. A model reviews each new listing and decides whether it goes live. A model rewrites the listing page so assistants can read it. The site asks AI assistants questions about the listing and shows the maker what they say. A spend gate limits every model call, and a monthly plan adds weekly tracking for makers who want more. I did not write code for it. A lead agent (Claude Code) read the plan, wrote a brief for each slice of work and reviewed what came back. Builder agents (Claude Code on Sonnet, a smaller model) wrote each slice in the product's own repository. I read the plans, answered the questions only I could answer, and corrected things I did not like when I saw them. The full test suite went from 2,643 green runs after the spend gate to 3,207 green runs at the final push. Over the same week the agents wrote 118 entries in the journal of my control plane, one for every session. ## Work is cut into slices before an agent writes code No agent got "build version 2.0" as a task. Each step started as a written plan in the product repository, a product requirements document (PRD) with its decisions, its pass marks and a numbered list of slices. A slice is small enough for one agent to finish in one run, such as the listing lifecycle, the re-review after an edit or the blind comparison of two models. For every slice the lead agent writes a brief to a file: what to build, which part of the PRD it covers, which tests prove it and what done means. A fresh builder agent gets that brief and the repository's own rules, and no history from earlier slices. When the slice comes back, the lead agent reviews it before the next one starts, and anything built differently from the plan is written into a "Built as" section of the PRD, so the plan and the code never quietly drift apart. ## The control plane is a folder of plain files An agent forgets everything between sessions, and the next session may be a different tool. So the state of my business lives in a Git repository of Markdown files that every agent reads first. I call it ashOS. The files are boring on purpose. `NOW.md` holds the current focus, the work in flight and a list called "Waiting on Ash" with everything only I can do. Each unit of work that spans sessions gets a task file with a "Done when" list and a Handoff section, which is overwritten at the end of every session with the state, the next step and the open questions. A journal gets one entry per session. Decisions get their own file with the reason. The rules sit in one instructions file that Claude Code, Codex and Gemini all read (the other names are symlinks to it). A few of them do most of the work: 1. Close out before the last reply, even when stopping halfway. Update the Handoff, the journal and `NOW.md` so the next agent can continue cold. 2. Numbers come from a source with the date they were read. Unknown is written as unknown, never guessed. 3. Anything outward-facing waits for my go-ahead, such as publishing, emailing a customer, deploying or spending money. 4. Dates are absolute and in Toronto time. "Yesterday" means nothing to the next agent. The Handoff is the piece I would copy first. A fresh agent that reads one paragraph and knows the next step is worth more than any amount of memory features. ## Agents build from the product's own repository I learned this one on 7 November, when I had a builder agent start from the control plane folder and work on the product repository from there. It wrote working code, and the tests passed. It also ignored the product's conventions, because its working directory decides which instructions, hooks and settings load, and it was in the wrong one. Now every code agent starts inside the product's repository, so the product's own rules win for code. The product's hooks run on every tool call (one of them refuses `grep` and points the agent at the project's own tools). The control plane is added as a second working directory, so the agent can still read the plan and write its journal entry. The product's own instruction file is twenty lines long. It points to one skill file per convention (coding, design, routes, tests and the PRD format), and hooks enforce the ones that matter most. One makes every new file come from the framework's generator, one blocks plain text search on code in favour of a structural search, and one lints each edit as it is made. The rule about where agents start is one line in the control plane's instructions, with the date and the reason beside it. The hooks, permission rules and sandbox that do this work are in [How do I set up guardrails for an AI coding agent?](/blogs/how-to-set-up-guardrails-for-ai-coding-agents). ## One gate for every change Every change in the product, from a person or an agent, passes `bin/ci`: the tests, the linters and the security checks. A pre-push hook runs it, so nothing reaches the shared repository red. The lead agent does not trust a builder's report that the suite passed. It reads the diff, runs `bin/ci` again itself, and walks the new screens in a real browser at a desktop width and a phone width, in light and dark themes. A second agent with no context then reviews the whole diff with fresh eyes. In one round that review found two forms on the landing page sharing the same ids. The browser walk found a Copy button 32 pixels tall beside a 40-pixel field, and a menu that faded over the page when the window crossed the phone breakpoint. The tests had passed on all of them. When I correct the same kind of mistake twice, the correction becomes a rule or a test. My sites, the page where makers prove they own a domain, grew a button that reloaded the page with the site in a query string, a redirect to an anchor and a value kept in the session. Each piece passed its tests, and together they were a mess. I asked for it as a proper step flow and for the convention to be written down, in the same style as [How should I name REST resources, routes and associations?](/blogs/how-should-i-name-rest-resources-routes-and-associations). Now a route test fails the build if a verb appears in a signed-in path, a flow's steps are not a nested resource, or a redirect lands on an anchor. Two fresh agents asked to design a new flow both followed the convention without being told about it. Both also used a hyphen where the convention wants an underscore, so that is now a rule as well. ## A model's judgment, measured against people The listing review is where a model makes a decision a person used to make, so it had to earn that. Before it touched a live listing, it ran in shadow mode against every listing a person had already decided, 761 of them (466 approved and 295 hidden). The model is a small OpenAI model. I wanted to measure larger models as well, but the judging endpoint the review uses refused both OpenAI Sol and OpenAI Astra, so the small model was the only one that could run. I set three pass marks before the run: - It agrees with at least 95% of the listings a person approved. - It sends fewer than 20% of listings to a person. - It approves zero unsafe listings. The first run missed. The model called 77 person-approved listings "not an AI agent", and reading them showed quiz makers, image generators and study tools. My past reviewers had approved any AI-powered product, and my prompt asked for an AI agent. The labels and the prompt measured two different rules, and no threshold could reconcile them. The fix was to write down the rule we followed in practice, widen the prompt to match it, and run again. The second run, with the wider rule, called 62 approved listings "not an AI agent" instead of 77 and told approved and hidden listings apart much better. Its best was still 83.8% against my 95%. Some of the rejects were sites our fetcher could reach but not read, and I decided those go to a person instead of being rejected. Then I accepted the measured figures as the marks, at least 92% agreement and under 30% sent to a person, and kept the safety mark at zero. At the chosen thresholds, the model approves 92.0% of the person-approved listings it decides on its own. It sends 27.2% of all listings to a person and approves none of the 27 listings in the unsafe set. The 25 approved listings it would now reject see the reason and can appeal. If no threshold had met the marks, the plan said the review would still ship with every listing going to a person and the model only sorting the queue. The rewrite got the same treatment. The small model and a larger OpenAI model each rewrote 20 approved listings. A nine-rule check (no dramatic colons, the product's name present, no banned phrases and so on) scored each page, and OpenAI Sol judged the pairs blind. The small model passed the check on 16 of 20 where I needed 18, so the larger model writes the pages, Sol takes over when it fails, and a better prompt for the small model is the first job after the deploy. The measured runs for the review and the rewrite cost a little over one US dollar in model calls. ## Every model call holds money before it runs A product that calls models for free visitors needs a hard limit, because a bug or an abusive user can turn into a bill overnight. Every call in AI Agents Live reserves an estimated amount before it starts and settles the real cost when it ends, and failed calls are charged too. Limits apply per account, per IP address, per domain and per listing, with a kill switch for the whole site. At launch every limit starts at one US dollar a day, and I raise them on an admin screen as real numbers come in. The client that calls the model refuses to run without a held reservation, so a new feature cannot skip the gate by accident. A cap set at each model provider is the second line, because my own code can have a bug. Setting those caps is on my list before the deploy. The same idea applied to an agent's tools, where each tool gets a narrow job and a limit on what it can reach, is in [How do I give an AI agent tools it can call safely?](/blogs/how-to-give-ai-agents-tools-safely). ## What went wrong These are the failures from this release, in the order I found them. - A builder agent started in the wrong folder and missed the product's conventions (above). It is now a rule. - A flow that passed every test was still badly shaped (above). It is now a convention and a test. - My first pass marks for the review measured labels that did not match my own rule (above). The rule is now written in the prompt, which an admin can edit and measure again. - On 9 November I ran the first real check of my own site through the new pipeline. The development OpenAI key had expired, so every model call failed. The run was still marked complete and sent "snapshot ready", which spent the listing's one free snapshot. The suggested actions were built from no data, and the questions to the assistants picked up my admin account's language setting, so Marathi words appeared in English questions. The tests had passed because they replace the model with a stub that always answers. These three get fixed before the deploy, and the lesson is that a run with zero answers is a failure, and a test has to say so. The last one is the kind of bug I now look for first in any team's AI features. Tests that stub the model prove the happy path. They say nothing about what the product does when the model is down, slow or wrong. A small set of hostile and broken cases run on every change catches most of it, which is the approach in [How do I test that my AI agent guardrails actually work?](/blogs/how-to-test-guardrails-for-ai-agents). ## What I do and what the agents do I decide what ships, at what price, with which pass marks. I do the things only a person can do, such as logins, keys, payment settings, posting from my own accounts and the deploy itself. The agents do not deploy, set keys or change payment settings, and every price is my decision. Everything I need to do waits in one list with the exact steps, so it takes minutes. ![Two lanes. Mine holds deciding what ships, prices and pass marks, my list of keys, payments, logins and posts, and one manual test before I deploy. The agents work along a line of stations, from planning a slice to building, the test suite, a second review, a browser walk and the records. Questions rise to me and decisions come back down. Deploys, keys and payments stay on my side.](/images/blogs/what-i-do-what-agents-do.svg) I also asked for one manual test script at the end instead of being asked to try things during the build. Every check I would have done mid-build goes into that script, and I run it once before the deploy. The agents do the rest, which includes planning slices, writing code and tests, reviewing each other, walking the screens in a browser, keeping the records and telling me what they could not verify. ## How to start with your own team You do not need my folder layout to try this. These are the pieces I would set up in the first week, in this order. 1. Write one file every agent reads first, with the current focus and the rules that matter most to you. 2. Give each piece of work that spans sessions a Handoff section that the agent overwrites at the end of every session. 3. Make one command the gate for every change, and run it from a hook so nobody can skip it, people included. 4. Before a model makes a decision a person used to make, run it against a few hundred of the decisions people already made, and write the pass marks down before you look. 5. Put a reservation and a hard cap in front of every model call, and set a cap at the provider too. ## Frequently asked questions ### Do the agents write all of the code? For this release, yes. I did not write code. I read plans, made decisions, answered questions and corrected what I saw. Before the deploy I run one manual test script myself. ### How much did the model calls cost? The measured runs for this release (the review in shadow mode and the rewrite comparison) cost a little over one US dollar together. The coding agents run on a subscription, which is a separate cost. ### Does this work for a team, or only for one person? The pieces are not specific to one person. A Handoff section, a single gate, a shadow run against past human decisions and a spend reservation all work the same way when several people and several agents share a repository. What changes is who answers the agents' questions and who owns the rules. ### Does a person review every pull request? No, and that is a deliberate trade. The lead agent reads every diff and runs the gate again, a second agent reviews with fresh eyes, and I read the plans and decisions rather than the code. I run one manual test of the whole release before it ships. On a team I would add human review where a mistake is expensive, such as payments, permissions and data migrations, and let the gate and the review agents carry the rest. ### What would I do differently? I would test the failure paths with real failures much earlier. Every bug that reached my first real run lived in what happens when the model does not answer. If you are working through the same questions with your team, I am happy to compare notes.