AmDo: Automating Ambrook

At Ambrook, we scaled human judgement through safe and reliable tools that can automate any task in our App.

aibrainstormtools Written for a (former) employer's engineering blog

This is a copy of a blog post I wrote for Ambrook, a company I worked for at the time. I think its core lessons are good ones for anyone thinking about ethical and safe automation.

TL;DR: Our agentic system for modifying customer data: (a) prompt an AI that has broad access to read-only tools, and have it draft a plan of how to change their books (b) review the pl plans in a comprehensible UI (c) execute the plans deterministically. This approach scales human judgement rather than replacing it, and is flexible to automations of every shape.

The Problem

After watching Haley click a new sequence of buttons for 8 opp, I knew we could do better. Haley is one of our (fabulous) Account Managers, and she was helping a customer with yet another (not so-fabulous) Loan Statement. She would create an account to track the loan, create a transaction for each payment on the loan statement for the same amount, and finally itemize each transaction using the loan statement PDF to figure out the percentage of each payment on the loan that should be tagged as principal or interest. Each of these operations was several clicks. It was painful to watch, and I’m sure even more painful to do.

In the weeks before, Haley and I had spent lots of time together as I tried to understand and accelerate her work as an Account Manager for Ambrook’s customers. Much of the day-to-day work of an Account Manager (AM) consists of helping customers by clicking around the product: setting up their account to reflect their farm operation, and helping them with some basic bookkeeping. Over the prior weeks, shadowing Haley and other AMs had been fruitful. I’d see a pattern of clicks that was inefficient or annoying and try to build automation to make it easier or obviate it. Working hand-in-hand like this, the AMs and I had reworked the authorization model, built many small product shortcuts, and eliminated a class of thorny bugs. Even though each optimization felt like progress, it increasingly felt like we were missing the forest for the trees. At the end of the day, AMs were still spending lots of time in the app, clicking buttons.

The core challenge was how heterogenous an Account Manager’s job can be:

• One customer might need Haley’s help itemizing loan statements by interest and principal amounts.
• Another might need help selectively changing the categorization of a set of transactions.
• A third might need check memos transcribed from a PDF of a bank statement, and applied to their relevant check.

In each instance, the actual work the AM performs for the customer is fairly mechanical, something that could be automated, but the range of tasks varies enough that automating all of it by creating new workflows or buttons wasn’t tractable. We’d toyed around with using LLMs for part of this job, connecting them to demo MCP servers, and found that they were too frequently wrong to be deployed in these accounting workflows where being exactly correct matters.

I spent a week noodling on this problem, and eventually prototyped and productionized a generic solution: PlanBrook and AmDo. In the last few months it has quietly revolutionized not only how AMs do their work, but what we can achieve for our customers, and it presents a hopeful vision for how AI might revolutionize work while centering and scaling human judgement .

The Solution

The core insight was thinking about why we engineers are so comfortable using coding agents extensively. Most of us use them to debug, to summarize, to propose changes, and even to do a first pass of code review - but we still recognize that AI is too often wrong to let it merge without human review. The layer of human review at code submission time is the fence that separates a low consequence playground and the high consequence factory. Account Managers needed a version of “code review”: they needed a step in the process where they could see what was being proposed by the LLM, and carefully consider whether it was good enough to execute.

Thus, we built this system with two halves: “PlanBrook” and “AmDo”, separated by a human in the loop:

PlanBrook is a planning agent based on Claude Code - one that has broad access to data in a read-only mode. This means it can search the codebase, Linear, a given customer’s data, and could even take in files like documents our customers submit to us. However, it doesn’t have any access to change any of that data. Our AMs have freeflowing conversations with PlanBrook - they provide it with instructions, meeting recordings, files, and context about the customer’s data setup. It uses its tools (including OCR, python scripts, SQL queries, and bash) to construct a “plan document” - a giant chunk of JSON that describes changes to be made on Ambrook. The AM and agent can go back and forth, repeatedly honing the plan.

AmDo is PlanBrook’s trustworthy sidekick. It is a deterministic piece of code that lives as a page in our frontend. It takes in PlanBrook’s JSON blob, and gives the human agent a comprehensive and clear review interface that lets them quickly understand the plan and interrogate its details. The plan is always easy to understand through the UI, even if it is big, or includes many nested or looped steps (more on that later). Beyond the review interface, AmDo’s other job is executing the plan that the user selects or approves, using the user’s credentials. Since this happens in the browser, on the user’s computer, this is all just straightforward API calls to our server using existing endpoints, with standard authorization (and thus, access controls).

I like to think of these two systems as Don Quixote (PlanBrook) and Sancho Panza (AmDo). PlanBrook is an inventive, flexible, and courageous explorer, unconstrained by consequence. AmDo is a trustworthy rate-limiter: informed, careful, and prudent.

A data model for anything

The core challenge of this project was building the data model (and corresponding review interface) to represent the variety of plans we wanted to support, and the user interface to allow an Account Manager to understand what would happen at each step. This required solving a variety of small problems:

Operation Set

The plan schema needed a registry of operations that could constitute individual steps. To achieve this we turned to our Frontend-to-backend API, GraphQL, and added some lightweight scaffolding to expose the mutations in that schema to PlanBrook. This scaffolding handled things like:

• hiding methods that should never be used,
• renaming methods to use more commonplace terms,
• adding hints about how each API should be called in brief documentation,
• winnowing the input arguments to the set that are most commonly needed

Each of these optimizations was found by looking at the stumbling blocks models ran into when interacting with our large and sedimentary GraphQL schema. The end result is that adding a new GQL mutation to this set just takes a line of configuration, and can be tweaked when agents get confused.

Loops and Data

PlanBrook and AmDo are only needed when AMs need to do large volumes of operations - if it’s just a few clicks, it’s still faster for them to do it in the App. To express this, we turned to loops and data sets - plans can include named tables that have columns and rows, and then have blocks in the plan that say “for each row in this table”. As an example, PlanBrook might use OCR to extract out the check numbers and memo lines from a PDF of a checking account statement, and then include in the plan both the data table that was extracted (column of Memo and Check Number), and then run an loop over an operation to set the memo for each check.

Dependency and Reference

If an AM specifies that first an invoice should be created, then it should be itemized, PlanBrook needs to describe (in a single plan document) how that second step relies on knowing the existence of the previous step (in order to know the ID that should be sent in the itemization request).

When this is only two steps, it seems straightforward, but we also have concrete use cases where dependencies are 1:N (i.e. create a single invoice and match it to multiple payments) and N:1 (i.e. create multiple transactions, then use a single batch tagging operation to update them).

Building this graph structure required careful tinkering with not only what was theoretically supportable, but what kinds of JSON agents could reliably produce. We found that data models that were too abstract took many more tokens for the agent to construct, and more frequently suffered from hallucinations.

Intentional Omissions

Even as important as what you build is what you intentionally choose not to build. We decided that in order for plans to be fully comprehensible they needed to be repeatable and deterministic, and so we fully skipped conditionality - the ability for a plan to specify a condition and two branches to take based on the evaluation of that condition. We first thought we would need conditionality to support the common “upsert” scenario, where the AM asks PlanBrook “to create or update the transaction”. We ended up finding we could work around this need by ensuring that PlanBrook had access to an internal Ambrook MCP that gives live read-access to the customer’s data. This allowed the planning agent to act conditionally to create plans that are based on the state of the data, rather than encoding that uncertainty into the plan.

This keeps our plans fully deterministic, and highly comprehensible to the AM reviewing them. For the same reason, we opted not to build in Lookups or Error Recovery. Making sure the human reviewer has confidence in what will happen when they press “Run Plan” is invaluable.

The Impact

AmDo has been launched for a few months, and we recently added logging to understand usage trends. For the five weeks we have data:

• Each week it performs 4k+ operations (each operation would have required someone to navigate to a page and click a button in the app)
• Every member of our Account Management team has used it at least once a week for the last five weeks (after vacations are taken into account)
• 11 team-members not on our AM team have used it more than once.

The numbers don’t capture the quality of life improvement - our AMs work so hard, and making their lives better has been sincerely rewarding. Some slack quotes that made me smile:

• “It helped me clean up convoluted COA and P&L in like half the time!”
• “This will save days of time for our customers!”
• “Consider it AmDONE!”

The things our AMs love about PlanBrook isn’t just the automation, it’s the way it allows them to “spend less time squinting” at their screen. With agents, AMs can trust that Claude will figure out the rote pieces of updating books. They can spend more time thinking about what they’re doing, and less time just trying to double check that the buttons they pressed were the right ones.

AmDo has become a primitive that we’re composing into more and more workflows - human and agentic. Several tasks that used to consume hours of AM time (including the original example, of itemizing loan statements by principal and interest) have now been turned into standalone Claude Skills, which ensure the creation of the AmDo plan for them. We’re planning on prototyping a version of AmDo to bring to users - so that they might have plain-text conversations with an agent to keep up with their books.

This process hasn’t been all roses - AMs often end up having to go back and forth with PlanBrook 2+ times in order to get to the plan that they want. Sometimes that’s due to a lack of specificity in the instructions, but more often than not, AMs have to correct misunderstandings the Agent makes about the scope of what it was tasked with, or even basic accounting concepts that the AM is fluent in but the agent isn’t even aware of.

While we’re working to make those kinds of misunderstandings less common through improved context engineering, this problem isn’t one we expect to be able to fully solve any time soon. Even though our AMs are fluent in this tool, and true experts in both the software and accounting, LLMs routinely misunderstand their directions. There’s a lesson in that.

Generalizable Lessons

Design Scaffolds Weaknesses

The lesson I pull out of this work is that when we’re building human-in-the-loop systems, we should design each phase to amplify the strengths of the actor, and scaffold/minimize their weaknesses.

In the planning phase, the actor is an LLM, and they can be tireless, diligent, and have almost any data thrown at them and make sense of it. However, they’re prone to hallucination, bad at math, susceptible prompt injection, and limited context routinely holds them back. The environment we built for this agent thus has access to tools that help it learn, get detailed context, do math, and have firewalls to ensure it can’t access the outside internet.

In the review phase, the actor is one of our AMs, and they are brilliant experts in both their customer’s operation, books, and Ambrook. However, they have limited attentional capacity to review plans, so we built the review interface so that it can help them quickly sample the data and confirm the operations that will take place, then see the results when they run.

By building software that helps us augment the strengths (and accommodate the weaknesses) of each part of the system, we’re getting more out of AI and helping our team do more.

Augment, don’t Replace

My experience with AI (including Planbrook and AmDo) has made me bullish on strategies for AI that aim for “scaled judgement” rather than “full automation”. This is a humanizing way to think about AI at work, because it appropriately recognizes the components of intellectual labor that aren’t captured by keystrokes and clicks.

The AMs that I’ve gotten to work with on our team (shoutout Aryn, Haley, Shannon, Taylor, and many others) are brilliant and kind people with deep experience. Many have worked in bookkeeping, many have run or live on a farm. The reason our customers like working with them isn’t because they can click around a website for them - it’s because they bring understanding and empathy with them when they jump in to help. Our full service program can’t be replaced by a setup wizard, because the conversation about what makes sense for each business is a fractal. Beyond experience, working with our AMs is better than working with a machine because it gives our customers someone they know and trust, a real relationship that provides some folks with accountability, others with encouragement to try something new, and still others with partnership in thinking about their business.

I’m proud that PlanBrook and AmDo allow these brilliant folks to spend more time with their customers. This experience has convinced me that some of the best use cases for AI will be the ones that help brilliant folks get more done, at a higher quality, rather than ones that try to distill their expertise into a button or a text box. Full automation is only possible when your objectives are unitary and easily described. Few problems in our complex world have that shape.