Agentic Scientific Publishing

I’m building software to replicate the calculation logic of Microsoft Excel. My code involves many steps that were (originally) very time and resource intensive. However, over the last few months, I’ve found a reliable approach for semi-automated performance optimization that works across wall-time, RAM, CPU, and probably any quantifiable metric of performance.

First some background: the unit of work for my code is a “case” - an Excel workbook and a series of mutations of the values in it, and a JSON file that describes the authoritative behavior of how Windows Excel responds to those series of mutations and calculations. Cases can cover a wide or narrow range of behavior - some are like integration tests, which evaluate large sets of functionality in concert, others are closer in construction to unit tests, evaluating all of the nuances of function behavior. When a case is evaluated, it results in many code-generation and compilation steps, each of which can have independent performance bottlenecks.

After carefully instrumenting the code so that the resource usage and runtime of each step is evaluated, I set about working on automating performance optimization loops. In the first version of this automated process, I used basic queries to find the worst performing cases in my corpus and then used a standard prompt to ask an agent to:

1. Read through the case, then find and understand the functionality it exercises
2. Use telemetry and code to understand why overall performance was so much worse on this case than others
3. Come up with 3-5 different hypotheses on how performance might be improved while maintaining computational equivalence
4. Test those hypotheses by implementing fixes/changes for each, and evaluating performance against baseline. Then, keep any positive changes
5. Create a PR that gives me a clear high level understanding of what the changes were, why they are safe, what assumptions they made.
6. Kick off a broad validation run to compare performance on other metrics across many other workbooks to ensure we don’t regress other kinds of performance.

Then, often many hours later, I would review the result. This process worked decently well, but many runs failed to make significant progress. When I dug into why by reading a bunch of transcripts, it was clear agents were often hitting the same dead ends, or making similar incorrect assumptions about what might work. Each agent was still moving the ball forward a bit in their session, but they weren’t benefiting from knowledge of the past attempts.

What revolutionized this loop was a simple directive to write down hypotheses and experimental results (positive and negative) as a persistent part of the repository. This allowed subsequent agents, during hypothesis generation, to vet their proposed experiments against the outcomes of prior ones, allowing them to avoid known dead ends, share methods and harnesses, and exposed each agent to a set of ideas it hadn’t come up with but were aimed in the same direction.

Writing things down completely revolutionized the performance optimization efforts. As one example, there is a very complex multi-megabyte spreadsheet that (at the beginning of my optimization efforts) I used as a benchmark because it could just barely compile using 90+ GB of RAM in about two hours. Today, its compilation uses under 8GB of ram, and takes just over one minute forty seconds. It went from barely being able to run on my Mac Studio to being able to run in an ephemeral cloud run function! Incredible.

The big lesson for me here is that agents, like people, can learn from prior experience if they are given the chance to. Next time an agent of yours is spinning on an enormous, hard to solve problem, instead try to decompose that problem into incremental (and evaluable) experimental steps, and have each agent write down what it learned as it goes. Having 40 small sessions that try new things and report back is often preferable to having one gargantuan session to try many things.

Agents, like humans, can achieve more when they publish the results of their experiments - positive and negative.