The Iceberg of Work

A job is more than meets the eye - assume otherwise at your own peril

Thesis

Proximity is a fantastic heuristic for authority. The trust you should have in someone’s opinion should generally be proportional to how well they understand the idea or problem they are engaging with. This is why I’m optimistic about how AI will reshape white-collar work.

I’ve noticed that people tend to think AI will displace their coworkers but not themselves. This is because our interface to our teammates and their work is mediated through mechanical signals - pixels and keystrokes - but our understanding of our own work is far richer than what we express through those physical inputs and outputs. I’m a software engineer, and my work is experienced by others as chat messages and code changes, but the quality and importance of these isn’t the text itself. A coworker that sees my work as interchangable with text generation is demonstrating they do not understand my job. It would be silly to see this iceberg (the hidden mass that supports the whole) within my own work and fail to assume it about the work of others. Extrapolating from that principle:

• When I find myself thinking of fully replacing a human function with a machine to emulate it, I try to think through what pieces of that job aren’t encoded in inputs and outputs. Judgement, discretion, knowing when to break rules, etc.
• There are piece of my job I do think are automatable, and I’m sure this is true for others too. I deeply trust a laborer’s instincts about what pieces of their job could be automated, and which cannot.
• Companies should be optimistic about bottoms-up automations, and should be skeptical of top-down automation initiatives.
• While AI will make individuals in every field more efficient, I won’t worry about wholesale replacement of a field by AI until I hear the experts - the people who do that job - express that forecast.

It would be idiotic of me to look at what a CEO does as text generation, yet many of them seem to see the rest of us as input-output text generation cost centers. I marvel at the arrogance it takes to assert “AI can replace this job” without having ever done that job.

My Experience

I’m a software engineer. I use AI in unfathomable volumes and varieties - it makes me radically more effective. Through that use, I’ve started to see clearly the pieces of my job that I expect AI systems will not be able to replace, they fall into a few categories:

Judgement

• Our profession oscillates between identifying as engineers and hackers. Much of what differentiates a good SWE from a great one is fluidity between these poles. The best engineers know when to go with the quick hack and when to build it right. Folks that haven’t paid the price for bad choices (in both directions) rarely have that insight.
• Similarly, much of software engineering is about balancing between goods, not achieving some maximal outcome along one dimension. Most of the time this means balancing complexity against some other goal like velocity, consistency, performance, or simplicity. LLMs may be great at lifting a concrete implementation into an abstraction, or decomposing an abstraction to concrete implementation, but they rarely demonstrate the prudence to know when to do each.
• Some of the biggest pieces of impact in my career have been pushing back on our leadership when our current prioritization and approach may lead to technical problems in sequencing, timeline, or bitrot, and knowing how forcefully to make the push. AI systems are good at doing, they rarely display strategic reasoning in what to do or when to do it.
• Though my job is profoundly technical, it often involves aesthetics - amorphous senses of what makes a UI, an algorithm, or an API beautiful. These fleeting moments of inspiration often end up being profoundly impactful on the trajectory of our products and teams. This is an area I have yet to see LLMs express sound judgement in, in large part (I suspect) because most of us rarely have the vocabulary to express and explain our aesthetic preferences in the thin pipe of language.

Human Management

• I mentor our junior engineers, which both helps them become more effective over time, and lets me more deeply understand and revisit our infrastructure, and learn new things from them and the experiences they bring with them. AI makes for a great tutor, but it’s unidirectional, and can only answer the questions you know to ask.
• I temper the instincts of our PMs and designers, often translating their vision through the lens of technical feasibility and maintenance - “if we are OK with slightly different functionality, it will be 10x easier to maintain” is a series of tokens I regularly emit by vibrating air, but is hard for agents to do precisely because they’re optimized to achieve the task as the user has asked for it.
• I conduct interviews, and have developed intuitions for where candidates should be challenged, assisted, and how a candidate’s internal model of the problem evolves over time. The mirrored state of the candidate’s mind that lives within mine gives me a much more reliable signal about how someone will perform in a job than externally visible signals.

Context

• While solving problems in the codebase and infrastructure, I accumulate whiffs of unrelated data - sometimes it’s a log that looks out of place, sometimes its a file that looks too big, or a test that takes longer to run than I expect. It’s rarely the right choice for me to dive head-first into solving or understanding these surprising tidbits, but their accumulation is at the heart of my ability to hear about a customer complaint or strange failure mode and quickly identify hypotheses about how it might have come to be. LLMs can solve any problem that is specifically stated, but they don’t do well to make connections between sparse signals that cannot all fit into a context window.
• If I had a dollar for every time an LLM tried to reverse an architectural decision (or bug fix), I’d go out to a really nice dinner - one of those ones where the food is tiny. LLMs can be reminded to check git history, they can be given detailed explanations and admonitions, they can have documentation of everything, but at the end of the day, LLM performance is trained against task execution, and the fastest way of accomplishing task X is often deleterious to the thousand other tasks and goals that came before it. I hold more context in my head about bugs that I have fixed than can fit into 1M tokens.
• The experience that has given me the best insight into our customers has been visiting them in person. Spending two days on a family ranch in South Dakota taught me more about the needs of ranchers than had months of prior reading. Experiencing the frustrations of a vegetable farmer as we watched her click through her invoicing flow in her wash facility in Vermont gave me more insights about how to change the product than had months of playing with the website in my office. Embodied and interpersonal experience is at the heart of figuring out what product should be built.

Note: in these problems of context there are technical ways you could imagine scaffolding an agentic workstream such that they can mimic or even surpass human skill on these kinds of tasks. While the other two categories I’m fairly confident are not going away, I do think AI will find ways of chipping away at these advantages.

The Gap is terrifying

The thing I fear is not AI capable of these parts of my job, it’s the delusion that these gaps don’t exist or matter.

That delusion ends poorly for the workers, who face unemployment and systematic de-valuing of their labor and humanity. It ends poorly for the companies - who will eventually fail or pay enormous costs for shortsighted layoffs and automations. It ends poorly for our culture, which further reduces the vibrancy and value of human life and thought to its measurable constituents.

I fear our society will steer for this iceberg because it looks so small.