successfully failed · essay
How to turn an analysis you've done twice into a system your team runs — and checks — without you.
Years ago, at a consulting firm, a fintech client wanted to know whether they would reach their yearly revenue targets. They were a growing startup with no real BI layer — and they weren't asking for one. They wanted a decision, and neither their timeline nor their budget had room for a data-platform project. So we answered the question — and alongside the report, we left behind a set of spreadsheets modeling the critical analyses, so their team could re-run them on fresh data. Because a data-driven decision has a shelf life, set by the freshness of what fed it. The report expired on schedule. The sheets kept answering. My first machine was a Google Sheet.
Years later, on our finance team, someone spent about four hours every week matching thousands of transactions across bank statements, payment-provider reports, and internal records — a person, a spreadsheet, and an afternoon of hunting for the rows that didn't tie. I watched it happen twice. The third time, I did the work myself, by hand, start to finish — and then I built a system that does it. The same reconciliation now takes minutes, and the people who used to do the hunting review exceptions instead.
Somewhere between that spreadsheet and this system, the lesson became a rule, and it fits in one sentence:
An analysis you do twice is a product in disguise.
The second time you produce the same deliverable — same structure, same logic, same output format, only the data renewed — you are not doing analysis anymore. You are operating a machine by hand. The memo doesn't know it's software yet. This essay is about how to notice, when to act on it, when not to, and how to hand the result over so completely that it survives your absence — which, I'll argue, is the only honest test of the whole exercise.
The instinct here is not new, and pretending otherwise would be the fastest way to lose your trust. Programmers have the Rule of Three — refactor on the third repetition (Don Roberts, via Martin Fowler). Tobi Lütke's version is folk wisdom by now: do it three times, then automate it. Paul Graham told founders to do things that don't scale precisely so they'd learn what the scalable version must do; Eric Ries called it the concierge MVP. Google's Site Reliability Engineering built an entire doctrine around eliminating "toil" — repetitive, automatable operational work — including a rule I love for its bluntness: a well-written runbook is essentially pseudocode, one step from being a program. And organizational researchers mapped the economics two decades ago: Zollo and Winter treated codifying knowledge as an investment whose payoff turns on a task's frequency, uniformity, and causal clarity — and found, against instinct, that deliberate codification earns its keep most on infrequent tasks, where repetition alone teaches nothing. That is good news for this method: an analysis produced only a few times a year is exactly where writing the machine down beats re-learning it by hand.
So the instinct is old. Two things are different here.
First, the unit. The Rule of Three is about code. SRE's toil doctrine explicitly scopes out work that involves judgment. RPA automates tasks; data teams productionize pipelines. What none of them take as their unit is the thing knowledge workers actually deliver: the analysis — the investment memo, the feasibility study, the credit assessment, the reconciliation report. A deliverable with judgment inside it. That's the unit here.
Second, the exit. Every adjacent practice keeps the machine: the firm keeps the methodology, the vendor keeps the software, the data team keeps the pipeline. This method ends with a handoff — the system, its documentation, and the ability to check it, transferred to the people who need the answers. I build it, I hand it over, I leave. Later I'll explain why that inversion matters more than it sounds, and why "leaving" turns out to be the hard technical problem, not the easy contractual one.
Candidates usually announce themselves in one of three ways: a task is eating someone's week, the same errors keep surfacing in it, or the function has stopped scaling — every new case costs the same human hours. When one shows up, I run three questions. All three must be yes.
1. Does it recur? Same judgment structure, new data. Twice is enough. Programmers wait for the third repetition because rerunning code is nearly free; rerunning an analysis costs someone a day. When each repetition is that expensive, two occurrences is a pattern, and the third one should already be the machine's job.
2. Is the judgment separable from the mechanics? Every analysis is part mechanics — gathering, cleaning, computing, formatting — and part judgment: what matters, what's anomalous, what to recommend. In the systems I've built this way, mechanics ran between 70 and 90 percent of the hours — and there's a simple recipe for measuring yours, below.
That ratio is the point of this question, and it does two jobs. It tells you whether there's enough automatable mass to bother. And it places the work on a spectrum that keeps you honest about what you're doing. At one end sits nearly pure mechanics — we built a system that reads printed invoices and fills the operations team's spreadsheets; call it 95 percent mechanics, and at that end you don't need this framework, you need automation discipline of the kind SRE has taught for a decade. At the other end sits nearly pure judgment, and there you shouldn't build at all — more on that graveyard in a moment. The interesting territory is the middle: the investment memo, the credit assessment, the feasibility study, where the judgment has structure — criteria, thresholds, a shape it takes every time — and that structure can be encoded while the judgment calls themselves stay human. This distinction is what the expert-systems era of the 1980s got wrong, at the cost of the whole field: they tried to encode the judgment itself, through specialist "knowledge engineers" interviewing experts, and the systems were brittle, unmaintainable, and commercially collapsed by the mid-1990s. Automate the mechanics. Encode the structure. Leave the judgment where it lives.
3. Who catches it when it's wrong — after you're gone? I call this the orphan question, and it's where most automation dreams should die. Not "who runs it": running is pressing a button. Catching it being wrong requires a named person who has done the work by hand at least once and can recognize a bad output when the system produces one with a straight face. If the honest answer is "nobody — they'd call me," then shipping the system isn't a transfer of capability. It's a transfer of liability, gift-wrapped.
A note for the skeptics of question 2, because they have a fifty-year literature on their side: Meehl's 1954 review found that simple models beat expert judgment on recurring structured decisions, and Noise (Kahneman, Sibony, Sunstein) renewed the argument recently. Nothing here replaces the judge — it automates the judge's clerical staff.
Sit with the task, break it into steps, time each one, and mark every step that needs no judgment — gathering, querying, entering, checking, simple calculations and reformatting; anything a pipeline, a deterministic rule, or an LLM could do. One pass of the reconciliation looked like this: gathering the sources, ~15 minutes; cleanup, ~15; transformation and matching, ~30; running the pre-built matching rules, ~15 — and then the one genuinely human part, working through the mismatches one by one. Repeat that per account and platform and the mechanics add up to most of the four hours, with every repeated step another chance for human error — which is its own argument for the machine: it removes error surface, not just hours. The investment memo was more extreme: gathering, entering, calculating, and filling the same slide template consumed the preparation entirely. The judgment didn't live in the memo. It lived in the committee room the memo was feeding.
A test that can only say yes is a funnel, not a test. Three disqualifiers override everything:
The invariant isn't stable. Regulatory reporting passes all three questions beautifully — recurs monthly, mechanics-heavy, named owners everywhere. But the rules move faster than anyone revisits the system, and the failure is silent: the machine keeps producing plausible, compliant-looking numbers long after the rules changed underneath it. The team that can run it cannot revalidate it. Unless a revalidation trigger is wired to every rule change, don't build it.
The scored population fights back. Freeze a scoring model into software and the people being scored adapt to it. This isn't statistical drift; it's an adversary. I know this one from the inside: my credit-scoring engine scores against an industry-specific framework we developed in-house — and we still don't let it run free. Its calibration segment is reviewed by human analysts every quarter, and some weights get re-tuned. That review isn't bureaucracy; it's the price of a silent failure mode. Decay is invisible from the operator's seat, and no amount of smooth operation proves the scores are still right. Unless the review that keeps it honest is funded and staffed past your departure, don't hand it off.
The process is the product. The clearest case in my own backlog is an annual group strategy review I once considered building. It recurs yearly, mechanics dominate the hours, and on paper anyone could push the buttons — three clean yeses. But the numbers in that document are not computed; they are negotiated, shaped through rounds between stakeholders until the analysis says what the room can live with. The "judgment" isn't a stable invariant — it's that year's politics wearing an analytical costume. Productize it and the organization either ignores the instant output or thrashes the requirements forever, because nobody was buying the document. They were buying the ritual. Don't build it — you cannot automate a negotiation by automating its paperwork.
And underneath all three runs the single most useful screening property I know: failure loudness. A reconciliation fails loud by nature — the numbers don't tie, everyone sees it. A credit score is the quietest instrument here by nature — a wrong score looks exactly like a right one. Loud failures are safe to hand off. Quiet ones are safe only with verification designed around them.
But loudness has two layers, and the second is the useful one. What I just described is intrinsic loudness — a property of the deliverable. Operational loudness is a property of the workflow wrapped around it, and it can be designed. The invoice reader's mechanics run on models that can misread a poor scan, and the stakes are financial — so a human check is built into every run, and its output feeds the same kind of tie-out checks the reconciliation lives under. My credit engine's scores are presented case by case in weekly sessions, where experts compare them against independent sources — external credit reports, financials, industry data — and a wrong-looking score triggers investigation and reconfiguration.
An instrument that is silent by nature ends up, in practice, failing semi-loud to loud.
That choice, though, is one you pay for. Every step up the axis is verification someone must design, staff, and keep staffed after you leave. A system that needs weekly expert sessions to fail loudly hasn't escaped the silent band — it has bought its way out, and the bill arrives every week. So the screen still screens: ask whether the team will still be paying that bill in a year, gladly, as part of the job. If the honest answer is no, the disqualifier stands — quiet failure plus unfunded verification is exactly the system you should not hand off.
The two questions that survive contact with a real backlog are how mechanical is it? and how loudly does it fail? Put them on two axes and the whole framework becomes one picture you can draw on a whiteboard in thirty seconds:
Those are six of mine, plotted honestly. Two sit at the pure-toil edge — the invoice reader and a regulator's template that grew out of the memo system — where you don't need this framework, just automation discipline. And the silent band is empty by design, not by nature: the one instrument that is born silent, the credit scorer, was moved up the axis by the reviews wrapped around it — the paid-for kind. When you run the inventory at the end of this essay, plot your own rows here; where a dot lands tells you what to do with it, and the three disqualifiers can veto it wherever it lands.
Five steps. I've run them many times now; the scars are where the details come from.
1. Do it by hand first. Two doors have to open before anything else: the sponsor's — for access, data, and cover — and the operator's, which only trust opens. Get authority from the managers, then sit with the person who actually does the work and watch a few cases end to end before touching anything. Then do it yourself. Graham and Ries own the underlying idea and I'm only applying it: you cannot automate an analysis you haven't personally performed. The manual pass is where you discover which steps are real and which are ritual — in the reconciliation, roughly a dozen of the two hundred spreadsheet operations turned out to carry all the logic; the rest were ceremony accumulated over years of nobody asking why. And do one thing the folklore misses: save your worked examples. The instances you solve by hand, with their known-correct outputs, are about to become the most durable asset in the whole project. That's ammunition for everything that follows — the testing, the parallel run, and the handoff kit — produced free of charge in step one.
2. Extract the invariant. Separate what's identical every run (the structure, the logic, the output format — the spec) from what varies (the data). Write the spec down in plain language before touching code. If you can't state the invariant in a paragraph, you haven't finished doing it by hand.
3. Build thin. The first version handles the common case and nothing else; exceptions stay human. With AI-assisted development, my builds have taken days to a few weeks — I'm an analyst who builds, not a software engineer, and that's rather the point: the person who has the judgment can now afford to build the machine. Resist every temptation toward completeness. Scope is how these projects die.
4. Ship to the operator, not the sponsor. The first user is the person who types the invoices, not the executive who approved the project. When we built the investment-memo generator, the first version pulled company data from the formal source — the official one any sponsor would have specified. The analyst who actually prepared the memos pushed back almost immediately: that source ran days behind — and in a live deal, days matter; the numbers a committee votes on can move inside a week. They knew a fresher source. We switched. No requirements meeting would have caught it, because freshness wasn't the sponsor's daily problem — it was the operator's. Their complaints are the roadmap; their relief is the metric.
Then run it both ways. Each of these systems started from cases the operator and I solved by hand together — finding the logic, the rules, the invariant — and the first builds were tested against those solved cases with the operator watching the comparison. After that came the parallel run: the operator kept doing the job the old way and through the system, on the same live cases, comparing outputs and reporting every gap. In the reconciliation this forced several rounds of fixes — and then something better than sign-off happened: the operator migrated on their own. Confidence isn't declared in a meeting. It accumulates until the old way quietly stops.
5. Hand off the catch, not just the keys. Don't hand off at the moment of migration. Let the system live in production for a while with you still nearby — the first weeks of real operation surface what the parallel run didn't, because edge cases arrive on their own schedule. Then write the documentation last, from what it actually took to operate, not from what you designed. Documentation, training, a named owner — the standard checklist, necessary and famously insufficient, because handed-off systems rot. Data formats drift, upstream systems change, owners move on. So the handoff includes what I've come to think of as the falsification kit: the golden set — five to ten of those hand-solved instances from step one, with known-correct outputs — and one standing ritual: once a quarter, the owner re-performs a single instance fully by hand and diffs it against the machine. Agreement means the invariant still holds. Divergence means it moved, and it's time to rebuild from the spec. The ritual costs a few hours a quarter, keeps the by-hand skill alive inside the team instead of severing it, and converts "trust the system" from a hope into a checkable claim. This is what the end state actually is — not a system that runs without you, but a system whose errors get caught without you. Anything less, and "capability transfer" is a phrase on a slide.
The systems in the table are the ones I can describe cleanly, built inside organizations I've worked with. The RFM one began as work for a client and later became a free public Looker Studio connector — the version Google's 2023 hackathon shortlisted. They are not the whole population — more of these run than I can unpack here, including the measurement and strategy-review layer of a market-making operation I run, where the analysis of the system's own behavior got done twice and so, by its own rule, became a system. The before-and-after, with one row (†) marked because its failures are the quietest by nature — and therefore carry the heaviest designed reviews:
| System | Runs | Before (per run) | After (per run) | Mechanics share | Failure loudness |
|---|---|---|---|---|---|
| Reconciliation | weekly | ~4 hrs | minutes + exception review | ~90% | loud |
| Invoice reader | daily | ~1 hr | automatic + human review | ~95% | semi-loud–loud (tie-out checks) |
| Investment memo | per case | ~8 hrs | ~5 min | ~70% | semi-loud |
| Regulator's template (a clerical follow-on to the memo) | per case | ~2 hrs | minutes | ~95% | semi-loud |
| Credit scoring † | per batch | ~3 hrs of data entry | minutes | ~75% | semi-loud–loud (weekly expert sessions) |
| RFM connector | on demand | ~2–3 hrs of data work | instant | ~85% | semi-loud |
What this evidence is and isn't. The numbers are rounded estimates of time reclaimed, not value created — the value lives in the judgment the reclaimed hours now go to. Several systems use LLMs for the reading layer — parsing documents, extracting fields — and that is no accident: document mechanics are exactly the layer LLMs unlocked, which is half of the "why now." Four of the six are fully handed off — named owners run them day to day, and when someone reaches me now it's to commission the next analysis, not to rescue the last one. Three limits are substance rather than modesty, so I'll state them plainly. First, the strictest orphan test — what happens after the builder has truly left the building — is untested for all of them. Second, the handoff evidence is four cases, not six: the credit scorer and the RFM connector still live with me. Third, this is one practitioner's portfolio, built inside organizations that gave me access — and the method's numbers (twice, quarterly, five to ten golden cases) are working defaults from that portfolio, not laws of nature. Tune the numbers; keep the structure.
Against the first limit, the falsification kit is not theoretical: every system was tested against its hand-solved golden cases with the operator watching, then proven in a parallel run, and the standing checks keep running without me — per-run reviews on the loud systems, weekly expert sessions and quarterly recalibration on the scorer (†). The best evidence arrived uninvited: after handoff, the memo operator noticed the LLM leaning too hard on the source document in one section, fed the system more context, and fixed it. I only heard about it afterward. An owner who catches the machine being wrong and repairs it without calling the builder is what capability transfer looks like.
The taper is schematic — the point is the direction, not the slope. The loop doesn't disappear; it moves from the consultant's cash register to the client's calendar.
Five neighboring practices, each real and each doing a different job — the differences are mostly about the unit of work, and in one case about who ends up owning the machine. RPA and workflow automation automate defined tasks and process steps; the unit here is the analytical deliverable and the judgment structure inside it. Data products and pipelines productionize data flows, usually inside a data team; this starts one level up, from a document a person is accountable for. AI agents from vendors overlap with roughly my step three — the build — while the surrounding steps, the manual pass and the handoff with its checks, remain yours either way. Productized services standardize a service to sell it repeatedly, which is a fine business; this method points the other way — the client ends up owning the machine. And expert systems, the 1980s ancestor, tried to encode the judgment itself; this encodes only its structure, and leaves the judgment where it was.
It's worth being plain about what this method refuses, because the refusal is the least fashionable part. Consulting firms have codified their knowledge for decades — the classic strategy is literally called codification — but the firm keeps the asset and bills the reuse. The venture-funded version is grander: it is now called Service-as-Software — AI-native firms that sell you the work, keep the machine, and meter the outcomes. What a vendor calls a "reporting agent" is step three of my five; the other four steps are why the thing survives you leaving, and no vendor ships those, because their business model is the dependency. Productized services, the indie-consulting version, packages a service to sell it many times. All three keep the machine. This method gives the machine away — and the market currently votes against me: AI-consulting retainers are growing, not shrinking. I think the retainer model is harvesting a temporary asymmetry, and the clients who learn to own their machines will stop renewing. That is a prediction. What I can report is narrower: the systems I've handed to their operators are still running.
Two documented failures explain the timing. Expert systems collapsed commercially by the mid-1990s because extracting judgment required a knowledge engineer standing between the expert and the machine — the bottleneck was the interview. And when Clayton Christensen predicted in 2013 that consulting would be disrupted by productized substitutes, the disruption largely stalled — building bespoke systems stayed too expensive to undercut the billable hour. AI removed both constraints in the same stroke. The person who has the judgment can now build the system, in days, without the intermediary — and they were always the right person for the job, because the hard part was never the code. It was knowing which twelve of the two hundred steps matter. One more thing changed with them: the newly automatable layer — reading documents — is probabilistic, and when mechanics are probabilistic, verification has to be part of the machine. That is why this method ships with golden sets and standing checks instead of confidence. The automation SRE perfected was deterministic; this wave isn't, and the frameworks that survive it will be the ones built around that fact.
If any of this is true for your team, it's checkable before lunch. List the analytical deliverables of your last twelve months — calendar and sent-mail are better sources than memory. Then fill this in:
| Deliverable | Times produced (12 mo) | Hours per run | % mechanics | Who catches it wrong without you? | Disqualifiers (D1 rules-drift / D2 adversarial / D3 ritual)? | Verdict |
|---|---|---|---|---|---|---|
Score everything. While you're there, estimate one number for yourself: of your total analytical hours last quarter, what share went to deliverables whose structure your team had already produced at least twice? I call it repeat share, and it takes about an hour to estimate honestly. I won't give you a threshold — one team's systems can teach a method, not set a benchmark. My own before-and-after: the reconciliation was four hours of skilled labor every week; it is now minutes plus an exception review. Circle the row with the highest surviving score. That's your first build. Do it by hand once more, and save your work.
One more thing, because it decides whether this method ages like advice or like infrastructure. Within a few years, agents will plausibly perform analyses end-to-end on demand, and every bespoke system built this year will look like the recorded macros of a previous automation age. If this method were about the software, it would expire with the software. It isn't. The durable asset is the spec and the golden set — the explicit statement of which steps matter and the worked instances with known-correct answers. Today, the cheapest binding of that spec is a thin app. Tomorrow it may be an agent you hand the spec to. The spec ports unchanged; the golden set becomes the agent's exam. If anything, better models raise the value of the method, because unspecified judgment becomes the only bottleneck left. The software is a rental. The extracted invariant is the property.
Which is, I've come to think, what this method was really about all along — not automation, but memory. An organization's analytical expertise usually lives in whoever happens to hold it, and leaves in the same taxi. A spec, a golden set, and an owner who can catch the machine being wrong: that is expertise made durable — organizational memory that runs.
That's the whole method. And this essay follows its own rules: it recurs (I had explained the method five times in conversation before writing it once), the judgment was separable from the mechanics of explaining, and it's built to run without me — the test, the table, and the method are all here, ungated, and you don't need to contact me to use them. The essay is the handoff.
Text, table, and diagram: CC BY 4.0 — reuse freely with attribution to Ehsan Shahabi, eshahabi.com.