Can We Trust the AI to Do the Math?

The One-Sentence Version

We tested three ways AI can be involved in producing a number for a loan application — computing it directly, computing it with code, and choosing which pre-built calculation to use — and compared all three against Eliza’s actual baseline: code built by the SigmaCore team for every one of Eliza’s compute requirements, with no AI involved in the calculation itself. The baseline never got a single case wrong across 3,400 tests; letting AI choose the right calculation came very close at 99.8%, while letting AI compute an answer directly was far less reliable.

Why We Did This At All

Eliza produces documents that go to banks — 3-year financial projections, loan payment schedules, break-even analysis. Every number in those documents is a claim. A loan officer reading “this business’s monthly payment is $3,845.65” doesn’t re-calculate that by hand — they trust the document.

The question this project set out to answer is simple to state and hard to answer well: when an AI system like Claude is asked to compute a number, how often does it get that number exactly right? And just as important — when it’s wrong, is it wrong in a way anyone would notice, or does it look just as confident as when it’s right?

Part 1 — Description of Terms

These terms come up throughout this report.

LLM (Large Language Model) — This is what “AI” usually means here, and we use the two terms interchangeably throughout this report — “LLM” where precision matters, “AI” as the more familiar general term for the same thing. It’s the technology behind Claude, ChatGPT, and similar tools. The key thing to understand for this report: an LLM doesn’t “calculate” the way a calculator does. It generates its answer one word (or number) at a time, based on patterns learned from enormous amounts of text — closer to predicting the next word in a sentence than computing it. This turns out to matter enormously, and we come back to it throughout.

Ground Truth — The actual, correct answer for a given calculation. If we’re testing whether the AI can calculate a loan payment correctly, the ground truth is what that payment actually is, according to a method we trust. Part 7 below covers exactly how we established this.

Test Harness — The actual computer program built to run this whole test automatically: it generates the test cases, sends each one to the AI (or to Eliza’s own engine, depending on which method is being tested), records the answer, and checks it against the correct value. Every number in this report came from this same harness, run the same way, so results are directly comparable across methods.

Error — In this report, an error means the AI’s answer didn’t match the ground truth. We measured errors two ways, covered in Part 6 — because “wrong by one cent” and “wrong by five thousand dollars” are different kinds of wrong, even though both are technically errors.

Deterministic — A process that gives the exact same answer every time, no matter who runs it or when. A calculator is deterministic: 47 × 12 is always 564.

Probabilistic — A process that doesn’t guarantee the same answer every time — it produces the most likely answer based on patterns, closer to a knowledgeable person doing mental math and stating their best answer out loud. Most of the time they’re right. Occasionally, especially on a hard problem, they slip — and the confidence in how they say the answer doesn’t tell you whether this was one of those times.

This is the core idea behind the whole report: an LLM is probabilistic. A calculator, or any ordinary program doing arithmetic, is deterministic. Financial documents need deterministic answers. The question is whether an LLM, used the right way, can deliver that.

Token — The unit LLMs like Claude are billed by, roughly comparable to a word or part of a word. It’s also the practical measure of how much room an AI has to reason through a problem before giving an answer, referenced in Part 9. The cost figures in Part 11 are driven by token usage.

Sandbox — An isolated space where a computer program can run without affecting anything else. When we describe the AI “running code” later, it wrote a small program and that program executed inside a sandbox.

Stateless — Having no memory of anything that happened before. Each request to the AI in this report was handled entirely on its own, using only what was included in that specific request — nothing carried over from any earlier test case. Every AI call in this report, across every method that uses one, was stateless: this rules out one possible explanation for a wrong answer being that the AI was somehow influenced by a previous case, correct or incorrect. Every result reflects that one case, and that case alone.

Tool Call — A structured request the AI makes to run a specific, pre-written function — for example, “run the loan payment calculation with these numbers” — rather than computing an answer in its own reasoning. Part 3 and Part 4 cover where and how Eliza uses this.

Part 2 — Why Financial Math Needs Decimal Arithmetic

Ordinary computers don’t store numbers with perfect precision by default. If you ask a typical computer program to add 0.1 + 0.2, it can come back with 0.30000000000000004 instead of exactly 0.3 — this is simply how computers commonly represent decimal numbers internally (called “floating-point” arithmetic), and for most everyday uses it doesn’t matter at all.

It matters in a financial system. If a system tracks a business’s cash balance across 36 months, tiny errors like this can quietly accumulate, and by the end of a 3-year projection the numbers can be meaningfully wrong, with no visible sign of where it happened.

This is why Eliza’s own calculation engine uses decimal arithmetic instead — a mode of calculation designed to avoid this problem entirely, handling numbers the way a person doing math on paper would, with no hidden rounding drift. Every dollar figure Eliza produces is computed this way.

An LLM computing math in its own reasoning has a related problem, for a different underlying reason — covered next.

Part 3 — Four Ways We Tested the AI Doing Math, and One We Haven’t Yet

Method 1 — Unaided

Claude was given the numbers and asked to work out the answer directly, showing its work in sentences — no calculator, no code, only its own reasoning.

This tells us how the AI performs on math when left to its own devices, without any additional safeguard.

Every one of the 3,400 test cases sent to the AI — under both this method and Method 2 below — was stateless: a completely independent request, with nothing carried over from any other case, in either direction. No case could have been made easier or harder by whatever happened before it.

Method 2 — Code-Enabled (also called Code-Assisted)

Claude was given the same problems, but with access to a tool that let it write a small computer program (in Python) and run it in a sandbox to get the answer, rather than reasoning it out in sentences.

This is a common way AI systems are made “better at math” — let the AI write code instead of guessing at arithmetic, since code executes with perfect precision. The interesting question this report answers is whether that actually solves the problem, or only part of it.

Method 3 — Python Calls (also called Baseline)

For Eliza’s most critical calculations — the ones that go directly into a loan document — there is no AI reasoning involved in the math at all. A human engineer already wrote and tested the exact calculation in advance (the loan payment formula, the break-even formula, whatever it is), and Eliza runs that pre-written, pre-tested code directly, the same way every time.

This isn’t quite the whole story of how Eliza handles every number it produces. A small, deliberately limited number of decision points still involve the AI — not in doing math, but in recognizing which pre-written calculation applies. Part 4 covers exactly where, and why that’s a different and much narrower risk than anything Methods 1 and 2 measure.

Part 7 covers an important limitation in what comparing against this method can and can’t prove.

Method 4 — Tool-Mediated

In Method 3, the specific calculation to run is decided by our own system’s logic — a fixed rule like “once all loan details are collected, run the loan payment calculation.” In a tool call setup, the AI itself is given a list of available pre-written calculations and has to decide which one applies and correctly supply the right numbers to it, based on understanding a conversation rather than a fixed rule.

This is a different question from anything else tested — not “can the AI do math” (Methods 1 and 2), and not “is our code correct” (Method 3). It’s “can the AI reliably pick the right calculation and the right inputs for it.” Part 4 identifies exactly where this decision comes up in Eliza, and Part 5 reports the full results now that this has been tested.

The One Thing Still Not Tested: Does the Same Question Get the Same Answer Twice?

Every test case in this report was run exactly once. Because of what Part 1 explains about the AI being probabilistic rather than deterministic, it’s possible — in principle — for the exact same question, with the exact same numbers, to produce a slightly different answer on a different occasion, not because anything about the setup changed, but simply because of how the AI generates an answer in the first place.

We have not directly tested this. Doing so would mean taking the same test case and asking it many times — for example, 100 times in a row — to see how much the answer actually varies, and whether a case reported as “wrong” in this report is reliably wrong every time, or was one unlucky result among several possible answers. This is a real, open question this report doesn’t answer, and it’s the one thing deferred to the next phase of this work.

Part 4 — Eliza’s Hybrid Architecture: Where AI Judgment Is, and Isn’t, Involved

How much of Eliza’s real, production system actually matches Method 3 deserves a precise answer, since it’s not quite “all of it.”

The overwhelming majority of what Eliza computes works exactly like Method 3 above — pre-written, pre-tested code, with no AI reasoning involved in the math at all. But Eliza’s system does still ask the AI to make one kind of decision in a small number of places: given a conversation with a business owner, which pre-written calculation applies right now, and what numbers should be fed into it. This is Method 4 from Part 3 — a genuinely different question from anything else tested in this report, since it isn’t about whether AI can do math, it’s about whether AI can correctly recognize the right math to reach for.

We’ve identified where this happens. Directly reading the current system, rather than relying on an earlier summary, surfaces two separate groups worth keeping distinct: the places where the AI actually performs or triggers a calculation, and the places where it makes a real selection or classification decision that carries risk even though no math is involved. Several other candidates were considered and specifically excluded, covered at the end of this section.

Every Place Eliza’s AI Touches a Calculation

Eliza has exactly 5 calculations the AI can call at all. Here’s every one of them, in one place, ranked from safest to least settled:

1. Calculating startup costs — takes no AI-supplied input at all. The AI only decides when to trigger it; every figure comes from what’s already been collected. Fully safe. Not part of Method 4 testing below — there’s no AI-supplied input for a test to meaningfully exercise.
2. Calculating a loan payment — same: zero AI-supplied input. Fully safe. Not tested for the same reason.
3. Totaling a personal financial statement — same again: zero AI-supplied input. Fully safe. Not tested for the same reason.
4. Converting an equity percentage into a dollar figure — the one exception among the safe group. The AI has to correctly read a percentage out of natural phrasing (“50%,” “about half”) and has to correctly recognize this is the right tool to reach for, rather than simply recording a dollar amount if the business owner had stated one instead.

This one has real history behind it. The original interview simply asked for a dollar amount. When a business owner answered with a percentage instead, the AI would try to be helpful and calculate the dollar figure itself, in its own reasoning — the same failure mode described throughout this whole report, just happening during data collection instead of plan generation — and it got that conversion wrong in some cases. The fix, built before this report’s testing ever began, was exactly the principle this report has been building toward: don’t let the AI do the arithmetic itself, give it a dedicated calculation to call instead, so its only remaining job is recognizing which format the user answered in. Now tested: 100 of 100 correct. Full detail in Part 5.

5. Running the full financial plan — the single most arithmetically demanding calculation in the entire system (the full profit, cash flow, and balance sheet chain), yet the least settled of the 5. Not because that math is simple — it’s the hardest math Eliza does — and not only a matter of timing, either: a number of critical inputs are supplied directly by Eliza’s own system every time, with no AI involvement, but several other figures collected earlier in the conversation still have to be correctly carried forward by the AI into this one call. A dropped or mistyped figure here wouldn’t throw an error — it would just quietly produce a wrong plan.

Engineering note: eliminating the remaining transcribed fields — applying the same treatment already given to the loan-payment and startup-cost calculations, so every input to this call is server-supplied rather than AI-transcribed — is on the roadmap. This is deliberately not being changed while active testing of the current system is underway.

Now tested: 100 of 100 correct. Full detail, including a real cost finding specific to this one, in Part 5. One honest note: a clean 100-of-100 result narrows the risk described above but doesn’t retire it — the failure mode is a silent, unthrown error on a dropped field, which a 100-case sample could plausibly miss even if it occurs at a low rate in much larger real-world volume. The roadmap fix above remains the more permanent mitigation.

So the complete, current state: the two calculations that carry any real, identified risk — equity-percentage conversion and running the full financial plan — have now been tested, 100 real cases each, and both resolved every case correctly. The three fully-safe-by-design calculations weren’t retested here, since there’s no AI-supplied input for a test to meaningfully exercise.

Three More Places With Real Risk — But Not From Computation

Separate from the 5 calculations above, three more tools carry a genuine selection or classification risk, even though none of them involve any math at all:

1. Recording an individual answer during the interview. This is the single most frequently used AI decision in the whole system — it fires on nearly every turn of the conversation. It looks nearly identical, from the AI’s point of view, to a second, separate tool used for a different kind of note-taking, so there’s a real risk of an answer being filed under the wrong category. Because this happens so often, even a small per-turn error rate adds up. Now tested: 99 of 100 correct. Full detail in Part 5.
2. Classifying a business type. Which specific sub-type a business falls into isn’t always obvious from an owner’s own description, and this decision determines which set of interview questions and industry benchmarks get used for the rest of the conversation. This risk isn’t hypothetical: in earlier testing, the AI correctly recognized a genuinely ambiguous real case — a boutique clothing store, which could reasonably fall into more than one sub-category — and asked a clarifying question instead of guessing. Now tested: 100 of 100 correct. Full detail in Part 5.
3. Choosing between writing the next business-plan section or rewriting an existing one. Two very similarly-named, similarly-structured options exist for this, differing mainly in whether an already-written section gets skipped or redone, and the AI has to judge from context which one the business owner actually wants. Now tested: 100 of 100 correct. Full detail in Part 5.

Between the 2 compute-related risk points above and these 3 tools, that’s the 5 places that carry any real, identified risk in how Eliza’s AI touches a calculation or classification decision — all 5 now tested, 100 cases each, 500 total, at 499 of 500 correct. Part 5 has the complete results, including six real issues found and fixed in how the test itself was built before these numbers were trusted.

Considered and excluded, each for a specific reason: setting a business’s seasonal sales pattern (a real but narrow, contained decision), looking up a specific number from an already-generated plan (constrained to a fixed list of options, and a wrong pick returns a real number in the wrong place rather than corrupting anything), and recording the purpose of a loan (a real but lower-stakes judgment call).

Beyond all of this, Eliza’s AI has several other tools available to it, covering things like recording narrative answers and confirming extracted data — none carrying enough risk to have warranted testing priority. A complete inventory of all 16 tools, including which carry any kind of risk and which have been confirmed as actually used in real conversations, is in the Addendum.

There’s a Second Reason to Keep This Number Small, Beyond Accuracy

Correctness isn’t the only cost of asking an AI to choose among many pre-written calculations. There’s a separate, purely practical cost: every calculation the AI has to be able to recognize and choose between has to be described to it, and that description takes up space in every single request — whether or not that particular calculation ends up being used.

We measured this directly during the Code-Enabled testing in this report. Simply making one additional capability available to the AI added a fixed charge to every single question asked — not because the capability was used, but just because it existed as an option. That cost applies per capability, and it stacks: an AI system that has to keep track of all 34 of Eliza’s calculations as live options, rather than 5, would pay that overhead 34 times over, on every single request, for capabilities that are relevant to that specific request only a small fraction of the time.

This is a genuine, industry-wide problem for AI systems built to juggle many tools or capabilities at once, and it isn’t something that gets solved automatically just by using better AI or newer software — it’s a direct, mechanical consequence of how many options an AI has to be told about before it can choose between them. Part 5’s cost results, now that Method 4 has actually been tested at scale, confirm this same pattern held in practice.

Why We Think This Is the Right Way to Build a System Like This

Putting the two reasons together — correctness and cost — leads to a specific, deliberate architectural principle, and we’d rather state it plainly than leave it implicit:

Don’t ask an AI to choose between many options as a matter of course. Identify precisely where a real choice exists, keep that list as short as possible, and test those specific points thoroughly. Everywhere else, decide the calculation in advance with ordinary, fixed logic, so the AI is never given a decision it doesn’t actually need to make.

This isn’t the same as trying to eliminate AI judgment entirely — that’s not fully possible, since understanding what a business owner actually means in a real conversation requires interpretation, somewhere. And it isn’t the same as trusting AI judgment everywhere either, which this whole report shows carries real, measurable risk. It’s a middle position: make the set of AI-mediated decisions as small and as explicitly identified as possible, and put real testing effort into exactly that small set, rather than spreading trust thin across everything a system does. Having built the testing approach described in this report meant we could check the 5 specific places identified above directly, rather than assume they were fine because everything else tested well. Part 5 reports exactly what we found.

Part 5 — Tool-Mediated Testing: Full Results

Part 4 identified 5 places in Eliza where the AI makes a real selection or extraction decision rather than following a fixed rule — 2 involving triggering a calculation (converting an equity percentage, running the full financial plan) and 3 involving classification or selection with no math involved (recording an interview answer, classifying a business type, choosing between writing or rewriting a business-plan section). This section reports what happened when all 5 were actually tested.

How This Was Tested

Each of the 5 was tested with 100 realistic, varied scenarios — 500 test cases in total. Unlike Methods 1 and 2, which handed the AI a pre-formatted math problem, these scenarios presented the AI with something closer to a real conversation: a business owner’s actual phrasing, including cases deliberately built to be ambiguous, alongside a set of plausible alternative tools the AI could have reached for instead of the correct one — testing genuine selection risk, not just whether there was only one button to press.

A case was scored correct only if the AI both picked the right tool and supplied the right arguments to it.

The Results

The one exception isn’t a mistake — and a closer look afterward revealed something more precise than originally reported here. In that single case, the AI asked a crisp, directly answerable question about a genuinely confusable term (“utilization rate” could mean a billable-hours ratio or a staff-cost percentage) — but the scripted reply our test gave in response didn’t actually answer the specific question the AI asked, so the AI correctly kept asking rather than guess, and the case ran out of the clarifying rounds we’d budgeted.

We followed up on this one case afterward, separately from the official result above. With a scripted reply that actually answered the AI’s question, the case resolved correctly — though in a single round this time, not the third round we expected to need. This test runs with no temperature setting, deliberately, to match production (see the Addendum) — which means the exact same scenario can genuinely behave somewhat differently from one attempt to the next, the same probabilistic property defined in Part 1, showing up directly in a live example. The honest reading isn’t “the issue is fixed” — it’s that the original non-resolution was one legitimate sample of normal variance, on a scenario where our own scripted reply happened to be flawed, not evidence of a hard limit in the AI’s reasoning. This is also a small, concrete preview of the one thing Part 3 flags as still untested — whether the same question reliably gets the same answer twice.

This doesn’t change the official result. 499/500 stands exactly as reported, under the same fixed methodology applied uniformly across all 500 cases. It’s disclosed here because a more complete, more accurate explanation became available afterward, and leaving the earlier, less complete account standing unchanged would have been less honest than updating it.

Cost

The full 500-case test cost $25.71 — measured directly from the actual run, not a projection. An earlier, much rougher estimate for testing this kind of risk had suggested a cost closer to $200; the real, carefully-scoped test came in at roughly one-eighth of that.

Running the full financial plan was the clear cost outlier — several times more expensive than the other four tools, for a real, structural reason rather than a test artifact. That calculation returns a large, complete result (the full computed plan), and the next step of the conversation has to carry that entire result forward — the same way a real Eliza conversation would. This isn’t a cost that could be engineered away in the test without also changing how the real system behaves; it’s an accurate reflection of what running this specific calculation actually costs.

What We Found and Fixed Before Trusting These Numbers

The first pass through this test did not look this clean — one tool scored as low as 35 of 100 correct on the initial run. Rather than report that number, we investigated every failure before trusting it, the same discipline applied to every other result in this report. Six real issues were found, all in how the test scenarios were built, not in the AI’s actual behavior:

1. Some scenarios never gave the AI a chance to receive a user’s confirmation before scoring the case — the AI was correctly asking a clarifying question, then being marked wrong for not completing a call it was never given the opportunity to complete.
2. A small number of scenarios needed a second round of confirmation, and the test initially only allowed one.
3. One test case’s expected answer was written in a truncated form (“33”) that didn’t match a mathematically correct, more precise answer the AI reasonably gave instead (“a third,” meaning 33.33…%) — the AI was right; the answer key was wrong.
4. One scenario used a genuinely ambiguous phrase that a reasonable person could read two different ways; the AI’s “incorrect” answer was a reasonable interpretation of confusing wording.
5. A handful of scenarios asked for a vague edit (“make it punchier”) and the AI reasonably asked for more specific direction rather than guessing what that meant — appropriate caution, not an error.
6. At one specific boundary case (a business with a 0% equity contribution), two different tools could reasonably record the correct answer; both were counted as correct rather than penalizing a defensible choice.

Each of these was fixed at its source — the test scenario or the answer key, never the AI’s actual response — and every affected case was re-run for real. The results above reflect that corrected, re-run data. We’re disclosing this the same way we disclosed the similar issues found earlier in this report’s original benchmark (Part 8): a test that quietly corrects its own mistakes without saying so is less trustworthy than one that shows its work.

Part 6 — How We Decided Whether an Answer Was Wrong

We used two standards.

Exact Match — Did the answer match the correct value to the penny (for dollars) or exactly (for other figures)? The strict standard.

Within Tolerance — Was the answer close — within $1, or within 1%, whichever is more forgiving? This catches cases where the AI clearly understood the problem and used the right method, but slipped somewhere in the arithmetic.

We report both numbers throughout, and never counted a close-but-wrong answer as a pass — both figures are always shown side by side. An AI that’s frequently close but rarely exact tells a different, more specific story than one that’s just randomly wrong, and Part 10 explains why that distinction turned out to matter.

Why does “close” deserve its own category at all, rather than simply being wrong? The natural reaction here — especially from anyone with an accounting background — is that a number is either correct or it isn’t, and there’s no partial credit in a financial statement. We agree with that instinct for anything that actually reaches a real document: Eliza’s own output is held to the exact standard, every time, with no tolerance applied anywhere — that’s what Method 3 delivers, and it’s the only standard Eliza’s real output is ever measured against.

The tolerance measurement in this report serves a narrower, different purpose: not lowering the bar for what’s acceptable, but diagnosing why an AI-computed answer was wrong when it was. An error that’s off by a few cents because of a precision issue in the underlying math is a fundamentally different problem than an error that’s off by a large amount because the AI misunderstood the calculation entirely — and as Part 10 shows, these two kinds of errors respond very differently to being given better tools. Knowing which kind of wrong you’re looking at is genuinely useful diagnostic information, even though, for anything that actually matters, wrong is wrong.

Part 7 — How We Built the Test, and How We Know the Answer Key Is Right

How many calculations did we test? Eliza performs 34 distinct types of financial calculations — a monthly loan payment, a restaurant’s projected monthly revenue, a 3-year cash flow, and so on. For each of the 34, we generated 100 different test scenarios — different loan amounts, different interest rates, different business sizes — for 3,400 individual test cases in total.

How did we make sure the scenarios were realistic? Each set of 100 included a mix of typical, realistic values (the kinds of numbers a real business owner would enter) and boundary or edge cases worth stress-testing specifically (a $0 interest rate, a business with barely enough income to qualify).

How did we compute the correct answer for each test case? Two different ways, and both deserve explaining, since relying on only the first would have left a real gap.

The first, and the source of every ground truth figure used to score every result in this report, was running Eliza’s own real calculation engine directly — the same Method 3 logic described earlier. This is fast and complete (all 3,400 test cases), but has a limitation we’d rather state directly than gloss over: since Method 3 is also Eliza’s own code, comparing Method 3 against this ground truth only proves the code is consistent — it behaves the same way every time it runs — not that it’s correct against some outside standard. If Eliza’s engine had a genuine error in how a calculation is defined, this approach alone would never catch it, because every comparison would still ultimately be checking against Eliza’s own logic as the reference.

To close that gap, we built a second, completely independent reference — using only outside sources (spreadsheet software’s own trusted, built-in financial functions, and standard accounting definitions), never looking at Eliza’s own code while building it — and checked Eliza’s engine, along with both AI methods, against it.

The result, for everything checked so far: complete agreement. Every one of the most universal calculations — loan payments, break-even points, basic ratios, and similar — plus every longer calculation we were able to independently reconstruct, including the debt service coverage ratio, matched Eliza’s engine exactly. In total, well over a thousand independent comparisons, with only a small handful of apparent disagreements found along the way — and every one of those turned out to be a mistake in the new independent reference itself, not in Eliza, caught and corrected before being reported as a finding.

What’s not yet covered by this independent check: revenue and cost formulas for service-based businesses, the exact shape of a retail business’s ramp-up curve (limited by how much data was available to reconstruct it from, not a disagreement), one figure on the balance sheet — owner’s equity, which didn’t resolve cleanly in either of two independent attempts and is being looked at further — and the exact mechanics of scenario scaling for one calculation. These remain checked only against Eliza’s own code, with the limitation described above.

One more point deserves plain statement: even this independent check was built and run by the same team that built Eliza, not by an outside party with no stake in the result. That’s a real, permanent limitation on how much confidence any of this can buy — a genuinely independent third party reviewing this from scratch would be a stronger form of assurance than anything described in this report.

Part 8 — Keeping the Test Itself Honest

Two more items belong here, briefly.

While building this test, we found and fixed four cases where the instructions given to the AI were missing a rule Eliza’s real system actually follows — for example, one calculation’s instructions initially left out a business’s seasonal sales pattern entirely, so the AI reasonably assumed flat sales all year and got that calculation wrong every time until we caught it. In every one of these four cases, this was a gap in our own test’s instructions, not a mistake by the AI, and all four were fixed before the results in this report were produced.

Separately, a handful of Eliza’s calculations involve a deliberate, considered simplification rather than one single textbook-correct approach — for instance, loan interest is recalculated once a year rather than every month, and a loan payment too small to cover the interest owed simply stops the balance from moving further, rather than letting it grow. These aren’t mistakes; they’re intentional design choices, and the independent verification work in Part 7 confirmed several of them are implemented exactly as designed.

Part 9 — How We Grouped the 34 Calculations by Difficulty

Not all 34 calculations are equally hard. Some are a single, quick calculation. Others require reasoning across a full 36-month projection, with numbers that build on each other month after month.

We classified difficulty using a precise, measurable standard: the actual number of tokens each calculation used, on average, when the AI computed it Unaided — not a judgment made in advance about which calculations seemed likely to be harder. Calculations averaging 4,000 tokens or fewer were classified as Simple. Calculations averaging between 4,001 and 12,000 tokens were classified as Medium. Calculations averaging more than 12,000 tokens were classified as Complex.

Simple Calculations (21)

• Calculating a monthly loan payment from the loan amount, interest rate, and term
• Sizing a loan amount for a new business
• Converting a percentage (“I’ll put in 50%”) into an exact dollar amount
• Loan interest tiers, recalculated annually
• Restaurant daily covers, derived from seating and turnover
• Multi-year growth factor
• Adding up a business’s regular monthly bills into one fixed-cost total
• Restaurant break-even analysis
• Restaurant startup costs
• Services and retail startup costs
• Services accounts-receivable working capital
• Retail inventory and payables working capital
• Adding up a person’s personal assets, debts, and net worth for a loan application
• Turning figures from an uploaded financial statement into percentages (such as “food costs are 31% of revenue”)
• Figuring out how many months old an uploaded financial statement is
• Converting figures reported yearly into a monthly figure, or vice versa
• Operating-business revenue and costs
• Sizing a loan for an already-operating business based on its stated need
• Break-even analysis and capital-expense sizing for an already-operating business
• Scenario modeling for an already-operating business
• Operating-business debt service coverage

Medium Calculations (6)

• The first-year debt service coverage ratio for a new business — the number lenders use to judge whether a business can afford its payments, requiring a full year of income totaled first
• Month-by-month restaurant revenue, including seasonal swings and a new restaurant’s ramp-up period
• Restaurant cost of goods sold
• The complete profit-and-loss chain — from revenue down through costs, taxes, and loan payments, to a business’s actual take-home profit, for each of 36 months
• Services and retail break-even analysis
• Balance sheet for an already-operating business

Complex Calculations (7)

• Month-by-month services-business revenue
• Month-by-month retail revenue
• Services cost of goods sold
• Retail cost of goods sold
• Full 36-month cash flow tracking
• Balance sheet for a brand-new business
• Scenario modeling (better-case/worse-case) for a brand-new business

Part 10 — The Results

The Headline Numbers

Two patterns stand out here.

The gap between exactly right (63.5%) and close (93.8%) for Unaided is itself informative. Most of the time, the AI understood the problem and used the right approach — it lost precision somewhere along the way doing the arithmetic, similar in spirit to the rounding problem in Part 2, for a different underlying reason. It wasn’t confused about what to calculate. It wasn’t precise enough calculating it.

Writing and running code closes most of that gap, but not all of it — and the reason why matters more than the number.

What “Wrong” Actually Looked Like, in More Detail

Of the roughly 36% of Unaided answers that weren’t exactly right, that number breaks down further, because “wrong” wasn’t one single thing.

91% of those wrong answers were still close — within the $1-or-1% tolerance described in Part 6. The AI’s approach and understanding were sound; the final number was just slightly off.

Only about 3% of all 3,400 test cases were genuinely, meaningfully wrong — not close by any reasonable standard.

A separate 3% never finished at all, running out of the tokens we allowed for the AI to work through its reasoning before giving a final answer. This is a distinct thing from being wrong — it’s more like someone still doing long division on paper when time runs out, rather than confidently writing down an incorrect final answer. Interestingly, this happened even on some of the simpler calculations, not only the hardest ones — a reminder that even ordinary compound-interest math can occasionally push the AI into extensive manual double-checking.

Why Code-Assist Doesn’t Fully Close the Gap

We found three distinct ways the AI got a wrong answer.

Way 1 — Losing precision on hard math. In one clean example, the AI was calculating a loan payment involving a difficult multiplication step. In one version, it kept enough decimal places and was off by 1 cent per month — a $3 error over the life of the loan. In an almost identical problem, it kept fewer decimal places and was off by $16.91 per month — over $5,000 over the loan’s life. Same AI, same type of problem, same method — a small difference in how carefully it carried the numbers produced a large difference in the final answer. Writing and running code mostly fixes this, since code doesn’t lose precision the way manual calculation does.

Way 2 — A single wrong addition or multiplication that then repeats. In one case, a $12,010.85 error in a business’s annual profit traced back to exactly two small slips: adding up eight cost items and getting a total $1,000 too high (repeated across all 12 months), plus one multiplication off by $15. Everything else was correct — two specific, isolated slips. This is also mostly fixed by code execution.

Way 3 — Using the wrong formula entirely, while sounding completely confident. This is the one code execution doesn’t fix, and it’s the most important finding in this report. In one case, a growing business’s numbers should increase in clean, yearly steps — like a staircase. Instead, the AI’s code computed a smooth, continuously-growing curve — a plausible-looking idea, but not the actual rule. At the exact points where the staircase and the smooth curve happened to cross, the wrong method coincidentally produced the right number — meaning someone skimming the figures might never notice anything was off.

This is not a math mistake. It’s a misunderstanding, expressed with total confidence. Giving the AI a code-writing tool made this specific kind of mistake worse, not better: two calculations that got the exact right answer about a third of the time under Method 1 dropped to landing genuinely, meaningfully wrong more than 80% of the time once the AI could write code — because the same wrong idea, once turned into working code that ran without error, looked more trustworthy and got used more consistently than the same wrong idea expressed in uncertain sentences. Both affected calculations landed at exactly the same result under Code-Enabled testing (18% exactly right, 82% genuinely wrong), which is itself informative — it’s a strong sign of one shared, well-understood mechanism, not two unrelated coincidences.

The core lesson: giving an AI a tool to execute math precisely doesn’t guarantee it picked the right math to execute. Those are two separate problems, and only one is solved by letting the AI write code.

Could We Have Predicted, in Advance, Which Answers Would Be Wrong?

A system’s mistakes matter less if you can spot them coming, so we checked directly. We checked several possible warning signs against the “wrong formula” errors (Way 3 above) specifically: did the AI’s response happen to be unusually long or short, did it get cut off before finishing, was the calculation one of the harder ones.

None of these turned out to be a reliable warning sign. Wrong answers weren’t noticeably longer or shorter than correct ones. None of them were cases that ran out of room — they finished normally, confidently, and were still wrong. There was no clear pattern by calculation type either. In other words: there was no shortcut available, like “just double-check the short answers” or “just double-check the complicated ones,” that would have reliably caught these specific mistakes in advance. This is a more honest and more useful finding than either “it’s unpredictable” (too strong) or “we can flag it” (not true, at least not with anything we checked) — the accurate statement is that we looked for a warning sign and didn’t find one.

Did These Errors Actually Matter for Real Lending Decisions?

We checked this for the two figures that most directly decide whether a business qualifies for a loan.

Debt service coverage ratio — the number a lender uses to judge whether a business can afford the loan. Even when the AI’s exact number was slightly off (Unaided method), we checked whether it landed on the correct side of the pass/fail line lenders actually use. Across 187 test cases, this never went wrong — 0 times out of 187 — even though the exact number was sometimes imprecise.

Break-even revenue versus a business’s actual projected revenue — a second, related check on whether a business appears financially viable. Across 200 test cases, this matched the correct side 199 times. The one exception was traced back to the same “wrong formula” issue described above (Way 3), not a new or different kind of mistake.

On both of the numbers that most directly affect whether a business appears to qualify, being a little off almost never changed the actual outcome.

Does Difficulty Matter? Simple, Medium, and Complex Calculations, Compared

Part 9 classified all 34 calculations into three difficulty tiers, based on the actual number of tokens each one used, on average, under the Unaided method — not a judgment made in advance. Averaging the exact-match results across each tier separately shows a clear, graded pattern:

The gradient under Unaided is exactly what the rest of this report would predict: accuracy drops steadily as a calculation requires more reasoning steps. What’s more interesting is what happens once code execution is introduced, because it doesn’t lift all three tiers by the same amount. Medium calculations get almost fully rescued — from well under half right to essentially matching the Simple tier. Complex calculations get a large, genuine improvement, but a real gap remains — more than 1 in 5 still comes back wrong, even with code doing the arithmetic.

This fits directly with the three failure mechanisms described earlier in this report. Precision decay and flat arithmetic slips (Ways 1 and 2) are execution problems, and code execution is very good at eliminating both — that’s most of what closes the Medium tier’s gap almost completely. But the wrong-formula problem (Way 3) is an understanding problem, not an execution problem, and it has more room to occur the more steps a calculation involves. The Complex tier — by definition the calculations with the most steps and the most places for a wrong idea to take hold — is exactly where that third mechanism has the most opportunity to appear, and the numbers here bear that out directly.

Part 11 — Cost and Time

Cost, precisely measured:

• Unaided method, all 3,400 test cases: $283.81
• Code-Enabled method, all 3,400 test cases: $300.25
• Python Calls (baseline), all 3,400 test cases: $0 — this method ran entirely on our own computer, using ordinary code with no AI service involved, so there was nothing to bill.
• Tool-Mediated method, all 500 test cases: $25.71 — Part 5 has the full breakdown.

One honest surprise here: we expected code to be cheaper than reasoning in sentences, since code is a much shorter way to express a calculation. It came out roughly the same instead. We traced this to a fixed cost — simply making the code-writing tool available adds a flat charge to every question asked, whether or not the AI ends up using it. We identified a technique (“caching”) that would likely reduce this, but haven’t run a second test to verify the actual savings, so we aren’t reporting a number for it here.

Time: Methods 1 and 2 (Unaided, Code-Enabled) each took several hours to work through their full 3,400 test cases, run overnight. The baseline completed all 3,400 test cases in under 5 minutes. The Tool-Mediated method completed all 500 test cases in about 5.5 minutes. This is the same finding as the rest of this report, in a different form: ordinary code that already knows the calculation runs enormously faster, as well as more reliably, than asking an AI to work it out each time.

Total across all four methods: 3,900 test cases, at a combined cost of $609.77.

Part 12 — Bottom Line

1. An AI left to do math on its own gets the exact right answer roughly two times out of three. Most mistakes are small — 91% of the wrong answers were still close, not wildly off. Only about 3% of all test cases were genuinely, meaningfully wrong, and a separate 3% ran out of room before finishing at all — a different and more visible kind of failure than being confidently wrong.
2. Letting the AI write and run real code fixes most, but not all, of that — pushing accuracy to about 95%. What it doesn’t fix is when the AI misunderstands the underlying rule in the first place, and in those specific cases, code execution can make it more confidently wrong, not less.
3. We could not find any reliable way to predict, in advance, which specific answers would be wrong — not response length, not whether the calculation was simple or complex. This means the safest assumption is that any individual unaided or code-assisted answer could be the wrong one, regardless of how it looks.
4. On the two numbers that most directly decide whether a business qualifies for a loan — debt service coverage and break-even viability — imprecision almost never changed the actual pass/fail outcome, even when the exact figure was off. This is a genuinely reassuring finding, specific to those two numbers, and shouldn’t be read as extending to every possible use of an imprecise figure.
5. Eliza’s most critical calculations — the ones that go directly into a loan document — never ask AI to compute the answer at all. That’s Method 3, and it’s why it matched every one of the 3,400 test cases. A small, explicitly identified set of 5 decision points, where AI has to recognize which pre-written calculation applies rather than compute anything itself, have now been tested with the same rigor as everything else in this report, at 499 of 500 (99.8%) correct — Part 4 covers exactly where these are, and Part 5 has the full results.
6. We’ve now independently verified the great majority of Eliza’s calculations against an outside reference, not just against Eliza’s own code — including the debt service coverage ratio specifically. What’s not yet independently checked: service-business formulas, one retail ramp-curve detail, one balance sheet figure (owner’s equity), and one scenario-scaling detail — all still verified only against Eliza’s own code, with the limitation that implies. Part 7 has the full detail.
7. One thing remains deliberately untested: whether asking the exact same question, with the exact same numbers, reliably produces the exact same answer every time. Every result in this report reflects each test case being run once. Tool-selection and argument-extraction reliability — previously the other open item on this list — has now been tested; see item 5 above and Part 5.

Addendum — Test Methodology and Reproducibility

This section summarizes the technical configuration behind every result in this report, for anyone wanting to understand exactly how these tests were run.

Model. All AI-based testing (Methods 1, 2, and 4) used the same Claude model Eliza runs in production — no different or newer model was introduced specifically for testing.

Temperature. Methods 1 and 2 (Unaided, Code-Enabled) were run with temperature set to 0 — the setting that produces the most consistent, least varied output, appropriate for isolating computational reliability specifically. Method 4 (Tool-Mediated) was run with no temperature parameter set at all, matching production Eliza’s actual configuration, since this method specifically measures judgment consistency and needed to reflect what real customers experience rather than an artificially stabilized setting.

Token budgets. Methods 1 and 2 were allotted either 4,000 tokens (for calculations classified as Simple) or 32,000 tokens (for Medium and Complex calculations) per case, based on the actual reasoning room each calculation required. Method 4 cases were allotted 2,048 tokens per turn.

Software development kit. Testing used Anthropic’s official Python SDK, version 0.116.0.

Test harness execution environment. All tests were orchestrated by a custom Python test harness, running inside Eliza’s own test infrastructure, with test cases executed in parallel (6 concurrent workers for Methods 1 and 2; 10 for Method 4).

Scoring standards. Methods 1 and 2 were scored against two standards — an exact match to the correct value, and a secondary “within $1 or 1%” tolerance band (Part 6 has the full detail on why both are reported). Method 4 was scored as correct only if both the right tool was selected and the right arguments were supplied to it.

Scope. 3,900 individual test cases in total — 3,400 for Methods 1 through 3 (34 calculations × 100 cases each), and 500 for Method 4 (5 decision points × 100 cases each).

What this doesn’t cover. This summary describes how our own tests were configured. It does not describe Anthropic’s own infrastructure for running the model itself (server hardware, data center location, and similar), which isn’t visible to us as an API customer and isn’t required to reproduce these results — reproducing this testing requires the model, the parameters above, and access to the same API, not information about the infrastructure behind it.

Addendum — Complete Tool Inventory

Part 4 covers the 5 places where the AI performs or triggers a calculation, and the 5 places where it makes a genuine selection or extraction decision (with some overlap between those two groups). This table is broader: every tool the AI has available to it during the interview, for completeness, whether or not it’s part of either group above.

Of the 16, 15 are confirmed used through direct evidence in real session data. The one exception — rewriting an existing business-plan section — isn’t confirmed unused; no record in this system distinguishes it from writing a new section, so its usage genuinely can’t be determined either way with the data currently available.

Addendum — Formula-by-Formula Results

This table shows the exact-match result for every one of the 34 calculations, under both AI methods, from the final, fully corrected dataset. A more detailed version — including the “close” and “did not finish” breakdowns for every formula, not just exact-match — is available as a separate spreadsheet on request.

The two bolded rows above call for direct explanation. Retail’s revenue and cost-of-goods calculations are the sole exception to the pattern in this entire table: every other calculation improves substantially, or is already perfect, once the AI can write and run code. These two get worse — a documented finding, not a data error, and explained in Part 10 above (the “Way 3” example — the AI’s wrong conceptual understanding of one specific calculation gets expressed more confidently as code than as prose, not corrected by it).

Operating-business balance sheet (86% exact under Code-Enabled, versus 100% for nearly everything else) is a smaller, separate imperfection, though far less severe than the retail pattern above.