Spot-Checking AI in Accounting Software Is Not Testing It
A read-through that looks right is a check, not a test. The difference, what NIST says about measuring AI, and where the testing burden lands.
An AI assistant drafts a reconciliation note, codes a batch of transactions, or summarises a stack of invoices. Somebody reads the output, it looks right, and the work moves on. That review is a check. A check is not a test, and the difference is simple: a test defines pass and fail before the run, against answers you already hold.
A check samples, a test compares
A check samples one output. You read what the model produced this once, on this file, and judge it against your own sense of plausibility. NIST's Generative AI Profile, a companion resource published in July 2024, lists confabulated and unfaithful output among the risks specific to generative AI. In plain terms, output that reads fluently and is nonetheless wrong is a documented failure shape, not an edge case. A reader's sense of plausibility is therefore a weak instrument.
A test differs in kind, not degree. It is a fixed set of inputs whose correct outputs you hold in advance. You run the tool against that set and compare the two before anything touches a client ledger.
Run it again and the answer can change
The second weakness is nondeterminism. The same prompt can return a different answer on a different run. OpenAI's API documentation notes that even with its randomness setting at zero, outputs are only mostly deterministic, and a small amount of variability remains.
An output that passes review on Monday therefore tells you little about Tuesday's batch. One clean run is a sample of one, from a process that does not promise to repeat itself.
How does NIST define assessment?
The US standards body NIST released its AI Risk Management Framework on 26 January 2023. The framework is voluntary. It still supplies a usable definition: four core functions, named govern, map, measure and manage.
Measurement sits among them. In the framework's own terms, a system that has never been measured has not been assessed. A firm relying on an AI feature without any measurement is outside that definition, whatever the review felt like on the day.
Balanced totals can hide wrong answers
Ledger work adds a trap that other domains lack. Arithmetic can reconcile while the substance is wrong. A mis-coded batch still foots. A transaction posted to a valid but wrong account passes every validation rule the software owns. A summarised invoice can drop a line and still read cleanly.
The checks you lean on mid-close, footing, cross-casting and tie-outs, are precisely the checks this class of error survives. Reviewing AI output with those habits imports a false sense of comfort.
Which tasks can be tested first?
Some work is naturally testable. Bank reconciliations, recurring coding decisions and prior-period returns all have known right answers, and you already hold them in your own history. That history is the test set.
Work without a single knowable answer, such as drafting narrative or summarising correspondence, resists testing. It can still be delegated, but with narrower expectations, because nobody can define a pass condition.
Ask what the vendor measured
A release note may describe what a feature does without saying how well it does it. Four questions separate a capability claim from a measured one:
- accuracy on ledger tasks like yours
- the data the evaluation used
- the stated failure rate
- how a regression would be caught
The practical reading, as we see it, is simple. If the vendor cannot point to its own evaluation, the testing burden sits with the firm. Silence is not evidence of failure, but it moves the work to your side of the table.
The line that matters
Checking asks whether this output looks right. Testing asks whether the tool reliably produces the right answer on work like yours. Only the second question can be answered before a deadline is riding on it.