← Back to notes

2026.01.12

Why most product audits miss the actual product

Most "technical due diligence" reports are a checklist wearing a suit. Dependency versions, header configs, a CVE scan — useful, but none of it answers the question an acquirer or a founder actually has: does this thing work the way it's supposed to, under the conditions that matter?

That question doesn't live in any single layer of a codebase. It lives in the gap between what the code does and what it was supposed to do — and that gap is invisible to any tool that only looks at one layer.

Layer 1: static analysis tells you about the code, not the product

Semgrep, dependency scanners, secret scanners — these are pattern matchers. They're good at what they do: an outdated JWT library, a hardcoded API key, a SQL string built with concatenation instead of parameters. All real, all worth catching, all catchable by a machine that has never run the application once.

What they can't catch is anything that requires knowing what the code is for. A discount-code redemption function that never checks whether a code has already been used isn't a syntax problem — every line of it is syntactically fine. It's a missing business rule, and a static scanner has no model of what business rules should exist.

// passes every static scanner. still loses money.
function applyDiscount(order: Order, code: string) {
  const discount = findDiscount(code);
  // no check that `code` hasn't already been redeemed by this user
  return applyTotal(order, discount);
}

Layer 2: dynamic crawling tells you what happens, not what should happen

This is the part most audits skip because it's expensive: actually driving the product like a user would. AudiX does this with Playwright — real sessions, real forms, real multi-step flows, not a smoke test that checks whether the homepage returns a 200.

But a crawl on its own has the opposite problem from a static scanner: it generates an enormous amount of true observations ("clicking here does this", "this endpoint returns that") with no way to know which of those observations is a bug. Behavior without a spec to compare it against is just a transcript.

Layer 4: synthesis is where the finding actually happens

This is the part that's easy to describe and hard to build well: taking the Playwright execution trace and cross-referencing it against what the code claims it should do — inferred from the schema, the validation logic, the naming, the comments, sometimes the commit history — using an LLM as the reconciliation layer, not the source of truth. The model isn't inventing findings from nothing; it's flagging the specific places where observed behavior and declared intent disagree, and it has to cite the exact request/response pair and code path that shows the mismatch. A finding without a reproduction path is a guess, and guesses don't survive a client pushing back on a report.

[FILL IN: a specific finding or bug class the pipeline caught in a real audit that a single-layer tool (just Semgrep, just a crawler, just an LLM reading the code) would have missed — the more concrete and embarrassing for the target product, the better the essay lands.]

What this means practically

If you're evaluating a product — as an acquirer, an investor, or the team about to inherit it — a report that's 100% static analysis or 100% "we poked around and it seemed fine" is answering a different, easier question than the one you're actually asking. The real answer requires all three layers talking to each other, and most audits stop after the first one because it's the cheapest to automate.

Evidence beats assumption. A finding without a reproduction path is a guess.

That's the whole thesis behind AudiX's L1 → L4 pipeline. More on this once the synthesis layer has a public track record to point to.