Why detection has no LLM in it

The claim, narrowed to something checkable

No model in this product decides that a signal fires. No model decides that signals correlate. No model computes the score. No model suppresses, closes or re-ranks a finding. Those four sentences are the claim, and they are verifiable by reading the code rather than the documentation.

What follows is why, in the order the problems actually bite: reproducibility first, because it poisons everything downstream, then explainability, then the input channel you would be opening, then cost, then testability.

A score you cannot recompute is not a score

Take a finding that misordered a queue for a quarter and ask a reviewer to check it. With arithmetic they can: a CRITICAL base of 9.0, two contributing signals adding 0.5, a crown-jewel multiplier of 1.6, a cap at 10.0 — the argument is legible and the counter-argument is a specific input they can disagree with.

With a model in that seat, the stored artefact is a number and a model version. The same request tomorrow may return something else, and the number that misordered the queue has nothing to say for itself. Worse, a provider's model update is not an event in your incident record. Your queue changes shape without a changelog entry you can point at, and the only way you notice is that triage volume looks different this month.

This is the argument that usually gets answered with a temperature setting. Temperature zero makes a single request more stable; it does not make an endpoint reproducible across model revisions, providers, quantization or serving stacks, and it does not give you a way to audit last quarter's ordering.

An unexplainable queue item is a queue nobody trusts

The engineer who owns the affected system asks one question: why is mine at the top. A workable answer names the mechanism — a workflow in this repository funnelled a deploy credential to an address no job here has ever contacted, and both signals are attached to the incident. An unworkable answer is a probability with no inputs.

The consequence is not hurt feelings. A detection team has exactly one currency: the belief that the top of the queue deserves attention. Once people conclude the ordering is arbitrary, they work the queue in arrival order, and a ranked queue has quietly become a log with extra steps.

There is a subtler version of the same problem. A model's explanation for its own ranking is generated text, not the reason. It reads plausibly whether or not it corresponds to what happened, which is worse than no explanation for a reviewer who cannot tell the difference.

Triage is an input channel, and the attacker writes to it

Look at the fields a detector handles. Repository and branch names, workflow filenames, commit messages and author strings, package names and versions, tags, certificate subject fields, log lines, tool descriptions, email bodies. Every one of them is attacker-influenced, and several are attacker-authored.

October's survey recorded exactly this in the wild: phishing mail carrying hidden prompt injections — HTML comments, invisible CSS, zero-width characters — aimed not only at the recipient but at the assistant summarising their inbox. One message, both targets. The technique transfers to any text a triage model reads.

If the model can change what fires, what is suppressed or what is ranked first, the injection is not a content problem. It is a write path into your decision process, supplied by the adversary, and the only stable defence is that the decision does not read text. That the product's own advisory prompt builder splices finding metadata into a prompt is precisely why that module is advisory and why nothing in the correlation path imports it.

Token cost scales with the attacker's activity

Detection has an unusual cost profile: the expensive case is the noisy one. A quiet estate needs almost no compute, and the week a campaign produces thousands of events is the week you need the pipeline to be cheap, fast and boring. Per-token inference inverts that, charging the most at exactly the moment you are least able to slow down.

Then the inversion becomes a lever. If your own triage burns tokens per finding, then making findings is a way to spend your money. The pattern is already established against model budgets — the x47.c botnet shipped an AI API drain command, and a Symantec honeypot capture instructed a hijacked endpoint to work through up to 100,000 steps while the model's owner paid for the compute. Adding a metered model to the detection path hands the same lever to anyone who can generate signal.

Caching, batching and provider-side caps reduce the bill. None of them change the shape: cost proportional to activity rather than to risk.

A rule has a regression test. A prompt has a sample.

A rule here ships with an id, a matcher or a chain function, its evidence fields, its ATT&CK mapping, a place in the index array the engine iterates, and a unit test. The registration guard fails the build if a rule was authored but never wired in, and it fails on duplicate ids — so the dead-code failure class, a perfectly good detector that silently never loads, is a red build rather than a coverage gap discovered during an incident.

Model behaviour cannot be pinned that way. You can measure an accuracy rate over a fixture set; you cannot assert that these inputs produce this output and hold that assertion for a year, because the thing behind the endpoint is not yours and is not versioned in your repository. That is an acceptable guarantee for a summary a human will read and a bad one for a decision.

Where a model genuinely helps

Being against a model in the decision is not being against models. There are tasks here where a model is better than anything deterministic, and they share one property: a wrong output costs a person a minute, not an ordering.

• Summarising an evidence trail someone is about to read. A chain with nine steps and three providers compresses to three lines a shift engineer can scan, and a bad summary wastes the time it saves.
• Drafting remediation text. The Jira description, the Slack message, the first paragraph of the change request. An engineer edits it, and if the draft is wrong the deterministic finding underneath is unchanged.
• Explaining. Turning a rule, a score or a risk narrative into language for someone who does not live in the console, which is a real and recurring need.
• Proposing candidate rules for a human to review and test. A model is a reasonable source of hypotheses and a poor source of verdicts. The month's own AI-discovered vulnerabilities are an argument for models in discovery, not in the runtime decision.

The test that separates the two columns: if the output is wrong, does the damage land on a person's reading time, or on the ordering of the queue. The first is fine. The second is the thing this article is about.

What actually ships, precisely

There is an optional advisory module with two model call sites. It resolves a provider configuration — the tenant's own key, or a managed one — and generates advisory text from a finding. Nothing under the correlation layer imports it, and the scoring function takes numbers and booleans.

The boundary is worth stating at the level of fields, because that is where boundaries are actually enforced. One route will write generated prose into a risk record, replacing the narrative field, and it returns an error unless a model provider has been configured for that tenant. The score, the rule id, the attack chain and the status are not touched by it. Nothing consumes the module's output as an input signal.

The check for any of this is a diff, not a promise. Turn the module off and compare: no detection disappears, no score changes, no finding moves in the queue. If that ever stops being true, the prioritisation claim stops being true with it.

Questions