Blog How Knolt works

Security-Bench 1.2 to 1.3: how we test Knolt against frontier AI

What our benchmark asks, what changed from 1.2 to 1.3, and how Knolt-Talos compares with frontier AI models on security incidents.

Knolt engineering 5 min read

Security-Bench is the test we give Knolt and frontier AI models before every release. It asks one question, the way a bank’s security team asks it: is this a security incident? This post covers both versions so far, what we changed between them, and how Knolt-Talos did.

Why our own benchmark

Public video benchmarks test general understanding: what is in a scene, what happens next. None of them test the call a security operations centre makes many times a day, on a fixed camera, from a high corner, in a branch. So we built one. Every model gets the same images and the same question, word for word.

Security-Bench 1.2: robbery, asked twice

Security-Bench 1.2 asked about robbery only. Each model saw frames from public security footage and said whether a robbery was happening. Then we showed every model the same footage again, to see whether it gave the same answer.

Security-Bench 1.2November 2026
62%41% GPT-6.1 Sol61%43% Sonnet 5.557%61% Haiku 5.564%100%Knolt-Talos 1.0Correct robbery callsSame answer on replay

On accuracy, everyone was close: Knolt-Talos 1.0 made the right robbery call 64% of the time, against 62% for GPT-6.1 Sol. The gap was consistency. Shown the same footage twice, Knolt gave the same answer every time. GPT-6.1 Sol did 41% of the time. For a security team, a system that changes its mind on the same footage is a system it cannot rely on.

It also showed the test’s limits. Robbery is one of many incidents a branch sees. A plain yes or no cannot show how sure a model is. And it did not measure how long each decision takes.

What changed in Security-Bench 1.3

  • Six kinds of incident. Robbery, violence, theft and concealment, camera tampering, a person down, and smoke or fire.
  • Lookalikes, on purpose. Ordinary moments that only resemble an incident, such as a hug or a handful of dropped papers. A model that flags everything scores badly.
  • Real footage and staged scenes, both labelled. Real security-camera frames, plus short staged scenes filmed the way a security camera sees them, so every kind of incident is covered.
  • A probability, not a yes or no. Every model says how likely an incident is. That lets us measure ranking quality (AUC), and count the alerts that were real and the ordinary moments left alone, not only correct calls.
  • Speed, end to end. The median time to one decision. The other models answer through their makers’ APIs, network included.
  • A stronger field. Claude Opus 5.5 joined GPT-6.1 Sol and Claude Sonnet 5.5.

Security-Bench 1.3 results

Knolt-Talos 1.1 made the right call 83% of the time and caught 75% of incidents. Claude Opus 5.5, the best of the rest, made the right call 80% of the time and caught 69%. The gap opens on incidents caught.

Security-Bench 1.3November 2026
69%80% Opus 5.554%74% GPT-6.1 Sol51%72% Sonnet 5.575%83%Knolt-Talos 1.1Incidents caughtCorrect incident calls
MeasureKnolt-Talos 1.1Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5
Correct incident calls83%80%74%72%
Incidents caught75%69%54%51%
Alerts that were real88%85%86%86%
Ordinary moments left alone91%89%92%92%
Ranking quality (AUC)91%87%85%84%
Time to one decision0.24 s1.71 s2.81 s1.91 s

Knolt decides in 0.24 s: 7× faster than Claude Opus 5.5 and 11× faster than GPT-6.1 Sol. Where we lose, we say so: two models leave slightly more ordinary moments alone, 92% against our 91%.

What changed in Knolt-Talos 1.1

  • Catches more. Better at spotting incidents across all six kinds, including someone behind the counter who should not be there.
  • Decides in a quarter of a second. Every decision takes 0.24 s, so an alert reaches your team while the incident is still unfolding.
  • Steadier on clear incidents. When an incident is clear, Talos 1.1 keeps the alert instead of second-guessing it.

The full comparison is on Why Knolt.