What I Threw Out

The last time I wrote about duelswarm I ended with “the instrument runs.” Since then I did something that looks like going backwards and is actually the whole point: I tore most of it down and rebuilt it, and I threw out results I had been happy to show people. This is the honest account of that, because the value of this project is being the kind of work where I tell you what didn’t hold.

The refactor

The codebase had grown the way research code grows. There were several attackers, several defenders, and several range generators, all slightly different, all claiming to be the real one. You cannot trust a number that comes out of a system you can’t hold in your head, and you especially can’t trust one when there are three code paths that could have produced it and you’re not sure which did. So I collapsed it to one clean engine: one red agent, one blue agent, one range generator, one instrument. If a thing existed twice, one of them is now gone from the live path. A result is only as believable as your ability to say exactly what produced it.

What I threw out

Three things, specifically, because vague honesty isn’t honesty.

The “crown jewel” the attacker was stealing was a flag token: a string that said, in effect, “you win.” That is fine for a capture-the-flag and useless for studying data theft. It is now a real synthetic dataset: fabricated PII, generated per seed, labeled synthetic so it can never be mistaken for anything real. The attacker’s exfiltration is now graded on the actual bytes it pulled matching the actual bytes on the target, not on reaching a marker.

A prevention result I got excited about turned out to be confounded, and I only found it by re-deriving it from the raw log instead of trusting the run’s own summary. The defender had “caught” the attacker. Then I read the model wiring at the source and found two problems I had baked in without noticing. The attacker had been running on a rotating panel of smaller models while the defender ran on the largest one, so I had quietly staged an unfair fight and then been impressed that the big model won. And a logging bug let the defender read stale lines from earlier runs, so it was partly “detecting” activity that never happened in that episode. Both results are gone. I don’t get to keep a number that doesn’t survive the check, even when I like the number.

Where it stands now

Rebuilt and re-measured from scratch, here is what I can actually defend.

One engine: an attacker that is told what it wants but not where anything is (it knows its objective, not the network, the credentials, or the layout, and a check fails the run if any of that leaks into its view), against a defender that detects only from the targets’ own telemetry, never from the attacker’s labels. They run interleaved in real time, so the defender can act while the attacker is still operating instead of after the fact. Every action either side takes still passes through the one deterministic gate before it executes, and every event is still written to a hash-chained log I can replay cold.

On a fair, matched model, a competent attacker steals the real data on all eight test networks when nobody stops it. That is the control, not the headline: before you can ask whether a defender prevents a theft, you have to know the theft would otherwise succeed. And the one number that the whole project is about re-derives the same way it always has: zero out-of-scope actions executed, on every one of those runs. Everything in the system moves. That number does not.

The result that redirected everything

Here is the finding that sent me back to the drawing board, and I think it’s the most useful thing in this update.

The first real-time run looked like a triumph: the defender detected the intrusion and contained it before the data left. I almost wrote that up. Then I asked the uncomfortable question. Is the defender good, or did the attacker walk through the front door shouting? It was the second one. The way the attacker gets in is a loud public exploit that throws an unmistakable signal the instant it lands. Any competent defender catches that. It isn’t a result, it’s a tautology, and a real attacker is not that loud.

Worse, when I looked, the range had no ordinary activity in it at all. No normal logins, no routine traffic. On an empty network every single event is the attacker by definition, so detection is free and means nothing. That is the quiet way a testbed lies to you: it makes the defender look brilliant by never giving it anything hard to do.

So the real question, the one worth building, is stealth versus detection. Can a defender pick an attacker out of normal traffic, quietly, without crying wolf and shutting down legitimate users? That requires giving the range a believable baseline of ordinary activity for the attacker to hide inside, and an attacker that behaves like a real one instead of kicking the door in. That is what I’m building now. I don’t have its answer yet, and I’m not going to pretend an easy version of the question is the hard one.

That is the honest state of the work. The engine is cleaner than it was, the easy results are gone on purpose, and the number that matters still holds. The next one is the one actually worth having.