Proving a Detection Actually Fires

The whole series has been building to this

#

Everything so far, from conversion through deployment, produces a rule that looks deployed. This post covers the step that turns “looks deployed” into “detects the attack”: run the attack, then ask the rule whether it noticed.

The loop, per rule, is four steps:

1. Look up the Atomic Red Team tests matching the rule’s ATT&CK technique.
2. Run them on the lab endpoint.
3. Wait for the endpoint’s logs to reach QRadar and get indexed.
4. Re-run the rule’s own AQL over the attack window and check whether it returns anything.

Step 4 is the decisive one, and it is a deliberate design choice.

Re-run the rule’s own query, not “did an offense appear”

#

The obvious validation is to run the atomic and check whether QRadar generated a new offense. The validator does record that, but it is not the main signal. The main signal is the rule’s own converted AQL, rerun over the attack window, and whether it returns rows:

# ---- 4: ask the detection itself whether it fired -------------------
scoped = scope_query_to_window(aql, start_ms, stop_ms)
search = client.run_search(scoped)

result.search_status = search.status
result.matched_events = search.record_count
result.offenses = len(client.offenses_since(start_ms))   # secondary signal

Why do it this way? Because it tests the thing I actually care about: did this translated query match the telemetry the attack produced? Not “did some rule somewhere generate an offense,” which could be a different rule entirely, or a coincidence. Re-running the converted query closes the loop precisely around the conversion, which is the step most likely to be quietly wrong.

It has a second advantage: it works before any CRE rule exists. A saved search doesn’t generate offenses on its own; it needs a scheduled-search-plus-threshold rule wrapped around it. Validating by re-running the query means Phase 4 is useful regardless of how the Phase 0 API question shakes out - I don’t need to have solved CRE-rule creation to know whether my detection logic matches real attack telemetry.

Scoping to the window is a small but necessary detail - AQL scans a default recent interval unless you pin it, and the backend-emitted query might already carry a time clause, so the validator only appends one if there isn’t one:

def scope_query_to_window(aql, start_ms, stop_ms):
    if TIME_WINDOW_RE.search(aql):      # backend already emitted START/LAST
        return aql
    return f"{aql.rstrip().rstrip(';')} START {start_ms} STOP {stop_ms}"

Finding and running the atomics

#

Atomic Red Team organises tests by technique. The validator loads the Windows-capable tests for the rule’s technique, falling back to the parent technique (that T1685.001 -> T1685 move from the v18 post) when the sub-technique has none of its own:

def load_atomic_tests(atomics_dir, technique):
    for candidate in (technique, parent_technique(technique)):
        definition = atomics_dir / candidate / f"{candidate}.yaml"
        ...
        tests = [t for t in doc.get("atomic_tests", [])
                 if "windows" in (t.get("supported_platforms") or [])]
        if tests:
            return tests
    return []

Execution goes over SSH, not WinRM. Modern Windows ships an OpenSSH server, it’s far easier to script from Linux CI, and credentials stay in the SSH agent rather than in the Python process:

remote = f"powershell -NoProfile -ExecutionPolicy Bypass -Command {shlex.quote(command)}"
subprocess.run(
    ["ssh", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=accept-new",
     f"{user}@{host}", remote],
    capture_output=True, text=True, timeout=timeout,
)

There’s also a --manual mode that just prompts you to run the simulation by hand and press Enter - useful when the endpoint isn’t scriptable, or when you want a human in the loop for a destructive test.

not_triggered is a first-class, honest result

#

This status is the ethic of the whole project in miniature. When the re-run query comes back empty, the rule is reported as not_triggered, and the detail message names the three usual suspects, because all three genuinely happen:

else:
    result.status = NOT_TRIGGERED
    result.detail = (
        "no matching events - check Sysmon coverage, the DSM field mapping, "
        "or whether the atomic actually exercises this rule's logic"
    )

The three usual suspects:

1. Sysmon isn’t capturing the event. The rule needs process-access or image-load telemetry that Sysmon isn’t configured to log.
2. The DSM maps the field differently than the backend assumed - QRadar parsed the event, but into a property name the converted query doesn’t look at.
3. The atomic doesn’t actually exercise the rule’s logic. The test and the rule are both tagged with the same technique, but the specific behaviour they touch doesn’t overlap.

A rule that deploys cleanly and never fires is reported as not_triggered, not deployed. That distinction is the reason this project exists. A rule that looks deployed but detects nothing creates false confidence, so the pipeline marks it red instead of letting it hide behind an orange “deployed” badge.

The device-type gotcha, cashed in

#

Remember from the conversion post that every query begins WHERE devicetype=12. This is where that bites. If Sysmon events land under a different device type - very likely with a generic syslog forwarder instead of WinCollect - every rule returns zero and the whole run comes back not_triggered while looking completely fine. If Phase 4 ever reports a clean sweep of nothing-fired, the device type is the first thing I’ll check, before I touch a single rule.

The lab endpoint setup script (lab/setup-endpoint.ps1) is built to avoid this: it installs Sysmon with the SwiftOnSecurity config, turns on command-line auditing (EID 4688), lays down the atomics, and forwards logs so they land under the right device type. Sysmon is what makes most Sigma rules detectable at all - it supplies process creation with full command lines (EID 1), process access (EID 10) and image loads (EID 7), none of which the default Windows audit policy provides.

Safety, because this runs real attacks

#

This phase executes actual attack techniques on a real machine, so the guardrails aren’t optional:

• Lab only. The simulation runs against an isolated VM I own, never against anything that isn’t mine.
• Snapshot before every run, so a destructive atomic is one rollback away from undone.
• The lab endpoint script deliberately lowers defences (Defender exclusions for the atomics folder) so attacks generate telemetry instead of being blocked before they do, which is exactly why it must never run on a machine that matters.
• In CI, the job that touches the lab runs on a self-hosted runner inside the lab network, behind a GitHub environment approval, and only on manual dispatch. A plain git push can never kick off an attack simulation.

The honest status, again

#

Phase 4 is code-complete and blocked on the lab. QRadar CE isn’t deployed yet (the disk-space problem from Phase 0), so the validator hasn’t run end to end against a live box. But it’s built, it’s structured so it works the moment a SIEM and an endpoint exist, and because the pipeline never counts unproven work as proven, the coverage map correctly shows zero validated techniques today rather than a wall of optimistic green.

That is the map I will cover next: folding three phases into one ATT&CK coverage layer.