An Honest Coverage Map (and a Console That Never Makes Up a Number)

Where all the phases land

#

Three phases each leave a JSON file behind - manifest.json (converted), deployments.json (live in QRadar), validation.json (fired or didn’t). The final phase folds all three into a single ATT&CK Navigator layer, colour-coded so one glance tells you the true state of coverage:

green   validated     simulated the attack and the detection fired
orange  deployed      live in QRadar, not yet proven by simulation
grey    converted     AQL exists, not deployed
red     not_triggered simulated but silent  /  failed conversion or deployment

A coverage map is one of the easiest artefacts in a detection programme to misread. It usually fails in the dangerous direction, showing green where there is only hope. Two rules keep this one honest. Mix them up and the map becomes wrong in exactly the way you cannot afford.

Rule one: within a rule, later phases win

#

A single rule moves through phases, and a later phase always overrides an earlier one. A rule that was deployed and then failed to fire ends up not_triggered, even though it once made it to deployed:

STATUS_ORDER = ["failed", "not_triggered", "converted", "deployed", "validated"]

def upsert(key, status, title, techniques):
    entry = rules.setdefault(key, {"title": title, "techniques": []})
    entry["status"] = status      # each phase overwrites the last for this rule
    ...

The phases run in order - conversion, then deployment, then validation - so whatever validation decides is the verdict that sticks for that rule. A rule that’s deployed but silent shows up red rather than orange. Flip the priority and a detection that provably doesn’t fire would still glow “deployed” on the map, which is the exact false comfort this whole project is trying to get rid of.

Rule two: across rules, the best status wins

#

Several rules can target the same technique, though. There, the best status wins, so one noisy failing rule doesn’t hide a working detection sitting behind it:

for technique in entry["techniques"]:
    bucket = techniques.setdefault(technique, {"status": status, "rules": []})
    if _rank(status) > _rank(bucket["status"]):   # best-of across rules
        bucket["status"] = status

Get these two backwards and you either hide real failures (worst-within-rule becomes best) or hide real coverage (best-across becomes worst), so I’ve kept them very deliberately separate. Each Navigator cell also carries a comment listing every contributing rule and its individual status, so if you’re staring at a green square wondering why, the answer is one click away.

Dry runs don’t count, again

#

The reporter re-enforces the discipline from the deployment post: only a genuine API write (created, updated, unchanged) counts as deployed. A dry-run record proves the payload builds, not that QRadar received anything, so it’s skipped:

live_states = {"created", "updated", "unchanged"}
...
elif raw in live_states:
    status = "deployed"
else:
    continue   # dry-run carries no coverage; don't paint the matrix orange

With no SIEM deployed, every run is currently a dry run, so the layer renders converted (grey), with nothing deployed and nothing validated. That is an accurate picture of a machine without a SIEM, and it is more useful than a green wall I cannot defend.

The output is a layer v4.5 file stamped "attack": "18" (the renumbering again), droppable straight into the public Navigator via Open Existing Layer -> Upload from local.

The console: same files, no invented numbers

#

Reading raw Navigator JSON is nobody’s idea of a dashboard, so there’s a small static web console over the same committed artefacts, built in IBM Carbon v11 with g10/g100 themes - it felt right for a project aimed at IBM’s own ecosystem. It reads the JSON directly, no backend, no database, and the one design rule I held hardest was that nothing on screen is a number I made up.

Every figure is either a count over artefact records or a field read out of one. Each artefact has exactly three states, and there is no fourth path where a broken file quietly renders as zero:

• loaded - the file is there and matches its contract.
• absent - the phase hasn’t run; the empty state shows the real zero in muted ink and names the command that would produce the data.
• invalid - the file exists but violates its contract, and the error says what’s wrong.

That is what a fresh clone looks like: an empty state that tells you what to run next instead of a demo full of fabricated coverage.

Two more decisions behind the console. The status-folding logic in the browser (lib/status.ts) is a port of the Python reporter rather than a reimplementation - same precedence, same exclusion of dry-run, same handling of skipped - with 15 tests covering the case I care most about: a deployed-then-silent rule reads not_triggered in the rule fold, the technique fold, and every count. And the matrix recomputes from state/ instead of trusting the layer file, because a checked-in layer can easily be older than the state that produced it, and a stale layer would happily show coverage that’s no longer there. The layer is still offered as a straight download, for the real Navigator, which is what it’s actually for.

The console also takes Sigma uploads, but it doesn’t pretend to do more than it can: it parses and validates the YAML against what the converter will demand, stages the file in that one browser’s localStorage, and shows you the command to make it real (cp into rules/sigma/<category>/, then make convert). It can’t write to the repo or run pySigma from a browser, so it automates what it reasonably can and stops there. A staged rule is never counted as coverage, because it doesn’t have any yet.

CI/CD: hermetic by default, gated where it bites

#

The GitHub Actions workflow splits along the same SIEM boundary the whole project respects. The conversion and reporting stages are hermetic - they run on every push, on GitHub-hosted runners, with the tests and a --fail-on-error conversion so an unconvertible rule can’t sneak into the repo:

- name: Convert rules
  # if it cannot become AQL, it cannot become a detection
  run: python -m src.convert --fail-on-error
- name: Rehearse deployment payloads
  run: python -m src.deploy --dry-run
- name: Build ATT&CK Navigator layer
  run: python -m src.report_navigator

The deploy-and-validate job - the one that touches the lab and runs real attacks - is fenced off behind a self-hosted runner inside the lab network, a GitHub environment approval, and workflow_dispatch only. A plain push runs conversion and reporting; it can never start an attack.

What the series was really about

#

Seven posts came out of one idea: a detection you have not simulated is still a hypothesis, however confident it looks in the repository. Converting Sigma to AQL was the solved part, and IBM solved it. Everything built on top of that, the empirical API probe, the two-pipeline converter that flags its own performance risks, idempotent deployment, validation that reruns the rule’s own query and this coverage map, serves one claim: a converted rule should still catch the attack once it is live.

The thread running through the series is a refusal to let the tooling flatter me. Dry runs do not count as deployed, a rule that goes silent after deployment shows red instead of retaining its old orange, untestable techniques stay orange instead of being rounded up to green, and absent data shows the command to run rather than a zero dressed up as a fact. Green is supposed to mean proven, and the pipeline is designed to make that true.

If you want to run the part that needs no SIEM:

make install     # venv + deps
make offline     # tests, conversion, deploy rehearsal, coverage layer

The code, the honest status, and the Phase 0 open questions that only a live instance can settle are all there. The lab is next. Thanks for reading the series.