Two Pipelines, One Choice: Converting Sigma to QRadar AQL

Same rule, two very different queries

#

Once IBM’s backend is installed, sigma list pipelines shows two processing pipelines for QRadar AQL. They are not two flavours of the same thing. The queries they produce are materially different, and choosing the wrong one without noticing is a real risk.

The fields pipeline maps each Sigma field onto a real, indexed QRadar/Ariel property. The queries are fast, but the pipeline refuses any rule that uses a field without a mapping. The payload pipeline does not refuse. For an unmapped field, it falls back to LOWER(UTF8(payload)) LIKE '%...%', a full-text scan over the raw event payload. It converts almost anything, but on a busy SIEM that flexibility can consume substantial resources.

Run the same LSASS-access rule through each and you can see it:

-- qradar-aql-fields  (rejected: field 'SourceUser' is not supported)

-- qradar-aql-payload (converted, with a performance warning)
SELECT * FROM events WHERE devicetype=12
  AND (LOWER("Target Process Path") LIKE '%\lsass.exe' AND ...)
  AND (NOT((LOWER(UTF8(payload)) LIKE '%authori%' OR LOWER(UTF8(payload)) LIKE '%autori%')))

The auto strategy: strict first, fall back on purpose

#

The converter defaults to auto. It tries the strict fields pipeline first, and only falls back to payload when fields outright refuses the rule. That keeps the fast path fast while still converting rules with odd fields - and, critically, it records which pipeline each rule landed on:

attempts = ["fields", "payload"] if strategy == "auto" else [strategy]
for attempt in attempts:
    try:
        queries, messages = _convert_with(rule_path, attempt)
    except (SigmaError, Exception) as exc:
        last_error = exc
        if attempt == "fields" and strategy == "auto":
            result.fields_pipeline_error = f"{type(exc).__name__}: {exc}"
            continue          # strict pipeline refused -> try payload
        break
    ...
    result.pipeline = attempt
    result.performance_risk = attempt == "payload" or any(
        marker in message for message in messages for marker in PERFORMANCE_WARNING_MARKERS
    )
    return result

The performance_risk flag is what makes this useful. A payload scan is survivable on a lab box capped at 100 EPS, but it may be unacceptable in production. Instead of hiding that trade-off, the manifest records it so someone can review and reject it before the rule ships. A payload fallback should be a deliberate decision, not an invisible converter default.

For my curated 25-rule set, auto put all 25 on the fast fields path - zero payload scans, zero performance-risk flags. That’s not luck: the rule-selection tool (a topic for another day) specifically prefers rules that convert cleanly through fields.

What the backend does across the whole corpus

#

Run auto over the entire SigmaHQ corpus - 3144 rules - and you get a measure of the backend itself:

The 103 failures are all essentially the same event - the strict pipeline refusing a construct it has no mapping for: a null check, a boolean, a regex or a CIDR expression on an unsupported field. And the fields that most often push a rule onto the payload fallback are the generic, cloud/SaaS ones with no obvious Ariel equivalent - Details, Provider_Name, operationName, riskEventType (those last two are Azure/Entra fields).

Put plainly, the backend is strong on the Windows/Sysmon process telemetry this project cares about, and weak on cloud and SaaS sources. That’s not a defect - it lines up with what the tool is built for - but it does mean a QRadar detection programme built on Sigma would need custom Ariel properties before it could cover cloud sources properly. Something to know before you promise cloud coverage.

The warning I had to filter out

#

My first run reported that every single rule emitted a warning, which made “rules with warnings” a useless signal. The culprit was a pyparsing deprecation ('parseString' deprecated) firing from inside pySigma itself - nothing to do with any rule. So the converter drops library-internal deprecation noise and keeps only the pipeline warnings, the ones you can actually act on:

NOISE_WARNING_MARKERS = ("deprecated - use", "DeprecationWarning")
...
messages = sorted({
    str(w.message) for w in caught
    if not any(marker in str(w.message) for marker in NOISE_WARNING_MARKERS)
})

Once the noise is gone, the warnings that remain are meaningful:

The one that fails open

#

That last warning is the interesting one, and it’s the reason the manifest keeps every pipeline warning around.

When the backend meets a log-source condition it doesn’t recognise, it doesn’t break the query - it drops the condition and prints 'X' is not a supported log source type and therefore is being removed. Conversion still succeeds, but the query you get back is now wider than the rule’s author intended. A detection scoped to one product can end up matching events from a completely different one.

It fails open, not closed - a rule meant to watch one narrow slice of telemetry silently ends up watching more, and that’s the kind of thing that looks fine in review but generates noise, or false confidence, in production. 36 rules in the corpus hit this. It needs to be flagged before a rule goes live, which is why the pipeline treats warnings as first-class manifest data rather than console spam.

Every query scopes by device type - remember this one

#

One structural detail that will come back to bite the lab: every converted query starts the same way.

SELECT * FROM events WHERE devicetype=<n> AND ...

That devicetype comes out of the Sigma logsource block via the backend’s product/service mappings. Windows rules map to devicetype=12 (Microsoft Windows Security Event Log). This is probably the single most important thing to get right in the lab: if your Sysmon events land under a different device type - which is likely if you forward them with a generic syslog agent instead of WinCollect - every converted rule returns zero results while looking completely correct. When validation eventually comes back not_triggered for everything, this is the first place to look.

The manifest is the contract

#

Everything downstream reads exactly one file: converted/manifest.json. It holds far more than the query text - Sigma UUID, title, severity, ATT&CK techniques and tactics, which pipeline was used, the performance_risk flag, and, for failures, the exact error.

Two things fall out of that deliberately. A rule that fails to convert still appears in the manifest, metadata and all, because the metadata is parsed straight from the YAML and doesn’t depend on pySigma - so failures land on the coverage map instead of vanishing. And the deployer never touches Sigma at all: if I wanted a different backend, or a different SIEM entirely, convert.py is the only file that would change. That’s the payoff of making one JSON file the contract between phases.

Here’s what falls out for Renamed Whoami Execution (T1033), header and all:

-- Renamed Whoami Execution
-- sigma id : f1086bf7-a0c4-4a37-9102-01e573caf4a0
-- pipeline : qradar-aql-fields
-- attack   : T1033
SELECT * FROM events WHERE devicetype=12 AND Filename='whoami.exe'
  AND (NOT((LOWER("Process Path") LIKE '%\whoami.exe'
        OR LOWER("Process Name") LIKE '%\whoami.exe')))

Fast, mapped, indexed - and traceable straight back to the rule it came from.

Next: a taxonomy change that quietly breaks coverage tooling - the ATT&CK v18 renumbering trap.