Detection Engineering with Sigma: Detections That Survive Vendor Churn

TL;DR — Detection logic is the part of a security program that dies fastest: queries written in one SIEM’s language don’t run in another, and migrations or re-platforming silently delete years of hard-won tuning. Sigma fixes the portability half by expressing detections as vendor-neutral YAML, pySigma pipelines compile them into Splunk SPL, KQL, EQL and friends, and practices borrowed from software engineering — version control, CI validation, atomic testing — keep rules provably alive. This is detection engineering as a discipline, not a rules backlog.

Ask a detection engineer what their job actually is and you rarely hear “writing queries”. The queries are the artifact. The job is converting knowledge about attacker behavior into machine-evaluated logic, keeping that logic correct as data schemas evolve, and proving it still fires. The industry shorthand for that discipline is detection engineering, and its foundational open standard is Sigma — the format this post covers end to end.

The Problem: Detection Logic That Dies with Your SIEM

Most organizations express detections in the native query language of whichever SIEM they bought: Splunk SPL, Microsoft Sentinel KQL, Elastic’s KQL/EQL, proprietary rule builders. That logic encodes the most expensive knowledge a security team owns — which combinations of fields indicate credential dumping in their environment, which admin behaviors are normal at 3 a.m. — and it is all written in a language that runs nowhere else. Switch platforms, merge with a company that chose differently, or spin up a second tool for a cloud workload, and years of detection engineering either get rewritten by hand or quietly lost.

There is a second, quieter failure: even inside one platform, rules rot. Log sources change field names on upgrade (we looked at exactly this class of silent breakage from the time side in why time synchronization breaks — timestamps are just the most visible schema dependency). A rule that references a renamed field doesn’t error; it stops matching. Nothing alerts, because the thing that would alert is broken.

Sigma, created by Florian Roth and maintained by the SigmaHQ community, attacks the portability half directly: a generic, open signature format for detections, written in YAML, that describes what suspicious behavior looks like in neutral field names — then compiles down to whatever your platform speaks.

Anatomy of a Sigma Rule

A Sigma rule is a single YAML document with a fixed skeleton: metadata (title, unique id, status, description, references, author, date), a logsource describing the event type in neutral terms, a detection block of selections and filters, and operational metadata — false positives, severity level, and MITRE ATT&CK tags. A realistic example:

title: Suspicious PowerShell Encoded Command Execution
id: 00000000-0000-0000-0000-000000000000   # generate a UUID per rule
status: experimental
description: Detects PowerShell launched with an encoded command, a common
  defense-evasion pattern in droppers and post-exploitation tooling.
references:
  - https://attack.mitre.org/techniques/T1027/
author: hmmnm.com
date: 2026/09/16
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith: '\powershell.exe'
    CommandLine|contains:
      - ' -enc '
      - ' -encodedcommand '
  condition: selection
falsepositives:
  - Legitimate admin scripts that routinely pass encoded commands
level: high
tags:
  - attack.defense_evasion
  - attack.t1027
  - attack.execution
  - attack.t1059.001

Three design choices do the heavy lifting. Neutral field names (Image, CommandLine) belong to no vendor. Field modifiers (|endswith, |contains, |re for regex, |all, |base64offset) express matching semantics declaratively instead of in query syntax. And the condition field composes named selections with boolean logic (selection1 and not filter2), which keeps complex logic readable. Severity is a controlled vocabulary, because “priority: 87” means nothing across teams:

The ATT&CK tags are not decoration — they are what makes rule coverage measurable. Once rules carry technique identifiers, you can diff your rule set against the ATT&CK matrix and answer “which techniques do we detect, and where are the holes?” with data instead of intuition.

How Conversion Works: pySigma and Processing Pipelines

A YAML file doesn’t match anything by itself. The reference implementation is pySigma, a Python library (with the sigma CLI) that parses, validates, and converts rules through two pluggable pieces: backends produce the target query language (Splunk SPL, Elasticsearch/EQL, Microsoft Sentinel KQL, SQL, and more), and processing pipelines transform rules on the way in — the critical part.

Pipelines solve the two hard conversion problems. First, field naming: your Sigma rule says Image, your Elastic stack stores process.executable.name under Elastic Common Schema (ECS), your Sentinel workspace normalizes into ASIM fields. A pipeline for ECS or ASIM rewrites the rule’s fields to the target schema during conversion. Second, index/table routing and custom logic: pipelines can add index hints, rewrite values, or drop rules that don’t apply. Conversion becomes a repeatable, reviewed artifact rather than a hand-porting exercise — you keep one canonical rule set in git and compile per environment, the same way you keep one source and build per platform.

The Big Vendors Already Proved the Model

If “detections as text files in a repository” sounds idealistic, look at how the largest detection teams actually work. Elastic maintains its entire detection content — prebuilt rules shipped to every Elastic Security deployment — in the public elastic/detection-rules repository as TOML files containing KQL, EQL and machine-learning job definitions, with a CLI that validates rules in continuous integration before release. The format differs from Sigma (it’s native-first), but the engineering pattern is identical: rules are code, versioned, tested, and released.

The mature pattern uses both: Sigma as the portable source of truth for shareable logic, compiled into each platform, with a thin layer of native rules where a query language’s unique power is genuinely needed. Cloud audit logs fit this well — for example, detection logic over Kubernetes audit events, whose logging surface we covered in the Kubernetes security fundamentals post, can be expressed once in Sigma and deployed to whatever stores those events this year.

Testing Detections Like Code

A rule that compiles is not a rule that detects. Detection engineering borrows the full software lifecycle:

• Static validation in CI. Every commit runs rule linters, schema validation, unit tests against example event fixtures (“this event must match; this near-miss must not”), and conversion smoke tests for every backend you deploy. Broken rules fail the build, not the incident response.
• Live validation with Atomic Red Team. Red Canary’s open-source Atomic Red Team is a library of small, ATT&CK-mapped tests — each designed to run in about five minutes — that execute a technique (credential dumping, persistence, encoded PowerShell) so you can observe whether a control and your detections actually fire. It answers the false-negative question nothing else can: not “is the query syntactically valid” but “would we catch T1003 on a real host?”
• False-positive accounting. Every tuned rule records what it matched and why in the change note; weekly FP review feeds back into falsepositives documentation and thresholds. A rule whose alert volume nobody can explain is a finding about the program, not just the rule.
• Deprecation discipline. Rules carry a status lifecycle — experimental → stable, with deprecated and unsupported terminals — so stale detections are visibly retired rather than silently zombie-ing in production.

Key Takeaways

• Detection logic is expensive institutional knowledge; writing it in a single vendor’s query language is an unpriced liability.
• Sigma is the open, SIEM-neutral YAML standard for detections: neutral fields, modifier-based matching, named selections with a condition, ATT&CK tags, and a controlled severity vocabulary.
• pySigma plus processing pipelines (ECS, ASIM, Splunk CIM) is the compiler chain: one canonical rule set, converted per platform, re-converted when schemas move.
• Elastic’s public TOML detection repository shows detections-as-code working at industrial scale; Sigma and native rules complement each other.
• Conversion proves syntax, not detection: CI validation plus Atomic Red Team technique tests prove the rule still catches the behavior — and surface false negatives.
• ATT&CK tags turn a rules backlog into a measurable coverage map you can audit against the techniques that actually appear in threat reporting.

FAQ

What is a Sigma rule?
A YAML document in the open Sigma format describing a suspicious log pattern in vendor-neutral terms — log source, field conditions, boolean logic, severity, and MITRE ATT&CK tags — that tools can convert into Splunk SPL, KQL, EQL and other SIEM query languages.

Is Sigma a replacement for my SIEM’s rule engine?
No. Sigma rules don’t run anywhere directly; they are a source format. You compile them with pySigma and a backend for your platform, and the platform’s engine evaluates the result.

What is a pySigma processing pipeline?
A configurable transformation applied during conversion: mapping Sigma’s neutral field names to a target schema (ECS, ASIM, Splunk CIM), setting index or table routing, rewriting values, or excluding rules — so one rule set serves many environments.

Who maintains Sigma rules?
The SigmaHQ repository holds thousands of community rules, and vendors increasingly ship Sigma-compatible content; teams typically curate a fork, adding their environment-specific rules and tuning on top.

How do I test that a detection actually fires?
Two layers: CI tests against event fixtures for logic, and live technique execution with Atomic Red Team (ATT&CK-mapped tests that run in minutes) to validate end-to-end visibility and alerting.

Does detection-as-code mean full automation?
It means detections get the software lifecycle — review, versioning, testing, staged rollout — while humans stay in the loop for judgment. The automation removes the toil and the silent breakage, not the engineering.

References

1. SigmaHQ — Sigma rule format and main rule repository (GitHub)
2. Sigma documentation — Rule structure
3. Sigma documentation — Processing pipelines
4. SigmaHQ — pySigma library and sigma CLI (GitHub)
5. Elastic — detection-rules repository (TOML, KQL/EQL, ML jobs)
6. Elastic — Opening the public detection rules repository
7. Red Canary — Atomic Red Team test library (GitHub)
8. Red Canary — Atomic Red Team overview
9. MITRE — ATT&CK knowledge base
10. pySigma on PyPI

Current as of September 2026. Educational engineering reference — validate rule syntax against the current Sigma specification and your SIEM’s schema before production deployment.

Published by Hmmnm

Hands-on cybersecurity tutorials, CVE breakdowns, and guided learning paths — written and lab-tested by the Hmmnm team.

Learning Paths · About Hmmnm · Editorial policy