Detection Engineering

A SIEM full of logs does not detect anything by itself. Someone has to decide what suspicious looks like, write it as a rule, check that the rule fires when it should and stays quiet when it should not, and keep it working as systems change. That work is detection engineering, and it is one of the most valuable skills a SOC analyst can grow into.

In Logging, SIEM & Threat Detection you wrote your first detections. This lesson treats detections as a product with a life cycle: you will learn why behaviour beats indicators, write a behaviour-based Sigma rule, test it with Python against good and bad events, measure alert quality and map your coverage to MITRE ATT&CK.

The detection life cycle

Mature teams handle every detection like a piece of code:

1. Hypothesis: what attacker behaviour do we want to see? Ideas come from threat reports, ATT&CK techniques, past incidents and penetration test findings.
2. Data check: do we collect the logs that show it? Many ideas stop here and become a logging task instead.
3. Write the rule, usually in Sigma or the SIEM's query language.
4. Test it against examples of the bad behaviour (true positives) and normal activity (benign events).
5. Deploy with a severity, an owner and a short runbook: what the analyst should check first.
6. Tune and measure: track how many alerts were real, adjust, and retire rules that no longer help.

Keeping rules in a Git repository, with review and automatic tests before deployment, is called detection as code. It brings the same benefits as in Secure SDLC & DevSecOps: history, review and fewer broken rules.

Behaviour beats indicators

An indicator of compromise (IOC) is a specific artefact: a file hash, an IP address, a domain. IOCs are easy to match but also easy for attackers to change. The Pyramid of Pain, described by security researcher David Bianco, ranks what defenders can detect by how much it hurts the attacker to change it:

Use IOCs for quick wins, but invest in detections of behaviour near the top of the pyramid, which keep working when the attacker changes their tools.

A behaviour-based rule

A classic behaviour: a Word or Excel document that starts PowerShell or the command interpreter. Normal users almost never do this, but malicious documents do it to download their next stage. On Windows, Sysmon event ID 1 (or Security event 4688 with command lines enabled) records each new process with its parent:

title: Office application starts a script interpreter
status: test
description: Detects Word, Excel or PowerPoint launching PowerShell, cmd or wscript, a common result of a malicious document
logsource:
  category: process_creation
  product: windows
detection:
  parent:
    ParentImage|endswith:
      - '\WINWORD.EXE'
      - '\EXCEL.EXE'
      - '\POWERPNT.EXE'
  child:
    Image|endswith:
      - '\powershell.exe'
      - '\cmd.exe'
      - '\wscript.exe'
  condition: parent and child
falsepositives:
  - Rare business macros that call scripts; document them as exceptions
level: high
tags:
  - attack.execution
  - attack.t1059.001
  - attack.t1566.001

The rule also records known false positives and ATT&CK tags (T1059.001 is PowerShell, T1566.001 is a spearphishing attachment), so an analyst knows what it means when it fires.

Test the rule like code

Before deploying, prove that the logic catches the bad case and ignores normal ones. Here is the same logic in Python with a tiny test set:

PARENTS = ("\\winword.exe", "\\excel.exe", "\\powerpnt.exe")
CHILDREN = ("\\powershell.exe", "\\cmd.exe", "\\wscript.exe")

def office_spawns_script(event):
    parent = event.get("ParentImage", "").lower()
    child = event.get("Image", "").lower()
    return parent.endswith(PARENTS) and child.endswith(CHILDREN)

tests = [
    ("malicious doc starts PowerShell", True,
     {"ParentImage": r"C:\Program Files\Microsoft Office\root\Office16\WINWORD.EXE",
      "Image": r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"}),
    ("admin opens PowerShell from Explorer", False,
     {"ParentImage": r"C:\Windows\explorer.exe",
      "Image": r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"}),
    ("Excel opens a help page in Edge", False,
     {"ParentImage": r"C:\Program Files\Microsoft Office\root\Office16\EXCEL.EXE",
      "Image": r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"}),
    ("upper/lower case does not matter", True,
     {"ParentImage": r"c:\office\excel.exe", "Image": r"C:\WINDOWS\system32\CMD.EXE"}),
]

failures = 0
for name, expected, event in tests:
    result = office_spawns_script(event)
    failures += result != expected
    print(f"{'PASS' if result == expected else 'FAIL'}  {name}")
print("All tests passed" if failures == 0 else f"{failures} test(s) failed")

PASS  malicious doc starts PowerShell
PASS  admin opens PowerShell from Explorer
PASS  Excel opens a help page in Edge
PASS  upper/lower case does not matter
All tests passed

The case-insensitivity test is there because Windows paths appear in many cases, a real source of missed detections. In your lab you can produce genuine test events safely: open a test macro document you wrote yourself on a Windows VM with Sysmon installed, or use the open-source Atomic Red Team tests, which are small, documented actions mapped to ATT&CK, run only on your own lab machines.

Measure alert quality

Every alert an analyst closes gets an outcome:

• True positive: real malicious activity.
• Benign true positive: the rule worked as written, but the activity was allowed (an approved admin script).
• False positive: the rule matched something it should not have.
• A false negative is an attack the rule missed; you only learn about these from incidents, tests and threat hunting.

Precision is the share of alerts that were worth acting on. Low-precision rules cause alert fatigue, so measure every rule:

from collections import Counter

triage = [
    ("Office spawns script", "TP"), ("Office spawns script", "BTP"),
    ("Office spawns script", "TP"),
    ("Login from new country", "FP"), ("Login from new country", "FP"),
    ("Login from new country", "FP"), ("Login from new country", "TP"),
    ("Login from new country", "FP"), ("Login from new country", "FP"),
    ("Security log cleared", "TP"),
]

outcomes = {}
for rule, outcome in triage:
    outcomes.setdefault(rule, Counter())[outcome] += 1

for rule, c in outcomes.items():
    total = sum(c.values())
    precision = 100 * c["TP"] / total
    action = "keep" if precision >= 50 else "TUNE"
    print(f"{rule:<23} alerts {total:>2}  TP {c['TP']}  precision {precision:5.1f}%  {action}")

Office spawns script    alerts  3  TP 2  precision  66.7%  keep
Login from new country  alerts  6  TP 1  precision  16.7%  TUNE
Security log cleared    alerts  1  TP 1  precision 100.0%  keep

Tuning without going blind

When a rule is noisy, improve it instead of switching it off:

• Add context: "login from a new country" becomes useful when combined with "no MFA", "admin account" or "followed by a mailbox rule change".
• Narrow exceptions: exclude one approved script path on one server, not every script everywhere. Give exceptions an owner and an expiry date.
• Adjust severity: a low-precision signal can feed a risk score instead of paging someone at night.
• Enrich: add asset importance and user role so analysts triage faster.

Coverage against ATT&CK

To know what you can and cannot see, list which ATT&CK techniques have at least one tested rule, and which have no data source at all:

priority_techniques = {
    "T1566": "Phishing",
    "T1059": "Command and Scripting Interpreter",
    "T1110": "Brute Force",
    "T1078": "Valid Accounts",
    "T1486": "Data Encrypted for Impact",
    "T1070": "Indicator Removal",
}
rules = {"T1059": 2, "T1110": 1, "T1070": 1}
no_data = {"T1078"}

covered = 0
for tid, name in priority_techniques.items():
    if tid in rules:
        covered += 1
        status = f"{rules[tid]} rule(s)"
    else:
        status = "NO DATA SOURCE" if tid in no_data else "gap: write a rule"
    print(f"{tid}  {name:<34} {status}")
print(f"Coverage: {covered}/{len(priority_techniques)} priority techniques")

T1566  Phishing                           gap: write a rule
T1059  Command and Scripting Interpreter  2 rule(s)
T1110  Brute Force                        1 rule(s)
T1078  Valid Accounts                     NO DATA SOURCE
T1486  Data Encrypted for Impact          gap: write a rule
T1070  Indicator Removal                  1 rule(s)
Coverage: 3/6 priority techniques

Choose the priority list from threats that matter to your organisation, not the whole ATT&CK matrix. Coverage on paper is not protection either: a technique counts as covered only when a rule is tested and someone responds to it.

Common mistakes

• Writing rules for data you do not collect, so they can never fire.
• Only IOC rules, which break as soon as the attacker changes infrastructure.
• No tests, so a typo silently disables a rule for months.
• Disabling noisy rules instead of tuning them.
• No runbook, so every analyst handles the same alert differently.

Key points

• Detection engineering treats rules as code: hypothesis, data check, write, test, deploy with a runbook, tune, retire.
• The Pyramid of Pain shows why detecting behaviour (TTPs) lasts longer than hashes, IPs or domains.
• Sigma rules describe behaviour in a vendor-neutral way and carry ATT&CK tags and known false positives.
• Test every rule with true positive and benign examples, and generate real test events only in your own lab.
• Measure precision per rule; tune with context and narrow, expiring exceptions.
• Map tested rules to priority ATT&CK techniques to find gaps and missing data sources.

PARENTS = ("\\winword.exe", "\\excel.exe", "\\powerpnt.exe", "\\outlook.exe")
CHILDREN = ("\\powershell.exe", "\\cmd.exe", "\\wscript.exe")

def office_spawns_script(event):
    return (event["ParentImage"].lower().endswith(PARENTS)
            and event["Image"].lower().endswith(CHILDREN))

tests = [
    ("Outlook starts PowerShell", True, {"ParentImage": r"C:\Office\OUTLOOK.EXE", "Image": r"C:\Windows\powershell.exe"}),
    ("Excel opens another Excel", False, {"ParentImage": r"C:\Office\EXCEL.EXE", "Image": r"C:\Office\EXCEL.EXE"}),
]
for name, expected, event in tests:
    print("PASS" if office_spawns_script(event) == expected else "FAIL", name)