Logging, SIEM & Threat Detection

When something goes wrong, logs are how you find out what happened, when, and who did it. Without them, a breach can go unnoticed for months, and even when it is found nobody can say what was taken. In the OWASP Top 10:2025, "Security Logging and Alerting Failures" has its own category, and in India the CERT-In directions require organisations to keep logs for 180 days.

In this lesson you will learn what to log and what never to log, the most useful Windows and Linux events, how a SIEM collects and correlates logs, how detections are written and mapped to MITRE ATT&CK, and you will build two detections in Python: brute-force logins and scanner traffic in web server logs.

What to log, and what never to log

Log the events that tell a security story:

• Logins: successes, failures, lockouts, MFA challenges and password resets.
• Access control failures ("user 17 tried to open invoice 1042 and was denied").
• Administrative actions: new users, role changes, configuration changes, deleted data.
• Input validation failures and application errors.
• Security tool events: firewall denies, IDS alerts, antivirus detections.

Each entry needs when (an accurate timestamp with time zone), who (user and source IP), what (the action and its result) and where (system and component). Clocks must be synchronised; the CERT-In directions require NTP sync with NIC or NPL servers so times from different systems can be compared.

Never write secrets or sensitive personal data into logs: passwords (even wrong ones), OTPs, session tokens, API keys, full card numbers, or full Aadhaar numbers. Logs are copied to many places and read by many people. Mask such values, for example keeping only the last four digits.

Structured logs (one JSON object per line) are much easier for machines to search than free text:

import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({"level": record.levelname, "event": record.getMessage(), **record.__dict__.get("ctx", {})})

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("shop")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.warning("login_failed", extra={"ctx": {"user": "kabir@shop.example", "ip": "192.168.56.102"}})
log.info("refund_issued", extra={"ctx": {"admin": "priya", "order": 1042, "amount": 1299}})

{"level": "WARNING", "event": "login_failed", "user": "kabir@shop.example", "ip": "192.168.56.102"}
{"level": "INFO", "event": "refund_issued", "admin": "priya", "order": 1042, "amount": 1299}

Python's logging module is covered in detail in the Python Logging lesson.

Key system events

What a SIEM does

A SIEM (Security Information and Event Management) system is the central brain of a Security Operations Centre. It:

1. Collects logs from servers, endpoints, firewalls, cloud services and applications.
2. Normalises them into common fields (source IP, user, action) despite different formats.
3. Correlates events across sources, for example "a VPN login from a new country, followed by a mass file download".
4. Alerts analysts and shows dashboards.
5. Stores logs for investigations and compliance (180 days in India under the CERT-In directions, often longer by policy).

Well-known SIEMs include Splunk, Microsoft Sentinel, Elastic Security, Google Security Operations and the open-source Wazuh, which you can install in your lab. SOAR tools add automated responses, such as disabling an account when a high-confidence alert fires.

Detection rules, ATT&CK and Sigma

A detection is a rule that turns logs into an alert. Good teams describe what each detection catches using MITRE ATT&CK, a public knowledge base of attacker tactics and techniques with IDs, such as T1110 (Brute Force) and T1566 (Phishing). Mapping detections to ATT&CK shows which techniques you can see and which you are blind to.

Sigma is an open, vendor-neutral format for writing detections once and converting them for different SIEMs. This rule alerts when the Windows security log is cleared:

title: Windows security audit log cleared
status: experimental
description: Detects clearing of the Security event log, often done to hide intrusions
logsource:
  product: windows
  service: security
detection:
  selection:
    EventID: 1102
  condition: selection
level: high
tags:
  - attack.t1070.001

Detecting brute force in Python

A good brute-force detection counts failures per source within a time window, and raises the severity if a success follows the failures, because that may mean the attacker guessed the password:

from collections import defaultdict, deque
from datetime import datetime, timedelta

events = [
    ("10:00:01", "192.168.56.102", "root", "fail"),
    ("10:00:03", "192.168.56.102", "root", "fail"),
    ("10:00:05", "192.168.56.102", "admin", "fail"),
    ("10:00:08", "192.168.56.102", "meera", "fail"),
    ("10:00:10", "192.168.56.102", "meera", "fail"),
    ("10:00:12", "192.168.56.102", "meera", "success"),
    ("10:05:00", "192.168.56.1", "rohan", "fail"),
    ("10:05:09", "192.168.56.1", "rohan", "success"),
]
WINDOW, THRESHOLD = timedelta(seconds=60), 5
recent = defaultdict(deque)

for t, ip, user, result in events:
    now = datetime.strptime(t, "%H:%M:%S")
    q = recent[ip]
    while q and now - q[0] > WINDOW:
        q.popleft()
    if result == "fail":
        q.append(now)
        if len(q) == THRESHOLD:
            print(f"{t} MEDIUM brute force from {ip}: {THRESHOLD} failures in 60s (ATT&CK T1110)")
    elif len(q) >= THRESHOLD:
        print(f"{t} HIGH   success for '{user}' from {ip} right after {len(q)} failures: check this account now")

10:00:10 MEDIUM brute force from 192.168.56.102: 5 failures in 60s (ATT&CK T1110)
10:00:12 HIGH   success for 'meera' from 192.168.56.102 right after 5 failures: check this account now

Rohan's single typo followed by a success does not trigger anything. That is the art of detection: alert on what matters and stay quiet for normal life, or analysts will start ignoring alerts (alert fatigue).

Spotting scanners in web logs

Web server access logs show reconnaissance clearly: bursts of 404 Not Found responses and the user agents of common tools. Run your lab tools against the target, then analyse its log:

import re
from collections import Counter

LOG_LINE = re.compile(r'(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+) [^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')
access_log = """192.168.56.1 - - [03/Nov/2026:10:01:10 +0530] "GET /index.php HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (Windows NT 10.0)"
192.168.56.102 - - [03/Nov/2026:10:02:01 +0530] "GET /admin/ HTTP/1.1" 404 196 "-" "gobuster/3.6"
192.168.56.102 - - [03/Nov/2026:10:02:01 +0530] "GET /backup/ HTTP/1.1" 404 196 "-" "gobuster/3.6"
192.168.56.102 - - [03/Nov/2026:10:02:02 +0530] "GET /config/ HTTP/1.1" 404 196 "-" "gobuster/3.6"
192.168.56.102 - - [03/Nov/2026:10:03:15 +0530] "GET /vulnerabilities/sqli/?id=1%27 HTTP/1.1" 200 4380 "-" "sqlmap/1.8"
192.168.56.102 - - [03/Nov/2026:10:04:20 +0530] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine)\""""

TOOLS = ["gobuster", "sqlmap", "nmap", "nikto", "ffuf"]
not_found = Counter()
for line in access_log.splitlines():
    m = LOG_LINE.match(line)
    if not m:
        continue
    ip, when, method, path, status, agent = m.groups()
    if status == "404":
        not_found[ip] += 1
    tool = next((t for t in TOOLS if t in agent.lower()), None)
    if tool:
        print(f"{when[12:20]} {ip} tool={tool:<8} {method} {path}")
print("404s per IP:", dict(not_found))

10:02:01 192.168.56.102 tool=gobuster GET /admin/
10:02:01 192.168.56.102 tool=gobuster GET /backup/
10:02:02 192.168.56.102 tool=gobuster GET /config/
10:03:15 192.168.56.102 tool=sqlmap   GET /vulnerabilities/sqli/?id=1%27
10:04:20 192.168.56.102 tool=nmap     GET /
404s per IP: {'192.168.56.102': 3}

User agents are easy to fake, so real detections also look at request rates, error patterns and payloads. But this is exactly what a defender sees when you run your lab scans, which is a good reason to look at both sides of every exercise.

Common mistakes

• Logging nothing until an incident happens. You cannot go back in time to turn logs on.
• Logging passwords, OTPs or tokens, which turns your log store into a target.
• Unsynchronised clocks, which make timelines across systems impossible to build.
• Keeping logs only on the machine that produced them. An attacker with admin access can delete them; forward logs to a central, protected store.
• Too many noisy alerts. Tune detections until each alert deserves a human's attention.

Key points

• Log authentication, access control failures, admin actions, errors and security tool events, with who, what, when and where.
• Never log passwords, OTPs, tokens or full sensitive identifiers; mask them.
• Synchronise clocks and keep logs (at least 180 days in India under CERT-In directions), centrally and protected.
• A SIEM collects, normalises, correlates, alerts and stores; Wazuh is a free option for your lab.
• Map detections to MITRE ATT&CK and share them in the Sigma format.
• Good detections use thresholds and time windows and correlate failures with later successes.

from collections import defaultdict
from datetime import datetime, timedelta

events = [
    ("10:00:01", "192.168.56.102", "root", "fail"),
    ("10:00:03", "192.168.56.102", "root", "fail"),
    ("10:00:05", "192.168.56.102", "admin", "fail"),
    ("10:00:08", "192.168.56.102", "meera", "fail"),
]
WINDOW = timedelta(seconds=60)
seen = defaultdict(list)
alerted = set()

for t, ip, user, result in events:
    now = datetime.strptime(t, "%H:%M:%S")
    if result != "fail":
        continue
    seen[ip] = [(when, u) for when, u in seen[ip] if now - when <= WINDOW] + [(now, user)]
    users = {u for _, u in seen[ip]}
    if len(users) >= 3 and ip not in alerted:
        alerted.add(ip)
        print(f"{t} MEDIUM password spraying from {ip}: users {sorted(users)}")