Every detection capability in security operations -- every SIEM correlation rule, every SOAR playbook, every threat hunt -- ultimately depends on one thing: whether the right log existed, was collected reliably, and was still trustworthy and searchable by the time someone needed it. Logging is the foundation everything else sits on, and it is also one of the most frequently under-designed parts of a security program. This article walks through the entire pipeline, end to end.

What a Log Actually Is

A log is a timestamped record of an event: something happened, at a specific time, involving a specific actor and a specific object, with a specific outcome. A useful security log answers, at minimum: what happened, when, who or what caused it, where it happened, and what the result was. A log line missing any of those dimensions is significantly less useful during an investigation, no matter how much data volume surrounds it.

Where Logs Come From

  • Operating systems. Authentication events, process creation, privilege changes, file access, and system configuration changes from Windows Event Logs, Linux auditd/syslog, and equivalent sources.
  • Network devices. Firewalls, routers, switches, VPN concentrators, and wireless controllers generate connection, traffic, and policy-decision logs.
  • Applications. Web servers, databases, and business applications log requests, errors, and application-specific security events such as failed authorization checks.
  • Identity providers. Authentication, authorization, MFA challenges, and account changes from directory services and single sign-on platforms -- often the single richest source for detecting account compromise.
  • Cloud platforms. API-level audit trails recording every action taken against cloud infrastructure, frequently the only visibility into resource changes that never touch a traditional network boundary at all.
  • Security tooling itself. EDR, antivirus, DLP, and email security platforms generate their own detection and action logs, which need to flow into the same central pipeline as everything else rather than staying siloed in a separate console.
  • OT/ICS sources. Engineering workstation activity, PLC/RTU configuration changes, and industrial protocol traffic (Modbus, DNP3, OPC UA, and similar) -- frequently the weakest-covered log source in organizations that built their logging strategy around IT systems first and extended it to operational technology as an afterthought.

The Logging Pipeline, End to End

  • 1. Generation. The source system writes the raw event, in whatever native format it uses.
  • 2. Collection. An agent, forwarder, or the source system's native shipping mechanism (syslog, an API push, an agent-based collector) picks up the event and moves it toward central infrastructure.
  • 3. Transport. Events travel across the network to a central point -- directly, or via an intermediate message queue or buffering layer that smooths out spikes in volume and prevents a temporary downstream outage from causing permanent data loss.
  • 4. Parsing and normalization. Raw, source-specific formats are parsed into structured fields and mapped onto a common schema, so a "source IP" field means the same thing regardless of which of forty different log formats it originally came from.
  • 5. Enrichment. Additional context is attached automatically -- resolving an IP to a geographic location or known-malicious reputation, mapping a hostname to an asset owner and criticality rating, mapping a username to a department and role.
  • 6. Storage and indexing. Enriched events are written to a storage layer optimized for both ingest volume and fast search, and indexed so an analyst can query months of history in seconds rather than minutes.
  • 7. Retention and lifecycle management. Data moves through hot, warm, and cold storage tiers as it ages, balancing search performance against storage cost, until it is eventually purged according to a defined retention policy.
  • 8. Analysis and correlation. This is where the logging pipeline hands off to the SIEM's correlation engine and to analysts running manual investigative searches.

Log Formats and Standards

Consistency of format is what makes normalization possible at scale.

  • Syslog (RFC 5424 and the older RFC 3164) -- the long-standing standard transport format for network devices and Unix/Linux systems, structured but often loosely so in practice, which is exactly why parsing and normalization matter so much downstream.
  • CEF (Common Event Format) -- a structured, vendor-neutral format designed specifically to make cross-product SIEM ingestion easier by standardizing field names.
  • LEEF (Log Event Extended Format) -- a similar structured standard, most closely associated with IBM QRadar but broadly supported.
  • JSON -- increasingly the default for modern applications and cloud platforms, because its structured, self-describing format is naturally easy to parse without brittle pattern-matching.
  • Windows Event Log (EVTX) -- a structured, XML-backed binary format native to Windows, requiring a dedicated collector rather than plain-text syslog forwarding.

The practical reality in almost every real environment is a mix of all of the above simultaneously -- which is precisely why the normalization layer in the pipeline is not optional overhead, it is the single piece of infrastructure that makes cross-source correlation possible at all.

Why Raw Logs Are Never Enough: Normalization and Enrichment in Practice

A raw firewall log might record a source and destination IP with no indication of which internal system that IP belongs to, who owns it, or how critical it is. A raw authentication log might record a username with no indication of that user's department, role, or normal working hours. Enrichment closes that gap automatically, attaching business context to what would otherwise be a bare technical record -- and it is exactly that added context which turns "connection from 10.4.2.19" into "connection from an engineering workstation belonging to a user who does not normally access this system outside business hours," which is the difference between a log entry and an actionable signal.

Storage and Retention Strategy

Retention design has to balance three competing pressures: storage cost, search performance, and compliance or forensic requirements that may mandate keeping certain log categories for a year or longer. A practical tiered approach is standard:

  • Hot storage -- recent data (typically the most recent 30-90 days), kept fully indexed for fast, interactive search during active investigations.
  • Warm storage -- older data still searchable, but with somewhat slower query performance, at a lower storage cost.
  • Cold/archive storage -- long-term retention for compliance and forensic purposes, often compressed and not actively indexed, retrievable when specifically needed rather than searchable in real time.

Retention periods should be driven by a deliberate policy decision -- tied to applicable regulatory requirements, incident investigation needs (attackers frequently maintain persistence for months before detection, so retention shorter than that window can mean the earliest evidence of compromise is already gone by the time an investigation starts), and realistic storage budget -- not simply defaulted to whatever the platform ships with out of the box.

Log Integrity and Tamper-Evidence

A log an attacker can quietly edit or delete after the fact is far less useful as either a detection input or forensic evidence. Mature logging architectures protect integrity through several complementary mechanisms: centralizing collection quickly so logs leave the source system (and an attacker's local reach) as close to real time as possible; write-once storage (WORM) for archived logs so they cannot be silently altered after being written; and cryptographic hash-chaining, where each log entry's hash incorporates the previous entry's hash, so that tampering with any single historical record breaks the chain in a way that is immediately detectable. This last technique is particularly valuable for audit logs that may need to stand up as evidence -- in a security investigation, a regulatory audit, or, in the most serious cases, in front of a court.

Feeding Detection: How Logging Connects to SIEM

The entire logging pipeline exists, from a security standpoint, to feed the correlation engine described in this Knowledge Center's SIEM and SOAR article. Every design choice in the pipeline -- which sources are collected, how completely they are normalized, how richly they are enriched, how long they are retained -- directly determines what a SIEM's correlation rules and an analyst's investigative searches are actually capable of finding. A brilliant correlation rule against an incomplete or poorly normalized log source will simply never fire.

Common Logging Mistakes

  • Logging everything with no plan. Indiscriminately ingesting maximum verbosity from every possible source drives storage cost up sharply while burying the genuinely useful signal in noise -- log source selection should be a deliberate decision tied to actual detection use cases, not a default-everything setting.
  • No retention policy at all. Either logs get purged too early to support an investigation, or storage cost grows unmanaged with no clear decision behind what's being kept and why.
  • Unprotected log storage. If an attacker who compromises a system can also edit or delete that system's logs, the logs cannot be trusted as evidence of what actually happened.
  • Missing OT/ICS log sources. Industrial environments are frequently logged far more thinly than IT environments, creating exactly the kind of blind spot where a highest-consequence incident is most likely to go undetected the longest.
  • No normalization strategy. Dumping raw, unparsed logs from dozens of different formats into a central store without normalization makes cross-source correlation and search dramatically harder than it needs to be, even though all the data is technically "there."