Why teams drop the sources they need most
When the cost model charges per ingested gigabyte, the first sources cut are the high-volume, low-density ones: firewall and flow logs, DNS, cloud control-plane audit trails, and endpoint process telemetry. Those are precisely the sources that provide the connective tissue in a cross-domain investigation.
The outcome is a platform full of alerts that cannot be explained. Detection quality collapses not because rules are bad but because the context needed to qualify them was never retained.
Normalize at ingest, into one schema
Every event is projected into a single schema of subject, action, object, outcome and context, with the asset key derived from the provider's authoritative inventory. Late normalization — at query time, per analyst — produces detections that break whenever a vendor changes a field name.
Deterministic keys are what makes correlation deterministic. Joining on hostname or IP is a probabilistic operation in any environment with DHCP, NAT, ephemeral workloads or container networking.
- One schema for identity, endpoint, network, cloud control plane and application events
- Asset key from provider inventory; hostname and IP retained as attributes only
- Severity computed from business impact, never inherited from the source
- Full lineage preserved from raw event to derived finding for chain of custody
Retention in tiers, evidence immutably
Retention design has three tiers with different economics: a hot tier sized for active detection and investigation, a warm tier for correlation lookback and hunting, and an immutable archive for regulatory evidence with defined restore time.
The archive is a control, not a cost line. It must be write-once, independently verifiable, and restorable within the window that supervisory reporting requires — a requirement that should be tested, not assumed.
- Hot: 30 to 90 days, full-fidelity search for detection and triage
- Warm: 12 months, optimized for lookback and hunting queries
- Archive: 5 to 7 years, immutable, with tested restore procedure
Detection engineering as a lifecycle
Detections are software. They need version control, test data, a coverage map against MITRE ATT&CK per telemetry source, and a retirement path. Without measured coverage, adding rules feels like progress while leaving whole technique families unobserved.
The most useful weekly metric is not alert count but the ratio of detections that produced a decision. Rules that never change an outcome should be constrained or retired.
Acceptance criteria
Accept a SIEM migration against measurable behaviour, in your environment, with the previous platform still running for the comparison window.
- Parsing completeness: 100% of committed sources normalized with field-level validation
- Correlation determinism: cross-domain joins resolve on asset key in 100% of sampled cases
- Search performance: 95th percentile investigative query under an agreed threshold on hot tier
- Cost predictability: monthly variance under 10% at stable ingest volume
- Evidence integrity: archive restore exercise completed and documented before go-live