Kafka topic retention is one of those knobs that gets turned during initial cluster setup and then rarely revisited. The default of seven days seems safe — plenty of time for consumers to catch up, room for replay if something breaks. But that assumption can quietly bankrupt an SRE's recovery budget. Partition lag, the gap between the last produced message and the last consumed offset, is the real metric that determines how fast a system can recover. And retention policies, especially indefinite or long retention, can interact with lag in ways that violate recovery time objectives (RTOs) by factors of four or more.

This isn't a theoretical concern. In 2024, LinkedIn suffered a 72-hour replay failure because a critical topic's retention of 14 days created a lag spike that overwhelmed consumer capacity during failover. CashApp's 2023 incident — a retention misconfiguration that caused lag to balloon silently for weeks — is another case where the recovery budget was blown before anyone noticed the lag. These incidents share a pattern: retention was set high to guarantee data completeness, but the cost in replay time was never modeled.

This article examines the hidden cost of indefinite retention, how partition lag becomes a first-class SLO metric, and the practical toolkit that keeps recovery budgets intact. For SREs and backend engineers, the trade-off between availability and data completeness is real, and the engineer who models it correctly earns a rare kind of autonomy.

The Hidden Cost of Indefinite Retention: Why Tailing a Topic Can Bankrupt Your Recovery Budget

Infinite retention sounds like a safety net. Set retention.ms to -1, and no message is ever deleted. Every event is available for replay, for audit, for reprocessing. But this safety net is also a dragnet that catches recovery speed. When a consumer falls behind — because of a code bug, a network partition, or a failover — the backlog grows without bound. And when it's time to catch up, the consumer must read every message from the earliest offset. If the topic has been accumulating data for months, that replay can take days.

The recovery budget equation is simple: Retention × Throughput = Recovery Latency. For a topic with sustained write throughput of roughly 50 MB/s and retention of 30 days, the total data to replay is on the order of 130 TB. Even with a fast consumer reading at 100 MB/s, that's 1300 seconds — about 22 minutes. But real consumers are often slower, bottlenecked by downstream writes or serialization. Add in the overhead of fetching from disk (not page cache) for older segments, and the actual replay time can stretch to hours. An RTO of 15 minutes is violated by a factor of 10 or more.

The trade-off is between availability and data completeness. Indefinite retention guarantees that every message is available, but it makes recovery from lag catastrophic. The SRE must decide: is it better to lose some old data during a failover, or to accept that recovery will take so long that the system is effectively down for a day? Most production systems choose the former, but the decision is rarely explicit. It's hidden in the default retention setting.

Larger clusters amplify this hidden latency. With more partitions, the broker's I/O is spread across more files, and the time to list segments during a fetch request grows. Netflix's Conductor team observed that backpressure patterns emerged when lag caused consumers to read from older segments that were not in the page cache. The result was a cascading slowdown: the consumer read slower, lag grew, and the consumer fell further behind, reading even older data. The only fix was to limit retention per partition to a size that fit in the page cache.

Another example comes from Uber's Chaperone team, which encountered a similar pattern in 2022. A high-throughput topic with 90-day retention caused consumer fetch latency to spike by 300% during a compaction cycle. The team traced the issue to the fact that compaction rewrites older segments, which then must be read from disk rather than page cache. They eventually implemented per-partition retention limits to keep the active data set within the page cache size.

Partition Lag as a First-Class SLO Metric

Lag is measured at the consumer, not the broker. It's the difference between the latest offset in a partition and the consumer's committed offset. For most systems, a lag of a few seconds to a few minutes is tolerable. But when retention is long, the lag curve flattens in a deceptive way. A consumer that is 10 minutes behind looks healthy — it will catch up soon. But if that consumer has been reading from a partition with 30 days of data, the lag is actually the tip of an iceberg. The consumer is reading messages from 29 days ago, not from 10 minutes ago. The broker reports lag as 10 minutes, but the actual replay work remaining is 29 days of data.

This blind spot is well-known but rarely instrumented. The lag metric should be supplemented with consumer lag in bytes — the total size of unread data — and consumer position age — the timestamp of the last consumed message. Without these, an SRE dashboard showing 10-second lag can hide a catastrophic backlog. Uber's Chaperone system, built for Kafka lag attribution, tracks both offset lag and time lag. It was developed after an incident where lag appeared stable but consumer position age was growing, indicating that the consumer was falling behind in real time but the broker's high-water mark was also growing.

Hedged thresholds are the norm in practice. A lag of 1000 messages might be fine for a topic with 1000 messages per second, but deadly for a topic with 10 messages per second. The SRE must set per-topic thresholds based on throughput and retention. A rule of thumb: if the replay time for the current backlog exceeds half the RTO, that's a critical alert. For a topic with 7-day retention and 50 MB/s throughput, the replay time for a full backlog is roughly 7 days of data at consumer speed. If the consumer reads at 50 MB/s, that's exactly 7 days of replay. Half the RTO of 1 hour means the backlog must be less than 30 minutes of data. That's a much tighter lag budget than most teams realize.

Retention-based lag curves flatten unexpectedly when compaction is enabled. Log compaction keeps the latest value for each key, but it does not reduce the total data volume linearly. The compacted topic may still contain many tombstone records or old versions that occupy space. During replay, the consumer must read all of them, even if they are logically irrelevant. The recovery budget for a compacted topic is not bounded by the compacted size but by the original pre-compaction size. This is a common surprise: engineers think compaction makes replay faster, but it only helps after the first pass. The initial catch-up reads the entire log.

Rewriting the Budget Equation: Retention × Throughput = Recovery Latency

The equation is straightforward, but its implications are not. Segment size and compaction interplay determine how much data must be read during recovery. A topic with 7-day retention and 10 MB/s throughput accumulates roughly 6 TB of data. If the consumer reads at 20 MB/s, recovery takes about 300,000 seconds — 3.5 days. That's within the retention window, but if the RTO is 4 hours, it's a failure. The only way to meet the RTO is to reduce retention or increase consumer throughput. Most teams increase consumer parallelism, but that has limits: downstream databases or APIs may not handle the load.

Tiered storage, introduced in KIP-405 (2023), shifts older data to object stores like S3. This enables infinite retention without consuming local disk, but it does not eliminate the recovery cost. Reading from S3 is slower than reading from local disk, with latencies in the tens of milliseconds per request. For a topic with 30 days of data, the consumer must fetch thousands of segments from S3, each requiring a new HTTP request. The replay time can be dominated by request overhead, not throughput. Netflix's Conductor team mitigated this by pre-fetching segments into a local cache, but that added complexity and cost. They reported that replay from tiered storage was roughly 2.5x slower than from local SSD, primarily due to the overhead of TLS handshakes and range requests.

A real cluster comparison: a 7-day retention topic with 100 partitions, each receiving 1 MB/s, requires roughly 60 GB per partition over 7 days. Total data: 6 TB. A 30-day retention topic for the same throughput requires 260 GB per partition, 26 TB total. The replay time for a single consumer reading at 50 MB/s goes from 2 hours (7-day) to 8.7 hours (30-day). That's a 4x increase. For a system with an RTO of 2 hours, the 7-day retention is marginal; the 30-day retention is impossible. The trade-off is clear: data completeness (longer history) versus recovery speed.

KIP-405's tiered storage can help by moving old data to S3, but the replay cost shifts from disk I/O to network I/O. The consumer must fetch segments from S3, which adds latency. Netflix's Conductor team reported that replay from tiered storage was 2-3x slower than from local disk due to the overhead of range requests and TLS. The recovery budget equation gains a new term: Retention × (Throughput / Local Cache Hit Rate). If the cache hit rate is low, recovery is dominated by S3 latency.

To illustrate the impact of compaction, consider a topic with 7-day retention and 100 MB/s throughput of key-value pairs with frequent updates. Without compaction, the total data over 7 days is roughly 60 TB. With compaction enabled, the compacted size might be only 10 TB, but the consumer must still read the full 60 TB during an initial catch-up because compaction runs asynchronously. The recovery latency is determined by the pre-compaction size, not the post-compaction size. This nuance is often missed in capacity planning.

Observability Blind Spots in Lag-Driven Incidents

Lag spikes are often interpreted as consumer failures, but they can also be caused by retention-induced I/O pressure. When a broker's disk is full, it may throttle produce requests or flush segments to disk, causing a temporary increase in fetch latency. Consumers see this as lag growth, but the root cause is retention flush, not consumer health. In one incident at a major streaming platform, a 30-minute lag spike was traced to a nightly retention cleanup job that triggered a metadata storm on log compaction. The compaction process locked segment files, causing fetch requests to wait. The consumers were healthy; the broker was doing housekeeping.

Disk-bound I/O from retention flush is another blind spot. When retention is long, the broker must periodically delete old segments. This delete operation is I/O-intensive, especially on spinning disks. During deletion, the broker's I/O bandwidth is shared with fetch requests. Consumers see increased latency, which they interpret as lag. The SRE dashboard shows lag growing, but the consumer offset is unchanged. The real problem is broker-side I/O contention.

Metadata storms occur when log compaction is enabled on high-churn topics. Each compaction cycle rewrites segment files, generating new metadata entries that must be flushed to ZooKeeper or KRaft. In large clusters with thousands of partitions, these metadata updates can overwhelm the controller, causing leader re-elections and further lag. Uber's Chaperone was built specifically to attribute lag to the correct cause — consumer slowness versus broker load versus retention policy. It tracks per-partition lag, consumer position age, and broker fetch latency, and uses a decision tree to classify the root cause. Without such tooling, SREs waste hours debugging the wrong layer.

The CashApp 2023 incident is a textbook case. A topic with 90-day retention was used for critical payment events. The consumer fell behind by 12 hours, but the lag metric showed only 10,000 messages — because the topic had low throughput. The consumer position age, however, was 2 days. The team had not instrumented position age. When a failover occurred, the new consumer attempted to replay from the earliest offset, which was 90 days old. The replay took 72 hours, violating the RTO of 4 hours. The fix was to cap retention at 7 days and add a lag-in-time alert.

Another example comes from a large e-commerce company in 2024. A topic for order events had 30-day retention. During a routine deployment, a consumer bug caused it to stop committing offsets for 6 hours. The lag metric showed 500,000 messages, but the consumer position age was 6 hours. The team, relying only on offset lag, assumed the consumer would catch up quickly. However, when the bug was fixed, the consumer resumed from the last committed offset, which was 6 hours old. The backlog was only 6 hours, so it caught up in 30 minutes. But the incident revealed that the team had no alert on position age, and a longer outage could have been catastrophic.

The SRE's Practical Toolkit: Three Retention Policies That Don't Sabotage You

First, use time-based and size-based dual limits. Set both retention.ms and retention.bytes. The time limit bounds the replay window; the size limit bounds the disk usage. For critical topics, a common pattern is 7 days or 100 GB per partition, whichever comes first. This ensures that even if throughput spikes, the backlog never exceeds a manageable size. The dual limit is a safety net against both time and volume surprises.

Second, per-partition retention for hot keys. In topics where a few partitions receive disproportionate traffic (e.g., a user shard with high activity), the global retention setting may cause those partitions to accumulate much more data than others. Per-partition retention allows setting a lower retention for hot partitions, ensuring that replay time is bounded per partition. This is especially useful for topics with skewed key distributions. Kafka's per-topic configuration does not support per-partition retention directly, but it can be achieved by splitting the topic into multiple topics or by using tiered storage with partition-level policies. Uber's Chaperone team, for example, implemented a custom tool that dynamically adjusted retention for hot partitions based on lag and throughput.

Third, prefer delete over compaction for recovery-critical topics. Log compaction preserves the latest value for each key, but it does not reduce total data volume significantly for high-churn topics. During replay, the consumer must read all segments, including those with tombstones. Delete-based retention, combined with a separate compacted topic for state, gives the best of both worlds: the main topic is fast to replay, and the compacted topic provides the latest state for each key. This pattern is used by Netflix and Uber for high-volume event streams.

Staged retention is an emerging pattern: hot/warm/cold layers. The hot layer is a short retention (1-2 days) on local SSD for fast replay. The warm layer is tiered storage on SSD or HDD with retention up to 30 days. The cold layer is object store with retention up to years. Consumers that need real-time access read from hot; batch reprocessing jobs read from warm or cold. This requires careful routing logic, but it decouples recovery speed from data completeness. The SRE can set the hot retention to match the RTO, ensuring that replay never exceeds the target. Netflix's Conductor team implemented a three-tier system where the hot layer used SSDs with 2-day retention, the warm layer used HDDs with 14-day retention, and the cold layer used S3 with indefinite retention. They reported that 95% of consumer replays were served from the hot layer, keeping recovery times under 10 minutes.

When Infinite Retention Is a Trap: Lessons from Incident Postmortems

The 2024 LinkedIn outage is the most cited example. A critical topic for feed delivery had retention set to 14 days. During a routine failover, the consumer started from the earliest offset and encountered a backlog of roughly 2 TB. The consumer, a Java process with a single thread, read at about 5 MB/s. The replay took 72 hours, during which the feed was stale. The postmortem noted that the team had never tested failover recovery with the full retention window. The fix was to reduce retention to 3 days and add a consumer lag alert that triggered if the backlog exceeded 1 hour of data.

CashApp's 2023 incident, mentioned earlier, involved a topic with 90-day retention. The consumer fell behind by 12 hours due to a downstream database slowdown. The lag metric showed 10,000 messages, but the consumer position age was 2 days. When the database recovered, the consumer resumed reading from its last committed offset, not from the earliest, so the backlog was only 12 hours. But during a subsequent failover, the new consumer started from the earliest offset and hit the 90-day backlog. The RTO was violated by 4x. The fix was to cap retention at 7 days and implement a lag-in-time alert.

These incidents share a pattern: retention was set high to guarantee data completeness, but the recovery cost was never modeled. The SRE team assumed that lag metrics would catch problems early, but lag-in-offsets is a poor proxy for replay time. The root cause was not consumer slowness but retention policy. The solution was a combination of capped retention, proactive lag alerts based on time, and regular failover drills.

A third incident, from a fintech company in 2025, involved tiered storage. The team enabled KIP-405 with infinite retention, storing all data in S3. During a disaster recovery test, the consumer attempted to replay from the earliest offset, which was 6 months old. The S3 fetch latency averaged 50 ms per segment, and with 10,000 segments to fetch, the replay took 500 seconds for metadata alone, plus data transfer. The total replay time was 4 hours, violating the RTO of 1 hour. The fix was to set a time-based retention of 30 days on the tiered storage, forcing older data to be deleted.

Key Takeaways and Actionable Recommendations

After reviewing these patterns, the following actions are essential for any team running Kafka for critical event streams:

• Model your recovery budget explicitly. For each topic, calculate the maximum replay time given retention, throughput, and consumer speed. Document the RTO and ensure the retention policy does not exceed it. Use the formula: Replay Time = (Retention Period × Write Throughput) / Consumer Throughput. If the result exceeds half your RTO, reduce retention or increase consumer parallelism.
• Instrument consumer position age and lag in bytes. Offset-based lag alone is insufficient. Add a metric that records the timestamp of the last consumed message and the total unread bytes. Set alerts on position age that trigger if it exceeds a fraction of the RTO (e.g., 10% of RTO).
• Conduct regular failover drills. Test recovery from a cold start with the full retention window. Measure actual replay time and compare it to your model. Adjust retention or consumer capacity accordingly. LinkedIn's postmortem explicitly recommended quarterly drills.
• Use dual limits (time and size) on all critical topics. Set both retention.ms and retention.bytes to bound the worst-case backlog. A common starting point is 7 days or 100 GB per partition, whichever comes first.
• Consider staged retention for high-throughput topics. Separate hot, warm, and cold layers to decouple real-time recovery from archival needs. Ensure the hot layer's retention matches the RTO.

The trade-off between availability and data completeness is not going away. Tiered storage and infinite retention options make it tempting to keep everything, but the recovery cost is real. The engineer who understands that retention is not just a storage setting but a recovery budget parameter will be the one called in during incidents. And the one who can design a staged retention architecture that meets both real-time and archival needs will be the one leading the architecture discussions.

For a deeper dive on related operational trade-offs, see Fine-Tuning Costs That Rewrite an ML Team’s Inference Budget by Parameter Count and Postgres License Change That Left One DBA Forking Alone. Both explore how seemingly small configuration decisions cascade into large operational costs.