Kubernetes controllers don’t react to events. They react to state.
Every Kubernetes operator runs on the same four parts: one watches the cluster for changes, one keeps a local copy of what it saw, a queue decides what to work on next, and workers do the work.
I ran all four against pinned versions of client-go and controller-runtime, wrote a small program for each claim, and recorded what came out. Deduplication only begins once a reconciler falls behind its input; below that it does nothing at all. The resync period never contacts the API server, and a widely recommended predicate throws its events away before they reach a reconciler. The queue swap that most write-ups date to 1.36 landed in 1.33, and discards far less than they claim. And calling Forget without Done takes an object out of the control loop with no error and no log.
The one idea everything else follows from
Open controller-runtime, find pkg/reconcile/reconcile.go, and read the doc comment on Request. Three lines in there is a word in capital letters:
// Request contains the information necessary to reconcile a Kubernetes object. This includes the
// information to uniquely identify the object - its Name and Namespace. It does NOT contain information about
// any specific Event or the object contents itself.That capitalisation is load-bearing. Kubernetes controllers are level-triggered: they react to the current state of an object, never to the event that woke them. The event is a nudge meaning "go look again", and its contents are thrown away at the handler boundary. It is written down as an architectural rule, not a convention — the archived design principles say functionality "must be level-based … regardless of how many intermediate state updates may have been missed", and then, pointedly: "Edge-triggered behavior must be just an optimization."
1. The deduplication you rely on isn't always on
The standard explanation of the workqueue is that it merges duplicate events, so a hot object doesn't generate unbounded work. True — but I wanted the shape of it, so I wrote 200 updates to one ConfigMap, 1 ms apart, behind a reconciler that takes 20 ms, and counted.
events delivered : 200 reconciles executed : 24 collapse ratio : 8.3x values observed : [1 10 19 27 36 45 54 62 71 80 89 97 106 115 124 132 141 150 159 168 176 185 194 200] intermediate values never observed: 176 of 200
The reconciler skipped 88% of the states the object passed through and still finished on the correct one. Fine — that's level-triggering working. The interesting part is what happens when you sweep the reconcile cost:
Below the crossover there is no deduplication at all. At 0 ms and 1 ms reconcile cost I measured exactly 200 reconciles for 200 events — a 1.0× ratio, zero states skipped. Collapse is not a feature the queue applies; it is what falling behind looks like. A level-triggered controller under light load behaves identically to an edge-triggered one.
Push further and a cleaner law appears: the number of reconciles tracks the wall-clock duration of the burst divided by the cost of one reconcile, not the number of events. The same 200 events compressed into 0 ms produced 13 reconciles; spread 8 ms apart they produced 106. The queue converts an arrival rate you don't control into a service rate you do.
2. "RealFIFO removed deduplication" is wrong, and so is the version
The queue between the reflector and the informer cache was replaced. Nearly every write-up I found says this landed in 1.36 and removed per-key deduplication. Both halves are off.
Both queue types are exported, so you can drive them with an identical event sequence and compare. I interleaved two objects deliberately, because that's where they separate:
DeltaFIFO : 3 Pop(s), 6 delta(s) — grouped by key; alpha→beta interleaving lost RealFIFO : 6 Pop(s), 6 delta(s) — one delta per Pop, arrival order preserved
Six deltas in, six out of both. DeltaFIFO discarded nothing; it grouped deltas per key and returned each group in one Pop. What it destroyed was the cross-key interleaving — which is exactly what the feature gate's name says it fixes: InOrderInformers, "deliver watch stream events in order instead of out of order."
There is a real loss, and it's narrower than advertised. DeltaFIFO.Delete returns early when the object is in neither the queue nor the known-objects store:
DeltaFIFO yields: [] RealFIFO yields: [Deleted ghost]
A create-and-delete that fits between relists was reported to nobody. Harmless for a converging reconciler, a silent hole for anything doing per-deletion cleanup or metrics.
On the version: the gate registry is unambiguous, and it disagrees with the blog posts.
InOrderInformers: {
{Version: version.MustParse("1.33"), Default: true, PreRelease: Beta},
{Version: version.MustParse("1.36"), Default: true, PreRelease: GA, LockToDefault: true},
}Default-on since 1.33. What 1.36 added was LockToDefault — and that is executable, not documentation:
E0911 envvar.go:179] "Could not set feature gate, feature is locked" feature="InOrderInformers" desiredState="false" lockedState=true InOrderInformers enabled = true
So if your controller depended on DeltaFIFO's grouping — measuring queue depth in keys rather than deltas, say — that broke at 1.33, quietly, and from 1.36 there's no way back.
3. The ten-hour resync does not resync
This is the one I'd most like people to know. Most engineers treat SyncPeriod as the safety net that repairs a cache which has drifted from the API server. Read the doc comment to the end:
// SyncPeriod will locally trigger an artificial Update event with the same
// object in both ObjectOld and ObjectNew for everything that is in the cache.
//
// Predicates or Handlers that expect ObjectOld and ObjectNew to be different
// (such as GenerationChangedPredicate) will filter out this event, preventing
// it from triggering a reconciliation.
// SyncPeriod does not sync between the local cache and the server."The same object in both" is a strong claim, and it's testable. Running an informer with a resync period and comparing the two arguments handed to every OnUpdate:
OnUpdate invocations : 4 ... with a genuine payload change: 1 ... with old payload == new : 3 ... where old and new are the SAME pointer: 3 GenerationChangedPredicate.Update(resync event) = false — filtered out GenerationChangedPredicate.Update(gen 7->8) = true
Not merely equal — the same pointer. There is no comparison a handler could make to distinguish them, because there are not two objects. A resync issues no request; it cannot discover an object the cache never learned about, and it cannot correct a value that drifted.
Adding GenerationChangedPredicate to suppress status-write reconcile storms is standard advice. Lowering SyncPeriod believing it a safety net is common. Together the net is removed entirely and nothing reports it: the resync fires on schedule, the predicate discards it, the reconciler is never called. Any metric counting resync events still moves.
The upstream remedy is in the same comment, and it isn't a shorter period — it's returning reconcile.Result{RequeueAfter: t} per object, which enqueues a key downstream of every predicate and so cannot be filtered out.
4. Two small sets do all the work
The workqueue isn't a list with a dedup check bolted on. It's three structures held together by one invariant: queue (ordering), dirty (needs work), processing (someone's on it). Every element of queue must be in dirty and not in processing.
Add arriving while a worker holds the key re-enters dirty but is forbidden from touching queue. Done is the only thing that can move it back, and only if dirty still has it.The amber arc is the whole mechanism. An Add arriving while a worker holds the key re-enters dirty but is forbidden from touching queue; only Done can move it back, and only if it's still dirty. Measured:
Add("default/target") x 100000 → queue Len() = 1
worker holds the key, then 200 more Add()s arrive:
Len() while key is held : 0 — adds went to `dirty`, not the queue
Len() immediately after Done: 1 — Done() re-queued it, once
further Get()s to drain : 1 — 200 adds produced exactly one reconcile
Note the middle line: queue depth reads zero for the entire window in which 200 changes arrive. Len() counts the queue slice, not dirty. It's a reasonable signal for "work is backing up across many keys" and a poor one for "how much has changed".
The other half of the guarantee is that no two workers ever hold the same key. That's a concurrency claim, so it deserves a concurrency test — 16 workers, 64 keys, 320,000 adds, under the race detector, with a detector that flags any instant two workers hold the same key. Zero violations. But a detector that has never fired proves nothing, so I ran the identical harness on a plain buffered channel:
client-go workqueue : 148,497 reconciles, 0 same-key violations, 1.79 s plain channel : 320,000 reconciles, 21,940 same-key violations, 0.58 s
The channel is faster and wrong — it finished in a third of the time by doing 2.2× more work, 21,940 pieces of which were on an object another worker was already reconciling. That's what "we replaced the workqueue with a channel for simplicity" buys.
5. Forget is not Delete, and getting it wrong is silent
Forget sounds like removal. It isn't:
// Forget indicates that an item is finished being retried. ... This only clears the
// `rateLimiter`, you still have to call `Done` on the queue.It touches exactly one thing — the rate limiter's failure count. Done is what releases the key from processing. Call Forget and skip Done, and:
100 further Add("obj") calls, then Len()=0
Get() timed out after 500ms: the key is NEVER returned again
=> this object has silently stopped reconciling. No error. No log.
Every observable reads healthy: depth zero, no failing reconciles, nothing logged, rate limiter tracking nothing. One object has simply left the control loop. This is precisely why controller-runtime registers defer c.Queue.Done(obj) immediately after Get, before any work happens — an early return added years later can't skip it, and neither can a panic.
The mirror-image mistake, calling Done but forgetting Forget, is far kinder: the rate limiter tracks that key's failures forever, inflating later backoff, but the object keeps reconciling. One is a memory leak; the other is an object that leaves the cluster's control loop. Only one of them is protected by the language.
While I was in there I read the retry curve off the default rate limiter, since it explains a shape people see in incidents: delays double from 5 ms and hit the 1000 s ceiling at failure #19 — under an hour of wall-clock. After that a broken object is retried every 16 minutes 40 seconds, indefinitely, until one success calls Forget. So a dependency that's down for twenty minutes leaves your controller looking wedged long after the fault is gone.
How I worked, and why it changed the answers
I wrote a research note from source first, then built runnable demos for every claim — and the demos corrected the note three times. DeltaFIFO doesn't drop deltas. Collapse isn't a constant. And one measurement was vacuously true: I'd compared resourceVersion between an event object and the cached one to prove the cache is updated before handlers fire, and it reported a perfect result — because the fake clientset never sets resourceVersion at all, so both sides parsed to zero and every comparison trivially matched. The conclusion survived a rewrite against the data payload; the evidence for it didn't.
That's the argument for running things. Reading the source tells you what it says. Running it tells you what your test actually measured.