feat(queuev2): cached timer queue reader with prefetch and time-based eviction

[arzonus]

What changed?

Adds a cached layer over the timer (scheduled) queue reader that keeps a look-ahead window of upcoming tasks in memory, eliminating repeated DB reads on the hot path. Relates to #7953.

Replaces four stacked PRs (#7954 #7955 #7956 #7957) with a single consolidated draft for easier review iteration.

Components:

InMemQueue (queue_mem.go) — sorted in-memory task store with PutTasks, GetTasks, LookAHead, LTrim, RTrimBySize, Clear. Deduplicates inserts by task key. RTrimBySize nils evicted interface slots for GC.

CachedQueueReader (queue_reader_cached.go) — wraps any QueueReader with an in-memory look-ahead cache. Three rollout modes: disabled (default), shadow (cache runs, DB always wins, mismatches logged), enabled (cache serves reads). Key behaviours: prefetch loop, time-eviction loop, Inject for new tasks, UpdateReadLevel for ack-level eviction. injectAllowedAfter is computed once at construction (now + WarmupGracePeriod) — no mutex needed.

CachedScheduledQueue (queue_scheduled_cached.go) — thin wrapper wiring NotifyNewTask → Inject, updateQueueStateFn → UpdateReadLevel, and reader lifecycle into the existing scheduled queue.

Config — nine new global dynamic config properties (no ShardID filter, apply uniformly per host). Feature flag: TimerProcessorEnableCachedScheduledQueue (default: off).

Simulation scenarios — two new scenarios (enabled + shadow mode) requiring #7952 for infra.

Why?

The timer queue currently issues a DB round-trip for every virtual slice read. For scheduled tasks most reads land within a predictable look-ahead window. A pre-fetched in-memory cache removes those round-trips in the common case.

How did you test it?

go test -race -count=100 -timeout=600s ./service/history/queuev2/...
make pr GEN_DIR=service/history/queuev2
make build

Unit test coverage for new code: 87.4%. Tests cover prefetch, eviction, shadow mismatch, inject races, lifecycle, RTrim guard (both shrink and raise cases).

Simulation:

./simulation/history/run.sh --scenario queuev2_scheduled_cache_shadow --dockerfile-suffix .local
./simulation/history/run.sh --scenario queuev2_scheduled_cache_enabled --dockerfile-suffix .local

Potential risks

• Feature-flagged behind TimerProcessorEnableCachedScheduledQueue (default: false) and TimerProcessorCachedQueueReaderMode (default: disabled). Zero code path changes when flags are off.
• When enabled: two background goroutines per shard (prefetch + time-eviction). Memory bounded by TimerProcessorCacheMaxSize (default: 1000 tasks per shard).

Release notes

N/A — internal change behind a feature flag.

Documentation Changes

N/A

[gitar-bot]

⚠️ Bug: JitDuration panics on out-of-range dynamic config coefficient

backoff.JitDuration panics if the coefficient is outside [0, 1]. PrefetchJitterCoefficient is a dynamic config property that can be changed at runtime to any float64 value. A misconfiguration (e.g. setting it to 1.5 or -0.1) will cause a panic in the prefetch goroutine, killing the shard's timer queue processing.

Since this is a per-shard goroutine operating in a hot loop, the blast radius is significant — every shard on the host would be affected if the dynamic config applies uniformly.

Suggested fix:

coeff := q.options.PrefetchJitterCoefficient()
if coeff < 0 {
    coeff = 0
} else if coeff > 1 {
    coeff = 1
}
return backoff.JitDuration(delay, coeff)

_{Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion}

[gitar-bot]

Code Review ⚠️ Changes requested 10 resolved / 11 findings

Implements a cached layer over the timer queue to eliminate repeated DB reads, but requires a fix for potential panics in JitDuration due to out-of-range dynamic configuration coefficients. Several issues concerning cache integrity, state consistency, and efficiency were resolved during this session.

⚠️ Bug: JitDuration panics on out-of-range dynamic config coefficient

📄 service/history/queuev2/queue_reader_cached.go:273

backoff.JitDuration panics if the coefficient is outside [0, 1]. PrefetchJitterCoefficient is a dynamic config property that can be changed at runtime to any float64 value. A misconfiguration (e.g. setting it to 1.5 or -0.1) will cause a panic in the prefetch goroutine, killing the shard's timer queue processing.

Since this is a per-shard goroutine operating in a hot loop, the blast radius is significant — every shard on the host would be affected if the dynamic config applies uniformly.

Suggested fix

coeff := q.options.PrefetchJitterCoefficient()
if coeff < 0 {
    coeff = 0
} else if coeff > 1 {
    coeff = 1
}
return backoff.JitDuration(delay, coeff)

✅ 10 resolved

✅ Edge Case: putTasks doesn't reset exclusiveUpperBound when RTrim empties cache

📄 service/history/queuev2/queue_reader_cached.go:426-435 📄 service/history/queuev2/queue_mem.go:191-195
When RTrimBySize(maxSize) is called with maxSize <= 0 (e.g. a dynamic config misconfiguration), it empties the queue and returns (MinimumHistoryTaskKey, true). In putTasks, the guard trimmed && newUpper.Greater(persistence.MinimumHistoryTaskKey) evaluates to true && false = false, so exclusiveUpperBound is not reset. The cache then thinks it covers [inclusiveLowerBound, oldExclusiveUpperBound) but holds zero tasks, causing GetTask to return empty results for ranges that may contain tasks in the DB.

This only triggers with MaxSize <= 0, which is unlikely with the default of 1000, but a defensive check would prevent silent data loss on misconfiguration.

✅ Quality: Mode string comparison has no validation for unknown values

📄 service/history/queuev2/queue_reader_cached.go:271-272
isDisabled() checks Mode() == "disabled" and isShadow() checks Mode() == "shadow". Any typo or unknown mode value (e.g. "enabeld") silently falls through to the enabled code path, which serves reads from cache. This could cause unintended behavior if someone misconfigures the dynamic config string.

Consider adding a log warning or defaulting to disabled for unknown values, particularly in GetTask and LookAHead where the mode drives correctness.

✅ Edge Case: Shadow mode false mismatches from cache eviction between reads

📄 service/history/queuev2/queue_reader_cached.go:604-618
In getTaskInShadow, the initial cache snapshot is taken under RLock (line 588-589), then the lock is released before querying the DB. If a mismatch is found, the cache is re-read (line 641-647) to filter benign races. However, between the two cache reads, timeEvict or UpdateReadLevel can advance inclusiveLowerBound and LTrim the cache, removing tasks that were in the original snapshot. A DB task that was in the first snapshot but got evicted before the second read would appear as a realMissingKey, creating a false mismatch alarm.

Since shadow mode is observational-only (DB result is always returned), this doesn't affect correctness — only alert noise. But spurious mismatch counters could erode confidence in the cache during rollout.

✅ Quality: InMemQueue.PutTasks is O(n*m) for unsorted bulk inserts

📄 service/history/queuev2/queue_mem.go:98-102
PutTasks calls putTask in a loop, and each putTask for an out-of-order insert does a binary search + copy shift. When inserting m tasks into a queue of n elements, and the tasks aren't pre-sorted, this is O(m * (log n + n)) in the worst case. For prefetch pages (up to 100 tasks by default into a 1000-element queue), this is acceptable, but if page sizes or cache sizes grow it could become a bottleneck.

The fast-path append optimization handles the common case of sorted inserts well. This is just a note for future scaling — a batch sort-merge would be O((n+m) log(n+m)).

✅ Edge Case: Non-atomic dynamic config Mode() reads can cause inconsistency

📄 service/history/queuev2/queue_reader_cached.go:567 📄 service/history/queuev2/queue_reader_cached.go:594
In GetTask (line 567), isDisabled() is checked first, then isShadow() is checked at line 594. Both call q.options.Mode() independently. Since this is a dynamic config property that can change between calls, a mode transition (e.g., shadow→enabled) between the two reads could cause the code to serve a cache result directly when the operator intended shadow mode, or vice versa. The window is small and the impact is transient (one request), but it's worth noting.

Similarly, prefetch() checks isDisabled() at line 303, and Inject() checks it at line 529 — both independently reading the mode.

...and 5 more resolved from earlier reviews

🤖 Prompt for agents

Code Review: Implements a cached layer over the timer queue to eliminate repeated DB reads, but requires a fix for potential panics in `JitDuration` due to out-of-range dynamic configuration coefficients. Several issues concerning cache integrity, state consistency, and efficiency were resolved during this session.

1. ⚠️ Bug: JitDuration panics on out-of-range dynamic config coefficient
   Files: service/history/queuev2/queue_reader_cached.go:273

   backoff.JitDuration panics if the coefficient is outside [0, 1]. PrefetchJitterCoefficient is a dynamic config property that can be changed at runtime to any float64 value. A misconfiguration (e.g. setting it to 1.5 or -0.1) will cause a panic in the prefetch goroutine, killing the shard's timer queue processing.
   
   Since this is a per-shard goroutine operating in a hot loop, the blast radius is significant — every shard on the host would be affected if the dynamic config applies uniformly.

   Suggested fix:
   coeff := q.options.PrefetchJitterCoefficient()
   if coeff < 0 {
       coeff = 0
   } else if coeff > 1 {
       coeff = 1
   }
   return backoff.JitDuration(delay, coeff)

Rules ✅ All requirements met

Repository Rules

✅ GitHub Issue Linking Requirement: The PR description includes a reference to #7953, which satisfies the issue linking requirement for cadence-workflow.

✅ PR Description Quality Standards: The PR description is comprehensive, includes all required sections with substantive content, follows the template guidance, and clearly explains the 'why' behind the technical implementation.

_{1 rule not applicable. Show all rules by commenting gitar display:verbose.}

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}