GLM-5.3 · Heavy

dual-storage-architecture-grafana-codex

Grafana · Go · L · 300–799 lines · 4 files

The generalist plus 9 specialist lens agents in parallel, then the same verify and dedup steps. Judge: Claude Haiku 5.5.

Gold bugs found
8 of 9
Comments published
10
Comments matching a bug
8
Comments not matching
2
Cost of this PR
$2.623

Published comments (10)

highbugpkg/apiserver/rest/dualwriter_mode3.go:50-52

Async legacy writes use request context that cancels when handler returns — This defeats mode 3's stated purpose (legacy write as a safe measure) and only passes tests because local file/unified storage finishes within the response-write window. Apply the timeout to a detached context instead, e.g. `context.WithTimeoutCause(context.WithoutCancel(ctx), time.Second*10, ...)` (the same pattern Delete at line 110, Update at 135, and DeleteCollection at 163 must use).

highbugpkg/apiserver/rest/dualwriter_mode3.go:50-57

Fire-and-forget goroutines break legacy write ordering across dual stores — Mode 3 now performs every legacy write in a fire-and-forget goroutine (same pattern repeated at lines 108-114, 134-141 and 161-167). Dual-write correctness assumes operations reach LegacyStorage in the same order they were applied to Storage, but goroutine execution order is not guaranteed: a delete(name) followed by a create(name) can leave the object present in legacy storage while unified storage reports it deleted, and two sequential updates (A then B) can leave legacy holding A's stale state. **Also found in:** - `pkg/apiserver/rest/dualwriter_mode3.go:134-141`

highbugpkg/apiserver/rest/dualwriter_mode3.go:100-105

Delete returns early on NotFound from unified storage, orphaning legacy-only rows — Delete now returns immediately when `d.Storage.Delete` fails with any error, including NotFound, whereas the removed code tolerated NotFound and still called `d.Legacy.Delete` to clean up legacy-only rows. Objects that exist only in legacy storage (the migration drift this dual writer exists to reconcile) become undeletable: the caller gets a 404 and the legacy row is orphaned forever, reappearing if reads ever fall back to legacy.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:55

Data race: async legacy Create mutates obj during response serialization — Mode 1 explicitly guards this exact scenario with `created.DeepCopyObject()` before its async write (dualwriter_mode1.go:51). Pass `obj.DeepCopyObject()` into the goroutine.

mediumbugpkg/apiserver/rest/dualwriter_mode3_test.go:63-77

Mode 3 tests never assert the asynchronous legacy half of the dual write — dualwriter_mode3.go now performs Legacy Create/Update/Delete/DeleteCollection in fire-and-forget goroutines, but every TestMode3_* function returns after the synchronous Storage call and asserts only on its result — no AssertExpectations/AssertCalled or call-count check exists anywhere in the package, and nothing waits for the goroutines. If all the `go func() { d.Legacy.X(...) }` blocks were deleted from the implementation, every one of these tests would still pass (TestMode3_Delete/TestMode3_DeleteCollection don't even configure a legacy expectation).

mediumbugpkg/apiserver/rest/dualwriter_mode3_test.go:107-132

Shared mock.Mock makes Get/List tests unable to pin reads to Storage — TestMode3_Get (and TestMode3_List) cannot detect which backend served the read, because legacyStoreMock and storageMock wrap the same shared mock.Mock and dispatch to the same method names — the `m.On("Get", mock.Anything, name, ...)` expectation set via setupStorageFn is matched identically by d.Legacy.Get. If Mode3's Get/List were changed to read from LegacyStorage instead of Storage (mode 3's core read contract, and the difference between mode 2 and mode 3), these tests still pass.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:139

Async legacy Update re-derives object instead of writing unified storage result — Mode 2 still enforces replication with `&updateWrapper{upstream: objInfo, updated: obj}` (dualwriter_mode2.go:298), as did this file's previous implementation. Wrap the unified-storage result: `d.Legacy.Update(ctx, name, &updateWrapper{upstream: objInfo, updated: res}, ...)`.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:45

Create/Update storage error paths record duration under legacy metric — On unified-storage failure, Create records the failure with `d.recordLegacyDuration(true, ...)` using the storage start time — the error and elapsed time land in grafana_dual_writer_legacy_duration_seconds for a call that never touched legacy storage, and grafana_dual_writer_storage_duration_seconds never sees its own failures. Update's error path has the identical defect at line 129. **Also found in:** - `pkg/apiserver/rest/dualwriter_mode3.go:127-131` - `pkg/apiserver/rest/dualwriter_mode3.go:129-132`

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:166

DeleteCollection legacy goroutine records latency on the storage metric instead of the legacy metric — It must call `d.recordLegacyDuration(err != nil, ...)` like the legacy goroutines in Create (line 56), Delete (line 113), and Update (line 140).

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:106

Delete success path records object name instead of kind in storage duration metric — The Delete success path passes the object `name` where every other call site in this file passes `options.Kind`, so the kind label of grafana_dual_writer_storage_duration_seconds receives the object's name (e.g. "foo") instead of the resource kind. Per-kind aggregation breaks, and each distinct object name creates a new histogram series (buckets × is_error × mode × method), growing label cardinality without bound as objects are deleted. **Also found in:** - `pkg/apiserver/rest/dualwriter_mode3.go:2-7`

Gold bugs found (8)

mediumThe legacy DeleteCollection goroutine records its duration with recordStorageDuration instead of recordLegacyDuration, misattributing legacy latency and errors to the storage metric.
mediumDelete's success path records the storage metric with the object name in the kind label instead of options.Kind, creating per-object series and inconsistent labels.
mediumOn storage Create/Update failure the duration is recorded with recordLegacyDuration, attributing unified-store errors to the legacy metric.
highMode 3 legacy writes now run in background goroutines derived from the request context, which is cancelled when the request completes, so legacy writes can be cancelled and legacy storage falls out of sync.
highLegacy.Update now receives the original objInfo instead of a wrapper carrying the object stored in unified storage, so legacy recomputes the update and can diverge.
mediumThe Mode 3 tests never wait for or assert the asynchronous legacy writes, and legacy and storage share one mock, so removing the legacy calls would not fail them.
highIndependent per-request goroutines remove ordering between legacy writes to the same resource, so legacy can end up with an older version or a resurrected object.
highDualWriterMode3.Delete now returns on any unified-storage error, including NotFound, before deleting from legacy, so objects that exist only in legacy can no longer be deleted.

Gold bugs missed (1)

mediumMode 3 legacy-storage write errors (create/update/delete/delete-collection) are no longer logged after moving to async goroutines; only a duration metric with an error flag is recorded.