Summary

A response served from the semantic cache is priced and deducted from a managed key as if the provider had been called. RunPostPlugins skips the SkipOnCacheHit plugins (cost, credits) only when cache_status == "hit_exact", but the cache plugin also answers with hit_semantic and hit_conditional, and sets rc.Flags.CacheHit for all three. The cached response keeps the usage of the call that filled the cache, so a semantic hit is billed at the full price of that call. The engine now reads rc.Flags.CacheHit, which is what the rate limiter already reads for the same reason since 720ed01 (#3570).

Linked issues

Closes #3579

AI use

AI use: Claude Code found the bug while reading the gateway, and wrote the fix and both tests. I reviewed the diff, reproduced the failure on dev, and ran the checks below.

Type of change

• Bug fix (non-breaking change which fixes an issue)

E2E coverage

E2E: exempt (gateway-internal)

1) What changes were done

A. Engine

• internal/pipeline/engine.go: isCacheHit is rc.Flags.ShortCircuited && rc.Flags.CacheHit instead of matching one value of cache_status.

B. Comment

• internal/plugins/ratelimit/ratelimit.go: the comment said the engine only honours hit_exact. That is no longer true, so it now only states that every cache layer sets the flag. No code change there.

2) Why the changes were done

• User-facing: with the semantic cache on, a paraphrased question answered from the cache cost a managed key the full price of the original call. In the test below that is $0.26 per cached answer, which is also what the request log shows as cost.
• Technical: cache_status says which layer answered. CacheHit says whether any layer did, which is what the skip decision needs. Reading the flag also covers any future cache layer without touching the engine.

3) Tests written + scenarios each covers

internal/pipeline/engine_test.go

• TestRunPostPluginsSkipsEveryCacheHitKind: a SkipOnCacheHit plugin is skipped for hit_exact, hit_semantic and hit_conditional, and still runs on a miss.
• TestRunPostPluginsSkipOnCacheHitTimings: the existing fixture set only cache_status. It now also sets CacheHit, as the real cache plugin does.

internal/plugins/credits/cache_hit_test.go

• TestCacheHitDeductsNoCredits: the real cache plugin (exact and semantic store), cost plugin and credits plugin in one engine, with a managed key. A miss calls the provider and is deducted. An exact hit and a semantic hit make no provider call, set no cost and leave the balance unchanged.

Both new tests fail on dev:

--- FAIL: TestRunPostPluginsSkipsEveryCacheHitKind (0.00s)
    --- FAIL: TestRunPostPluginsSkipsEveryCacheHitKind/hit_semantic (0.00s)
        engine_test.go:362: cache_status=hit_semantic: plugin ran=true, want false
    --- FAIL: TestRunPostPluginsSkipsEveryCacheHitKind/hit_conditional (0.00s)
        engine_test.go:362: cache_status=hit_conditional: plugin ran=true, want false
--- FAIL: TestCacheHitDeductsNoCredits (0.00s)
    --- FAIL: TestCacheHitDeductsNoCredits/hit_semantic (0.00s)
        cache_hit_test.go:107: a hit_semantic was priced at cost=0.260000 although no provider call was made
        cache_hit_test.go:110: a hit_semantic deducted 260000 micros from the managed key

With the fix:

go test -count=1 ./...                                  82 packages ok, 0 FAIL
go test -race -count=1 ./internal/pipeline/ ./internal/plugins/...   all ok
go vet ./internal/pipeline/ ./internal/plugins/credits/ ./internal/plugins/ratelimit/   clean
gofmt -l on the four touched files                      clean

Not run: the backend bin/test stack and the e2e harness, since nothing outside agentcc-gateway changes and the e2e gateway config cannot enable the semantic cache or managed keys.

Generated with Claude Code