Ah interesting, we already have service_tier in the fixtures 👀

I'm not sure if this is correct 🤔 so we relay the chunks back to the client, and in each iteration we keep accumulating it?
For instance, assume 3 iterations:

• Iteration 1: previousUsage = 0, prc.lastUsage = u1 => cumulativeUsage: u1 => token usage u1 send to the client
• Iteration 2: previousUsage = u1, prc.lastUsage = u2 => cumulativeUsage: u1+u2 => token usage u1+u2 send to the client
• Iteration 3: previousUsage = u1+u2, prc.lastUsage = u3 => cumulativeUsage: u1+u2+u3 => token usage u1+u2+u3 send to the client

So basically the client would receive something like:

data: {... "usage": u1}
data: {... "usage": u1+u2}
data: {... "usage": u1+u2+u3}

This is WAI.
It used to work like that (or worked if it stream would only contain 1 usage chunk) until ad10045

Linked commit fixed issue where some OpenAI compatible providers return multiple usages for one request which resulted in old solution counting all usages from all chunks from all streams which was too much but also dropped the sum of all inner loop requests feature.

I see this was already before 👀 but could it happen that CompletionTokens == 0 but cache read/write > 0?

This test is kinda related to the above comment, it is only cumulative because the previous usage has a completion_tokens > 0. If all chunks have completion_tokens = 0, there would be no accumulation, which would also affect cost control, because we don't store RecordTokenUsage 🤔