Ruler: Prevent counting 2xx responses as failed writes by rajagopalanand · Pull Request #6785 · cortexproject/cortex · GitHub
Skip to content

Ruler: Prevent counting 2xx responses as failed writes#6785

Merged
yeya24 merged 3 commits into
cortexproject:masterfrom
rajagopalanand:ha-label-errors
Jun 3, 2025
Merged

Ruler: Prevent counting 2xx responses as failed writes#6785
yeya24 merged 3 commits into
cortexproject:masterfrom
rajagopalanand:ha-label-errors

Conversation

@rajagopalanand

@rajagopalanand rajagopalanand commented Jun 3, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this PR does:

In some cases, the Distributor returns a 2xx with error. On the ruler side, this causes failedWrites to be incremented. This PR prevents failedWrites counter from being incremented in cases where the distributor returns a 2xx with an error

@rajagopalanand rajagopalanand marked this pull request as ready for review June 3, 2025 18:49
@dosubot dosubot Bot added the component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. label Jun 3, 2025
@yeya24

yeya24 commented Jun 3, 2025

Copy link
Copy Markdown
Contributor

@rajagopalanand

rajagopalanand commented Jun 3, 2025

Copy link
Copy Markdown
Contributor Author

In some cases, the Distributor returns a 2xx with error

What would be the error in this case? If it returns an error then it sounds indeed an error. Maybe there is something wrong when returning the error and the error is wrapped with a 2XX not 5XX

Just for posterity, the error is rpc error: code = Code(202) desc = replicas did not mach, rejecting sample ...

@rajagopalanand

rajagopalanand commented Jun 3, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

danielblando
danielblando approved these changes Jun 3, 2025
View reviewed changes

@danielblando danielblando left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense as we already ignore 4xx as failure

danielblando
danielblando reviewed Jun 3, 2025
View reviewed changes
Comment thread pkg/ruler/compat.go
@rajagopalanand
Prevent counting 2xx responses as failed writes
fd780ea 
Signed-off-by: Anand Rajagopal <anrajag@amazon.com>
@yeya24 yeya24 changed the title Prevent counting 2xx responses as failed writes Ruler: Prevent counting 2xx responses as failed writes Jun 3, 2025
rajagopalanand and others added 2 commits June 3, 2025 22:30
@rajagopalanand
Adding changelog
c88103f 
Signed-off-by: Anand Rajagopal <anrajag@amazon.com>
@yeya24
Merge branch 'master' into ha-label-errors
75327d8 
Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24 yeya24 enabled auto-merge (squash) June 3, 2025 23:01
@yeya24 yeya24 merged commit c9fa217 into cortexproject:master Jun 3, 2025
5 of 7 checks passed
@CharlieTLe CharlieTLe mentioned this pull request Jul 26, 2025
Closed
@CharlieTLe CharlieTLe mentioned this pull request Aug 10, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danielblando danielblando danielblando approved these changes

Assignees

No one assigned

Labels

component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. size/XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rajagopalanand @yeya24 @danielblando