{{ message }}
Repository navigation
cluster/afr: prefer majority errno in afr_final_errno - #4854
Open
YifanZhang-311 wants to merge 1 commit into
Open
YifanZhang-311 wants to merge 1 commit into
YifanZhang-311 wants to merge 1 commit into
Conversation
Problem: On a replicated volume with sharding, when one node's data disk is pulled (brick process stays up), the brick becomes a silent failure: AFR still sees all bricks up (up_count == child_count), but backend I/O fails with EIO instead of ENOTCONN/ENOENT. When a shard file is first touched, several concurrent transactions may issue a concurrent mknod for the same shard. The mknod usually succeeds on the healthy majority (returning EEXIST because a sibling already created it), while the dead brick reports EIO. For the mixed result [EEXIST EEXIST EIO], afr_higher_errno() treats EEXIST and EIO as non-priority errnos and falls back to "last seen wins", so the final errno reported to the shard layer becomes EIO. The shard layer then treats any non-EEXIST failure as fatal and unwinds EIO to the client, causing spurious guest I/O errors even though the shard exists on a majority of healthy bricks. Fix: In afr_final_errno(), after aggregating the per-brick errnos with afr_higher_errno(), also determine the errno agreed on by a strict majority of the failed subvols (afr_majority_errno()). When such a majority exists, combine it with the aggregated result through afr_higher_errno() so that the majority signal wins over a single minority deviant errno (typically a dead brick's EIO), while the higher-priority errors (ENODATA/ENOENT/ESTALE/ENOSPC) are preserved. The new behavior is gated by a volume option "cluster.eexist-majority-fallback", enabled by default, which can be turned off at runtime to restore the previous behavior. This makes a single out-of-sync brick no longer flip the final errno away from the majority result, so I/O proceeds from a healthy replica instead of failing spuriously. fix gluster#4853
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Problem:
On a replicated volume with sharding, when one node's data disk is
pulled (brick process stays up), the brick becomes a silent failure:
AFR still sees all bricks up (up_count == child_count), but backend
I/O fails with EIO instead of ENOTCONN/ENOENT.
When a shard file is first touched, several concurrent transactions
may issue a concurrent mknod for the same shard. The mknod usually
succeeds on the healthy majority (returning EEXIST because a sibling
already created it), while the dead brick reports EIO. For the mixed
result [EEXIST EEXIST EIO], afr_higher_errno() treats EEXIST and EIO as
non-priority errnos and falls back to "last seen wins", so the final
errno reported to the shard layer becomes EIO. The shard layer then
treats any non-EEXIST failure as fatal and unwinds EIO to the client,
causing spurious guest I/O errors even though the shard exists on a
majority of healthy bricks.
Fix:
In afr_final_errno(), after aggregating the per-brick errnos with
afr_higher_errno(), also determine the errno agreed on by a strict
majority of the failed subvols (afr_majority_errno()). When such a
majority exists, combine it with the aggregated result through
afr_higher_errno() so that the majority signal wins over a single
minority deviant errno (typically a dead brick's EIO), while the
higher-priority errors (ENODATA/ENOENT/ESTALE/ENOSPC) are preserved.
The new behavior is gated by a volume option
"cluster.eexist-majority-fallback", enabled by default, which can be
turned off at runtime to restore the previous behavior.
This makes a single out-of-sync brick no longer flip the final errno
away from the majority result, so I/O proceeds from a healthy replica
instead of failing spuriously.
fix #4853