cluster/afr: prefer majority errno in afr_final_errno by YifanZhang-311 · Pull Request #4854 · gluster/glusterfs · GitHub
Skip to content

cluster/afr: prefer majority errno in afr_final_errno - #4854

Open
YifanZhang-311 wants to merge 1 commit into
gluster:develfrom
YifanZhang-311:fix/afr-majority-errno
Open

YifanZhang-311 wants to merge 1 commit into
gluster:develfrom
YifanZhang-311:fix/afr-majority-errno

Conversation

@YifanZhang-311

Copy link
Copy Markdown
Contributor

Problem:
On a replicated volume with sharding, when one node's data disk is
pulled (brick process stays up), the brick becomes a silent failure:
AFR still sees all bricks up (up_count == child_count), but backend
I/O fails with EIO instead of ENOTCONN/ENOENT.

When a shard file is first touched, several concurrent transactions
may issue a concurrent mknod for the same shard. The mknod usually
succeeds on the healthy majority (returning EEXIST because a sibling
already created it), while the dead brick reports EIO. For the mixed
result [EEXIST EEXIST EIO], afr_higher_errno() treats EEXIST and EIO as
non-priority errnos and falls back to "last seen wins", so the final
errno reported to the shard layer becomes EIO. The shard layer then
treats any non-EEXIST failure as fatal and unwinds EIO to the client,
causing spurious guest I/O errors even though the shard exists on a
majority of healthy bricks.

Fix:
In afr_final_errno(), after aggregating the per-brick errnos with
afr_higher_errno(), also determine the errno agreed on by a strict
majority of the failed subvols (afr_majority_errno()). When such a
majority exists, combine it with the aggregated result through
afr_higher_errno() so that the majority signal wins over a single
minority deviant errno (typically a dead brick's EIO), while the
higher-priority errors (ENODATA/ENOENT/ESTALE/ENOSPC) are preserved.

The new behavior is gated by a volume option
"cluster.eexist-majority-fallback", enabled by default, which can be
turned off at runtime to restore the previous behavior.

This makes a single out-of-sync brick no longer flip the final errno
away from the majority result, so I/O proceeds from a healthy replica
instead of failing spuriously.

fix #4853

Problem:
On a replicated volume with sharding, when one node's data disk is
pulled (brick process stays up), the brick becomes a silent failure:
AFR still sees all bricks up (up_count == child_count), but backend
I/O fails with EIO instead of ENOTCONN/ENOENT.

When a shard file is first touched, several concurrent transactions
may issue a concurrent mknod for the same shard.  The mknod usually
succeeds on the healthy majority (returning EEXIST because a sibling
already created it), while the dead brick reports EIO.  For the mixed
result [EEXIST EEXIST EIO], afr_higher_errno() treats EEXIST and EIO as
non-priority errnos and falls back to "last seen wins", so the final
errno reported to the shard layer becomes EIO.  The shard layer then
treats any non-EEXIST failure as fatal and unwinds EIO to the client,
causing spurious guest I/O errors even though the shard exists on a
majority of healthy bricks.

Fix:
In afr_final_errno(), after aggregating the per-brick errnos with
afr_higher_errno(), also determine the errno agreed on by a strict
majority of the failed subvols (afr_majority_errno()).  When such a
majority exists, combine it with the aggregated result through
afr_higher_errno() so that the majority signal wins over a single
minority deviant errno (typically a dead brick's EIO), while the
higher-priority errors (ENODATA/ENOENT/ESTALE/ENOSPC) are preserved.

The new behavior is gated by a volume option
"cluster.eexist-majority-fallback", enabled by default, which can be
turned off at runtime to restore the previous behavior.

This makes a single out-of-sync brick no longer flip the final errno
away from the majority result, so I/O proceeds from a healthy replica
instead of failing spuriously.

fix gluster#4853
@YifanZhang-311 YifanZhang-311 changed the title Fix/afr-majority-errno cluster/afr: prefer majority errno in afr_final_errno Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replica volume returns spurious EIO during concurrent shard creation when one brick is down (silent disk failure)

1 participant