[release/7.0-staging] Send connection WINDOW_UPDATE before RTT PING by github-actions[bot] · Pull Request #98384 · dotnet/runtime · GitHub
Skip to content

[release/7.0-staging] Send connection WINDOW_UPDATE before RTT PING#98384

Closed
github-actions[bot] wants to merge 8 commits into
release/7.0-stagingfrom
backport/pr-97881-to-release/7.0-staging
Closed

[release/7.0-staging] Send connection WINDOW_UPDATE before RTT PING#98384
github-actions[bot] wants to merge 8 commits into
release/7.0-stagingfrom
backport/pr-97881-to-release/7.0-staging

Conversation

@github-actions

@github-actions github-actions Bot commented Feb 13, 2024

Copy link
Copy Markdown
Contributor

Backport of #97881 to release/7.0-staging

Fixes #97131

Customer Impact

Problem: gRPC server-streaming connections may be closed by server when communicating with service on GCP behind Google Cloud L7 External Load Balancer. It depends on size and frequency of gRPC messages.
Reported by customers in grpc/grpc-dotnet#2361 and grpc/grpc-dotnet#2358

gRPC is built on top of HTTP/2 protocol. In 'HttpClient' we use PING frames to measure RTT (Round-Trip Time) to leverage connections efficiently. Our usage of PING frames triggers DoS protection in Google Cloud L7 External Load Balancer and they close the connection from the server side. They are unwilling to change their implementation.
We worked with them to design ordering of frames mixed with WINDOW_UPDATE frames in a way that will avoid triggering their DoS protection and will allow us to measure RTT.

Regression

No. The behavior is specific to Google Cloud L7 External Load Balancer.

Testing

  • A functional test has been added to emulate the customer scenario.
  • The change has been manually tested against customer endpoint (i.e. GCP-hosted service behind Google Cloud L7 External Load Balancer).

Risk

Low-Medium

We are now sending higher volume of WINDOW_UPDATE frames. In theory, some servers might have problem with that. Mitigations:

  • We have a pre-existing switch to turn off RTT measurements entirely - System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing
  • grpc-go client implementation has exactly same behavior which we are proposing for many years now (although it is gRPC only, not general HTTP/2 as our)
  • We did code review of popular HTTP/2 implementations -- nginx and nghttp2. We didn't find any DoS protection mechanism that would be triggered by more WINDOW_UPDATE frames.

@ghost ghost added the area-System.Net.Http label Feb 13, 2024
@ghost

ghost commented Feb 13, 2024

Copy link
Copy Markdown

@antonfirsov antonfirsov added the Servicing-consider Issue for next servicing release review label Feb 14, 2024
@antonfirsov antonfirsov requested a review from a team February 14, 2024 23:45
@antonfirsov antonfirsov removed the Servicing-consider Issue for next servicing release review label Feb 14, 2024
@antonfirsov antonfirsov added the Servicing-consider Issue for next servicing release review label Feb 15, 2024
@antonfirsov

Copy link
Copy Markdown
Contributor

/azp run runtime-libraries-coreclr outerloop

1 similar comment
@antonfirsov

Copy link
Copy Markdown
Contributor

/azp run runtime-libraries-coreclr outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@carlossanlop

Copy link
Copy Markdown
Contributor

Friendly reminder that Monday March 11th is the Code Complete date for the April Release. Please make sure to get the CI green or confirm the failures are unrelated and send an email to Tactics requesting approval.

@rbhanda rbhanda removed the Servicing-consider Issue for next servicing release review label Mar 28, 2024
@antonfirsov

Copy link
Copy Markdown
Contributor

@antonfirsov antonfirsov closed this Apr 9, 2024
@jkotas jkotas deleted the backport/pr-97881-to-release/7.0-staging branch April 17, 2024 01:27
@github-actions github-actions Bot locked and limited conversation to collaborators May 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants