fix: properly stream `uploadData` in `protocol.handle()` by BurningEnlightenment · Pull Request #41052 · electron/electron · GitHub
Skip to content

fix: properly stream uploadData in protocol.handle() - #41052

Merged
jkleinsc merged 6 commits into
electron:mainfrom
BurningEnlightenment:dev/protocol_handle_refactor
Feb 16, 2024
Merged

jkleinsc merged 6 commits into
electron:mainfrom
BurningEnlightenment:dev/protocol_handle_refactor

Conversation

@BurningEnlightenment

Copy link
Copy Markdown
Contributor

Description of Change

Fixes a few annoying bugs which prevented requests intercepted with protocol.handle() from being properly forwarded to net.fetch():
Fixes #39658
Fixes #40754
Fixes #40826

I considered submitting multiple pull requests, but the fixes would've generated conflicts with each other. In essence I've just streamlined the pull() implementation of convertToRequestBody() and added a test case for every fixed bug. I also added an explicit error condition in case an unknown/new chunk type gets pushed into the protocol handler. Otherwise it is very
unobvious and hard to figure out why your request has been mutilated.

If you like, I am willing to fix the chunk type definitions / remove the any casts. However, I'd need some guidance on whether I should introduce local types or add them to the public API.

Additionally I noticed while reviewing the C++ parts that the cb, which is passed to the actual handler function async (preq: ProtocolRequest, cb: any), never accesses the statusText property. Is there a reason for passing statusText in anyways?

cc: @nornagon, @codebytere

Checklist

Release Notes

Notes: Fix various bugs which could prevent forwarding requests intercepted with protocol.handle().

@welcome

welcome Bot commented Jan 18, 2024

Copy link
Copy Markdown

@electron-cation electron-cation Bot added the new-pr 🌱 PR opened recently label Jan 18, 2024
@codebytere

Copy link
Copy Markdown
Member

hey @BurningEnlightenment - thanks for this! We'll try to get to reviewing it soon - @nornagon is out until the end of the month so we might want to wait until he can take a look as well.

@BurningEnlightenment

Copy link
Copy Markdown
Contributor Author

No pressure, I'm applying these fixes currently via monkey patching protocol.handle().

Some additional thoughts:

  • It would probably be nicer to disentangle the stream-flattening from the chunk => stream conversion instead of doing it all in pull().
  • We should probably forward cancel events to the inner stream, otherwise a file handle might not get closed until GC kicks in.
  • Make a type: 'bytes' stream and supply an explicit backpressure strategy.
  • Adding support for BYOBReader could reduce memory pressure in certain scenarios

@ckerr

ckerr commented Jan 22, 2024

Copy link
Copy Markdown
Member

Additionally I noticed while reviewing the C++ parts that the cb, which is passed to the actual handler function async (preq: ProtocolRequest, cb: any), never accesses the statusText property. Is there a reason for passing statusText in anyways?

@codebytere and @nornagon know the context better than me but it looks like statusText was added in the initial feature commit @ #36674 but AFAICT isn't used or documented. So maybe we should remove it?

@electron-cation electron-cation Bot removed the new-pr 🌱 PR opened recently label Jan 25, 2024

@nornagon nornagon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, these are good changes (and good tests!)

Regarding the unused statusText property, good catch, that should probably be plumbed through to the underlying HttpResponseHeaders—currently we use net::GetHttpReasonPhrase to derive the status text from the status code instead.

Comment on lines +77 to +82
// Note that even though `getBlobData()` is a `Session` API, it doesn't
// actually use the `Session` context. Its implementation solely relies
// on global variables which allows us to implement this feature without
// knowledge of the `Session` associated with the current request by
// always pulling `Blob` data out of the default `Session`.
controller.enqueue(await session.defaultSession.getBlobData(chunk.blobUUID));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this should be fixed to not be an instance method then 😅 but won't block on it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you. I just wanted to sidestep a potential API design review process (and it would've bumped this from semver/patch to at least a semver/minor change).

Btw. if the getBlobData() API gets touched it might also be prudent to rename blobUUID to just blobId as the implementation hasn't used UUIDs for quite some time. Should I submit a follow up PR potentially including some further changes outlined in my previous comments?

@nornagon nornagon added target/28-x-y PR should also be added to the "28-x-y" branch. target/29-x-y PR should also be added to the "29-x-y" branch. semver/patch backwards-compatible bug fixes component/protocol labels Feb 2, 2024
@zcbenz
zcbenz force-pushed the dev/protocol_handle_refactor branch from 51031a5 to 50817e2 Compare February 8, 2024 01:49
@codebytere

Copy link
Copy Markdown
Member

@BurningEnlightenment would you mind rebasing this to fix CI? there was an issue with WOA CI that's since been addressed. Beyond that:

Should I submit a follow up PR potentially including some further changes outlined in my previous comments?

Yes, I think your best course of action is to open a follow-up! The scope of this PR is good to merge for now and i'd rather address the issues herein first.

@BurningEnlightenment
BurningEnlightenment force-pushed the dev/protocol_handle_refactor branch from 50817e2 to 885ae8c Compare February 14, 2024 16:47
@BurningEnlightenment

Copy link
Copy Markdown
Contributor Author

@codebytere done. LGTM

@jkleinsc
jkleinsc merged commit 80906c0 into electron:main Feb 16, 2024
@welcome

welcome Bot commented Feb 16, 2024

Copy link
Copy Markdown

Congrats on merging your first pull request! 🎉🎉🎉

@release-clerk

release-clerk Bot commented Feb 16, 2024

Copy link
Copy Markdown

Release Notes Persisted

Fix various bugs which could prevent forwarding requests intercepted with protocol.handle().

@trop

trop Bot commented Feb 16, 2024

Copy link
Copy Markdown
Contributor

I have automatically backported this PR to "28-x-y", please check out #41358

@trop trop Bot added in-flight/28-x-y and removed target/28-x-y PR should also be added to the "28-x-y" branch. labels Feb 16, 2024
@trop

trop Bot commented Feb 16, 2024

Copy link
Copy Markdown
Contributor

@trop trop Bot added in-flight/29-x-y merged/29-x-y PR was merged to the "29-x-y" branch. and removed target/29-x-y PR should also be added to the "29-x-y" branch. in-flight/29-x-y labels Feb 16, 2024
@trop trop Bot removed the in-flight/28-x-y label Mar 25, 2024
mnvr added a commit to ente/ente that referenced this pull request Apr 16, 2024
This was primarily for getting the latest Electron, but I ran `yarn
upgrade-interactive` and upgraded the other non-breaking deps (mostly dev) too.

Reason for wanting electron is to try and see if this backport fixes the issue
with our streams not getting faithfully written:
electron/electron#41052

In some ad-hoc and quick testing, I noticed that the new `writeStream` we've
implemented works fine for files up to 128 K, presumably some chunk size, but
then begins to diverge. Sounds similar (but not exactly the same) as this issue:
electron/electron#39658

Unfortunately, this didn't fix the issue we're facing, so our case is perhaps
different.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/protocol merged/29-x-y PR was merged to the "29-x-y" branch. semver/patch backwards-compatible bug fixes

Projects

None yet

6 participants