Support building `object_store` and `parquet` on wasm32-unknown-unknown target by jondo2010 · Pull Request #2896 · apache/arrow-rs · GitHub
Skip to content

Support building object_store and parquet on wasm32-unknown-unknown target#2896

Merged
tustvold merged 2 commits into
apache:masterfrom
rerun-io:build_for_wasm
Oct 26, 2022
Merged

Support building object_store and parquet on wasm32-unknown-unknown target#2896
tustvold merged 2 commits into
apache:masterfrom
rerun-io:build_for_wasm

Conversation

@jondo2010

@jondo2010 jondo2010 commented Oct 19, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #180.

Rationale for this change

My goal is to get datafusion building and running on the wasm32-unknown-unknown target, and getting object_store and parquet are the first steps.

What changes are included in this PR?

This PR adds some conditional compilation in object_store (mostly around disabling local filesystem access), as well as shifting some dependencies to dev, and removing un-used features in deps (i.e. tokio).

Are there any user-facing changes?

None expected, this is purely a build-support change.

@github-actions github-actions Bot added object-store parquet Changes to the parquet crate labels Oct 19, 2022
@jondo2010 jondo2010 mentioned this pull request Oct 24, 2022
Open
@jondo2010 jondo2010 changed the title Support building object_store on wasm32-unknown-unknown target Support building object_store and parquet on wasm32-unknown-unknown target Oct 24, 2022
@jondo2010 jondo2010 marked this pull request as ready for review October 24, 2022 09:04
tustvold
tustvold reviewed Oct 24, 2022
View reviewed changes
Comment thread .github/workflows/parquet.yml Outdated

@tustvold tustvold Oct 24, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI compiling with lz4 appears to fail

@tustvold

tustvold commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

Trying to compile object_store with non-default features returns an error about getrandom which appears related to https://docs.rs/getrandom/latest/getrandom/#webassembly-support

I'm not familiar enough with WASM to know if this is an issue or not?

@jondo2010
Support building object_store on wasm32-unknown-unknown target
b141ded 
* Added cargo check step to parquet workflow for wasm32-unknown-unknown
* Added compile-time warning for unsupported cloud features when compiling with wasm32
* Added cargo check features to the parquet github workflow.
* Added a section to the README.md for parquet
@jondo2010

jondo2010 commented Oct 25, 2022

Copy link
Copy Markdown
Contributor Author

Yep, wasm32-unknown-unknown is only going to support a limited set of features on parquet and object_store, in particular, the cloud storage features in object_store won't work, and some of the compression features on parquet won't work. I've documented everything in the respective READMEs now, and I also added a compile-time warning for object_store.

@tustvold tustvold merged commit f812d2c into apache:master Oct 26, 2022
@ursabot

ursabot commented Oct 26, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 4d5d10d and contender = f812d2c. f812d2c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@tustvold tustvold mentioned this pull request Oct 27, 2022
Merged
This was referenced Dec 5, 2022
Merged
Merged
@kylebarron kylebarron mentioned this pull request Feb 21, 2023
Open
@kylebarron

kylebarron commented Feb 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

alamb pushed a commit to alamb/arrow-rs that referenced this pull request Mar 20, 2025
@jondo2010
Support building object_store and parquet on wasm32-unknown-unkno…
e2e7dba 
…wn target (apache#2896)

* Support building object_store on wasm32-unknown-unknown target

* Added cargo check step to parquet workflow for wasm32-unknown-unknown
* Added compile-time warning for unsupported cloud features when compiling with wasm32
* Added cargo check features to the parquet github workflow.
* Added a section to the README.md for parquet

* * Added wasm32-unknown-unknown section to the object_store README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tustvold tustvold tustvold approved these changes

Assignees

No one assigned

Labels

parquet Changes to the parquet crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Parquet does not support wasm32-unknown-unknown target

4 participants

@jondo2010 @tustvold @ursabot @kylebarron