Ig/as completed read by d-v-b · Pull Request #194 · d-v-b/zarr-python · GitHub
Skip to content

Ig/as completed read#194

Merged
d-v-b merged 12 commits into
d-v-b:perf/prepared-write-v2from
zarr-developers:ig/as_completed_read
Jun 18, 2026
Merged

Ig/as completed read#194
d-v-b merged 12 commits into
d-v-b:perf/prepared-write-v2from
zarr-developers:ig/as_completed_read

Conversation

@d-v-b

@d-v-b d-v-b commented Jun 15, 2026

Copy link
Copy Markdown
Owner

[Description of PR]

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@ilan-gold

ilan-gold commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown

@ilan-gold

ilan-gold commented Jun 15, 2026

Copy link
Copy Markdown

I am not sure we could expect this to make a difference on the current benchmark suite where the latencies all appear to be identical - we would need some sort of randomness injected, no? Otherwise as_completed is the same as iterating independent of completion (I would think).

@d-v-b

d-v-b commented Jun 15, 2026

Copy link
Copy Markdown
Owner Author

we would need some sort of randomness injected, no? Otherwise as_completed is the same as iterating independent of completion (I would think).

yeah, as_completed will be wash without some real noise on the wire. we could consider allowing latencystore to take a parameter that defines the distribution to sample from, instead of a constant.

@ilan-gold

ilan-gold commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown

I ran a small subset of the benchmarks using np.random.normal (see commit before this comment) calls with loc=.2 and scale=.05:

This PR:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃                                                                                                 Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│      test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.06s │        0.0% │   20.06s │     1 │
│   test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.67s │        0.0% │   20.67s │     1 │
│ test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     307.0ms │       15.0% │    1.93s │     5 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────┘

and previous:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃                                                                                                 Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│      test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.41s │        0.0% │   20.41s │     1 │
│   test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.54s │        0.0% │   20.54s │     1 │
│ test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     428.6ms │        6.2% │    3.32s │     7 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────┘

So that's a ~33% reduction on this benchmark. Not bad.

@ilan-gold

ilan-gold commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown

This PR:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃                                                                                                  Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│      test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.32s │        0.0% │   20.32s │     1 │
│   test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.15s │        0.0% │   20.15s │     1 │
│ test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     469.9ms │       10.4% │    2.72s │     5 │
│       test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.12s │        0.0% │   20.12s │     1 │
│    test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.32s │        0.0% │   20.32s │     1 │
│  test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     357.4ms │        8.7% │    2.88s │     7 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────┘

Base branch:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃                                                                                                  Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃ Iters ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│      test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.24s │        0.0% │   20.24s │     1 │
│   test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.29s │        0.0% │   20.29s │     1 │
│ test_write_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     510.6ms │        5.6% │    2.71s │     5 │
│       test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] │      20.23s │        0.0% │   20.23s │     1 │
│    test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] │      20.50s │        0.0% │   20.50s │     1 │
│  test_read_array[fused-latency=0.2-memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] │     437.1ms │        1.7% │    2.67s │     6 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────┘

FWIW I noticed these are both single-threaded unless I am missing something.

@d-v-b d-v-b merged commit 83d2d85 into d-v-b:perf/prepared-write-v2 Jun 18, 2026
4 checks passed
@d-v-b d-v-b deleted the ig/as_completed_read branch June 18, 2026 13:12
@d-v-b

d-v-b commented Jun 18, 2026

Copy link
Copy Markdown
Owner Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

needs release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@d-v-b @ilan-gold