Activity

1. klaudworks

   changed the title [-]Parallelize S3 .index + .kfs downloads in processors[/-] [+]perf: parallelize S3 .index + .kfs downloads in processors[/+] on Feb 28, 2026
2. novatechflow commented on May 18, 2026

   

   novatechflow

   on May 18, 2026

   Collaborator

   More actions

   Took a pass on this since it's stale a few month - the best fix is a bit different from just parallelizing the two downloads.

   In both processor decoders, we're downloading .index and .kfs sequentially, but the decoder path is not actually using the parsed .index entries for decoding. It;s validating the index and then throwing that result away. So parallelizing those two GETs would reduce wall time a bit, but it would still keep an unnecessary S3 read in the hot path.

   Better approach is to remove the .index download from the processor decode path entirely and keeps segment/index pairing validation in discovery, where it already belongs. That gives us a cleaner fix than parallel fetches for this specific path.

   While tracing this, we also found a related storage-side issue: for PITR/restore, .index does matter, just not in the processor decoder. That bug will handled separately in the same branch by making the restore path truncate the final segment at the cutoff and regenerate a matching .index, instead of only restoring whole segments.

   So for this issue, the outcome in my local tests are:

   • processor hot path: stop downloading .index when decode does not use it
   • discovery: still requires valid .kfs + .index pairing
   • PITR/restore: keep .index as part of the exact recovery path where it is actually needed

   I’ll link the PR once it’s up.
3. novatechflow

   linked a pull request that will close this issueProcessor durability and processor parallelism #144on May 18, 2026
4. novatechflow

   self-assigned this

   on May 18, 2026
5. kamir

   closed this as completedin #144on May 21, 2026