{{ message }}
Repository navigation
Make Europeana fetch viable #224
Description
Problem
The Europeana data set is too big to fetch all at once, or in its entirety.
- data providers: 4206
- rights: 64
Assuming approximately 1 second per query, the expected duration is approximately 75 hours.
Description
- Add counts for rights only (not including data providers)
- Write a query plan on first run per quarter to map out which queries to make over 30 runs (30 days)
Alternatives
Parallel queries (probably considered impolite, Timid Robot will ask)
Additional context
Implementation
- I would be interested in implementing this feature.
Activity
oree-xx commentedon Nov 3, 2025
@TimidRobot This is interesting to see that Europeana has a lot of data providers.
But why is the status of the issue blocked?
Are you trying to identify or investigate something?
TimidRobot commentedon Nov 3, 2025
Yes
oree-xx commentedon Nov 5, 2025
@TimidRobot Is there a verdict yet on whether parallel queries can be considered?
TimidRobot commentedon Nov 5, 2025
This issue will be updated when there is more information.
TimidRobot commentedon Mar 6, 2026
TODO: