Every storage client re-implements logic that belongs to the storage frontends. DatasetClient has to provide iterate_items() and support clean, fields, omit, unwind, skip_hidden, flatten and view in get_data(), which the memory client only warns about and ignores. KeyValueStoreClient.set_value() takes an arbitrary Python value, so each backend serializes and parses by content type on its own (see _file_system/_key_value_store_client.py around line 271). RequestQueueMetadata carries the Apify-only had_multiple_clients field.

JS v4 made the backends plain transports and moved these semantics into Dataset and KeyValueStore. Port the same split, so a custom backend only moves bytes and pages.

Proposed solution

• DatasetClient: push_data() plus a single-page get_data(offset, limit, desc). Iteration and the Apify-style list options move to Dataset, which can apply them uniformly to every backend or pass them through where a backend supports them.
• KeyValueStoreClient: get_value / set_value deal in bytes plus a content type, and iterate_keys / iterate_entries become a single paged list_keys(prefix, exclusive_start_key, limit). KeyValueStore owns serialization, parsing and content-type inference, and a parse error surfaces to the caller.
• Drop Apify-specific fields such as had_multiple_clients from the shared metadata models (the Apify storage client can extend them).

Do this in the same release as the storage client to storage backend rename (#1592), so custom backends break once. Retry ownership between frontends and backends is out of scope here; JS has not settled it yet (apify/crawlee#4220).

Acceptance criteria

• The memory, file system, SQL and Redis clients contain no serialization or iteration logic.
• Every dataset list option behaves the same on every backend.
• Upgrade notes for custom storage clients.

JS references

• Dataset and KVS client interfaces: refactor!: Align DatasetClient and KeyValueStoreClient interfaces with their Python counterparts crawlee#3627
• KVS value semantics in the frontend: refactor!: Centralize KVS value semantics in @crawlee/core crawlee#3774
• Contract cleanup (list shapes, Apify-only fields): fix: Rectify inconsistencies in the StorageClient interface family crawlee#3800
• StorageClient overhaul: refactor!: Overhaul the StorageClient interface crawlee#3570
• Umbrella issue: Rework the storage client system crawlee#3075

This will be a breaking change.

✍️ Drafted by Claude Code