Hi @Alegrowin, thanks for posing the question and starting the discussion on this topic. Personally, I am pretty excited about it! Basically you are looking for ACID transaction support on the storage level that any process can concurrently work and modify the data. We have two options.

1. Short-term: Currently Deep Lake is limited to multi-branch concurrent writes. In other words, each process needs to checkout a new branch and write the data. There also needs to be a final process that merges all branches together. In terms of performance at the moment, it would be similar to creating different datasets and combining them together at a later time (without metadata handling). We are happy to nail down low-hanging fruit optimizations here to make this work for you. e.g. being able to create virtual views of multiple branch datasets as if it is a single branch dataset.

2. Long-term: We are considering to start an initiative for supporting ACID transactions on storage level to support your use case and we would love to collaborate to make this work for you. If you are interested, let's jump on a call and discuss this further.