How to Create a Deep Lake Vector Store

Let's create a Vector Store in LangChain for storing and searching information about the Twitter OSS recommendation algorithm.

Downloading and Preprocessing the Data

First, let's import necessary packages and make sure the Activeloop and OpenAI keys are in the environmental variables ACTIVELOOP_TOKEN, OPENAI_API_KEY.

from deeplake.core.vectorstore import VectorStore
import openai
import os

Next, let's clone the Twitter OSS recommendation algorithm and define paths for for source data and the Vector Store.

!git clone https://github.com/twitter/the-algorithm

vector_store_path = '/vector_store_getting_started'
repo_path = '/the-algorithm'

Next, let's load all the files from the repo into list of data that will be added to the Vector Store (chunked_text and metadata). We use simple text chunking based on a constant number of characters.

CHUNK_SIZE = 1000

chunked_text = []
metadata = []
for dirpath, dirnames, filenames in os.walk(repo_path):
    for file in filenames:
        try: 
            full_path = os.path.join(dirpath,file)
            with open(full_path, 'r') as f:
               text = f.read()
            new_chunkned_text = [text[i:i+CHUNK_SIZE] for i in range(0,len(text), CHUNK_SIZE)]
            chunked_text += new_chunkned_text
            metadata += [{'filepath': full_path} for i in range(len(new_chunkned_text))]
        except Exception as e: 
            print(e)
            pass

Next, let's define an embedding function using OpenAI. It must work for a single string and a list of strings, so that it can both be used to embed a prompt and a batch of texts.

Finally, let's create the Deep Lake Vector Store and populate it with data. We use a default tensor configuration, which creates tensors with text (str), metadata(json), id (str, auto-populated), embedding (float32). Learn more about tensor customizability here.

The Vector Store's data structure can be summarized using vector_store.summary(), which shows 4 tensors with 21055 samples:

To create a vector store using pre-compute embeddings, instead of embedding_data and embedding_function, you may run:

Performing Vector Search

Deep Lake offers highly-flexible vector search and hybrid search options. First, let's show a simple example of vector search using default options, which performs simple cosine similarity search in Python on the client (your machine).

The search_results is a dictionary with keys for the text, score, id, and metadata, with data ordered by score. By default, the search returns the top 4 results which can be verified using:

If we examine the first returned text, it appears to contain the text about trust and safety models that is relevant to the prompt.

Returns:

We can also retrieve the corresponding filename from the metadata, which shows the top result came from the README.

Customization of Vector Search

You can customize your vector search with simple parameters, such as selecting the distance_metric and top k results:

The search now returns 10 search results:

The first search result with the L2 distance metric returns the same text as the previous Cos search:

Returns:

Full Customization of Vector Search

Deep Lake's Compute Engine can be used to rapidly execute a variety of different search logic. It is available with !pip install "deeplake[enterprise]" (Make sure to restart your kernel after installation), and it is only available for data stored in or connected to Deep Lake.

Let's load a representative Vector Store that is already stored in Deep Lake Tensor Database. If data is not being written, is advisable to use read_only = True.

The query should be constructed using the Tensor Query Language (TQL) syntax.

Let's run the query, noting that the query execution happens in the Managed Tensor Database, and not on the client.

If we examine the first returned text, it appears to contain the same text about trust and safety models that is relevant to the prompt.

Returns:

We can also retrieve the corresponding filename from the metadata, which shows the top result came from the README.

Deep Lake also offers a variety of search options depending on where data is stored (load, cloud, Deep Lake storage, etc.) and where query execution should take place (client side or server side)