Let's improve graph.commit() performance using redis.pipeline() to sync state · Issue #82 · RedisGraph/redisgraph-py · GitHub
Skip to content

Let's improve graph.commit() performance using redis.pipeline() to sync state #82

Open

Description

@boris-42

Running commands one by one is slow as you have TCP overhead. Which can cause 10-100 performance degradation.
To avoid it there are redis pipelines https://redis.io/topics/pipelining

Redis pipeline in python client willreturn all results as array of responses, so change in redisgraph should be minimal to support them. Here is the bottleneck place https://github.com/RedisGraph/redisgraph-py/blob/master/redisgraph/graph.py#L127-L128

Are you OK if Implement this?

Activity

  1. gkorland commented on Jul 11, 2020

    @gkorland
    Contributor

    First, thanks for the suggestion to contribute the code!
    Second, you're right pipeline can improve performance dramatically and we already added this support in other clients.
    But, it can't replace the sync API but added as alternative API.

  2. boris-42 commented on Jul 11, 2020

    @boris-42
    ContributorAuthor

    @gkorland First of all great that this is something on mind ;).

    So you suggest to implement something like bulk_queries(queries) ?

    Another thing is this place of code (as well please take a look at #81 )

        def commit(self):
            """
            Create entire graph.
            """
            if len(self.nodes) == 0 and len(self.edges) == 0:
                return None
    
            query = 'CREATE '
            for _, node in self.nodes.items():
                query += str(node) + ','
    
            query += ','.join([str(edge) for edge in self.edges])
    
            # Discard leading comma.
            if query[-1] is ',':
                query = query[:-1]
    
            return self.query(query)
    

    The problem is that it only works for creation of things but not updates.
    As soon as we add functionality to update/delete we would switch to multiple Graph commands e.g.:
    MATCH + DELETE, MERGE + SET in such situation we would need to use multiple queries (that was sort of initial point, sorry for being unclear it was a bit late)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions