FilterPolicy.MERGE resolves a repeated field differently when both filters are AND · Issue #13208 · deepset-ai/haystack · GitHub
Skip to content

FilterPolicy.MERGE resolves a repeated field differently when both filters are AND #13208

Open
Harsh23Kashyap/haystack
#4

Description

@pcbeingused333

Describe the bug

Under FilterPolicy.MERGE, when the init and runtime filters constrain the same field, the result depends on how the filters are wrapped rather than on what they say.

  • If either side is a bare comparison filter, the runtime condition replaces the init condition on that field, and a warning is logged. This matches the docstring of FilterPolicy.MERGE: "Runtime filters are merged with init filters, with runtime filters overwriting init values."
  • If both sides are AND logical filters, combine_two_logical_filters concatenates the conditions. Both conditions on the field are kept, so the result is their intersection, and no warning is logged.

With the same intent (init meta.lang == "en", runtime meta.lang == "de"), three of the four shapes return the German document, and the fourth returns nothing without logging anything.

To Reproduce

from haystack import Document
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import FilterPolicy

store = InMemoryDocumentStore()
store.write_documents([
    Document(content="hello world", meta={"lang": "en"}),
    Document(content="hello welt", meta={"lang": "de"}),
])

en = {"field": "meta.lang", "operator": "==", "value": "en"}
de = {"field": "meta.lang", "operator": "==", "value": "de"}
AND = lambda *c: {"operator": "AND", "conditions": list(c)}

for init, runtime in [(en, AND(de)), (AND(en), de), (en, de), (AND(en), AND(de))]:
    retriever = InMemoryBM25Retriever(store, filters=init, filter_policy=FilterPolicy.MERGE)
    docs = retriever.run(query="hello", filters=runtime)["documents"]
    print([d.meta["lang"] for d in docs])

Output on main (bc4729f):

['de']   # init comparison + runtime AND   (warning: init filter ignored)
['de']   # init AND + runtime comparison   (warning: runtime filter overwrites)
['de']   # init comparison + runtime comparison   (warning)
[]       # init AND + runtime AND   (no warning)

Expected behavior

I'm not sure which behavior you want here, so I'm asking before proposing a change. #12816 settled that two AND filters should both narrow the result. That reading conflicts with "runtime filters overwriting init values" and with the three comparison paths. Two consistent options:

  1. In combine_two_logical_filters, when both are AND, drop init conditions whose field also appears in the runtime conditions, and log the same warning as combine_runtime_comparison_and_init_logical_filters.
  2. Keep the intersection, and at least log a warning when the two AND filters share a field, so an empty result is not silent. The FilterPolicy.MERGE docstring would also need to be updated.

I'm happy to open a PR for whichever you prefer.

System:

Activity

  1. added a commit that references this issue on Oct 25, 2026
    4205b94
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions