Describe the bug

DocumentSplitter(split_by="token") can emit empty documents from nonempty Unicode text despite the default skip_empty_documents=True.

A Unicode character can span several tiktoken tokens. decode_with_offsets then returns equal character offsets for adjacent tokens, so some token windows contain no text. The token path appends those windows without checking skip_empty_documents, unlike the character and custom-function paths. This can send empty chunks into document indexing/embedding pipelines.

Expected behavior

Skip empty token slices when skip_empty_documents=True, while preserving skip_empty_documents=False, source character offsets, page metadata, and overlap information.

To Reproduce

from haystack import Document
from haystack.components.preprocessors import DocumentSplitter

splitter = DocumentSplitter(
    split_by="token", split_length=1, tokenizer_encoding="cl100k_base"
)
chunks = splitter.run(documents=[Document(content="🍕🍣🚀")])["documents"]
print([ascii(chunk.content) for chunk in chunks])

On current main (c5e1335), this returns nine chunks: six empty strings and the three emoji. With a guard before appending zero-character windows, it returns only the three original emoji. The same text is reconstructed without replacement characters.

Additional context

The local change is one condition in _split_by_token, with an offline regression covering both values of skip_empty_documents and overlap on/off. The regression also checks reconstruction, character offsets, source IDs, contiguous split IDs, page numbers, and overlap ranges. All 88 unit tests in test_document_splitter.py pass; focused mypy and Ruff checks pass. No public API change is needed.

This report and the proposed patch were prepared with Codex; the reproduction was run using real tiktoken in a Hatch-managed environment. A PR will follow.

System:

• Windows, Python 3.13.2
• Haystack: main at c5e1335