feat: parse SearchApi AI answer engines in SearchApiWebSearch by hpdkhoa · Pull Request #4041 · deepset-ai/haystack-core-integrations · GitHub
Skip to content

feat: parse SearchApi AI answer engines in SearchApiWebSearch - #4041

Open
hpdkhoa wants to merge 3 commits into
deepset-ai:mainfrom
hpdkhoa:feat/searchapi-ai-answer-engines
Open

hpdkhoa wants to merge 3 commits into
deepset-ai:mainfrom
hpdkhoa:feat/searchapi-ai-answer-engines

Conversation

@hpdkhoa

@hpdkhoa hpdkhoa commented Oct 1, 2026

Copy link
Copy Markdown

SearchApiWebSearch read organic_results only, so with search_params such as {"engine": "perplexity"} the request succeeded but the component returned no documents and no links. SearchApi's answer engines (google_ai_mode, perplexity, chatgpt, gemini, bing_copilot) share one response shape: the answer as markdown plus typed text_blocks, the cited sources as reference_links, and for Google AI Mode the web_results it drew from.

When a response carries that shape, the component now returns the synthesized answer as the first Document (content from the markdown field, rendered from text_blocks when the API omits it), one Document per reference link, then the web results. links holds the reference links followed by the web result links. Classic engines are unchanged.

Tests cover the parser with and without markdown, with web results, with an empty answer, and one run through the component with a mocked httpx.get.

Related Issues

None open. SearchApiWebSearch reads organic_results only, so with search_params={"engine": "perplexity"} (or google_ai_mode, chatgpt, gemini, bing_copilot) it returns no documents and no links although the request succeeds.

Proposed Changes

When a response carries the shared AI answer shape (text_blocks, markdown, reference_links), the component now returns the synthesized answer as the first Document (content is the markdown field, rendered from text_blocks when the API omits it), one Document per cited reference link, and for Google AI Mode the web_results it drew from. links holds the reference links, then the web result links. Documents for the classic engines are unchanged. The class docstring documents the behaviour.

How did you test it?

Unit tests for the parser (markdown present, markdown absent, web results, empty answer) and one run through run with a mocked httpx.get. hatch run fmt-check, hatch run test:unit and hatch run test:types equivalents pass locally (ruff, pytest with 21 unit tests, mypy). The integration test still requires SEARCHAPI_API_KEY.

Notes for the reviewer

_text_blocks_to_markdown and its two helpers are module-level functions rather than methods so the rendering stays easy to unit test and under the ruff complexity limits.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes (no issue to update)
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: feat

SearchApiWebSearch read organic_results only, so with search_params such as
{"engine": "perplexity"} the request succeeded but the component returned no
documents and no links. SearchApi's answer engines (google_ai_mode, perplexity,
chatgpt, gemini, bing_copilot) share one response shape: the answer as markdown
plus typed text_blocks, the cited sources as reference_links, and for Google AI
Mode the web_results it drew from.

When a response carries that shape, the component now returns the synthesized
answer as the first Document (content from the markdown field, rendered from
text_blocks when the API omits it), one Document per reference link, then the
web results. links holds the reference links followed by the web result links.
Classic engines are unchanged.

Tests cover the parser with and without markdown, with web results, with an
empty answer, and one run through the component with a mocked httpx.get.
@hpdkhoa
hpdkhoa requested a review from a team as a code owner October 1, 2026 15:28
@hpdkhoa
hpdkhoa requested review from bogdankostic and removed request for a team October 1, 2026 15:28
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Oct 1, 2026
@CLAassistant

CLAassistant commented Oct 1, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @hpdkhoa, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from bogdankostic October 1, 2026 16:31
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 1, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 1, 2026 16:31
@hpdkhoa
hpdkhoa marked this pull request as ready for review October 2, 2026 14:20
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (searchapi)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/searchapi/src/haystack_integrations/components/websearch/searchapi
  websearch.py 317-318, 330
Project Total  

This report was generated by python-coverage-comment-action

@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 2, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @hpdkhoa! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@hpdkhoa

hpdkhoa commented Oct 8, 2026

Copy link
Copy Markdown
Author

@bogdankostic bogdankostic self-assigned this Oct 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:searchapi type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants