fix(linkedin): deduplicate only successfully parsed cards by O-marqs · Pull Request #426 · speedyapply/JobSpy · GitHub
Skip to content

fix(linkedin): deduplicate only successfully parsed cards - #426

Open
O-marqs wants to merge 1 commit into
speedyapply:mainfrom
O-marqs:fix/linkedin-dedup-after-parse-offline
Open

O-marqs wants to merge 1 commit into
speedyapply:mainfrom
O-marqs:fix/linkedin-dedup-after-parse-offline

Conversation

@O-marqs

@O-marqs O-marqs commented Oct 20, 2026

Copy link
Copy Markdown

LinkedIn marks a search card's ID as seen before parsing it. If that card is malformed, a later valid card with the same ID is discarded, including on a subsequent page. A synthetic response containing a card without a title followed by a complete card reproduces the loss on current main.

This change records an ID only after _process_job succeeds. A check immediately before parsing still skips IDs already successfully processed, so duplicate valid cards are returned and enriched once. The production diff is three added lines and one removed line in the existing loop. There is no architecture change, new public API, dependency or model change.

Validation against main 8dd83cdd194b6e27f6487eb09b2b2a10117d9a1c:

  • Before the production change: five test methods pass and three fail, with six failed assertions/subtests reproducing lost jobs.
  • With the change: all eight test methods pass. Coverage includes both existing job and jobPosting URNs, recovery across pages, one enrichment per valid ID, existing page bounds, result limits, request filters and public DataFrame output.
  • python -m unittest discover -s tests -p test_linkedin_dedup.py -v
  • ruff check jobspy tests and ruff format --check jobspy tests pass.
  • Strict mypy for jobspy/linkedin and full configured mypy both pass. Final validation used a clean isolated Python 3.11 environment with the upstream-declared MCP extra (mcp 2.3.0) and development dependencies installed. No versioned dependency or configuration was changed.
  • git diff --check passes.

Tests use standard-library unittest and block sockets and both HTTP transports. No real LinkedIn request was made. The fixtures demonstrate the parser contract, rather than compatibility with today's live HTML.

Pagination bounds and request parameters remain unchanged. A malformed repeated card may now be parsed again until a valid version is found or an existing stop condition is reached; this adds no retry request mechanism. Salary parsing and scraping policy are unchanged, and this does not duplicate or depend on salary PR #421.

@kirkwat

kirkwat commented Oct 20, 2026

Copy link
Copy Markdown
Member

@O-marqs

O-marqs commented Oct 20, 2026

Copy link
Copy Markdown
Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants