Sunbelt Computer Software

PL/B Language Development and Support

Issues · crawler-commons/crawler-commons · GitHub

Labels Milestones

Support LLM-Policy entry in robots.txt

#564

· lewismc opened

on Jan 13, 2026

BasicURLNormalizer out-of-bound exception when normalize query part and path contains percent-encoded

#559

· sebastian-nagel opened

on Dec 9, 2025

·

Provide variant API methods taking URL parameters as URI or String

#556

· sebastian-nagel opened

on Dec 2, 2025

·

[Sitemaps] Erroneous value "own" vs. "purchase" of video attribute "price type"

#543

· sebastian-nagel opened

on Oct 7, 2025

·

Ignore/parse incorrect news:access values

#542

· rajda opened

on Oct 7, 2025

·

Support for Cloudflare Content-Signals policy in robots.txt parser

#541

· rzo1 opened

on Sep 25, 2025

Port URL unit tests from Web Platform Tests

#535

· sebastian-nagel opened

on Aug 12, 2025

·

Simplify and automate release process

#523

· sebastian-nagel opened

on Jul 9, 2025

·

Google deprecates some Image & Video extension tags

#375

· Chaiavi opened

on May 10, 2022

[Github pages] Add organization home page

#315

· sebastian-nagel opened

on Aug 14, 2021

[Github pages] Add project home page

#314

· sebastian-nagel opened

on Aug 14, 2021

Should we reformat CHANGES.txt using markdown (CHANGES.md) ?

#269

· Chaiavi opened

on Sep 11, 2019