Summary

• apply URL rewrite expressions to every HTTP redirect target before requesting or evaluating skip rules
• support urlRewriteExpressions from JSON and JavaScript config files in the CLI
• preserve the effective final URL when redirects are followed manually so recursive relative links are resolved and crawled from the destination page
• isolate HTTP transport and redirect mechanics in a focused request module with one typed rewrite-and-skip policy
• document redirect rewrite behavior and JSON pattern syntax

Root cause

URL rewrites were only applied when a URL first entered crawl(). Redirect targets were followed inside makeRequest() without passing through the rewrite pipeline, so rewritten destinations could still hit the original host.

Manual redirect handling also relied on the fetch response's URL metadata for the effective destination. Tracking the final URL explicitly makes recursive parsing deterministic and ensures relative links discovered on the redirect destination use the correct base URL.

User impact

Redirected URLs now behave consistently with directly discovered URLs: rewrites run at every hop, skip rules see the rewritten destination, sensitive headers remain protected across origin changes, and destination documents continue to contribute recursively discovered links.

The maintainability review also reduced src/index.ts by 187 lines relative to main and kept redirect rewrite coverage out of the existing redirect test file, which would otherwise have crossed 1,000 lines.

Fixes #877.
Fixes #875.

Validation

• npm run build
• npm test -- --run — 259 tests passed
• npm run coverage — 94.03% statements, 89.06% branches, 98% functions
• npm run lint
• npm run docs-test — 17 links passed
• npm pack --dry-run --json
• npm run build-binaries
• Node CLI and macOS standalone binary --version smoke tests