Repository navigation
AdaptivePlaywrightCrawler: the inner crawlers' options are rejected, and the HTTP-only attempt always ignores TLS errors #4260
Description
Context
Migrating WCC to v4. We pass ignoreTlsErrors, additionalMimeTypes and ignoreIframes to every crawler type. The adaptive crawler rejects them, so we have to leave them out, and some of our settings quietly stop applying. #4219 handles navigationTimeoutSecs; this issue is about the rest.
Which package is this bug report for?
@crawlee/playwright (AdaptivePlaywrightCrawler)
Issue description
The constructor only accepts BasicCrawlerOptions, so options that belong to the inner crawlers fail with Unrecognized key: "…" in BasicCrawlerOptions:
ignoreTlsErrorsandadditionalMimeTypes(HTTP crawler);ignoreIframes(Playwright crawler).
ignoreTlsErrors can't be worked around either. The inner CheerioCrawler is built from ...rest, so it keeps the HTTP crawler's default ignoreTlsErrors: true and sends it with every request. A per-request flag can only turn ignoring on, so httpClient: new ImpitHttpClient({ ignoreTlsErrors: false }) has no effect. The HTTP-only attempt accepts a self-signed certificate that the same client rejects in a plain CheerioCrawler.
Expected behavior: these options are accepted and forwarded to the inner crawlers, the way #4088 forwarded the browser options and #4219 forwards navigationTimeoutSecs.
Code sample
import { readFileSync } from 'node:fs';
import { createServer } from 'node:https';
import { ImpitHttpClient } from '@crawlee/impit-client';
import { AdaptivePlaywrightCrawler } from '@crawlee/playwright';
for (const option of [{ ignoreTlsErrors: false }, { additionalMimeTypes: ['application/pdf'] }, { ignoreIframes: true }]) {
try {
new AdaptivePlaywrightCrawler({ ...option, requestHandler: async () => {} });
} catch (error) {
console.log(error.message.split('\n')[0]);
}
}
// openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 1 -subj "/CN=localhost"
const server = createServer({ key: readFileSync('key.pem'), cert: readFileSync('cert.pem') }, (req, res) => {
res.writeHead(200, { 'content-type': 'text/html' }).end('<html><body>ok</body></html>');
});
await new Promise((resolve) => server.listen(0, resolve));
const crawler = new AdaptivePlaywrightCrawler({
httpClient: new ImpitHttpClient({ ignoreTlsErrors: false }),
// Forces the HTTP-only attempt.
renderingTypePredictor: {
predict: () => ({ renderingType: 'static', detectionProbabilityRecommendation: 0 }),
storeResult: () => {},
initialize: async () => {},
},
maxRequestRetries: 0,
requestHandler: async () => console.log('handled over HTTP despite the self-signed certificate'),
failedRequestHandler: async () => console.log('failed, as expected'),
});
await crawler.run([`https://localhost:${server.address().port}/`]);
server.close();Output:
Unrecognized key: "ignoreTlsErrors" in `BasicCrawlerOptions`
Unrecognized key: "additionalMimeTypes" in `BasicCrawlerOptions`
Unrecognized key: "ignoreIframes" in `BasicCrawlerOptions`
handled over HTTP despite the self-signed certificate
A CheerioCrawler with the same httpClient and ignoreTlsErrors: false fails that request.
Package version
@crawlee/playwright@4.0.0-rc.1@crawlee/impit-client@4.0.0-rc.1impit@0.14.5playwright@1.62.1
Node.js version
v24.15.0
Operating system
macOS 26.6.2 (arm64)
reproduced locally.
Activity