Repository navigation
Send requests with human User-Agent header to reduce false positives - #134
Conversation
…tives. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36
|
I love the idea, but I'm just a little bit scared that it's going to get folks IPs banned :) Usually not respecting robots.txt, and faking out the user agent can get you into a little trouble. I'm not saying no, just saying I probably need to do a bit of research. |
|
If we add it as an argument, it gives the choice to the people to use it or not. |
JustinBeckwith
left a comment
JustinBeckwith
left a comment
There was a problem hiding this comment.
This generally looks good. Could I trouble you to add a test to validate the header is sent?
| let res = await gaxios.request<string>({ | ||
| method: opts.crawl ? 'GET' : 'HEAD', | ||
| url: opts.url.href, | ||
| headers: { |
There was a problem hiding this comment.
Let's define this once, globally, and then use it in both places where we need it.
|
👋 @jakejarvis this is something I'd love to land! Any way I could trouble you for a test? |
|
@JustinBeckwith Of course, really sorry for neglecting this! Will re-visit today 😊 |
There might be a better way to do this, but after some quick testing I've found specifying a User-Agent header that resembles a human (i.e. Chrome on macOS) drastically cut down on non-404 errors (401, 403, 502, etc.).
I added this quickly without much familiarity with the code (did run
testandfix, though) so feel free to make/suggest a better implementation. Maybe it's worth making the specific agent string customizable via an option/flag too, but I can't think of a scenario where that would really be necessary?Thanks! :)