http,https: add built-in proxy support in http/https.request and Agent · nodejs/node@7f654ce · GitHub
Skip to content

Commit 7f654ce

Browse files
joyeecheungaduh95
joyeecheung
authored and
aduh95
committed
http,https: add built-in proxy support in http/https.request and Agent
This patch implements proxy support for HTTP and HTTPS clients and agents in the `http` and `https` built-ins`. When NODE_USE_ENV_PROXY is set to 1, the default global agent would parse the HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy settings from the environment variables, and proxy the requests sent through the built-in http/https client accordingly. To support this, `http.Agent` and `https.Agent` now accept a few new options: - `proxyEnv`: when it's an object, the agent would read and parse the HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy properties from it, and apply them based on the protocol it uses to send requests. This option allows custom agents to reuse built-in proxy support by composing options. Global agents set this to `process.env` when NODE_USE_ENV_PROXY is 1. - `defaultPort` and `protocol`: these allow setting of the default port and protocol of the agents. We also need these when configuring proxy settings and deciding whether a request should be proxied. Implementation-wise, this adds a `ProxyConfig` internal class to handle parsing and application of proxy configurations. The configuration is parsed during agent construction. When requests are made, the `createConnection()` methods on the agents would check whether the request should be proxied. If yes, they either connect to the proxy server (in the case of HTTP reqeusts) or establish a tunnel (in the case of HTTPS requests) through either a TCP socket (if the proxy uses HTTP) or a TLS socket (if the proxy uses HTTPS). When proxying HTTPS requests through a tunnel, the connection listener is invoked after the tunnel is established. Tunnel establishment uses the timeout of the request options, if there is one. Otherwise it uses the timeout of the agent. If an error is encountered during tunnel establishment, an ERR_PROXY_TUNNEL would be emitted on the returned socket. If the proxy server sends a errored status code, the error would contain an `statusCode` property. If the error is caused by timeout, the error would contain a `proxyTunnelTimeout` property. This implementation honors the built-in socket pool and socket limits. Pooled sockets are still keyed by request endpoints, they are just connected to the proxy server instead, and the persistence of the connection can be maintained as long as the proxy server respects connection/proxy-connection or persist by default (HTTP/1.1) PR-URL: #58980 Refs: #57872 Refs: #8381 Refs: #15620 Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
1 parent 16dc53c commit 7f654ce

55 files changed

Lines changed: 3516 additions & 37 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎doc/api/errors.md‎

Lines changed: 12 additions & 0 deletions

‎doc/api/http.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,14 @@ http.get({
116116
<!-- YAML
117117
added: v0.3.4
118118
changes:
119+
- version:
120+
- REPLACEME
121+
pr-url: https://github.com/nodejs/node/pull/58980
122+
description: Add support for `proxyEnv`.
123+
- version:
124+
- REPLACEME
125+
pr-url: https://github.com/nodejs/node/pull/58980
126+
description: Add support for `defaultPort` and `protocol`.
119127
- version:
120128
- v15.6.0
121129
- v14.17.0
@@ -178,6 +186,20 @@ changes:
178186
**Default:** `'lifo'`.
179187
* `timeout` {number} Socket timeout in milliseconds.
180188
This will set the timeout when the socket is created.
189+
* `proxyEnv` {Object|undefined} Environment variables for proxy configuration.
190+
See [Built-in Proxy Support][] for details. **Default:** `undefined`
191+
* `HTTP_PROXY` {string|undefined} URL for the proxy server that HTTP requests should use.
192+
If undefined, no proxy is used for HTTP requests.
193+
* `HTTPS_PROXY` {string|undefined} URL for the proxy server that HTTPS requests should use.
194+
If undefined, no proxy is used for HTTPS requests.
195+
* `NO_PROXY` {string|undefined} Patterns specifying the endpoints
196+
that should not be routed through a proxy.
197+
* `http_proxy` {string|undefined} Same as `HTTP_PROXY`. If both are set, `http_proxy` takes precedence.
198+
* `https_proxy` {string|undefined} Same as `HTTPS_PROXY`. If both are set, `https_proxy` takes precedence.
199+
* `no_proxy` {string|undefined} Same as `NO_PROXY`. If both are set, `no_proxy` takes precedence.
200+
* `defaultPort` {number} Default port to use when the port is not specified
201+
in requests. **Default:** `80`.
202+
* `protocol` {string} The protocol to use for the agent. **Default:** `'http:'`.
181203

182204
`options` in [`socket.connect()`][] are also supported.
183205

@@ -4243,6 +4265,98 @@ added:
42434265
42444266
A browser-compatible implementation of {WebSocket}.
42454267
4268+
## Built-in Proxy Support
4269+
4270+
<!-- YAML
4271+
added: REPLACEME
4272+
-->
4273+
4274+
> Stability: 1.1 - Active development
4275+
4276+
When Node.js creates the global agent, it checks the `NODE_USE_ENV_PROXY`
4277+
environment variable. If it is set to `1`, the global agent will be constructed
4278+
with `proxyEnv: process.env`, enabling proxy support based on the environment variables.
4279+
4280+
Custom agents can also be created with proxy support by passing a
4281+
`proxyEnv` option when constructing the agent. The value can be `process.env`
4282+
if they just want to inherit the configuration from the environment variables,
4283+
or an object with specific setting overriding the environment.
4284+
4285+
The following properties of the `proxyEnv` are checked to configure proxy
4286+
support.
4287+
4288+
* `HTTP_PROXY` or `http_proxy`: Proxy server URL for HTTP requests. If both are set,
4289+
`http_proxy` takes precedence.
4290+
* `HTTPS_PROXY` or `https_proxy`: Proxy server URL for HTTPS requests. If both are set,
4291+
`https_proxy` takes precedence.
4292+
* `NO_PROXY` or `no_proxy`: Comma-separated list of hosts to bypass the proxy. If both are set,
4293+
`no_proxy` takes precedence.
4294+
4295+
If the request is made to a Unix domain socket, the proxy settings will be ignored.
4296+
4297+
### Proxy URL Format
4298+
4299+
Proxy URLs can use either HTTP or HTTPS protocols:
4300+
4301+
* HTTP proxy: `http://proxy.example.com:8080`
4302+
* HTTPS proxy: `https://proxy.example.com:8080`
4303+
* Proxy with authentication: `http://username:password@proxy.example.com:8080`
4304+
4305+
### `NO_PROXY` Format
4306+
4307+
The `NO_PROXY` environment variable supports several formats:
4308+
4309+
* `*` - Bypass proxy for all hosts
4310+
* `example.com` - Exact host name match
4311+
* `.example.com` - Domain suffix match (matches `sub.example.com`)
4312+
* `*.example.com` - Wildcard domain match
4313+
* `192.168.1.100` - Exact IP address match
4314+
* `192.168.1.1-192.168.1.100` - IP address range
4315+
* `example.com:8080` - Hostname with specific port
4316+
4317+
Multiple entries should be separated by commas.
4318+
4319+
### Example
4320+
4321+
Starting a Node.js process with proxy support enabled for all requests sent
4322+
through the default global agent:
4323+
4324+
```console
4325+
NODE_USE_ENV_PROXY=1 HTTP_PROXY=http://proxy.example.com:8080 NO_PROXY=localhost,127.0.0.1 node client.js
4326+
```
4327+
4328+
To create a custom agent with built-in proxy support:
4329+
4330+
```cjs
4331+
const http = require('node:http');
4332+
4333+
// Creating a custom agent with custom proxy support.
4334+
const agent = new http.Agent({ proxyEnv: { HTTP_PROXY: 'http://proxy.example.com:8080' } });
4335+
4336+
http.request({
4337+
hostname: 'www.example.com',
4338+
port: 80,
4339+
path: '/',
4340+
agent,
4341+
}, (res) => {
4342+
// This request will be proxied through proxy.example.com:8080 using the HTTP protocol.
4343+
console.log(`STATUS: ${res.statusCode}`);
4344+
});
4345+
```
4346+
4347+
Alternatively, the following also works:
4348+
4349+
```cjs
4350+
const http = require('node:http');
4351+
// Use lower-cased option name.
4352+
const agent1 = new http.Agent({ proxyEnv: { http_proxy: 'http://proxy.example.com:8080' } });
4353+
// Use values inherited from the environment variables, if the process is started with
4354+
// HTTP_PROXY=http://proxy.example.com:8080 this will use the proxy server specified
4355+
// in process.env.HTTP_PROXY.
4356+
const agent2 = new http.Agent({ proxyEnv: process.env });
4357+
```
4358+
4359+
[Built-in Proxy Support]: #built-in-proxy-support
42464360
[RFC 8187]: https://www.rfc-editor.org/rfc/rfc8187.txt
42474361
[`'ERR_HTTP_CONTENT_LENGTH_MISMATCH'`]: errors.md#err_http_content_length_mismatch
42484362
[`'checkContinue'`]: #event-checkcontinue

‎doc/api/https.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,14 @@ An [`Agent`][] object for HTTPS similar to [`http.Agent`][]. See
6565

6666
<!-- YAML
6767
changes:
68+
- version:
69+
- REPLACEME
70+
pr-url: https://github.com/nodejs/node/pull/58980
71+
description: Add support for `proxyEnv`.
72+
- version:
73+
- REPLACEME
74+
pr-url: https://github.com/nodejs/node/pull/58980
75+
description: Add support for `defaultPort` and `protocol`.
6876
- version: v12.5.0
6977
pr-url: https://github.com/nodejs/node/pull/28209
7078
description: do not automatically set servername if the target host was

‎lib/_http_agent.js‎

Lines changed: 95 additions & 15 deletions

0 commit comments

Comments
 (0)