Skip to content
Snippets Groups Projects
Commit d11b552e authored by Ilya Konyukhov's avatar Ilya Konyukhov Committed by Kirill Yukhin
Browse files

httpc: add MAX_TOTAL_CONNECTIONS option binding

Right now there is only one option which is configurable for http
client. That is CURLMOPT_MAXCONNECTS. It can be setup like this:

> httpc = require('http.client').new({max_connections = 16})

Basically, this option tells curl to maintain this many connections in
the cache during client instance lifetime. Caching connections are very
useful when user requests mostly same hosts.

When connections cache is full and all of them are waiting for response
and new request comes in, curl creates a new connection, starts request
and then drops first available connection to keep connections cache size
right.

There is one side effect, that when tcp connection is closed, system
actually updates its state to TIME_WAIT. Then for some time resources
for this socket can't be reused (usually 60 seconds).

When user wants to do lots of requests simultaneously (to the same
host), curl ends up creating and dropping lots of connections, which is
not very efficient. When this load is high enough, sockets won't be able
to recover from TIME_WAIT because of timeout and system may run out of
available sockets which results in performance reduce. And user right
now cannot control or limit this behaviour.

The solution is to add a new binding for CURLMOPT_MAX_TOTAL_CONNECTIONS
option. This option tells curl to hold a new connection until
there is one available (request is finished). Only after that curl will
either drop and create new connection or reuse an old one.

This patch bypasses this option into curl instance. It defaults to -1
which means that there is no limit. To create a client with this option
setup, user needs to set max_total_connections option like this:

> httpc = require('http.client').new({max_connections = 8,
                                      max_total_connections = 8})

In general this options usually useful when doing requests mostly to
the same hosts. Other way, defaults should be enough.

Option CURLMOPT_MAX_TOTAL_CONNECTIONS was added from 7.30.0 version, so
if curl version is under 7.30.0, this option is simply ignored.
https://curl.haxx.se/changes.html#7_30_0

Also, this patch adjusts the default for CURLMOPT_MAX_CONNECTS option to
0 which means that for every new easy handle curl will enlarge its max
cache size by 4. See this option docs for more
https://curl.haxx.se/libcurl/c/CURLMOPT_MAXCONNECTS.html

Fixes #3945
parent 14e9b43d
No related branches found
No related tags found
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment