continuwuation/continuwuity

Fork 117

TCP: too many orphaned sockets when server is running after `v26.6.0-alpha.1` #2005

New issue

Open

opened 2026-07-16 16:13:28 +00:00 by lippiece · 7 comments

lippiece commented

2026-07-16 16:13:28 +00:00

Thank you for your work.

I was using early v26 alpha then updated to 26.6.2. When I start the container, the whole system seems like it can't perform network requests. DNS resolution isn't working, pings drop, etc. dmesg says TCP: too many orphaned sockets.

Latest working version is v26.6.0-alpha.1.

The continuwuity server last log is about successful ICMP email test.

Thank you for your work. I was using early v26 alpha then updated to 26.6.2. When I start the container, the whole system seems like it can't perform network requests. DNS resolution isn't working, pings drop, etc. `dmesg` says `TCP: too many orphaned sockets`. Latest working version is `v26.6.0-alpha.1`. The continuwuity server last log is about successful ICMP email test.

stratself commented

2026-07-16 16:48:40 +00:00

Contributor

You probably ran out of memory for network (see this). This is normally quite low on Linux, especially for Continuwuity which can make a lot of calls. Try increasing them via tuning your /etc/sysctl.conf file. Example:

# increasing a bunch of network-related memory buffers,
# defaults are usually quite low, somewhere 1-2MB is good
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.core.rmem_default = 2097152
net.core.wmem_default = 2097152
# tcp-specific stuff
net.ipv4.tcp_mem=1048576 1048576 1048576 # min, default, max
net.ipv4.tcp_rmem=4096 262144 1048576
net.ipv4.tcp_wmem=4096 262144 1048576

Then do sysctl -p to apply changes

Do reply if the issue persists after.

Edit: before changing, you can check current values via cat /proc/sys/net/core/rmem_max and so on

You probably ran out of memory for network (see [this](https://serverfault.com/a/624951)). This is normally quite low on Linux, especially for Continuwuity which can make a lot of calls. Try increasing them via tuning your `/etc/sysctl.conf` file. Example: ``` # increasing a bunch of network-related memory buffers, # defaults are usually quite low, somewhere 1-2MB is good net.core.rmem_max = 2097152 net.core.wmem_max = 2097152 net.core.rmem_default = 2097152 net.core.wmem_default = 2097152 # tcp-specific stuff net.ipv4.tcp_mem=1048576 1048576 1048576 # min, default, max net.ipv4.tcp_rmem=4096 262144 1048576 net.ipv4.tcp_wmem=4096 262144 1048576 ``` Then do `sysctl -p` to apply changes Do reply if the issue persists after. Edit: before changing, you can check current values via `cat /proc/sys/net/core/rmem_max` and so on

nex added the

Status

Needs Investigation

label

2026-07-16 16:57:38 +00:00

nex commented

2026-07-16 16:59:11 +00:00

Owner

I've never encountered this before, some more information regarding your environment will required (fd limit, the limits stratself mentioned above, etc). 26.6 will make more network requests than 0.5.X since we improved federation reliability, which means more connection noise is expected, but this is the first report of it causing system problems

lippiece commented

2026-07-16 21:29:12 +00:00

Author

Thanks for investigating.

@stratself wrote in #2005 (comment):

(see this).

Yeah, I also stumbled upon that thread when this happened, but my settings are well above:

net.core.rmem_max = 7500000
net.core.wmem_max = 7500000
net.core.rmem_default = 212992
net.core.wmem_default = 212992
net.ipv4.tcp_mem = 91761        122351  183522
net.ipv4.tcp_rmem = 4096        131072  33554432
net.ipv4.tcp_wmem = 4096        16384   4194304

This one is ridiculously high, but I didn't set it, so it must be one of the NixOS module's work (and, strangely, hasn't caused any issues with other services):

❮ sysctl fs.file-max
fs.file-max = 9223372036854775807

I applied your changes, but it didn't help. I suspect, there are indeed too many open sockets. Normal state (/proc/net/sockstat):

sockets: used 1992
TCP: inuse 87 orphan 1 tw 101 alloc 1925 mem 2561
UDP: inuse 14 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0

1 minute after continuwuity starts (somewhere around used 40000 a second after it starts, then it gets worse):

sockets: used 99190
TCP: inuse 44830 orphan 24093 tw 343 alloc 45373 mem 63489
UDP: inuse 53763 mem 45028
RAW: inuse 0
FRAG: inuse 0 memory 0

When the alpha version is in use:

sockets: used 4835
TCP: inuse 1218 orphan 33 tw 59 alloc 2253 mem 1537
UDP: inuse 2608 mem 1919
RAW: inuse 0
FRAG: inuse 0 memory 0

Thanks for investigating. @stratself wrote in https://forgejo.ellis.link/continuwuation/continuwuity/issues/2005#issuecomment-32341: > (see [this](https://serverfault.com/a/624951)). Yeah, I also stumbled upon that thread when this happened, but my settings are well above: ``` net.core.rmem_max = 7500000 net.core.wmem_max = 7500000 net.core.rmem_default = 212992 net.core.wmem_default = 212992 net.ipv4.tcp_mem = 91761 122351 183522 net.ipv4.tcp_rmem = 4096 131072 33554432 net.ipv4.tcp_wmem = 4096 16384 4194304 ``` This one is ridiculously high, but I didn't set it, so it must be one of the NixOS module's work (and, strangely, hasn't caused any issues with other services): ``` ❮ sysctl fs.file-max fs.file-max = 9223372036854775807 ``` I applied your changes, but it didn't help. I suspect, there are indeed too many open sockets. Normal state (/proc/net/sockstat): ``` sockets: used 1992 TCP: inuse 87 orphan 1 tw 101 alloc 1925 mem 2561 UDP: inuse 14 mem 0 RAW: inuse 0 FRAG: inuse 0 memory 0 ``` 1 minute after continuwuity starts (somewhere around `used 40000` a second after it starts, then it gets worse): ``` sockets: used 99190 TCP: inuse 44830 orphan 24093 tw 343 alloc 45373 mem 63489 UDP: inuse 53763 mem 45028 RAW: inuse 0 FRAG: inuse 0 memory 0 ``` When the alpha version is in use: ``` sockets: used 4835 TCP: inuse 1218 orphan 33 tw 59 alloc 2253 mem 1537 UDP: inuse 2608 mem 1919 RAW: inuse 0 FRAG: inuse 0 memory 0 ```

lippiece commented

2026-07-16 21:55:43 +00:00

Author

Upon the advice from katie from the continuwuity's room, I ran netstat, which froze when continuwuity started. I had to stop it to see the logs, which are interesting. Seems like something makes the host try different DNS servers (one.one.one.one, etc), and sometimes even itself:

udp        0      0 cumulonimbus:22354      dns9.quad9.net:domain   ESTABLISHED                                                                                                                                      udp        0      0 cumulonimbus:38738      dns9.quad9.net:domain   ESTABLISHED                                                                                                                                      udp        0      0 cumulonimbus:63314      cumulonimbus:domain     ESTABLISHED
udp        0      0 cumulonimbus:34642      cumulonimbus:domain     ESTABLISHED                                                                                                                                      udp        0      0 cumulonimbus:30546      cumulonimbus:domain     ESTABLISHED                                                                                                                                      udp        0      0 cumulonimbus:26450      dns9.quad9.net:domain   ESTABLISHED

...it goes on. These are the servers from dnsproxy I have configured. When I set my Adguard Home's IP as DNS, I saw around 2K requests just for 5 seconds the server ran.

Upon the advice from katie from the continuwuity's room, I ran `netstat`, which froze when continuwuity started. I had to stop it to see the logs, which are interesting. Seems like something makes the host try different DNS servers (`one.one.one.one`, etc), and sometimes even itself: ``` udp 0 0 cumulonimbus:22354 dns9.quad9.net:domain ESTABLISHED udp 0 0 cumulonimbus:38738 dns9.quad9.net:domain ESTABLISHED udp 0 0 cumulonimbus:63314 cumulonimbus:domain ESTABLISHED udp 0 0 cumulonimbus:34642 cumulonimbus:domain ESTABLISHED udp 0 0 cumulonimbus:30546 cumulonimbus:domain ESTABLISHED udp 0 0 cumulonimbus:26450 dns9.quad9.net:domain ESTABLISHED ``` ...it goes on. These are the servers from dnsproxy I have configured. When I set my Adguard Home's IP as DNS, I saw around 2K requests just for 5 seconds the server ran.

stratself commented

2026-07-17 06:24:18 +00:00

Contributor

@lippiece

These are expected levels of DNS load for Continuwuity (and Matrix in general). There's not much one can do about it. Host --> host traffic is likely the server talking to AGH.

I have a few more advises you can employ:

Increase nofile descriptors limit for your continuwuity user in limits.conf, which should help per-user ulimits
Tune AdGuardHome according to instructions, or set up a dedicated resolver for Continuwuity
Also increase /proc/sys/net/ipv4/tcp_max_orphans in your sysctl as well

Let us know of further discoveries. I do think there are some noticeable DNS/system impact but not sure how

@lippiece These are expected levels of DNS load for Continuwuity (and Matrix in general). There's not much one can do about it. Host --> host traffic is likely the server talking to AGH. I have a few more advises you can employ: - Increase `nofile` descriptors limit for your continuwuity user in `limits.conf`, which should help per-user ulimits - Tune AdGuardHome according to [instructions](https://continuwuity.org/guides/dns#dnsproxy), or set up a dedicated resolver for Continuwuity - Also increase `/proc/sys/net/ipv4/tcp_max_orphans` in your sysctl as well Let us know of further discoveries. I do think there are some noticeable DNS/system impact but not sure how

lippiece commented

2026-07-17 09:39:57 +00:00

Author

I have increased tcp_max_orhpans to 400000 and Adguard's cache to 100 MB, but it still can't handle this. The server starts, but doesn't run more than a few seconds: it times out when checking for announcements, and fails to query DNS for the SMTP connection test. The system can handle this many orphans, but Adguard or dnsproxy can't handle this many requests.

I've had Synapse running for year or more, but never had this at least tenfold increase in DNS requests. Docs say "sometimes thousands per minute", but this is thousands per second, and I haven't yet joined that many rooms as I had when I had Synapse. I have increased `tcp_max_orhpans` to 400000 and Adguard's cache to 100 MB, but it still can't handle this. The server starts, but doesn't run more than a few seconds: it times out when checking for announcements, and fails to query DNS for the SMTP connection test. The system can handle this many orphans, but Adguard or dnsproxy can't handle this many requests.

lippiece commented

2026-07-18 14:59:18 +00:00

Author

There has been some discussion in continuwuity's Matrix room about this. In short (please correct if I missed something):

Increased DNS usage is expected due to recent federation resolver changes.
A spike this big is probably not expected.
This only happens on startup.
This can be mitigated with startup_netburst = false in config.
This can also be mitigated by following the promoted docs about DNS tuning. In particular, don't use filtering DNS for continuwuity, and use more upstreams, e.g. with dnsproxy.

There has been some discussion in continuwuity's Matrix room about this. In short (please correct if I missed something): 1. Increased DNS usage is expected due to recent federation resolver changes. 2. A spike this big is probably not expected. 3. This only happens on startup. 4. This can be mitigated with `startup_netburst = false` in config. 5. This can also be mitigated by following the promoted docs about [DNS tuning](https://continuwuity.org/guides/dns.html). In particular, don't use filtering DNS for continuwuity, and use more upstreams, e.g. with `dnsproxy`.