Possibly high amount of resolution failures after switch to resolvematrix

silversys commented

2026-07-17 21:59:13 +00:00

Member

Some users have reported abnormally high amounts of resolutions falling through to DefaultPort (6), sometimes as high as 70-80% of cache.

I have started investigations, will try to see why these numbers are so high. There is a strong possibility that the numbers themselves are the same, but with the changes they are being actually shown as DefaultPort (instead of not shown at all), i.e. no bug or regression at all.

First mention in support room: https://matrix.to/#/!c10y-t1HZB9jgYr9mmaKtMDsS19HXbWRFc6d0bWGVYU/%24hioIXq2q4gAA80ZotSwHh_JMgNbIhqxV-qJ1aYtUjsA?via=wolfgirl.pet&via=explodie.org&via=matrix.org

Some users have reported abnormally high amounts of resolutions falling through to DefaultPort (6), sometimes as high as 70-80% of cache. I have started investigations, will try to see why these numbers are so high. There is a strong possibility that the numbers themselves are the same, but with the changes they are being actually shown as DefaultPort (instead of not shown at all), i.e. no bug or regression at all. First mention in support room: https://matrix.to/#/!c10y-t1HZB9jgYr9mmaKtMDsS19HXbWRFc6d0bWGVYU/%24hioIXq2q4gAA80ZotSwHh_JMgNbIhqxV-qJ1aYtUjsA?via=wolfgirl.pet&via=explodie.org&via=matrix.org

silversys commented

2026-07-17 23:22:47 +00:00

Author

Member

I sent a curl "https://${DOMAIN}/.well-known/matrix/server" across 4733 domains to see status on those:

❯ head resolutions.txt
00ffff.eu: OK (0)
0520.ch: OK (0)
075-141-169-120.res.spectrum.com:8447: FAIL (6 - FAILED TO RESOLVE)
09e52.uk: FAIL (22 - BAD STATUS CODE)
0c0.ch: FAIL (22 - BAD STATUS CODE)
0m.ax: OK (0)
0px.no: FAIL (22 - BAD STATUS CODE)
0upti.me: OK (0)
0vice.reegargs.uk: OK (0)
0wnz.at: FAIL (22 - BAD STATUS CODE)
❯ cat resolutions.txt | rev | cut -d: -f1 | rev | sort | uniq -c | sort -nr
   3020  OK (0)
    671  FAIL (22 - BAD STATUS CODE)
    476  FAIL (6 - FAILED TO RESOLVE)
    223  FAIL (28 - TIMED OUT)
    133  FAIL (60 - SSL CONNECT ERROR)
    101  FAIL (7 - COULDNT CONNECT)
     98  FAIL (35 - SSL CONNECT ERROR)
      5  FAIL (92 - UNKNOWN ERROR)
      3  FAIL (56 - UNKNOWN ERROR)
      1  FAIL (52 - UNKNOWN ERROR)
      1  FAIL (47 - UNKNOWN ERROR)
      1  FAIL (16 - UNKNOWN ERROR)

Error codes are from https://everything.curl.dev/cmdline/exitcode.html, the unknown codes are

92: HTTP/2 stream error, server possibly doesn't talk HTTP/2
56: "Failure in receiving network data"
52: "The server did not reply anything"
47: Too many redirects (lol)
16: HTTP/2 error (of undefined type)

The various FAIL statuses would all fall over to step 4, which if SRV has not been set up would fall further to step 6, i.e. DefaultPort.
(The sum of the FAILs is 669+474+221+133+101+97+5+3+1+1+1 = 1706.)

My next step is to check the servers for SRV status, and correlate this with the fail statuses, and see which servers would correctly fall to DefaultPort status.

I sent a `curl "https://${DOMAIN}/.well-known/matrix/server"` across 4733 domains to see status on those: ``` ❯ head resolutions.txt 00ffff.eu: OK (0) 0520.ch: OK (0) 075-141-169-120.res.spectrum.com:8447: FAIL (6 - FAILED TO RESOLVE) 09e52.uk: FAIL (22 - BAD STATUS CODE) 0c0.ch: FAIL (22 - BAD STATUS CODE) 0m.ax: OK (0) 0px.no: FAIL (22 - BAD STATUS CODE) 0upti.me: OK (0) 0vice.reegargs.uk: OK (0) 0wnz.at: FAIL (22 - BAD STATUS CODE) ❯ cat resolutions.txt | rev | cut -d: -f1 | rev | sort | uniq -c | sort -nr 3020 OK (0) 671 FAIL (22 - BAD STATUS CODE) 476 FAIL (6 - FAILED TO RESOLVE) 223 FAIL (28 - TIMED OUT) 133 FAIL (60 - SSL CONNECT ERROR) 101 FAIL (7 - COULDNT CONNECT) 98 FAIL (35 - SSL CONNECT ERROR) 5 FAIL (92 - UNKNOWN ERROR) 3 FAIL (56 - UNKNOWN ERROR) 1 FAIL (52 - UNKNOWN ERROR) 1 FAIL (47 - UNKNOWN ERROR) 1 FAIL (16 - UNKNOWN ERROR) ``` Error codes are from <https://everything.curl.dev/cmdline/exitcode.html>, the unknown codes are - 92: HTTP/2 stream error, server possibly doesn't talk HTTP/2 - 56: "Failure in receiving network data" - 52: "The server did not reply anything" - 47: Too many redirects (lol) - 16: HTTP/2 error (of undefined type) The various FAIL statuses would all fall over to step 4, which if SRV has not been set up would fall further to step 6, i.e. DefaultPort. (The sum of the FAILs is `669+474+221+133+101+97+5+3+1+1+1 = 1706`.) My next step is to check the servers for SRV status, and correlate this with the fail statuses, and see which servers would correctly fall to DefaultPort status.

nex added the

labels

2026-07-25 13:15:19 +00:00

silversys commented

2026-07-26 10:00:32 +00:00

Author

Member

There is 100% something broken somewhere. I have now gone through the files I got sent, and multiple of them have DefaultPort as the final resolution target despite the host itself resolving just fine on my end.

However, I have no idea why this happens. Looking at the code and the files I've received, the best I can think of is some transient error that happened to make the resolution become DefaultPort.

Best solution I can think of as of right now is to give DefaultPort resolutions a lower TTL than anything else, just in case they are erroneous resolutions. However, this would also cause dead servers (domains with no server behind them anymore) to be re-resolved repeatedly, causing extra strain on everything.

There is 100% something broken somewhere. I have now gone through the files I got sent, and multiple of them have DefaultPort as the final resolution target despite the host itself resolving just fine on my end. However, I have no idea _why_ this happens. Looking at the code and the files I've received, the best I can think of is some transient error that happened to make the resolution become DefaultPort. Best solution I can think of as of right now is to give DefaultPort resolutions a lower TTL than anything else, just in case they are erroneous resolutions. However, this would also cause dead servers (domains with no server behind them anymore) to be re-resolved repeatedly, causing extra strain on everything.

silversys referenced this issue from a commit

2026-07-27 09:46:40 +00:00

chore: Add more trace and debug logs

silversys referenced this issue from continuwuation/resolvematrix

2026-07-27 09:47:28 +00:00

chore: Add more trace and debug logs #36

koen commented

2026-07-28 21:27:14 +00:00

Contributor

Maybe a slightly wild but perhaps good idea, if a certain server has incoming federation to continuwuity, but it failed to resolve (DefaultPort and a basic connectivity check fails?), attempt to resolve again? If it fails, store DefaultPort with a higher TTL, exponential with a maximum of 24 hours. This doesn't fix the underlying issue but likely mitigates some of the failures, especially those surfacing from transient network errors. On my own homeserver, federation often fails due to this issue, and I do not quite understand why.

silversys commented

2026-07-29 16:55:32 +00:00

Author

Member

discussion in in dev room:

@nex: re-resolving based on the result of the resolution being the fallback destination isn't a good idea
me: how do we define when a server comes back online though?
@nex: mark as offline when a federation request times out or returns a 5XX, mark as online when the server auth check in the router passes

I agree with the assessment, re-resolving purely based on DefaultPort would lead to too many re-resolutions and higher server load with very little benefit.

discussion in [in dev room](https://matrix.to/#/!ksTlboXVgcyWjv5GrlEeKyQuJ8ZCprnwQx2b6-BQ44Q/%24THIoE4rMX8hgtgyDdo-ZFdhtZx-qdY_MtvhG9L0-U8E?via=wolfgirl.pet&via=explodie.org&via=matrix.org): > **@nex:** re-resolving based on the result of the resolution being the fallback destination isn't a good idea > **me:** how do we define when a server comes back online though? > **@nex:** mark as offline when a federation request times out or returns a 5XX, mark as online when the server auth check in the router passes I agree with the assessment, re-resolving purely based on DefaultPort would lead to too many re-resolutions and higher server load with very little benefit.

koen commented

2026-07-29 19:34:46 +00:00

Contributor

I didn't exactly mean immediately re-resolving, just increasing/doubling the TTL each time a failure occurs and it is clear that resolution failed (for example, /_matrix/federation/v1/version does not result in 200), but then again it might depend on whether the resolution was successful or not in the first place, but this might be hard to distinguish in some cases.

I didn't exactly mean immediately re-resolving, just increasing/doubling the TTL each time a failure occurs and it is clear that resolution failed (for example, `/_matrix/federation/v1/version` does not result in 200), but then again it might depend on whether the resolution was successful or not in the first place, but this might be hard to distinguish in some cases.

nex commented

2026-07-29 19:36:31 +00:00

Owner

server resolution can't fail

koen commented

2026-07-29 19:37:25 +00:00

Contributor

Sure it can? No delegation, no SRV, default port closed?

silversys commented

2026-07-29 19:45:46 +00:00

Author

Member

if there's no delegation, srv, and default port is closed then it didnt fail, its the server which has closed their ways of communicating, resolution still succeeded as per the spec:

If the /.well-known request returned an error response, and no SRV records were found, an IP address is resolved using CNAME, AAAA and A records. Requests are made to the resolved IP address using port 8448 and a Host header containing the . The target server must present a valid certificate for .

This is the last step of resolution, which doesn't care if the port is closed

if there's no delegation, srv, and default port is closed then it didnt fail, its the server which has closed their ways of communicating, resolution still succeeded as per the spec: > 6. If the /.well-known request returned an error response, and no SRV records were found, an IP address is resolved using CNAME, AAAA and A records. Requests are made to the resolved IP address using port 8448 and a Host header containing the <hostname>. The target server must present a valid certificate for <hostname>. This is the last step of resolution, which doesn't care if the port is closed

👍 1

koen commented

2026-07-29 20:09:46 +00:00

Contributor

That is spec, but obviously, your policy on how to then deal with the result of the resolution may still depend on implementation details.

silversys commented

2026-07-30 17:52:00 +00:00

Author

Member

Our policy on how to deal with it will probably be "mark as failed on federation failure, unmark when they send us a request again or after a set time period"

silversys commented

2026-08-27 15:44:00 +00:00

Author

Member

Yeah I've tried to dig more into this issue, and I really don't understand why it's happening. With the addition of #2097, hopefully errored servers should re-resolve correctly if they got incorrectly resolved in the first place, but that remains to be seen. Closing this issue for now.

silversys closed this issue

2026-08-27 15:44:00 +00:00

Rows
Columns

Possibly high amount of resolution failures after switch to resolvematrix #2019