feat: Server health tracking #2097

Merged
Aranjedeath merged 28 commits from nex/feat/server-health-tracking into main 2026-08-20 00:57:12 +00:00
Owner

The FitnessGram Pacer test is a multistage aerobic capacity test that progressively gets more difficult as it continues. The 20 meter Pacer test will begin in 30 seconds. Line up at the start. The running speed starts slowly, but gets faster each minute after you hear this signal boop. A single lap should be completed each time you hear this sound ding. Remember to run in a straight line, and run as long as possible. The second time you fail to complete a lap before the sound, your test is over. The test will begin on the word start. On your mark, get ready, start.


Real description: Introduces "health tracking" that enables us to mark a remote server as healthy/unhealthy, allowing the server to intelligently decide whether it wants to send a request to a remote, or just skip over it. This should improve the speed of some federation dependent operations, like invites, joins, encryption, and federation transaction sending.

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
The FitnessGram Pacer test is a multistage aerobic capacity test that progressively gets more difficult as it continues. The 20 meter Pacer test will begin in 30 seconds. Line up at the start. The running speed starts slowly, but gets faster each minute after you hear this signal *boop*. A single lap should be completed each time you hear this sound *ding*. Remember to run in a straight line, and run as long as possible. The second time you fail to complete a lap before the sound, your test is over. The test will begin on the word start. On your mark, get ready, start. --- Real description: Introduces "health tracking" that enables us to mark a remote server as healthy/unhealthy, allowing the server to intelligently decide whether it wants to send a request to a remote, or just skip over it. This should improve the speed of some federation dependent operations, like invites, joins, encryption, and federation transaction sending. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages
nex added this to the 26.8.0 milestone 2026-08-03 15:26:09 +00:00
nex self-assigned this 2026-08-03 15:26:09 +00:00
Otherwise federation would have to depend on sender
feat: Refuse to send requests to unhealthy servers
Some checks failed
Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 3s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 9s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m11s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m5s
51ba0fc30b
Also marks them as online if a request succeeds
feat: Clear remote health tracking map when cache is cleared
Some checks failed
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m12s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
f0ccfe1003
fix: Don't honour unhealthy hits unless the block period has already expired
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m1s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m12s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m10s
faba6811f9
style: Remove unsafe arithmetic
Some checks failed
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m15s
8a3f7cde55
feat: Hook transaction senders into health tracking
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 9s
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Checks / Prek / Check changed files (pull_request) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Documentation / Build and Deploy Documentation (pull_request) Has been cancelled
85064c9390
nex force-pushed nex/feat/server-health-tracking from 85064c9390
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 9s
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Checks / Prek / Check changed files (pull_request) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Documentation / Build and Deploy Documentation (pull_request) Has been cancelled
to 44abc4e8dd
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m7s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m14s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m24s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m34s
2026-08-06 14:21:25 +00:00
Compare
feat: Use an explicit backoff base value
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 8s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m9s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m11s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m26s
f635fd19be
feat: Add debug command to extract backoff info
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m9s
e75fc04c3f
feat: Clear destinations cache when a remote is potentially stale
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Has been cancelled
Documentation / Build and Deploy Documentation (pull_request) Has been cancelled
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Checks / Prek / Check changed files (pull_request) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
1734baf440
nex force-pushed nex/feat/server-health-tracking from 1734baf440
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Has been cancelled
Documentation / Build and Deploy Documentation (pull_request) Has been cancelled
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Checks / Prek / Check changed files (pull_request) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
to b8576e974f
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 11s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m25s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 4m45s
2026-08-11 14:44:15 +00:00
Compare
style: Centralise decision for stale marker
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m5s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m8s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m7s
8610d921f4
fix(sender): retry_after duration can be negative
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m6s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
448e783634
style: Reformat
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m1s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m8s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m14s
e8ca2edb2a
fix: Correctly insert running status only if the sender should attempt to send
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m18s
612d313dc0
fix: Mark even more error codes as stale based on observed data
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m14s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m11s
a012a42981
fix: Treat any non-ok response as a server error in sender
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m33s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m12s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 12s
38133db05b
Otherwise we never back off sending failed requests
@ -1240,0 +1246,4 @@
let backoff_map = self.services.federation.remote_health();
if backoff_map.is_empty() {
return Err!("No servers in backoff.");
Owner

this probably should be a write! and an return Ok

this probably should be a write! and an return Ok
nex marked this conversation as resolved
@ -25,4 +12,1 @@
) -> bool {
let min = min.saturating_mul(tries).saturating_mul(tries);
let min = cmp::min(min, max);
elapsed < min
Owner

my single line function?

my single line function?
Author
Owner

there's like 5 references I couldn't be bothered refactoring out, I'll do that later

there's like 5 references I couldn't be bothered refactoring out, I'll do that later
nex marked this conversation as resolved
@ -33,0 +83,4 @@
/// Marks or updates a remote's health status as unhealthy. If the remote is
/// not already marked as unhealthy, a new entry is created. Otherwise, the
/// retry count is incremented and
Owner

and... what?

and... what?
Author
Owner

wdym? the remote is unhealthy if the next retry is after the current timestamp, otherwise it's healthy

wdym? the remote is unhealthy if the next retry is after the current timestamp, otherwise it's healthy
Author
Owner

there's no and in the docstring either

there's no and in the docstring either
nex marked this conversation as resolved
@ -157,2 +174,3 @@
| TransactionStatus::Failed(..) => {
panic!("Request that was not even running failed?!")
panic!(
"{}",
Owner

panic! can do formatting on its own...?

panic! can do formatting on its own...?
Author
Owner

this is load bearing actually

i have no idea how this happened, I don't remember writing it, I remember clippy complaining at some point but then it went away before I looked at it soooo... clippy?

this is load bearing actually i have no idea how this happened, I don't remember writing it, I remember clippy complaining at some point but then it went away before I looked at it soooo... clippy?
nex marked this conversation as resolved
nex force-pushed nex/feat/server-health-tracking from 38133db05b
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m33s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m12s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 12s
to 7e3e01e731
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m22s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m10s
2026-08-19 12:48:55 +00:00
Compare
feat: Lower default backoff base for a gentler backoff curve
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m11s
c04c732b22
fix: Address review comments
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m5s
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
a108208274
chore: Add newsfrag
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m5s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
331025e0fd
nex changed title from WIP: feat: Server health tracking to feat: Server health tracking 2026-08-19 13:37:11 +00:00
nex requested review from Owners 2026-08-19 13:37:19 +00:00
chore: Re-generate example config
All checks were successful
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m5s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m8s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m0s
84343001a4
eleboucher approved these changes 2026-08-19 13:42:02 +00:00
Author
Owner

Just watched this PR successfully mark a server as being in backoff, with a stale destination, and then successfully re-resolve it after I flushed my DNS resolver's cache (unbound had a stale entry itself) and getting a request from the unhealthy server, leading to it immediately receiving a complete flush of all of my previous messages and encryption keys. Truly brings a tear to one's eye, the continuwuity indefinite federation cache and sender backoff is dead!

Just watched this PR successfully mark a server as being in backoff, with a stale destination, and then successfully re-resolve it after I flushed my DNS resolver's cache (unbound had a stale entry itself) and getting a request from the unhealthy server, leading to it immediately receiving a complete flush of all of my previous messages and encryption keys. Truly brings a tear to one's eye, the continuwuity indefinite federation cache and sender backoff is dead!
nex requested review from ginger 2026-08-19 23:35:08 +00:00
Jade approved these changes 2026-08-20 00:32:09 +00:00
Jade left a comment

Seems like an improvement to me. 🚢 it :drgn_3c_evil:

Seems like an improvement to me. 🚢 it :drgn_3c_evil:
Aranjedeath deleted branch nex/feat/server-health-tracking 2026-08-20 00:57:12 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
continuwuation/continuwuity!2097
No description provided.