continuwuation/continuwuity
continuwuation/continuwuity
13
35
Fork 34
Code Issues 103 Pull requests 19 Releases 20 Packages 3 Activity Actions

WIP: Improve federation backoff logic #1182

Draft
nex wants to merge 1 commit from nex/fix/backoff into main
pull from: nex/fix/backoff
merge into: continuwuation:main
Conversation 2 Commits 1 Files changed 5 +34 -7
nex commented 2025-11-19 19:15:10 +00:00
Owner
Copy link

This PR will fix up some of the federation sender code to fix the issue where servers get banished to backoff purgatory for an undefined amount of time which may also be different depending on which sender worker fails.
This will not fix issues like lookup errors poisoning the destinations cache, that is a separate issue.

Things to do:

  • Expose the transaction statuses to the entire sender service so that external callers can mutate them
  • Reset backoffs when clearing caches
  • Command to force reset specific server name's backoff & destination?
  • Reset server backoff when we receive a successful transaction from it (mirrors highly desired Synapse behaviour)
This PR will fix up some of the federation sender code to fix the issue where servers get banished to backoff purgatory for an undefined amount of time which may also be different depending on which sender worker fails. This will not fix issues like lookup errors poisoning the destinations cache, that is a separate issue. Things to do: - [ ] Expose the transaction statuses to the entire sender service so that external callers can mutate them - [ ] Reset backoffs when clearing caches - [ ] Command to force reset specific server name's backoff & destination? - [ ] Reset server backoff when we receive a successful transaction from it (mirrors highly desired Synapse behaviour)
nex added this to the 0.5.0 milestone 2025-11-19 19:15:10 +00:00
nex self-assigned this 2025-11-19 19:15:10 +00:00
feat(wip;sender): Centralise status registries
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Failing after 12s
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 18s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m10s
Details
Release Docker Image / Build linux-amd64 (release) (pull_request) Successful in 3m1s
Details
Release Docker Image / Build linux-arm64 (release) (pull_request) Successful in 3m47s
Details
Release Docker Image / Create Multi-arch Release Manifest (pull_request) Failing after 15s
Details
Release Docker Image / Build linux-amd64 (max-perf) (pull_request) Successful in 2m36s
Details
Release Docker Image / Build linux-arm64 (max-perf) (pull_request) Successful in 2m31s
Details
Release Docker Image / Create Max-Perf Manifest (pull_request) Failing after 13s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 18m6s
Details
97c13d8c77
nex reviewed 2025-11-19 19:17:03 +00:00
Cargo.toml Outdated
@ -556,4 +556,1 @@
[workspace.dependencies.resolv-conf]
version = "0.7.5"
nex commented 2025-11-19 19:17:03 +00:00
Author
Owner
Copy link

where'd resolv-conf go wtf

where'd resolv-conf go wtf
Jade reviewed 2025-11-19 19:18:04 +00:00
src/service/sending/mod.rs Outdated
@ -101,6 +103,7 @@ impl crate::Service for Service {
federation: args.depend::<federation::Service>("federation"),
},
channels: (0..num_senders).map(|_| loole::unbounded()).collect(),
statuses: vec![sender::CurTransactionStatus::new(); num_senders],
Jade commented 2025-11-19 19:18:04 +00:00
Owner
Copy link

This is still splitting into multiple statuses different per worker?

This is still splitting into multiple statuses different per worker?
nex commented 2025-11-19 19:20:21 +00:00
Author
Owner
Copy link

Yes, we need that for concurrent sending. The status registry tracks the state of a destination per-worker, if one is actively sending and the registry is global, then all other workers will refuse to send until that one completes

Yes, we need that for concurrent sending. The status registry tracks the state of a destination per-worker, if one is actively sending and the registry is global, then all other workers will refuse to send until that one completes
ginger force-pushed nex/fix/backoff from 97c13d8c77
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Failing after 12s
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 18s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m10s
Details
Release Docker Image / Build linux-amd64 (release) (pull_request) Successful in 3m1s
Details
Release Docker Image / Build linux-arm64 (release) (pull_request) Successful in 3m47s
Details
Release Docker Image / Create Multi-arch Release Manifest (pull_request) Failing after 15s
Details
Release Docker Image / Build linux-amd64 (max-perf) (pull_request) Successful in 2m36s
Details
Release Docker Image / Build linux-arm64 (max-perf) (pull_request) Successful in 2m31s
Details
Release Docker Image / Create Max-Perf Manifest (pull_request) Failing after 13s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 18m6s
Details
to ae595dd0d1
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Successful in 57s
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 17s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m31s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 10m15s
Details
2025-11-21 17:11:15 +00:00 Compare
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Successful in 57s
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 17s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m31s
Required
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 10m15s
Required
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin nex/fix/backoff:nex/fix/backoff
git switch nex/fix/backoff
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Bug

Something isn't working as intended

  
Cherry-picking

Commits picked from other conduit projects

  
Database

This requires or includes changes to the database

  
Dependencies

Something dependency related

  
Dependencies/Renovate

Automatic dependency upgrades by Renovate

  
Difficulty
Easy
Low difficulty to implement - touches few parts of the codebase, low complexity

  
Difficulty
Hard
High difficulty to implement - touches many parts of the codebase, high complexity

  
Difficulty
Medium
Medium difficulty to implement - touches more parts of the codebase, higher complexity

  
Documentation

Improvements or additions to documentation

  
Enhancement

New feature or request

  
Good first issue

Good for newcomers

  
Help wanted

Additional eyes and keyboards are required for this one

  
Inherited

Issues that have been inhereted from the project pre-fork

  
Matrix/Administration

Features pertaining to homeserver administration

  
Matrix/Appservices

Features pertaining to the appservice API

  
Matrix/Auth

Features pertaining to authentication

  
Matrix/Client

Features pertaining to client-to-server interactions

  
Matrix/Core

Issues relating to core matrix functionality, such as state resolution and PDU formats

  
Matrix/Federation

Features pertaining to server-to-server interactions

  
Matrix/Hydra

Issues related to room version 12 and related changes (temporary label)

  
Matrix/MSC

Features pertaining to unstable matrix features

  
Matrix/Media

Features pertaining to media interactions

  
Meta

Related to housekeeping, maintenance, or other repo-meta.

  
Meta/CI

Issues related to CI changes

  
Meta/Packaging

Packaging

  
Priority
Blocking
This issue is blocking the next release

  
Priority
High
This issue is very important

  
Priority
Low
This issue is of a rather low priority

  
Security

This item is related to general security

  
Status/Blocked

This pull request or issue is currently blocked from being merged/closed

  
Status
Confirmed
This issue has enough information and is confirmed

  
Status
Duplicate
This issue or pull request already exists

  
Status
Invalid
This issue doesn't seem right

  
Status
Needs Investigation
This issue needs further investigation

  
Support

Questions or support requests

  
To-Merge

   
Wont fix

This will not be worked on

  
old/ci/cd

Ci/CD

Archived

  
old/rust

Pull requests that update Rust code

Archived

No labels
Bug
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status/Blocked
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
To-Merge
Wont fix
old/ci/cd
old/rust
Milestone
Clear milestone
No items
No milestone
0.5.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
nex
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!1182
No description provided.