fix: Re-introduce event fetch backoff #2201

nex · 2026-08-28T15:00:38Z

nex commented

2026-08-28 15:00:38 +00:00

Some prior versions of continuwuity (I forget when it was removed) had a per-event backoff system that prevented the server constantly asking for the same events it's never going to see. This was removed at some point, likely in #1818, which now means continuwuity does not attempt to slow down when asking for events it isn't allowed to get, resulting in #2090. As I run several servers behind the same reverse proxy that happen to be backfill candidates, I have seen my request rate surge from 20 rps average to 100 rps average, which obviously isn't great.

This PR re-introduces backoff on event fetches using the same algorithm as federation transaction sending.

Fixes: #2090

Pull request checklist:

This pull request targets the main branch, and the branch is named something other than
main.
I have written an appropriate pull request title and my description is clear.
I understand I am responsible for the contents of this pull request.
I have followed the contributing guidelines:
- My contribution follows the code style, if applicable.
- I ran pre-commit checks before opening/drafting this pull request.
- I have tested my contribution (or proof-read it for documentation-only changes)
  myself, if applicable. This includes ensuring code compiles.
- My commit messages follow the commit message format and are descriptive.

Some prior versions of continuwuity (I forget when it was removed) had a per-event backoff system that prevented the server constantly asking for the same events it's never going to see. This was removed at some point, likely in #1818, which now means continuwuity does not attempt to slow down when asking for events it isn't allowed to get, resulting in #2090. As I run several servers behind the same reverse proxy that happen to be backfill candidates, I have seen my request rate surge from 20 rps average to 100 rps average, which obviously isn't great. This PR re-introduces backoff on event fetches using the same algorithm as federation transaction sending. Fixes: #2090     **Pull request checklist:**  - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [ ] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive.  [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages

nex added the

Bug

Matrix/Federation

labels

2026-08-28 15:00:38 +00:00

nex self-assigned this

2026-08-28 15:00:38 +00:00

nex added 3 commits

2026-08-28 15:00:38 +00:00

feat: Add pdu pull backoff fafb6af877

feat: Plumb event backoff into auth and prev fetchers d431f75107

style: Remove unused import and reformat

Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 4s

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s

Details

Checks / Prek / Check changed files (pull_request) Successful in 6s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m28s

Details

Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

84b974579e

forgejo-actions added the

Changelog

Missing

label

2026-08-28 15:00:51 +00:00

nex force-pushed nex/fix/state-fetch-backoff from 84b974579e

Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 4s

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s

Details

Checks / Prek / Check changed files (pull_request) Successful in 6s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m28s

Details

Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

to 6e30efb64f

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 13s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled

Details

Checks / Prek / Check changed files (pull_request) Has been cancelled

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

Update flake hashes / update-flake-hashes (pull_request) Has been cancelled

Details

Documentation / Build and Deploy Documentation (pull_request) Has been cancelled

Details

2026-08-28 15:08:05 +00:00

Compare

forgejo-actions added

and removed

labels

2026-08-28 15:08:18 +00:00

nex force-pushed nex/fix/state-fetch-backoff from 6e30efb64f

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 13s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled

Details

Checks / Prek / Check changed files (pull_request) Has been cancelled

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

Update flake hashes / update-flake-hashes (pull_request) Has been cancelled

Details

Documentation / Build and Deploy Documentation (pull_request) Has been cancelled

Details

to 6cfbb41a42

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 1m22s

Details

Checks / Prek / Check changed files (pull_request) Successful in 7s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m27s

Details

Update flake hashes / update-flake-hashes (pull_request) Successful in 1m31s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 12m14s

Details

2026-08-28 15:08:34 +00:00

Compare

nex requested review from Owners

2026-08-28 15:17:01 +00:00

nex reviewed

2026-08-28 15:22:48 +00:00

src/service/rooms/event_handler/fetch_and_handle_outliers.rs

					
				@ -485,3 +475,2 @@

								}

								debug!(elapsed=?start.elapsed(),"Fetching {target_id} over federation");

								self.ensure_can_pull_event(&target_id).inspect_err(|e| {

Not sure if this warrants a comment explicitly calling it out, but failing the entire fetch routine immediately here is deliberate - from my logs, it looks like the 5 retry attempts are generally wasted, and errors are usually final (e.g. 403), meaning we ended up sending 5 requests per failed event to every backfill remote, exponentially.

If we can't pull a missing auth event at any point, it is impossible for us to safely continue. The old backoff system simply dropped events which couldn't be fetched, which would lead to incorrect behaviour on pdu check 4. If we can't fetch auth events, we can't auth the event, even if only one is missing.

As for why it doesn't matter for prev events - if we are missing some prev events, we won't be able to calculate the state before the incoming event locally, and will fall back to asking the remote server for the state/_ids. From there, we either get the whole state, or atomically fetch using fetch_and_handle_auth_events, meaning we always have all events necessary to calculate PDU check 5 (unless, of course, FAHAE fails to fetch an event, bailing out the entire thing yet again)

Not sure if this warrants a comment explicitly calling it out, but failing the entire fetch routine immediately here is deliberate - from my logs, it looks like the 5 retry attempts are generally wasted, and errors are usually final (e.g. 403), meaning we ended up sending 5 requests per failed event to every backfill remote, exponentially. If we can't pull a missing auth event at any point, it is impossible for us to safely continue. The old backoff system simply dropped events which couldn't be fetched, which would lead to incorrect behaviour on pdu check 4. If we can't fetch auth events, we can't auth the event, even if only one is missing. As for why it doesn't matter for *prev* events - if we are missing some prev events, we won't be able to calculate the state before the incoming event locally, and will fall back to asking the remote server for the state/_ids. From there, we either get the whole state, or atomically fetch using `fetch_and_handle_auth_events`, meaning we always have all events necessary to calculate PDU check 5 (unless, of course, FAHAE fails to fetch an event, bailing out the entire thing yet again)

nex added 1 commit

2026-08-28 15:38:27 +00:00

fix: Don't increment backoff if we're still in backoff

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 14s

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 1m23s

Details

Checks / Prek / Check changed files (pull_request) Successful in 8s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m26s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m8s

Details

ffff986763

eleboucher requested changes

2026-08-28 16:04:28 +00:00

Dismissed

src/service/rooms/event_handler/fetch_and_handle_outliers.rs Outdated

					
				@ -664,3 +667,1 @@

									} else {

										Some(event_id.to_owned())

									}

									(discovered_events.contains_key(event_id)

This is inverted. missing_prev now collects the events we already have.

I've got a knack for doing this

nex marked this conversation as resolved

nex added 1 commit

2026-08-29 13:40:22 +00:00

fix: Re-invert prev events filter in fetch_prev_events

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s

Details

Checks / Prek / Check changed files (pull_request) Successful in 7s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m18s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m1s

Details

36abe7ba66

eleboucher approved these changes

2026-08-29 14:23:03 +00:00

ginger approved these changes

2026-09-01 16:13:17 +00:00

nex merged commit 03d669594e into main

2026-09-01 16:24:18 +00:00

nex deleted branch nex/fix/state-fetch-backoff

2026-09-01 16:24:18 +00:00

Sign in to join this conversation.

No reviewers

No labels

Dependencies/Renovate

Matrix/Administration

Rows
Columns