continuwuation/continuwuity

Fork 123

bug: New atomic state fetch does not appropriately throttle requests for events #2090

New issue

Open

opened 2026-08-02 09:18:46 +00:00 by nex · 5 comments

nex commented

2026-08-02 09:18:46 +00:00

Owner

pub(super) async fn fetch_and_handle_auth_events<Pdu>(

There is no backoff in event fetching here, which means c10y will attempt to fetch an auth event 5 times as fast as possible, and then moves on. However, it will then do this dance again if it receives yet another event that references a bad event. This is particularly damaging during catchup.

The same failure can be observed for the atomic fallback for prev events,

pub(super) async fn fetch_prev_events<Pdu>(

, however this is significantly less frequent as this is a fallback behaviour that is usually not reached due to get_missing_events.

When requesting events atomically to fill gaps in missing state before, continuwuity does not attempt to back off or prevent further requests for an event if an initial request fails. This can happen when all remote servers (for example) return a 403 M_FORBIDDEN on the state event: https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/4146e0f1daf7d82a58d6a99f177462c69fe059ae/src/service/rooms/event_handler/fetch_and_handle_outliers.rs#L432. There is no backoff in event fetching here, which means c10y will attempt to fetch an auth event 5 times as fast as possible, and then moves on. However, it will then do this dance again if it receives yet another event that references a bad event. This is particularly damaging during catchup. The same failure can be observed for the atomic fallback for *prev* events, https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/4146e0f1daf7d82a58d6a99f177462c69fe059ae/src/service/rooms/event_handler/fetch_and_handle_outliers.rs#L600, however this is significantly less frequent as this is a fallback behaviour that is *usually* not reached due to `get_missing_events`.

nex added this to the 26.8.0 milestone

2026-08-02 09:18:46 +00:00

nex added the

labels

2026-08-02 09:18:46 +00:00

nex self-assigned this

2026-08-02 09:18:46 +00:00

nex commented

2026-08-02 11:02:44 +00:00

Author

Owner

After further investigation:

fetch_and_handle_auth_events is only called during the PDU upgrade process
Misleadingly, it is only used to recursively fetch the auth chain for the state before the incoming event
It might be worth just fetching the entire auth chain via fetch_and_persist_event_auth rather than continuing to attempt to fetch recursively if we encounter an event we cannot fetch, as the chances of a retry succeeding are also slim (UNLESS the error is a 429)
It actually looks like it might be possible for there to be an infinite loop here

After further investigation: 1. `fetch_and_handle_auth_events` is only called during the PDU upgrade process 2. Misleadingly, it is only used to recursively fetch the auth chain for the state before the incoming event 3. It might be worth just fetching the entire auth chain via `fetch_and_persist_event_auth` rather than continuing to attempt to fetch recursively if we encounter an event we cannot fetch, as the chances of a retry succeeding are also slim (UNLESS the error is a 429) 4. It actually looks like it might be possible for there to be an infinite loop here

Aranjedeath commented

2026-08-02 15:18:14 +00:00

Owner

does the old "event backoff" need to be stubbed back in here?

nex commented

2026-08-02 15:19:39 +00:00

Author

Owner

most likely, since fetch_and_persist_event_auth is slow. But also, it might just be worth aborting the entire fetch if we can't get a single event, because we can't upgrade the PDU without the entire auth chain. and like I said, subsequent retries are just as likely to fail anyway. Upgrading PDUs is retried when we receive another one referencing the failed one anyway

most likely, since `fetch_and_persist_event_auth` is slow. But also, it might just be worth aborting the entire fetch if we can't get a single event, because we can't upgrade the PDU without the entire auth chain. and like I said, subsequent retries are just as likely to fail anyway. Upgrading PDUs is retried when we receive another one referencing the failed one anyway

nex commented

2026-08-18 20:36:17 +00:00

Author

Owner

Fixing this will depend on #2097 (for the new backoff stuff)

nex added a new dependency

2026-08-18 20:36:25 +00:00

#2097 feat: Server health tracking

nex commented

2026-08-19 18:42:54 +00:00

Author

Owner

gonna make this blocking because my reverse proxy is sustaining a request load of 100 requests per second just because I'm a backfill candidate in some rooms, and a trusted server, and this is an absurd waste of compute and network

nex added

and removed

labels

2026-08-19 18:43:00 +00:00

nex modified the milestone from 26.8.0 to 26.9.0

2026-08-22 19:20:37 +00:00