WIP: perf: Improve gapfill logic #1818

Draft

nex wants to merge 49 commits from nex/perf/get-missing-events into main

pull from: nex/perf/get-missing-events

merge into: continuwuation:main

continuwuation:main

continuwuation:renovate/rust-non-major

continuwuation:renovate/rust-zerover-patch-updates

continuwuation:renovate/ruma-digest

continuwuation:aranje/illegal-car-mods

continuwuation:ginger/oauth

continuwuation:nex/feat/admin-api

continuwuation:renovate/serde-saphyr-0.x

continuwuation:ginger/kill-sync-tokens

continuwuation:renovate/rand_core-0.x

continuwuation:nex/feat/deprecated-room-versions

continuwuation:release/v0.5.9

continuwuation:nex/feat/enable-debug-log-release-builds

continuwuation:nex/feat/room-purging

continuwuation:nex/feat/room-shutdown

continuwuation:ginger/ruma-upstreaming

continuwuation:jade/tls-backends

continuwuation:ginger/email-fixes

continuwuation:jade/changelog-labels

continuwuation:nex/fix/v12-publishing

continuwuation:jade/build-info

continuwuation:jade/purge-sync-tokens

continuwuation:ginger/terms-and-conditions

continuwuation:ginger/remove-sliding-sync-proxy

continuwuation:nex/fix/pusher-association

continuwuation:ginger/email-support

continuwuation:jade/community-guidelines

continuwuation:nex/fix/federation-format

continuwuation:jade/git-deps-updates

continuwuation:jade/changelog-check

continuwuation:jade/rust-1-92

continuwuation:ginger/password-reset

continuwuation:nex/experiment/push-gateway-logs

continuwuation:ginger/msc3575-obliteration

continuwuation:nex/feat/block-busted-rooms

continuwuation:nex/fix/informative-startup-errs

continuwuation:ginger/no-left-room-initial-sync

continuwuation:jade/docker-entrypoint

continuwuation:jade/dehydrated-devices

continuwuation:ginger/complement-fixes

continuwuation:nex/fix/stale-destination-cache

continuwuation:nex/experiment/sync-mutex

continuwuation:tcpipuk/docker-docs

continuwuation:jade/snafu

continuwuation:jade/rand-update

continuwuation:nex/stateres-refactor

continuwuation:ginger/779-in-troubleshooting

continuwuation:jade/liveit-guide

continuwuation:jade/http3

continuwuation:nex/feat/admin-hide-empty-rooms

continuwuation:ginger/oobe

continuwuation:nex/fix/debian-thingy

continuwuation:jade/ldap-admin-check

continuwuation:nex/fix/remote-restricted-joins

continuwuation:nex/feat/msc4406-sender-ignored

continuwuation:jade/deadlock-detection

continuwuation:jade/get-started

continuwuation:jade/docs-guide

continuwuation:ginger/fix-local-invites

continuwuation:nex/fix/tpi

continuwuation:nex/feat/room-deletion

continuwuation:nex/feat/msc4322-media-redaction

continuwuation:ginger/stitched-order

continuwuation:ginger/deps/update-rspress

continuwuation:jade/admin-announce-improvements

continuwuation:ginger/xtask-improvements

continuwuation:jade/improve-admin-config-display

continuwuation:nex/fix/better-stateres-error-logs

continuwuation:jade/sender-timeouts

continuwuation:nex/feat/custom-v12-room-ids

continuwuation:ginger/update-metadata

continuwuation:nex/feat/admin-force-logout

continuwuation:tom/max-perf-docs

continuwuation:nex/fix/invalid-appservice-reg

continuwuation:nex/feat/antispam

continuwuation:nex/feat/account-locking

continuwuation:jade/logging-cleanup

continuwuation:jade/remove-legacy-appservice-auth

continuwuation:nex/fix/key-query

continuwuation:jade/update-prek

continuwuation:nex/fix/room-summaries

continuwuation:ginger/restrict-admin-commands

continuwuation:ginger/enable-console-by-default

continuwuation:jade/tag-fixes

continuwuation:jade/otlp

continuwuation:nex/meta/pull-req-template

continuwuation:nex/fix/fed-invite-compliance

continuwuation:nex/feat/build-commit

continuwuation:nex/feat/join-logging

continuwuation:jade/mailmap-updates

continuwuation:jade/hack-ci-tmp

continuwuation:jade/v12-stable

continuwuation:jade/relations

continuwuation:ginger/database-refactor

continuwuation:jade/fix-ldap-uiaa

continuwuation:nex/fix/validation

continuwuation:ginger/nuke-invalid-msc4133-fields-in-migration

continuwuation:ginger/downgrade-artifact-actions

continuwuation:oddlid/reload-fix

continuwuation:jade/fix-assert

continuwuation:ginger/sync-v3-cleanup

continuwuation:ginger/remove-absolute-action-urls

continuwuation:jade/website

continuwuation:nex/fix/backoff

continuwuation:ginger/fix-mdbook-for-0.5

continuwuation:ginger/no-docker-on-prs

continuwuation:backport/v0.5.0-rc.8-1

continuwuation:nex/fed-improvements

continuwuation:jade/rust-1.90

continuwuation:jade/mirror-dockerhub

continuwuation:jade/clippy-fixes

continuwuation:jade/fix-support

continuwuation:jade/clean-images

continuwuation:jade/wal-compression-type

continuwuation:jade/flake-clone

continuwuation:ginger/upload-rpms-on-schedule

continuwuation:nex/fix/incoming-fetch

continuwuation:nex/fix/upgrade

continuwuation:tom/ci-fedora-rpm

continuwuation:jade/ci-release-fix

continuwuation:jade/rocksdb-10-5

continuwuation:ginger/fix-msc4133-migration

continuwuation:ginger/migrate-busted-tz

continuwuation:hydra/public

continuwuation:nex/feat/manual-extremities

continuwuation:nex/feat/async-media

continuwuation:nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE

continuwuation:nex/feat/better-logging

continuwuation:trigger-ci-so-latest-isnt-on-illegal-car-mods

continuwuation:nex/feat/pins-backfill

continuwuation:jade/tuwunel-2025-06-old

continuwuation:jade/ai-slop-db-docs

continuwuation:nex/fix-create-auth

continuwuation:jade/version-stats

continuwuation:jade/read-receipts

continuwuation:jade/rust-toolchain-no-targets

continuwuation:jade/logging-features

continuwuation:jade/syncv5-typing

continuwuation:jade/msc2815

continuwuation:morguldir/see-eye

continuwuation:jade/css-small-screen

continuwuation:nex/wip-751

continuwuation:tuwunel-rebase

continuwuation:test

continuwuation:oddlid/rename-admin-room-bot

continuwuation:strawberry/nix-ci-stuff

continuwuation:strawberry/valgrind

continuwuation:phonemain

continuwuation:strawberry/morgs-snake-sync-jason-main

continuwuation:newer-media-endpoints

continuwuation:folly-coroutines-async-io

continuwuation:federation-retry-timer-port

continuwuation:bad-attempt-at-extracting-homeserver-signing-key

continuwuation:room-deletion-attempt-do-not-use

Conversation 1 Commits 49 Files changed 20 +1252 -626

nex commented 2026-05-26 19:16:29 +00:00

Owner

Copy link

Right now our gapfilling code is pretty awful for performance: for each event we are missing, we make an individual GET /_matrix/federation/v1/event/$eventID call, which is incredibly slow. This also happens for fetching auth events we are missing. Luckily, Matrix defines two bulk-fetch endpoints that help us. So, this PR updates our gapfilling code to be more reliable and effective.

Warning

This pull request is UNSTABLE and may result in "PERMANENT" DATA LOSS. Do NOT deploy to a production server until it is confirmed to be stable. I know "we can now persist >1,000 events in under 5 seconds" sounds tantalising, but there's no guarantees half of those events aren't just being silently dropped.

Pull request checklist:

• This pull request targets the main branch, and the branch is named something other than
  main.
• I have written an appropriate pull request title and my description is clear.
• I understand I am responsible for the contents of this pull request.
• I have followed the contributing guidelines:
  • My contribution follows the code style, if applicable.
  • I ran pre-commit checks before opening/drafting this pull request.
  • I have tested my contribution (or proof-read it for documentation-only changes)
    myself, if applicable. This includes ensuring code compiles.
  • My commit messages follow the commit message format and are descriptive.

Right now our gapfilling code is pretty awful for performance: for each event we are missing, we make an individual `GET /_matrix/federation/v1/event/$eventID` call, which is *incredibly* slow. This *also* happens for fetching auth events we are missing. Luckily, Matrix defines two bulk-fetch endpoints that help us. So, this PR updates our gapfilling code to be more reliable and effective. > [!WARNING] > This pull request is **UNSTABLE** and may result in **"PERMANENT" DATA LOSS**. Do NOT deploy to a production server until it is confirmed to be stable. I know "we can now persist >1,000 events in under 5 seconds" sounds tantalising, but there's no guarantees half of those events aren't just being silently dropped. **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages

nex added the

Matrix/Core

Enhancement

Matrix/Federation

labels 2026-05-26 19:16:29 +00:00

nex self-assigned this 2026-05-26 19:16:29 +00:00

nex added 8 commits 2026-05-26 19:16:29 +00:00

feat: Add backfill_missing_events helper 882d6485f8

feat: Better prev event fetching 54fbcbca34

feat: More logging and fix some handling a93ac0a2bc

fix: Don't panic in debug mode when making an empty notary query 1a98507944

fix: This is some bullshit I tell you 40a75d307f

perf: Improve gap filling, handle missing auth events better e5e0949be4

style: Fix some lints 1983872ed4

perf: Make max gap depth fetch configurable a1eeef95e0

forgejo-actions added the

Changelog

Missing

label 2026-05-26 19:16:35 +00:00

nex force-pushed nex/perf/get-missing-events from a1eeef95e0 to d15064871e 2026-05-26 19:22:51 +00:00 Compare

nex added 1 commit 2026-05-26 19:27:57 +00:00

perf: Increase default max_fetch_prev_events to 256 c4d297ae3b

nex added 1 commit 2026-05-26 20:22:37 +00:00

feat: Keep track of a min_depth value ... 0d4bbe612d

Should prevent weird situations where we accidentally gapfill into backfill territory

nex added 1 commit 2026-05-26 20:27:04 +00:00

chore: Add newsfrag c7f8eec282

forgejo-actions added

Changelog

Added

and removed

Changelog

Missing

labels 2026-05-26 20:27:11 +00:00

nex added 5 commits 2026-05-26 23:45:46 +00:00

fix: Only increment mindepth on state events 9132e98236

fix: Lower floor for min depth 8be7739c9a

style: Improve logging 40770cbd08

perf: Don't re-process events as outliers 8ce6ca50dd

fix: Be smarter when re-receiving already-seen PDUs 58b2af3f45

nex added 1 commit 2026-05-26 23:51:23 +00:00

fix: Don't lie about using already-known content a229961c1d

nex added 3 commits 2026-05-28 17:38:38 +00:00

perf(wip): Improve individual events fetcher dc5505f25a

fix: Calculate max iterations dynamically, and bump max prevs e437fc2351

feat: Enhance reliability by fetching full state when we're missing a lot of auth events 94fb7e6f84

nex added 3 commits 2026-05-28 19:04:01 +00:00

fix: Don't repeat already-included metadata in fetch_state instrument de06b7ccf8

fix: Don't try to fetch the same event endlessly 060d0718f1

fix: Remove short-term memory loss ... 4edc4269e7

I keep writing forgetful code, it's a problem

nex added 1 commit 2026-05-28 20:13:51 +00:00

fix: Fall back to atomic fetch when full-state fetch fails 9daf828724

nex added 1 commit 2026-05-29 13:22:16 +00:00

fix: Don't try and fetch zero events c6785befa9

nex added 4 commits 2026-05-29 16:30:51 +00:00

fix: Replace our local extremity tracking when joining a disconnected room remotely 0fd7194084

fix: Hold a federation room lock while remotely joining a room f1456e28f4

fix: Properly remove event_id from the PDU JSON before upgrading it 8b80d64bce

style: Rename gapfill helpers instruments 2eeddc3c14

nex added 1 commit 2026-05-29 19:57:15 +00:00

fix: Don't silence PDU handle logs 701d2aae08

nex added 1 commit 2026-05-29 20:47:43 +00:00

fix: Make dedupe noisy, don't allow non-create event as create event 7ff499a78d

nex added 1 commit 2026-05-30 15:55:39 +00:00

fix: Make PDU handle errors noisier & correct error types 48c182d05d

nex added 1 commit 2026-05-30 16:07:31 +00:00

fix: Remove redundant check that accidentally banned everyone 22abf0acfe

nex added 1 commit 2026-05-30 16:12:44 +00:00

fix: Remove redundant check 2 ... 17d0e3edab

This may look scary, but this is safe because event auth performs the same check, and will reject the event if it doesn't reference the create event correctly.

nex added 1 commit 2026-05-30 16:41:30 +00:00

fix: Ask more servers for state_ids when origin fails to provide ... f06997ba93

Some servers reference events in prev_events that they might not yet have finished processing, so this allows us to at least attempt to get the state from another trustworthy server in the room that might be faster. I don't think this is too effective, however it's more effective than giving up immediately.

nex added 1 commit 2026-05-30 17:02:13 +00:00

fix: Don't treat prev outlier upgrades as fetch failures 82616ed9be

nex added 1 commit 2026-05-30 17:33:53 +00:00

feat: Include timing information in debug logs 186cd149f4

nex added 1 commit 2026-05-30 18:24:56 +00:00

feat: Make logging more verbose to diagnose the aranjesplosion 60e4fe0bed

nex added 1 commit 2026-05-30 18:30:37 +00:00

fix: Don't download the world 110c482122

nex added 1 commit 2026-05-30 19:29:23 +00:00

fix: Downgrade safe assert to debug assert f8ebd0a107

nex added 1 commit 2026-05-30 20:03:09 +00:00

fix: Progress log in fetch_prev 39866f87d6

nex added 1 commit 2026-05-30 20:57:37 +00:00

fix: Elide auth chain from fetch_and_handle_outliers 90c33b6ab2

nex added 1 commit 2026-05-30 21:05:31 +00:00

fix: Be noisy when there's no incoming state ade214cb2b

nex added 1 commit 2026-05-30 21:48:30 +00:00

perf: Don't add trees we already have to latest boundary e0359ad7cb

nex added 1 commit 2026-05-30 21:56:12 +00:00

perf: Don't try to re-persist non-outliers we already have d5eb48e2a2

nex added 1 commit 2026-05-30 23:17:36 +00:00

fix: Friendly assertations 883adcef9e

nex added 1 commit 2026-05-31 14:55:15 +00:00

fix: Correctly pre-populate state events vec with known events 784ae234ca

nex added 2 commits 2026-05-31 15:35:14 +00:00

perf: Always fetch at least N events per GME ef3a54cbe2

fix: Correct inverted boolean condition, add explicit timeout on /state fetch 0acd9819a8

nex added 1 commit 2026-05-31 16:26:33 +00:00

fix: Correctly handle still-missing state, always fetch full state atomically if regular fetch fails 7fe8d95fa1

nex added 1 commit 2026-05-31 16:41:19 +00:00

style: Resolve lint complaints 5290b36f4b

nex added 1 commit 2026-05-31 16:47:26 +00:00

fix: Make fetch_state_ids_from_backfill_servers candidate-free safe c8e1c657ae

nex force-pushed nex/perf/get-missing-events from c8e1c657ae to b7e0d96b91 2026-05-31 17:51:53 +00:00 Compare

nex force-pushed nex/perf/get-missing-events from b7e0d96b91 to c8e1c657ae 2026-05-31 17:52:52 +00:00 Compare

nex commented 2026-05-31 19:09:41 +00:00

Author

Owner

Copy link

So this PR yields unparalleled speed and consistency improvements over the base branch, even in this unoptimised state, however it highlights a problem. Filling gaps can take an extraordinarily long time, especially if servers pushing events to us are unreachable (e.g. ipv6 only server pushes to a server without ipv6 egress), or the room graph is really complex and actively moving (e.g. the ping room). As a result, the median PDU handle time with this PR can easily go from ~4ms to >500ms, with a few minutes being the worst-case observed thus far, since we now try significantly more aggressively to pull in events that we missed. The best-case remains unchanged, and generally speaking consistency is improved too, but in the future, processing missing events and upgrading outlier PDUs to timeline PDUs will probably need to be cast into a background task, as right now the entire room becomes locked while prev events are being fetched, which as stated, can take minutes. Not sure if that'll get squeezed into this PR's scope or whether it'll end up being a separate job.

So this PR yields unparalleled speed and consistency improvements over the base branch, even in this unoptimised state, however it highlights a problem. Filling gaps can take an extraordinarily long time, especially if servers pushing events to us are unreachable (e.g. ipv6 only server pushes to a server without ipv6 egress), or the room graph is really complex and actively moving (e.g. the ping room). As a result, the median PDU handle time with this PR can easily go from ~4ms to >500ms, with a few minutes being the worst-case observed thus far, since we now try significantly more aggressively to pull in events that we missed. The best-case remains unchanged, and generally speaking consistency is improved too, but in the future, processing missing events and upgrading outlier PDUs to timeline PDUs will probably need to be cast into a background task, as right now the entire room becomes locked while prev events are being fetched, which as stated, can take minutes. Not sure if that'll get squeezed into this PR's scope or whether it'll end up being a separate job.

1

All checks were successful

Hide all checks

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s

Required

Details

Checks / Prek / Check changed files (pull_request) Successful in 5s

Required

Details

Documentation / Build and Deploy Documentation (pull_request) Successful in 59s

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m28s

Required

Details

Update flake hashes / update-flake-hashes (pull_request) Successful in 1m35s

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m3s

Required

Details

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin nex/perf/get-missing-events:nex/perf/get-missing-events

git switch nex/perf/get-missing-events

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

Blocked

This pull request or issue is currently blocked from being merged/closed

  

Bug

Something isn't working as intended

  

Changelog

Added

Changelog entry added

  

Changelog

Missing

No changelog when one is needed

  

Changelog

None

Changelog is unnecesary for this change

  

Cherry-picking

Commits picked from other conduit projects

  

Database

This requires or includes changes to the database

  

Dependencies

Something dependency related

  

Dependencies/Renovate

Automatic dependency upgrades by Renovate

  

Difficulty

Easy

Low difficulty to implement - touches few parts of the codebase, low complexity

  

Difficulty

Hard

High difficulty to implement - touches many parts of the codebase, high complexity

  

Difficulty

Medium

Medium difficulty to implement - touches more parts of the codebase, higher complexity

  

Documentation

Improvements or additions to documentation

  

Enhancement

New feature or request

  

Good first issue

Good for newcomers

  

Help wanted

Additional eyes and keyboards are required for this one

  

Inherited

Issues that have been inhereted from the project pre-fork

  

Matrix/Administration

Features pertaining to homeserver administration

  

Matrix/Appservices

Features pertaining to the appservice API

  

Matrix/Auth

Features pertaining to authentication

  

Matrix/Client

Features pertaining to client-to-server interactions

  

Matrix/Core

Issues relating to core matrix functionality, such as state resolution and PDU formats

  

Matrix/E2EE

Issues related to end to end encryption

  

Matrix/Federation

Features pertaining to server-to-server interactions

  

Matrix/Hydra

Issues related to room version 12 and related changes (temporary label)

  

Matrix/MSC

Features pertaining to unstable matrix features

  

Matrix/Media

Features pertaining to media interactions

  

Matrix/T&S

Changes or issues related to trust & safety tooling

  

Merge

This PR is ready to be merged

  

Merge/Manual

This PR should be manually merged

  

Merge/Squash

This PR should be squashed when it is merged

  

Meta

Related to housekeeping, maintenance, or other repo-meta.

  

Meta/CI

Issues related to CI changes

  

Meta/Packaging

Packaging

  

Priority

Blocking

This issue is blocking the next release

  

Priority

High

This issue is very important

  

Priority

Low

This issue is of a rather low priority

  

Security

This item is related to general security

  

Status

Confirmed

This issue has enough information and is confirmed

  

Status

Duplicate

This issue or pull request already exists

  

Status

Invalid

This issue doesn't seem right

  

Status

Needs Investigation

This issue needs further investigation

  

Support

Questions or support requests

  

Wont fix

This will not be worked on

  

old/ci/cd

Ci/CD

Archived

  

old/rust

Pull requests that update Rust code

Archived

No labels

Blocked

Bug

Changelog

Added

Changelog

Missing

Changelog

None

Cherry-picking

Database

Dependencies

Dependencies/Renovate

Difficulty

Easy

Difficulty

Hard

Difficulty

Medium

Documentation

Enhancement

Good first issue

Help wanted

Inherited

Matrix/Administration

Matrix/Appservices

Matrix/Auth

Matrix/Client

Matrix/Core

Matrix/E2EE

Matrix/Federation

Matrix/Hydra

Matrix/MSC

Matrix/Media

Matrix/T&S

Merge

Merge/Manual

Merge/Squash

Meta

Meta/CI

Meta/Packaging

Priority

Blocking

Priority

High

Priority

Low

Security

Status

Confirmed

Status

Duplicate

Status

Invalid

Status

Needs Investigation

Support

Wont fix

old/ci/cd

old/rust

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

nex

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

continuwuation/continuwuity!1818

No description provided.