WIP: perf: Improve gapfill logic #1818

Draft

nex wants to merge 49 commits from nex/perf/get-missing-events into main

Author	SHA1	Message	Date
timedout	c8e1c657ae	fix: Make `fetch_state_ids_from_backfill_servers` candidate-free safe All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s Details Checks / Prek / Check changed files (pull_request) Successful in 5s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m28s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m35s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m3s Details	2026-05-31 17:47:22 +01:00
timedout	5290b36f4b	style: Resolve lint complaints Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m15s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m18s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m25s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-31 17:41:15 +01:00
timedout	7fe8d95fa1	fix: Correctly handle still-missing state, always fetch full state atomically if regular fetch fails Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 6s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m14s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m29s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m6s Details	2026-05-31 17:26:26 +01:00
timedout	0acd9819a8	fix: Correct inverted boolean condition, add explicit timeout on /state fetch Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 7s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m18s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m12s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m6s Details	2026-05-31 16:35:09 +01:00
timedout	ef3a54cbe2	perf: Always fetch at least N events per GME	2026-05-31 16:24:03 +01:00
timedout	784ae234ca	fix: Correctly pre-populate state events vec with known events Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m12s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m14s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m58s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m5s Details	2026-05-31 15:55:08 +01:00
timedout	883adcef9e	fix: Friendly assertations Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m0s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m15s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m27s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m7s Details	2026-05-31 00:17:30 +01:00
timedout	d5eb48e2a2	perf: Don't try to re-persist non-outliers we already have Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m1s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m15s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m30s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m14s Details	2026-05-30 22:56:00 +01:00
timedout	e0359ad7cb	perf: Don't add trees we already have to latest boundary Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 57s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m27s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m8s Details	2026-05-30 22:48:23 +01:00
timedout	ade214cb2b	fix: Be noisy when there's no incoming state All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s Details Checks / Prek / Check changed files (pull_request) Successful in 9s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 56s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m41s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m40s Details	2026-05-30 22:05:19 +01:00
timedout	90c33b6ab2	fix: Elide auth chain from fetch_and_handle_outliers Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m0s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m18s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m28s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-30 21:57:30 +01:00
timedout	39866f87d6	fix: Progress log in fetch_prev Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 7s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 55s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 57s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m35s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m5s Details	2026-05-30 21:03:01 +01:00
timedout	f8ebd0a107	fix: Downgrade safe assert to debug assert All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 7s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 56s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m0s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m37s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m42s Details	2026-05-30 20:29:16 +01:00
timedout	110c482122	fix: Don't download the world All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 59s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m1s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m41s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m34s Details	2026-05-30 19:30:35 +01:00
timedout	60e4fe0bed	feat: Make logging more verbose to diagnose the aranjesplosion Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 6s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 55s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 58s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m35s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-30 19:24:48 +01:00
timedout	186cd149f4	feat: Include timing information in debug logs All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 4s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m13s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m30s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m42s Details	2026-05-30 18:33:44 +01:00
timedout	82616ed9be	fix: Don't treat prev outlier upgrades as fetch failures All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 54s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m15s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m31s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m30s Details	2026-05-30 18:02:09 +01:00
timedout	f06997ba93	fix: Ask more servers for state_ids when origin fails to provide All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 57s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m10s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m18s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m44s Details Some servers reference events in prev_events that they might not yet have finished processing, so this allows us to at least attempt to get the state from another trustworthy server in the room that might be faster. I don't think this is too effective, however it's more effective than giving up immediately.	2026-05-30 17:41:23 +01:00
timedout	17d0e3edab	fix: Remove redundant check 2 All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s Details Checks / Prek / Check changed files (pull_request) Successful in 6s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 55s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 58s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m36s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m48s Details This may look scary, but this is safe because event auth performs the same check, and will reject the event if it doesn't reference the create event correctly.	2026-05-30 17:12:40 +01:00
timedout	22abf0acfe	fix: Remove redundant check that accidentally banned everyone Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 6s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 55s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m37s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-30 17:07:27 +01:00
timedout	48c182d05d	fix: Make PDU handle errors noisier & correct error types All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m11s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m57s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m3s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m44s Details	2026-05-30 16:55:32 +01:00
timedout	7ff499a78d	fix: Make dedupe noisy, don't allow non-create event as create event All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Checks / Prek / Check changed files (pull_request) Successful in 6s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 56s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m35s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m47s Details	2026-05-29 21:47:36 +01:00
timedout	701d2aae08	fix: Don't silence PDU handle logs All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 55s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m12s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m32s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m46s Details	2026-05-29 20:57:05 +01:00
timedout	2eeddc3c14	style: Rename gapfill helpers instruments All checks were successful Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 59s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m12s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m32s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m40s Details	2026-05-29 17:30:46 +01:00
timedout	8b80d64bce	fix: Properly remove `event_id` from the PDU JSON before upgrading it	2026-05-29 17:15:07 +01:00
timedout	f1456e28f4	fix: Hold a federation room lock while remotely joining a room	2026-05-29 14:53:15 +01:00
timedout	0fd7194084	fix: Replace our local extremity tracking when joining a disconnected room remotely	2026-05-29 14:45:37 +01:00
timedout	c6785befa9	fix: Don't try and fetch zero events All checks were successful Checks / Prek / Check changed files (pull_request) Successful in 5s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 53s Details Update flake hashes / update-flake-hashes (pull_request) Successful in 1m12s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m21s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m45s Details	2026-05-29 14:22:08 +01:00
timedout	9daf828724	fix: Fall back to atomic fetch when full-state fetch fails Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 5s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Failing after 2m15s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 4m26s Details Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 5m18s Details	2026-05-28 21:13:42 +01:00
timedout	4edc4269e7	fix: Remove short-term memory loss Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 5s Details Checks / Prek / Check changed files (pull_request) Successful in 5s Details Documentation / Build and Deploy Documentation (pull_request) Failing after 2m17s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 4m27s Details Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 5m17s Details I keep writing forgetful code, it's a problem	2026-05-28 19:59:33 +01:00
timedout	060d0718f1	fix: Don't try to fetch the same event endlessly	2026-05-28 19:30:17 +01:00
timedout	de06b7ccf8	fix: Don't repeat already-included metadata in fetch_state instrument	2026-05-28 18:46:27 +01:00
timedout	94fb7e6f84	feat: Enhance reliability by fetching full state when we're missing a lot of auth events Some checks failed Checks / Prek / Check changed files (pull_request) Successful in 4s Details Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Failing after 2m17s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 4m27s Details Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 6m0s Details	2026-05-28 18:38:27 +01:00
timedout	e437fc2351	fix: Calculate max iterations dynamically, and bump max prevs	2026-05-28 17:42:01 +01:00
timedout	dc5505f25a	perf(wip): Improve individual events fetcher	2026-05-27 02:43:08 +01:00
timedout	a229961c1d	fix: Don't lie about using already-known content Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Check changed files (pull_request) Successful in 6s Details Documentation / Build and Deploy Documentation (pull_request) Failing after 2m18s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m52s Details Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 25m17s Details	2026-05-27 00:51:12 +01:00
timedout	58b2af3f45	fix: Be smarter when re-receiving already-seen PDUs Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Check changed files (pull_request) Successful in 33s Details Documentation / Build and Deploy Documentation (pull_request) Failing after 2m18s Details Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-27 00:43:37 +01:00
timedout	8ce6ca50dd	perf: Don't re-process events as outliers	2026-05-27 00:33:57 +01:00
timedout	40770cbd08	style: Improve logging	2026-05-27 00:25:56 +01:00
timedout	8be7739c9a	fix: Lower floor for min depth	2026-05-27 00:08:11 +01:00
timedout	9132e98236	fix: Only increment mindepth on state events	2026-05-26 23:29:31 +01:00
timedout	c7f8eec282	chore: Add newsfrag Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s Details Checks / Prek / Check changed files (pull_request) Successful in 7s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m12s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 21m16s Details	2026-05-26 21:26:57 +01:00
timedout	0d4bbe612d	feat: Keep track of a min_depth value Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Failing after 8s Details Checks / Prek / Check changed files (pull_request) Successful in 32s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m14s Details Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details Should prevent weird situations where we accidentally gapfill into backfill territory	2026-05-26 21:22:32 +01:00
timedout	c4d297ae3b	perf: Increase default `max_fetch_prev_events` to 256 Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Failing after 8s Details Checks / Prek / Check changed files (pull_request) Successful in 27s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s Details Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m32s Details Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 34m17s Details	2026-05-26 20:25:37 +01:00
timedout	d15064871e	perf: Make max gap depth fetch configurable Some checks failed Checks / Changelog / Check changelog is added (pull_request_target) Failing after 33s Details Checks / Prek / Check changed files (pull_request) Successful in 10s Details Documentation / Build and Deploy Documentation (pull_request) Successful in 1m58s Details Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details	2026-05-26 20:22:45 +01:00
timedout	b925936195	perf: Improve gap filling, handle missing auth events better	2026-05-26 20:22:45 +01:00
timedout	56feba0ea0	fix: This is some bullshit I tell you	2026-05-26 20:22:40 +01:00
timedout	8d89ba94d5	feat: Better prev event fetching fix: Don't panic in debug mode when making an empty notary query	2026-05-26 20:22:10 +01:00
timedout	0b135c7717	feat: Add backfill_missing_events helper	2026-05-26 20:22:10 +01:00