perf: Improve gapfill logic #1818

Merged

Jade merged 82 commits from nex/perf/get-missing-events into main 2026-07-01 18:45:08 +00:00

Conversation 14 Commits 82 Files changed 28 +1712 -717

82 commits

Author	SHA1	Message	Date
timedout	62e0b53f52	feat: Prioritise sending dummy events when the extremity count reaches 20 or more	2026-07-01 19:15:35 +01:00
timedout	5fa3d5f6c8	fix: Don't create fake objects when defaulting a partial PDU build	2026-07-01 19:15:35 +01:00
timedout	9b80a99aa7	style: De-duplicate build_local_dag definitions, replace outdated comment on unwrap safety	2026-07-01 19:15:22 +01:00
timedout	19a83c6891	style: Add docstrings to squasher functions	2026-06-29 18:37:04 +01:00
timedout	2bfd678c71	feat: Add jitter to extremity squasher	2026-06-29 18:29:04 +01:00
timedout	e936d18324	feat: Don't panic in build_local_dag ... Previously the function assumed the caller had performed proper validation on the inputs (and all current callers do), but this is a poor reason to panic when sane error handling is available. Events with no prev events now return an error, and prev events which are illegal are simply skipped.	2026-06-29 18:12:16 +01:00
timedout	90797fa3cd	fix: Avoid string roundtrip when fetching timestamps in build_local_dag	2026-06-29 17:44:13 +01:00
Jade Ellis	5199cde870	feat: Allow sending dummy events to clients	2026-06-28 03:44:23 +01:00
Jade Ellis	b6bc7dfc16	feat: Debounce extremity squashing ... Additionaly circuit-breaks it if the squash would have only b triggered by dummy events / other squashes	2026-06-28 01:29:52 +01:00
Jade Ellis	945ea5a78a	chore: Box large futures	2026-06-28 01:29:52 +01:00
timedout	d719fe2048	chore: Merge origin/main into nex/perf/get-missing-events	2026-06-28 01:29:23 +01:00
timedout	3b6858e936	perf: Throttle dummy events to prevent stampeding	2026-06-23 15:26:28 +01:00
timedout	7ca00e4ab9	chore: Drop unused param from handle_outlier_pdu	2026-06-20 17:08:00 +01:00
timedout	b8ca06029f	perf: Use a hashmap for full-state filtering	2026-06-20 17:08:00 +01:00
timedout	2efe8f2ec0	fix: Inverted power level check in extremity squash	2026-06-20 17:08:00 +01:00
timedout	d1aa911739	fix: Rename variables in auth event fetcher ... Also fixes a couple bugs where events were being misattributed	2026-06-20 17:08:00 +01:00
timedout	4673282ca1	perf: Don't try to fetch prevs we already fetched ... graphs are hard	2026-06-20 17:08:00 +01:00
timedout	0c03195aec	fix: Fall back to legacy behaviour when prev events are missed from get_missing_events	2026-06-20 17:08:00 +01:00
timedout	439bc2784d	style: Use more explicit variable names	2026-06-20 17:08:00 +01:00
timedout	689a1ce59b	style: Use user_can_send_message	2026-06-20 17:08:00 +01:00
timedout	c141503ccb	style: Re-use GET_MISSING_EVENTS_MAX_BATCH_SIZE	2026-06-20 17:08:00 +01:00
timedout	bcadecdc3b	feat: Add !admin debug rooms-by-extremity-count command	2026-06-20 17:08:00 +01:00
timedout	ab3be337cb	chore: Reformat	2026-06-20 17:08:00 +01:00
timedout	aa1281a9e0	fix: Prevent arbitrary state injection attack	2026-06-20 17:08:00 +01:00
timedout	115a2e802e	style: Check power levels before attempting to send extremity squashes ... Solves a problem where the console screams in agony when local users can't send dummy events	2026-06-20 17:08:00 +01:00
timedout	968328d788	perf: Squash weird mutable variable	2026-06-20 17:08:00 +01:00
timedout	f1cde5f323	style: Fix up some TODOs	2026-06-20 17:08:00 +01:00
timedout	832ee8650b	style: Adjust docstrings and dodgy comment	2026-06-20 17:08:00 +01:00
timedout	c0666a1793	fix: Default PDU content to empty object instead of literal NULL	2026-06-20 17:08:00 +01:00
timedout	ddb4ef539f	fix: un-forget how streams work	2026-06-20 17:08:00 +01:00
timedout	3faedc4581	perf: Remove huge clone and tackle TODOs	2026-06-20 17:08:00 +01:00
timedout	a3544353ba	feat: Automatically squash extremities when they exceed a threshold ... Attempts to tackle #1844	2026-06-20 17:08:00 +01:00
timedout	b0612397d3	style: Tidy up	2026-06-20 17:08:00 +01:00
timedout	31737e127e	fix: Make fetch_state_ids_from_backfill_servers candidate-free safe	2026-06-20 17:08:00 +01:00
timedout	486dcd208c	style: Resolve lint complaints	2026-06-20 17:08:00 +01:00
timedout	a945a4b2ad	fix: Correctly handle still-missing state, always fetch full state atomically if regular fetch fails	2026-06-20 17:08:00 +01:00
timedout	c28ea44e11	fix: Correct inverted boolean condition, add explicit timeout on /state fetch	2026-06-20 17:08:00 +01:00
timedout	3e4d6b2565	perf: Always fetch at least N events per GME	2026-06-20 17:08:00 +01:00
timedout	29ce21cd2e	fix: Correctly pre-populate state events vec with known events	2026-06-20 17:08:00 +01:00
timedout	d461c6977a	fix: Friendly assertations	2026-06-20 17:08:00 +01:00
timedout	5a0d6461d1	perf: Don't try to re-persist non-outliers we already have	2026-06-20 17:08:00 +01:00
timedout	2ec7394785	perf: Don't add trees we already have to latest boundary	2026-06-20 17:08:00 +01:00
timedout	8a5708b9f9	fix: Be noisy when there's no incoming state	2026-06-20 17:08:00 +01:00
timedout	cd46070bd3	fix: Elide auth chain from fetch_and_handle_outliers	2026-06-20 17:08:00 +01:00
timedout	9930e549a0	fix: Progress log in fetch_prev	2026-06-20 17:08:00 +01:00
timedout	36ed20cb04	fix: Downgrade safe assert to debug assert	2026-06-20 17:08:00 +01:00
timedout	3907589b6c	fix: Don't download the world	2026-06-20 17:08:00 +01:00
timedout	2a598af888	feat: Make logging more verbose to diagnose the aranjesplosion	2026-06-20 17:08:00 +01:00
timedout	f34f84832b	feat: Include timing information in debug logs	2026-06-20 17:08:00 +01:00
timedout	0efa6ed1f2	fix: Don't treat prev outlier upgrades as fetch failures	2026-06-20 17:08:00 +01:00
timedout	2bf5876778	fix: Ask more servers for state_ids when origin fails to provide ... Some servers reference events in prev_events that they might not yet have finished processing, so this allows us to at least attempt to get the state from another trustworthy server in the room that might be faster. I don't think this is too effective, however it's more effective than giving up immediately.	2026-06-20 17:08:00 +01:00
timedout	31982e84de	fix: Remove redundant check 2 ... This may look scary, but this is safe because event auth performs the same check, and will reject the event if it doesn't reference the create event correctly.	2026-06-20 17:08:00 +01:00
timedout	970958652a	fix: Remove redundant check that accidentally banned everyone	2026-06-20 17:08:00 +01:00
timedout	c94a395bf0	fix: Make PDU handle errors noisier & correct error types	2026-06-20 17:08:00 +01:00
timedout	43adff926f	fix: Make dedupe noisy, don't allow non-create event as create event	2026-06-20 17:07:59 +01:00
timedout	04fce56381	fix: Don't silence PDU handle logs	2026-06-20 17:07:59 +01:00
timedout	4df2097e6c	style: Rename gapfill helpers instruments	2026-06-20 17:07:59 +01:00
timedout	8b85b04d10	fix: Properly remove event_id from the PDU JSON before upgrading it	2026-06-20 17:07:59 +01:00
timedout	9e0bcd3be8	fix: Hold a federation room lock while remotely joining a room	2026-06-20 17:07:59 +01:00
timedout	9509080e0d	fix: Replace our local extremity tracking when joining a disconnected room remotely	2026-06-20 17:07:59 +01:00
timedout	61066bb0c6	fix: Don't try and fetch zero events	2026-06-20 17:07:59 +01:00
timedout	573d5bc50e	fix: Fall back to atomic fetch when full-state fetch fails	2026-06-20 17:07:59 +01:00
timedout	162e6eb92f	fix: Remove short-term memory loss ... I keep writing forgetful code, it's a problem	2026-06-20 17:07:59 +01:00
timedout	cef4ebe38e	fix: Don't try to fetch the same event endlessly	2026-06-20 17:07:59 +01:00
timedout	316a0b7d58	fix: Don't repeat already-included metadata in fetch_state instrument	2026-06-20 17:07:59 +01:00
timedout	2bcc56704b	feat: Enhance reliability by fetching full state when we're missing a lot of auth events	2026-06-20 17:07:59 +01:00
timedout	7f64de9727	fix: Calculate max iterations dynamically, and bump max prevs	2026-06-20 17:07:59 +01:00
timedout	aea03f2f99	perf(wip): Improve individual events fetcher	2026-06-20 17:07:59 +01:00
timedout	8edf9552b8	fix: Don't lie about using already-known content	2026-06-20 17:07:59 +01:00
timedout	d5f69c8a31	fix: Be smarter when re-receiving already-seen PDUs	2026-06-20 17:07:59 +01:00
timedout	9ea9b0e04c	perf: Don't re-process events as outliers	2026-06-20 17:07:59 +01:00
timedout	e8db01fc8d	style: Improve logging	2026-06-20 17:07:59 +01:00
timedout	f80e1e89a5	fix: Lower floor for min depth	2026-06-20 17:07:59 +01:00
timedout	b9dca84acf	fix: Only increment mindepth on state events	2026-06-20 17:07:59 +01:00
timedout	e3ec1066c4	chore: Add newsfrag	2026-06-20 17:07:59 +01:00
timedout	1445a8d446	feat: Keep track of a min_depth value ... Should prevent weird situations where we accidentally gapfill into backfill territory	2026-06-20 17:07:59 +01:00
timedout	9547c438d6	perf: Increase default max_fetch_prev_events to 256	2026-06-20 17:07:59 +01:00
timedout	51d0e615f5	perf: Make max gap depth fetch configurable	2026-06-20 17:07:59 +01:00
timedout	eeb937416c	perf: Improve gap filling, handle missing auth events better	2026-06-20 17:07:59 +01:00
timedout	0d5aa7ede1	fix: This is some bullshit I tell you	2026-06-20 17:07:59 +01:00
timedout	ba9dc27773	feat: Better prev event fetching ... fix: Don't panic in debug mode when making an empty notary query	2026-06-20 17:07:59 +01:00
timedout	abf5a155ba	feat: Add backfill_missing_events helper	2026-06-20 17:07:59 +01:00