continuwuation/continuwuity
continuwuation/continuwuity
16
112
Fork 123
Code Issues 136 Pull requests 24 Releases 40 Packages 3 Activity Actions 2

fix: Properly track orphaned extremities #2164

Merged
nex merged 7 commits from nex/fix/extremity-tracking-please-please-please-im-so-sick-of-fixing-extremity-tracking into main 2026-08-22 19:19:50 +00:00
Conversation 9 Commits 7 Files changed 2 +70 -36
nex commented 2026-08-21 04:55:19 +00:00
Owner
Copy link

This pull request fixes extremity tracking once again, this time fixing the issue where soft-failed/rejected events fail to adequately mark their predecessors as referenced, which resulted in a build-up of orphaned extremities. This PR solves that by reworking how extremities are calculated to use this new algorithm:

  1. Starting from the current extremity set,
  2. Add the incoming event to the extremity set
  3. Remove all events from the extremity set that have themselves been marked as referenced
  4. Then, for each ancestor, starting with the incoming event's prev_events:
    i. Mark the ancestor as referenced and drop it from the extremity set
    ii. Gather all prev_events of the ancestor that are soft-failed OR rejected OR an outlier for another reason, AND not yet marked as referenced
    iii. Add each ancestor from the previous step to the queue
    iv. Repeat until there are no more such events from step ii

this ensures events which have actually been referenced but perhaps only by an event which was not added to our timeline tree is appropriately marked as such, preventing a build-up of soft-failed and rejected events.

Fixes: #2123

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
<!-- In order to help reviewers know what your pull request does at a glance, you should ensure that 1. Your PR title is a short, single sentence describing what you changed 2. You have described in more detail what you have changed, why you have changed it, what the intended effect is, and why you think this will be beneficial to the project. If you have made any potentially strange/questionable design choices, but didn't feel they'd benefit from code comments, please don't mention them here - after opening your pull request, go to "files changed", and click on the "+" symbol in the line number gutter, and attach comments to the lines that you think would benefit from some clarification. --> This pull request fixes extremity tracking once again, this time fixing the issue where soft-failed/rejected events fail to adequately mark their predecessors as referenced, which resulted in a build-up of orphaned extremities. This PR solves that by reworking how extremities are calculated to use this new algorithm: 1. Starting from the current extremity set, 2. Add the incoming event to the extremity set 3. Remove all events from the extremity set that have themselves been marked as referenced 4. Then, for each ancestor, starting with the incoming event's `prev_events`: i. Mark the ancestor as referenced and drop it from the extremity set ii. Gather all `prev_events` of the ancestor that are soft-failed OR rejected OR an outlier for another reason, AND not yet marked as referenced iii. Add each ancestor from the previous step to the queue iv. Repeat until there are no more such events from step ii this ensures events which have actually been referenced but perhaps only by an event which was not added to our timeline tree is appropriately marked as such, preventing a build-up of soft-failed and rejected events. Fixes: #2123 <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages
nex added this to the 26.8.0 milestone 2026-08-21 04:55:19 +00:00
nex self-assigned this 2026-08-21 04:55:19 +00:00
perf: Remove unnecessary retain
Some checks failed
Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 4s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m9s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m18s
Details
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m22s
Details
39b01da74f
 
The remove() at the start of the function already does this
nex requested reviews from Jade, ginger 2026-08-21 04:57:07 +00:00
chore: Add newsfrag and re-run cargo fmt
All checks were successful
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m4s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m13s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m11s
Details
5061512fee
eleboucher approved these changes 2026-08-22 10:03:34 +00:00
Dismissed
Jade commented 2026-08-22 10:06:14 +00:00
Owner
Copy link

Uh are rejected events supposed to mark things as referenced? I don't think they are given they are excluded from the DAG

Uh are rejected events supposed to mark things as referenced? I don't think they are given they are excluded from the DAG
eleboucher requested changes 2026-08-22 10:24:18 +00:00
Dismissed
src/service/rooms/timeline/append.rs Outdated
@ -221,1 +250,4 @@
if forward_extremities.is_empty() {
return Err!("Resolved extremities cannot be empty");
}
Ok(forward_extremities)
eleboucher commented 2026-08-22 10:21:00 +00:00
Contributor
Copy link

i guess we should return new_extremities there

i guess we should return new_extremities there
nex marked this conversation as resolved
src/service/rooms/timeline/append.rs Outdated
@ -191,3 +192,2 @@
.await
.eq(&false)
.then_some(event_id)
.then_some(extremity)
eleboucher commented 2026-08-22 10:21:42 +00:00
Contributor
Copy link

i guess this is not what we want, this keeps the referenced events and drops the unreferenced ones. Needs the .eq(&false) back

i guess this is not what we want, this keeps the referenced events and drops the unreferenced ones. Needs the .eq(&false) back
nex marked this conversation as resolved
src/service/rooms/timeline/append.rs Outdated
@ -213,0 +223,4 @@
.map(ToOwned::to_owned)
.stream()
.broad_filter_map(|prev_id| async move {
let room_id = incoming_pdu.room_id_or_hash();
eleboucher commented 2026-08-22 10:22:34 +00:00
Contributor
Copy link

we have it already on 210

we have it already on 210
nex commented 2026-08-22 16:15:34 +00:00
Author
Owner
Copy link

Already have what? 210 is the incoming event's prev_events (initial work set), this line adds any prev events of the prev event we're working on that is unreferenced and a local outlier/soft-failed/rejected, so that their prev events will be walked until we reach the root of the unaccepted tree and reach the first accepted event (or create event)

Already have what? 210 is the incoming event's `prev_events` (initial work set), this line adds any prev events of the prev event we're working on that is unreferenced and a local outlier/soft-failed/rejected, so that their prev events will be walked until we reach the root of the unaccepted tree and reach the first accepted event (or create event)
eleboucher commented 2026-08-22 17:01:09 +00:00
Contributor
Copy link

we already have the room_id we don't need to create it again no ?

we already have the room_id we don't need to create it again no ?
nex commented 2026-08-22 17:19:23 +00:00
Author
Owner
Copy link

Oh yeah, so we do, I forgot I put that up there

Oh yeah, so we do, I forgot I put that up there
ginger commented 2026-08-22 18:29:36 +00:00
Owner
Copy link

@nex fix

@nex fix
eleboucher commented 2026-08-22 18:30:21 +00:00
Contributor
Copy link

She just want to eat our memory like a cookie monster

She just want to eat our memory like a cookie monster
nex marked this conversation as resolved
nex commented 2026-08-22 16:08:34 +00:00
Author
Owner
Copy link

Sorry this wasn't quite ready for review, I had changed staged but couldn't push them as the forge went down

Sorry this wasn't quite ready for review, I had changed staged but couldn't push them as the forge went down
nex commented 2026-08-22 16:11:24 +00:00
Author
Owner
Copy link

@Jade wrote in #2164 (comment):

Uh are rejected events supposed to mark things as referenced? I don't think they are given they are excluded from the DAG

They aren't marking things as referenced, this is only if they're ancestors of a legal event (or an event referenced by the ancestor that previously would've been unreferenced). Synapse also does this

@Jade wrote in https://forgejo.ellis.link/continuwuation/continuwuity/pulls/2164#issuecomment-34625: > Uh are rejected events supposed to mark things as referenced? I don't think they are given they are excluded from the DAG They aren't marking things as referenced, this is only if they're ancestors of a legal event (or an event referenced by the ancestor that previously would've been unreferenced). [Synapse also does this](https://github.com/element-hq/synapse/blob/3debaff0f86c1f34d5c5970bd5a10bea8a67b266/synapse/storage/databases/main/events.py#L979)
👍 1
fix: Return correct new extremities set
All checks were successful
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m6s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m14s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m17s
Details
4db94bbe17
ginger requested changes 2026-08-22 18:35:51 +00:00
Dismissed
src/service/rooms/timeline/append.rs Outdated
@ -180,0 +182,4 @@
current_extremities.into_iter().collect();
// Add the incoming event
new_extremities.insert(incoming_pdu.event_id().to_owned());
// Remove any extremities that have since been referenced but not removed(?)
ginger commented 2026-08-22 18:29:03 +00:00
Owner
Copy link

that "(?)" fills me with confidence

that "(?)" fills me with confidence
nex commented 2026-08-22 18:50:06 +00:00
Author
Owner
Copy link

I don't think it's possible but it's a good check to have anyway to be safe, no?

I don't think it's possible but it's a good check to have anyway to be safe, no?
ginger marked this conversation as resolved
src/service/rooms/timeline/append.rs Outdated
@ -219,3 +250,2 @@
);
assert!(!forward_extremities.is_empty(), "resolved extremities cannot be empty");
Ok(forward_extremities)
debug_assert!(!new_extremities.is_empty(), "resolved extremities cannot be empty");
ginger commented 2026-08-22 18:35:32 +00:00
Owner
Copy link

me when I check the same thing twice

me when I check the same thing twice
nex commented 2026-08-22 18:49:35 +00:00
Author
Owner
Copy link

In debug this will panic (incredibly noisy), in release this will simply return an error (still noisy but unlikely to Fuck Shit Up)

In debug this will panic (incredibly noisy), in release this will simply return an error (still noisy but unlikely to Fuck Shit Up)
ginger commented 2026-08-22 18:51:02 +00:00
Owner
Copy link

this could be one if with a #[cfg(debug_assertions)] { panic!() } but ok 🤓

this could be one `if` with a `#[cfg(debug_assertions)] { panic!() }` but ok 🤓
nex commented 2026-08-22 18:53:35 +00:00
Author
Owner
Copy link
if new_extremities.is_empty() {
    panic!("Resolved extremities cannot be empty");
    Err!("Resolved extremities cannot be empty")
} else {
    Ok(new_extremities.into_iter().collect())
}

I could do this but my IDE insists that Err!() is unreachable because it lints with the debug profile 😭

```rs if new_extremities.is_empty() { panic!("Resolved extremities cannot be empty"); Err!("Resolved extremities cannot be empty") } else { Ok(new_extremities.into_iter().collect()) } ``` I could do this but my IDE insists that Err!() is unreachable because it lints with the debug profile 😭
nex commented 2026-08-22 18:54:02 +00:00
Author
Owner
Copy link

I could make a new debug_assert_or_error macro 🤔

I could make a new `debug_assert_or_error` macro 🤔
ginger commented 2026-08-22 18:54:19 +00:00
Owner
Copy link

it is unreachable because that panic! doesn't have a #[cfg] on it

it is unreachable because that `panic!` doesn't have a `#[cfg]` on it
nex commented 2026-08-22 18:54:55 +00:00
Author
Owner
Copy link

even with the cfg, I apologise for writing code in the forgejo comment box it wont happen again

even with the cfg, I apologise for writing code in the forgejo comment box it wont happen again
ginger commented 2026-08-22 18:55:41 +00:00
Owner
Copy link

add a #[cfg(not())] on the err return then

add a` #[cfg(not())]` on the err return then
nex commented 2026-08-22 18:57:55 +00:00
Author
Owner
Copy link

But then I'd be writing

if new_extremities.is_empty() {
    #[cfg(debug_assertions)]
    panic!("Resolved extremities cannot be empty");
    #[cfg(not(debug_assertions))]
    Err!("Resolved extremities cannot be empty")
} else {
    Ok(new_extremities.into_iter().collect())
}

which is insane, and the current

		debug_assert!(!new_extremities.is_empty(), "resolved extremities cannot be empty"); 
		if new_extremities.is_empty() {
		    return Err!("Resolved extremities cannot be empty");
		}
		Ok(new_extremities.into_iter().collect())

is just better atp

But then I'd be writing ```rs if new_extremities.is_empty() { #[cfg(debug_assertions)] panic!("Resolved extremities cannot be empty"); #[cfg(not(debug_assertions))] Err!("Resolved extremities cannot be empty") } else { Ok(new_extremities.into_iter().collect()) } ``` which is insane, and the current ```rs debug_assert!(!new_extremities.is_empty(), "resolved extremities cannot be empty"); if new_extremities.is_empty() { return Err!("Resolved extremities cannot be empty"); } Ok(new_extremities.into_iter().collect()) ``` is just better atp
ginger commented 2026-08-22 19:01:55 +00:00
Owner
Copy link

no it's much clearer

no it's much clearer
🧐 1
nex marked this conversation as resolved
fix: Address review comments
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m2s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Details
6a58539caf
eleboucher approved these changes 2026-08-22 19:14:16 +00:00
nex force-pushed nex/fix/extremity-tracking-please-please-please-im-so-sick-of-fixing-extremity-tracking from 6a58539caf
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m2s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled
Details
to 103dafd529
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m5s
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m15s
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m20s
Details
Documentation / Build and Deploy Documentation (push) Successful in 1m6s
Details
Checks / Prek / Pre-commit & Formatting (push) Successful in 1m15s
Details
Checks / Prek / Check changed files (push) Successful in 6s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m7s
Details
Release Docker Image / Build linux-amd64 (release) (push) Failing after 3m15s
Details
Release Docker Image / Build linux-arm64 (release) (push) Successful in 10m13s
Details
Release Docker Image / Create Multi-arch Release Manifest (push) Has been skipped
Details
Release Docker Image / Build linux-amd64 (max-perf) (push) Has been skipped
Details
Release Docker Image / Build linux-arm64 (max-perf) (push) Has been skipped
Details
Release Docker Image / Create Max-Perf Manifest (push) Has been skipped
Details
Release Docker Image / Mirror Images (push) Has been skipped
Details
Release Docker Image / Release Binaries (push) Has been skipped
Details
Checks / Prek / Clippy and Cargo Tests (push) Successful in 9m46s
Details
2026-08-22 19:14:48 +00:00 Compare
nex requested review from ginger 2026-08-22 19:16:00 +00:00
ginger approved these changes 2026-08-22 19:17:29 +00:00
nex merged commit 103dafd529 into main 2026-08-22 19:19:50 +00:00
nex deleted branch nex/fix/extremity-tracking-please-please-please-im-so-sick-of-fixing-extremity-tracking 2026-08-22 19:19:50 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Jade
Aranjedeath
eleboucher
ginger
Labels
Clear labels   
Abandoned
This pull request appears to be abandoned by its author.

  
Blocked
This pull request or issue is currently blocked from being merged/closed

  
Bug
Something isn't working as intended

  
Changelog
Added
 Changelog entry added

  
Changelog
Missing
 No changelog when one is needed

  
Changelog
None
 Changelog is unnecesary for this change

  
Cherry-picking
Commits picked from other conduit projects

  
Database
This requires or includes changes to the database

  
Dependencies
Something dependency related

  
Dependencies/Renovate
Automatic dependency upgrades by Renovate

  
Difficulty
Easy
 Low difficulty to implement - touches few parts of the codebase, low complexity

  
Difficulty
Hard
 High difficulty to implement - touches many parts of the codebase, high complexity

  
Difficulty
Medium
 Medium difficulty to implement - touches more parts of the codebase, higher complexity

  
Documentation
Improvements or additions to documentation

  
Enhancement
New feature or request

  
Good first issue
Good for newcomers

  
Help wanted
Additional eyes and keyboards are required for this one

  
Inherited
Issues that have been inhereted from the project pre-fork

  
Matrix/Administration
Features pertaining to homeserver administration

  
Matrix/Appservices
Features pertaining to the appservice API

  
Matrix/Auth
Features pertaining to authentication

  
Matrix/Client
Features pertaining to client-to-server interactions

  
Matrix/Core
Issues relating to core matrix functionality, such as state resolution and PDU formats

  
Matrix/E2EE
Issues related to end to end encryption

  
Matrix/Federation
Features pertaining to server-to-server interactions

  
Matrix/Hydra
Issues related to room version 12 and related changes (temporary label)

  
Matrix/MSC
Features pertaining to unstable matrix features

  
Matrix/Media
Features pertaining to media interactions

  
Matrix/T&S
Changes or issues related to trust & safety tooling

  
Merge
This PR is ready to be merged

  
Merge/Manual
This PR should be manually merged

  
Merge/Squash
This PR should be squashed when it is merged

  
Meta
Related to housekeeping, maintenance, or other repo-meta.

  
Meta/CI
Issues related to CI changes

  
Meta/Packaging
Packaging

  
Priority
Blocking
 This issue is blocking the next release

  
Priority
High
 This issue is very important

  
Priority
Low
 This issue is of a rather low priority

  
Security
This item is related to general security

  
Status
Confirmed
 This issue has enough information and is confirmed

  
Status
Duplicate
 This issue or pull request already exists

  
Status
Invalid
 This issue doesn't seem right

  
Status
Needs Investigation
 This issue needs further investigation

  
Support
Questions or support requests

  
bob the builder
pdu logic rewrite starring nexus nicholson

No labels
Abandoned
Blocked
Bug
Changelog
Added
Changelog
Missing
Changelog
None
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/E2EE
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Matrix/T&S
Merge
Merge/Manual
Merge/Squash
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
bob the builder
Milestone
Clear milestone
No items
No milestone
26.8.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
nex
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!2164
No description provided.