Legacy sync cleanup, removal of legacy SS #1132
Labels
No labels
Bug
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status/Blocked
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
To-Merge
Wont fix
old/ci/cd
old/rust
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
continuwuation/continuwuity!1132
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ginger/sync-v3-cleanup"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes !361, fixes !839, generally improves performance and fixes weird edge cases in clients. Please test this PR out if you can!
state8cc0c8a6838cfd0f56bfThis breaks SSS right now -- initial syncs load the timeline from some random point months back in the room's history.
SSS is working again yippee
Not sure how you feel about incorporating #842/files ? There isn't too much code, although it does rely on
continuwuation/ruwuma@a3a59ebdfd@Jade wrote in #1132 (comment):
Incorporate in what sense?
WIP: Sync v3 cleanupto WIP: Legacy sync cleanup, removal of legacy SSMerging it, basically (given it conflicts with the refactoring here)
@Jade wrote in #1132 (comment):
I can do that yeah
@ -0,0 +76,4 @@a token which identifies the state of the room at a point in time.2. `load_timeline` is called to fetch timeline events that happened since `since`.3.*/i forgor 💀
limitedtotruefor newly joined rooms again@ginger wrote in #1132 (comment):
This is now fixed, but clients sometimes get confused when leaving a room -- there are situations where we can send a
joinmembership event for the user instatebut aleavemembership event intimeline, which is correct behavior becausestateshould only go up to the start oftimeline, but Cinny (at least) gets confused and doesn't remove the room from its UI.Additionally, if an invite is cancelled by the user that sent it the room will still appear as
inviteduntil an initial sync is performed.@ginger wrote in #1132 (comment):
I don't think there's anything we can do about this -- the S2S spec doesn't seem to include a way to retract invites over federation.
520705163525211ec56dprepare_lazily_loaded_membersfor joined rooms2a0beac32ba6e92eff11WIP: Legacy sync cleanup, removal of legacy SSto Legacy sync cleanup, removal of legacy SSpretty solid honestly, been running for most of the week and haven't run into any issues myself. Just a couple things and I think it should be good to go really.
@ -1199,0 +1202,4 @@# The default of 10 is reasonable for most use cases.##incremental_sync_max_timeline_size = 10Why 10? The specced limit is 100 and most clients take 20-50 at a time, 10 seems quite low especially for high-traffic rooms (such as the ping room)
10 was the old hard-coded value and I assumed the Ancient Maintainers knew what they were doing when they selected it 🧌 what do you think a more reasonable value would be?
@ -480,0 +480,4 @@// TODO: this call does not appear to do anything because `update_membership`// doesn't call `mark_as_knock`. investigate further, ideally with the aim of// removing this call entirely -- Ginger thinks `update_membership` should only// be called from `force_state` and `append_pdu`.federated knocks and invites don't give you real PDUs so I'm not entirely sure the named functions can be used
mark_as_invitedandmark_as_knockedboth (iirc) take only the stripped state as an argument.update_membershipwas previously heavily overloaded to handle all of that in one function, hence me attempting to only use it in the two places where the caches need to be updated from an arbitrary PDU.@ -0,0 +184,4 @@.into();let is_encrypted_room = services.roomsIt's worth noting that this only happens when every membership for the user has been state reset out of the room entirely, as if they were never in it in the first place. This should be incredibly rare, but [M]
@ -0,0 +201,4 @@timeline_start_shortstatehash,is_encrypted_room,).await;nitpick: This is one hell of a statement, could it be made more readable? for maintainability's sakes
@ -0,0 +262,4 @@origin_server_ts: utils::millis_since_unix_epoch().try_into().expect("Timestamp is valid js_int value"),kind: TimelineEventType::RoomMember,Any reason the event ID can't be calculated? We know from the policy server bugs that calculating on a partial PDU is totally viable
The PDU we're returning on this code path doesn't actually exist in the room's DAG or our database, so we have to make up an ID. This should only happen if the room was banned or the user left it before this PR was merged (so there's no cached leave state). I'm going to investigate trawling the database to fix up old leaves as part of a migration, but there's no easy way around it for banned rooms.
Might be worth omitting offline users in the initial sync since
a. most people dont care about offline uesrs
b. I think offline is the default state?
Initial sync already sends pretty few users with LL enabled, I doubt this would be much of a performance win.
@ -445,1 +440,3 @@.and_then(|val: Raw<Vec<AnyStrippedStateEvent>>| val.deserialize_as().map_err(Into::into))// old databases may have garbage data as values in the `userroomid_leftstate` table from before// the leave event was stored there. they still need to be included, so we return Ok(None) for deserialization failures..unwrap_or(None)no migration instead?
Oh also clippy nightly needs running (see:
Checks / Prek / Clippy and Cargo Testsfailing)marking as blocking since this is one of the few remaining major changes before a release can be considered
@ -0,0 +451,4 @@}lazily_loaded_members}Clippy wants you to skip the assignment here, although I think the variable name makes it clearer, so feel free to
#[allow(clippy::let-and-return)]userroomid_leftstatetableLegacy sync cleanup, removal of legacy SSto WIP: Legacy sync cleanup, removal of legacy SSre-WIPed because I'm working on a refactor but it's busted rn
stale
Re-request review from me when you've unbusted it 🫶
load_joined_roominto smaller functions623f138f6e0f36806b19WIP: Legacy sync cleanup, removal of legacy SSto Legacy sync cleanup, removal of legacy SSshould be ready for review again, I'm going to avoid making any more big changes because this diff is already huge and if I continue tweaking shit it'll never get merged
TODO review (forgejo won't let me re-request my own review)
@ginger wrote in #1132 (comment):
dw about the diff too much, the important thing is that the commits make sense so that a blame doesn't follow back to one gigantic commit saying "merged the thing". More than happy to review gigantic diffs lol
d29ad99f8d779be80928779be809286925b9380d2c6ac4bf7079dc97390079dc97390051aa173a6ea6e5dfcd505a3446641c5a3446641cb16a1615d5b16a1615d51a2547541c1a2547541c38aa83307738aa833077bfa03b7c82bfa03b7c8215196ad24915196ad249b70d6487d5b70d6487d517346ff77817346ff778c376fce725View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.