feat: Support omitting members in the send_join response #977

Merged
nex merged 5 commits from nex/feat/provide-fast-joins into main 2025-09-07 21:01:28 +00:00
Owner

This PR allows us to omit membership events in our /send_join endpoint, allowing us to provide fast/partial joins. Joining a room via a continuwuity server with this change should be significantly faster and weigh less on the server itself.

This does not allow us to use fast/partial joins, that is a feature being kinda worked on in nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE (but as the branch name says do not use it)

This PR allows us to omit membership events in our `/send_join` endpoint, allowing us to *provide* fast/partial joins. Joining a room via a continuwuity server with this change should be significantly faster and weigh less on the server itself. This does not allow us to *use* fast/partial joins, that is a feature being kinda worked on in [nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE](https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE) (but as the branch name says *do not use it*)
@ -238,6 +258,17 @@ async fn create_join_event(
.rooms
.auth_chain
.event_ids_iter(room_id, starting_events)
// .broad_filter_map(|event_id| async {
Author
Owner

This was commented out as for whatever reason, not including events in the auth chain that are already in the state object caused synapse to explode. Still, without this, this is already a large improvement. Remove commented code before merge?

This was commented out as for whatever reason, not including events in the auth chain that are already in the state object caused synapse to explode. Still, without this, this is already a large improvement. Remove commented code before merge?
Author
Owner

just removed it, there's already a huge performance gain without it

just removed it, there's already a huge performance gain without it
nex marked this conversation as resolved
@ -257,0 +296,4 @@
.await;
// If there's no servers, just add us
let servers = if servers.is_empty() {
vec![services.globals.server_name().to_string()]
Author
Owner

This should never happen, but also bees shouldn't be able to fly (this did happen in my testing)

This should never happen, but also bees shouldn't be able to fly (this did happen in my testing)
nex marked this conversation as resolved
nex changed title from WIP: feat: Support omitting members in the send_join response to feat: Support omitting members in the send_join response 2025-09-05 01:37:01 +00:00
nex requested review from Owners 2025-09-05 01:37:15 +00:00
@ -211,3 +215,3 @@
drop(mutex_lock);
debug!("Fetching current state IDs");
Author
Owner

should be trace, it contains no valuable timing information or whatever

should be trace, it contains no valuable timing information or whatever
nex marked this conversation as resolved
@ -219,9 +223,25 @@ async fn create_join_event(
.collect()
.await;
#[allow(clippy::unnecessary_unwrap)]
Author
Owner

This wrap is in fact necessary otherwise the warning turns into an error

This wrap is in fact necessary otherwise the warning turns into an error
Owner

could use expect to have this comment in the code

could use expect to have this comment in the code
nex marked this conversation as resolved
@ -257,0 +286,4 @@
let servers_in_room: Option<Vec<_>> = if !omit_members {
None
} else {
debug!("Fetching list of servers in room");
Author
Owner

Should also be a trace

Should also be a trace
nex marked this conversation as resolved
@ -257,0 +302,4 @@
};
Some(servers)
};
debug!("Returning send_join data");
Author
Owner

Another one for trace

Another one for trace
nex marked this conversation as resolved
nex force-pushed nex/feat/provide-fast-joins from 8bff0179fe
All checks were successful
Release Docker Image / define-variables (push) Successful in 11s
Documentation / Build and Deploy Documentation (pull_request) Successful in 42s
Checks / Prek / Pre-commit & Formatting (push) Successful in 1m6s
Checks / Prek / Clippy and Cargo Tests (push) Successful in 5m24s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (push) Successful in 12m18s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (push) Successful in 12m51s
Release Docker Image / merge (push) Successful in 11s
to a0d373bba9
Some checks failed
Release Docker Image / define-variables (push) Successful in 2s
Documentation / Build and Deploy Documentation (pull_request) Successful in 50s
Checks / Prek / Clippy and Cargo Tests (push) Successful in 14m57s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (push) Successful in 17m32s
Checks / Prek / Pre-commit & Formatting (push) Failing after 55s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (push) Successful in 19m37s
Release Docker Image / merge (push) Successful in 17s
2025-09-07 17:50:58 +00:00
Compare
tcpipuk force-pushed nex/feat/provide-fast-joins from a0d373bba9
Some checks failed
Release Docker Image / define-variables (push) Successful in 2s
Documentation / Build and Deploy Documentation (pull_request) Successful in 50s
Checks / Prek / Clippy and Cargo Tests (push) Successful in 14m57s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (push) Successful in 17m32s
Checks / Prek / Pre-commit & Formatting (push) Failing after 55s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (push) Successful in 19m37s
Release Docker Image / merge (push) Successful in 17s
to 8c0a54e4ba
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Successful in 34s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 57s
Release Docker Image / define-variables (pull_request) Successful in 5s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m7s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (pull_request) Successful in 9m44s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (pull_request) Successful in 7m50s
Release Docker Image / merge (pull_request) Successful in 9s
2025-09-07 18:00:32 +00:00
Compare
Author
Owner

Deploying this one to my main srv now, will report any issues

Deploying this one to my main srv now, will report any issues
nex self-assigned this 2025-09-07 18:19:03 +00:00
nex force-pushed nex/feat/provide-fast-joins from fb6a9a8cb7
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Successful in 39s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 49s
Release Docker Image / define-variables (pull_request) Successful in 2s
Release Docker Image / merge (pull_request) Has been cancelled
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (pull_request) Has been cancelled
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (pull_request) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
to cf7da3a9b4
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Successful in 42s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m13s
Release Docker Image / define-variables (pull_request) Successful in 2s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (pull_request) Successful in 6m12s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (pull_request) Successful in 6m25s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m51s
Release Docker Image / merge (pull_request) Successful in 10s
2025-09-07 19:51:06 +00:00
Compare
Author
Owner

Can confirm this works amazingly now.

Synapse log:

synapse_main | 2025-09-07 20:17:36,864 - synapse.handlers.room_summary - 529 - INFO - GET-7628357 - Requesting summary for !itCOeHerJPucwbZJDz:element.io via ['nexy7574.co.uk']
synapse_main | 2025-09-07 20:17:40,885 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369 - {GET-O-3106645} [nexy7574.co.uk] Completed request: 200 OK in 0.20 secs, got 916 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/make_join/%21itCOeHerJPucwbZJDz%3Aelement.io/%40nex%3Asynapse.nexy7574.co.uk?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11
synapse_main | 2025-09-07 20:17:41,491 - synapse.federation.federation_server - 474 - INFO - PUT-7628373 - Ignoring PDU for unknown room_id: !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:42,251 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369 - {PUT-O-3106646} [nexy7574.co.uk] Completed request: 200 OK in 1.36 secs, got 829942 bytes - PUT matrix-federation://nexy7574.co.uk/_matrix/federation/v2/send_join/%21itCOeHerJPucwbZJDz%3Aelement.io/%24gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs?omit_members=true
synapse_main | 2025-09-07 20:17:55,439 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs - {GET-O-3106658} [nexy7574.co.uk] Completed request: 200 OK in 0.16 secs, got 16276 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/event_auth/%21itCOeHerJPucwbZJDz%3Aelement.io/%24gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs
synapse_main | 2025-09-07 20:17:55,720 - synapse.handlers.sync - 2130 - INFO - GET-7628629 - User membership change between getting rooms and current token: @nex:synapse.nexy7574.co.uk join !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:55,721 - synapse.handlers.federation - 1917 - INFO - sync_partial_state_room-27 - Syncing state for room !itCOeHerJPucwbZJDz:element.io via nexy7574.co.uk
synapse_main | 2025-09-07 20:17:55,730 - synapse.handlers.sync - 2130 - INFO - GET-7628626 - User membership change between getting rooms and current token: @nex:synapse.nexy7574.co.uk join !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:55,732 - synapse.handlers.room_member - 1348 - INFO - POST-7628369 - Transferring room state from !iyIlInqJyxXrRmRHFx:matrix.org to !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:55,733 - synapse.storage.controllers.state - 471 - INFO - GET-7628626 - Failed to find any events in room !itCOeHerJPucwbZJDz:element.io at RoomStreamToken(stream: 2342766, topological: None, instances: {})
synapse_main | 2025-09-07 20:17:55,736 - synapse.storage.controllers.state - 471 - INFO - GET-7628629 - Failed to find any events in room !itCOeHerJPucwbZJDz:element.io at RoomStreamToken(stream: 2342766, topological: None, instances: {})
synapse_main | 2025-09-07 20:17:56,249 - synapse.storage.util.partial_state_events_tracker - 179 - INFO - GET-7628634 - Awaiting un-partial-stating of room !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:56,297 - synapse.storage.util.partial_state_events_tracker - 179 - INFO - GET-7628635 - Awaiting un-partial-stating of room !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:17:56,522 - synapse.http.matrixfederationclient - 364 - INFO - sync_partial_state_room-27-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs-$FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 - {GET-O-3106659} [nexy7574.co.uk] Completed request: 200 OK in 0.78 secs, got 81107 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/state_ids/%21itCOeHerJPucwbZJDz%3Aelement.io?event_id=%24FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08
synapse_main | 2025-09-07 20:17:57,394 - synapse.http.matrixfederationclient - 364 - INFO - sync_partial_state_room-27-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs-$FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 - {GET-O-3106660} [nexy7574.co.uk] Completed request: 200 OK in 0.78 secs, got 1382136 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/state/%21itCOeHerJPucwbZJDz%3Aelement.io?event_id=%24FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08
synapse_main | 2025-09-07 20:18:12,803 - synapse.handlers.federation - 1925 - INFO - sync_partial_state_room-27 - Updating current state for !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:18:14,389 - synapse.handlers.federation - 1939 - INFO - sync_partial_state_room-27 - Clearing partial-state flag for !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:18:14,412 - synapse.handlers.federation - 1943 - INFO - sync_partial_state_room-27 - State resync complete for !itCOeHerJPucwbZJDz:element.io
synapse_main | 2025-09-07 20:18:14,415 - synapse.storage.util.partial_state_events_tracker - 156 - INFO - sync_partial_state_room-27 - Notifying 2 things waiting for un-partial-stating of room !itCOeHerJPucwbZJDz:element.io

Continuwuity log:

2025-09-07T20:17:41.213830Z  INFO create_join_event{room_id="!itCOeHerJPucwbZJDz:element.io" origin="synapse.nexy7574.co.uk"}: conduwuit_api::server::send_join: Sending join event to other servers fast_join=true
2025-09-07T20:17:41.215335Z  INFO conduwuit_api::server::send_join: Finished sending a join for synapse.nexy7574.co.uk in !itCOeHerJPucwbZJDz:element.io in 287.90184ms
Can confirm this works amazingly now. Synapse log: ``` synapse_main | 2025-09-07 20:17:36,864 - synapse.handlers.room_summary - 529 - INFO - GET-7628357 - Requesting summary for !itCOeHerJPucwbZJDz:element.io via ['nexy7574.co.uk'] synapse_main | 2025-09-07 20:17:40,885 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369 - {GET-O-3106645} [nexy7574.co.uk] Completed request: 200 OK in 0.20 secs, got 916 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/make_join/%21itCOeHerJPucwbZJDz%3Aelement.io/%40nex%3Asynapse.nexy7574.co.uk?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11 synapse_main | 2025-09-07 20:17:41,491 - synapse.federation.federation_server - 474 - INFO - PUT-7628373 - Ignoring PDU for unknown room_id: !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:42,251 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369 - {PUT-O-3106646} [nexy7574.co.uk] Completed request: 200 OK in 1.36 secs, got 829942 bytes - PUT matrix-federation://nexy7574.co.uk/_matrix/federation/v2/send_join/%21itCOeHerJPucwbZJDz%3Aelement.io/%24gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs?omit_members=true synapse_main | 2025-09-07 20:17:55,439 - synapse.http.matrixfederationclient - 364 - INFO - POST-7628369-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs - {GET-O-3106658} [nexy7574.co.uk] Completed request: 200 OK in 0.16 secs, got 16276 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/event_auth/%21itCOeHerJPucwbZJDz%3Aelement.io/%24gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs synapse_main | 2025-09-07 20:17:55,720 - synapse.handlers.sync - 2130 - INFO - GET-7628629 - User membership change between getting rooms and current token: @nex:synapse.nexy7574.co.uk join !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:55,721 - synapse.handlers.federation - 1917 - INFO - sync_partial_state_room-27 - Syncing state for room !itCOeHerJPucwbZJDz:element.io via nexy7574.co.uk synapse_main | 2025-09-07 20:17:55,730 - synapse.handlers.sync - 2130 - INFO - GET-7628626 - User membership change between getting rooms and current token: @nex:synapse.nexy7574.co.uk join !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:55,732 - synapse.handlers.room_member - 1348 - INFO - POST-7628369 - Transferring room state from !iyIlInqJyxXrRmRHFx:matrix.org to !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:55,733 - synapse.storage.controllers.state - 471 - INFO - GET-7628626 - Failed to find any events in room !itCOeHerJPucwbZJDz:element.io at RoomStreamToken(stream: 2342766, topological: None, instances: {}) synapse_main | 2025-09-07 20:17:55,736 - synapse.storage.controllers.state - 471 - INFO - GET-7628629 - Failed to find any events in room !itCOeHerJPucwbZJDz:element.io at RoomStreamToken(stream: 2342766, topological: None, instances: {}) synapse_main | 2025-09-07 20:17:56,249 - synapse.storage.util.partial_state_events_tracker - 179 - INFO - GET-7628634 - Awaiting un-partial-stating of room !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:56,297 - synapse.storage.util.partial_state_events_tracker - 179 - INFO - GET-7628635 - Awaiting un-partial-stating of room !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:17:56,522 - synapse.http.matrixfederationclient - 364 - INFO - sync_partial_state_room-27-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs-$FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 - {GET-O-3106659} [nexy7574.co.uk] Completed request: 200 OK in 0.78 secs, got 81107 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/state_ids/%21itCOeHerJPucwbZJDz%3Aelement.io?event_id=%24FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 synapse_main | 2025-09-07 20:17:57,394 - synapse.http.matrixfederationclient - 364 - INFO - sync_partial_state_room-27-$gtV1c044SfLVkRRq3pc7k-yt6pdpXMQQeHWhG-dqjFs-$FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 - {GET-O-3106660} [nexy7574.co.uk] Completed request: 200 OK in 0.78 secs, got 1382136 bytes - GET matrix-federation://nexy7574.co.uk/_matrix/federation/v1/state/%21itCOeHerJPucwbZJDz%3Aelement.io?event_id=%24FGIMBT-KTDggYeCioi86jwQmdAXVNhza80ayamlcR08 synapse_main | 2025-09-07 20:18:12,803 - synapse.handlers.federation - 1925 - INFO - sync_partial_state_room-27 - Updating current state for !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:18:14,389 - synapse.handlers.federation - 1939 - INFO - sync_partial_state_room-27 - Clearing partial-state flag for !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:18:14,412 - synapse.handlers.federation - 1943 - INFO - sync_partial_state_room-27 - State resync complete for !itCOeHerJPucwbZJDz:element.io synapse_main | 2025-09-07 20:18:14,415 - synapse.storage.util.partial_state_events_tracker - 156 - INFO - sync_partial_state_room-27 - Notifying 2 things waiting for un-partial-stating of room !itCOeHerJPucwbZJDz:element.io ``` Continuwuity log: ``` 2025-09-07T20:17:41.213830Z INFO create_join_event{room_id="!itCOeHerJPucwbZJDz:element.io" origin="synapse.nexy7574.co.uk"}: conduwuit_api::server::send_join: Sending join event to other servers fast_join=true 2025-09-07T20:17:41.215335Z INFO conduwuit_api::server::send_join: Finished sending a join for synapse.nexy7574.co.uk in !itCOeHerJPucwbZJDz:element.io in 287.90184ms ```
Jade approved these changes 2025-09-07 20:52:00 +00:00
nex force-pushed nex/feat/provide-fast-joins from cf7da3a9b4
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Successful in 42s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m13s
Release Docker Image / define-variables (pull_request) Successful in 2s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (pull_request) Successful in 6m12s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (pull_request) Successful in 6m25s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m51s
Release Docker Image / merge (pull_request) Successful in 10s
to 09de586dc7
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Successful in 41s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 49s
Release Docker Image / define-variables (pull_request) Successful in 7s
Documentation / Build and Deploy Documentation (push) Successful in 41s
Checks / Prek / Pre-commit & Formatting (push) Successful in 1m13s
Release Docker Image / define-variables (push) Successful in 1s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 6m1s
Checks / Prek / Clippy and Cargo Tests (push) Successful in 5m43s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (pull_request) Successful in 7m45s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (pull_request) Successful in 8m23s
Release Docker Image / merge (pull_request) Successful in 55s
Release Docker Image / build-image (linux/amd64, release, linux-amd64, base) (push) Successful in 7m53s
Release Docker Image / build-image (linux/arm64, release, linux-arm64, base) (push) Successful in 6m58s
Release Docker Image / merge (push) Successful in 9s
2025-09-07 21:01:11 +00:00
Compare
nex merged commit 09de586dc7 into main 2025-09-07 21:01:28 +00:00
nex deleted branch nex/feat/provide-fast-joins 2025-09-07 21:01:28 +00:00
nex added this to the v0.5.0-rc.8 milestone 2025-09-21 17:12:28 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!977
No description provided.