perf: msc3902 partial joins and memoization for total joins #1537

Closed

gamesguru wants to merge 1 commit from gamesguru/continuwuity:guru/feat/impl/msc3902-partial-joins-and-caching-optimizations into main

gamesguru commented

2026-03-12 18:32:24 +00:00

Contributor

I advise no one test this out yet. Just something I scrapped together on my lunch break in response to #1503.
Likely needs further work.

commit msg log:

perf: msc3902 partial joins and memoization for total joins
impl partial join worker. batch server discovery and PDU queries
enable complement partial join tests

memoize full join auth chains (very basic moka LRU impl)

unit tests; refactor server query func for testability
enable `fast_joins=true` by default in config

fix lint with `url_preview` feature. TODO: suggest partial lints to avoid differential results.

Pull request checklist:

This pull request targets the main branch, and the branch is named something other than
main.
I have written an appropriate pull request title and my description is clear.
I understand I am responsible for the contents of this pull request.
I have followed the contributing guidelines:
- My contribution follows the code style, if applicable.
- I ran pre-commit checks before opening/drafting this pull request.
- I have tested my contribution (or proof-read it for documentation-only changes)
  myself, if applicable. This includes ensuring code compiles.
- My commit messages follow the commit message format and are descriptive.
- I have written a news fragment for this PR, if applicable.

I advise no one test this out yet. Just something I scrapped together on my lunch break in response to #1503. Likely needs further work. ------------------------------------------------ **commit msg log:** ``` perf: msc3902 partial joins and memoization for total joins impl partial join worker. batch server discovery and PDU queries enable complement partial join tests memoize full join auth chains (very basic moka LRU impl) unit tests; refactor server query func for testability enable `fast_joins=true` by default in config fix lint with `url_preview` feature. TODO: suggest partial lints to avoid differential results. ``` **Pull request checklist:**  - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [ ] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. - [ ] I have written a [news fragment][n1] for this PR, if applicable.  [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages [n1]: https://towncrier.readthedocs.io/en/stable/tutorial.html#creating-news-fragments

gamesguru added 1 commit

2026-03-12 18:32:24 +00:00

perf: msc3902 partial joins and memoization for total joins

Documentation / Build and Deploy Documentation (pull_request) Has been cancelled

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

Update flake hashes / update-flake-hashes (pull_request) Has been cancelled

Details

443267ff74

I advise no one test this out yet. Just something I scrapped together on my lunch break in response to #1503.
Likely needs further work.

------------------------------------------------

**commit msg log:**

```
perf: msc3902 partial joins and memoization for total joins
impl partial join worker. batch server discovery and PDU queries
enable complement partial join tests

memoize full join auth chains (very basic moka LRU impl)

unit tests; refactor server query func for testability
enable `fast_joins=true` by default in config

fix lint with `url_preview` feature. TODO: suggest partial lints to avoid differential results.
```

nex commented

2026-03-13 11:45:28 +00:00

Owner

I have said repeatedly we are not implementing partial join support as it leads to inconsistencies and a terrible user experience, especially when it (very commonly) goes wrong. I recommend instead working with the Matrix spec team to improve the spec for joins so that these problems are fixed at the source.

I have said repeatedly we are not implementing partial join support as it leads to inconsistencies and a terrible user experience, *especially* when it (very commonly) goes wrong. I recommend instead working with the Matrix spec team to improve the spec for joins so that these problems are fixed at the source.

nex closed this pull request

2026-03-13 11:45:29 +00:00

gamesguru commented

2026-03-15 03:46:15 +00:00

Author

Contributor

I think it leads to a terrible developer experience. Users are already affected by the ecosystem rot, as Synapse defaults to partial joins and so it's everywhere.

Regardless, there are some other basic speedups here for full joins.

I don't think implementing a ZK-SNARK msc proposal is going to happen any time soon, so doing a rigorous partial join implementation is probably the more reasonable short-term goal.

I think it leads to a terrible developer experience. Users are already affected by the ecosystem rot, as Synapse defaults to partial joins and so it's everywhere. Regardless, there are some other basic speedups here for full joins. I don't think implementing a ZK-SNARK msc proposal is going to happen any time soon, so doing a rigorous partial join implementation is probably the more reasonable short-term goal.

nex commented

2026-03-15 04:08:57 +00:00

Owner

Synapse can reap what they sow, and recently have been doing so more and more often - I've had to patch out partial joins on my Synapse deployments because they're so horribly broken. There's no way you can make this resilient and reliable without looping back to square one and having joins as slow as non-partial joins. Genuinely, writing a proposal that doesn't involve using room inconsistency as an "optimisation" will be the faster route here.

Feel free to open another PR or re-push to this branch with just the other optimisations if you want, but MSC3902 support isn't going to be accepted due to aforementioned fundamental flaws.

nex commented

2026-03-15 04:10:48 +00:00

Owner

Also for context - the only reason we support providing partial joins is because it's a bit of a bandwidth saver in the sense that it distributes the payload over multiple requests rather than uploading the entire state in one big response. We've got no incentive to support utilising them.

Also for context - the only reason we support *providing* partial joins is because it's a bit of a bandwidth saver in the sense that it distributes the payload over multiple requests rather than uploading the entire state in one big response. We've got no incentive to support utilising them.