feat: Add support for state_after, stop sync token accumulation #1795

Open

ginger wants to merge 9 commits from ginger/kill-sync-tokens into main

pull from: ginger/kill-sync-tokens

merge into: continuwuation:main

continuwuation:main

continuwuation:renovate/ruma-digest

continuwuation:renovate/rust-non-major

continuwuation:renovate/rust-zerover-patch-updates

continuwuation:aranje/illegal-car-mods

continuwuation:nex/perf/get-missing-events

continuwuation:ginger/oauth

continuwuation:nex/feat/admin-api

continuwuation:renovate/serde-saphyr-0.x

continuwuation:renovate/rand_core-0.x

continuwuation:nex/feat/deprecated-room-versions

continuwuation:release/v0.5.9

continuwuation:nex/feat/enable-debug-log-release-builds

continuwuation:nex/feat/room-purging

continuwuation:nex/feat/room-shutdown

continuwuation:ginger/ruma-upstreaming

continuwuation:jade/tls-backends

continuwuation:ginger/email-fixes

continuwuation:jade/changelog-labels

continuwuation:nex/fix/v12-publishing

continuwuation:jade/build-info

continuwuation:jade/purge-sync-tokens

continuwuation:ginger/terms-and-conditions

continuwuation:ginger/remove-sliding-sync-proxy

continuwuation:nex/fix/pusher-association

continuwuation:ginger/email-support

continuwuation:jade/community-guidelines

continuwuation:nex/fix/federation-format

continuwuation:jade/git-deps-updates

continuwuation:jade/changelog-check

continuwuation:jade/rust-1-92

continuwuation:ginger/password-reset

continuwuation:nex/experiment/push-gateway-logs

continuwuation:ginger/msc3575-obliteration

continuwuation:nex/feat/block-busted-rooms

continuwuation:nex/fix/informative-startup-errs

continuwuation:ginger/no-left-room-initial-sync

continuwuation:jade/docker-entrypoint

continuwuation:jade/dehydrated-devices

continuwuation:ginger/complement-fixes

continuwuation:nex/fix/stale-destination-cache

continuwuation:nex/experiment/sync-mutex

continuwuation:tcpipuk/docker-docs

continuwuation:jade/snafu

continuwuation:jade/rand-update

continuwuation:nex/stateres-refactor

continuwuation:ginger/779-in-troubleshooting

continuwuation:jade/liveit-guide

continuwuation:jade/http3

continuwuation:nex/feat/admin-hide-empty-rooms

continuwuation:ginger/oobe

continuwuation:nex/fix/debian-thingy

continuwuation:jade/ldap-admin-check

continuwuation:nex/fix/remote-restricted-joins

continuwuation:nex/feat/msc4406-sender-ignored

continuwuation:jade/deadlock-detection

continuwuation:jade/get-started

continuwuation:jade/docs-guide

continuwuation:ginger/fix-local-invites

continuwuation:nex/fix/tpi

continuwuation:nex/feat/room-deletion

continuwuation:nex/feat/msc4322-media-redaction

continuwuation:ginger/stitched-order

continuwuation:ginger/deps/update-rspress

continuwuation:jade/admin-announce-improvements

continuwuation:ginger/xtask-improvements

continuwuation:jade/improve-admin-config-display

continuwuation:nex/fix/better-stateres-error-logs

continuwuation:jade/sender-timeouts

continuwuation:nex/feat/custom-v12-room-ids

continuwuation:ginger/update-metadata

continuwuation:nex/feat/admin-force-logout

continuwuation:tom/max-perf-docs

continuwuation:nex/fix/invalid-appservice-reg

continuwuation:nex/feat/antispam

continuwuation:nex/feat/account-locking

continuwuation:jade/logging-cleanup

continuwuation:jade/remove-legacy-appservice-auth

continuwuation:nex/fix/key-query

continuwuation:jade/update-prek

continuwuation:nex/fix/room-summaries

continuwuation:ginger/restrict-admin-commands

continuwuation:ginger/enable-console-by-default

continuwuation:jade/tag-fixes

continuwuation:jade/otlp

continuwuation:nex/meta/pull-req-template

continuwuation:nex/fix/fed-invite-compliance

continuwuation:nex/feat/build-commit

continuwuation:nex/feat/join-logging

continuwuation:jade/mailmap-updates

continuwuation:jade/hack-ci-tmp

continuwuation:jade/v12-stable

continuwuation:jade/relations

continuwuation:ginger/database-refactor

continuwuation:jade/fix-ldap-uiaa

continuwuation:nex/fix/validation

continuwuation:ginger/nuke-invalid-msc4133-fields-in-migration

continuwuation:ginger/downgrade-artifact-actions

continuwuation:oddlid/reload-fix

continuwuation:jade/fix-assert

continuwuation:ginger/sync-v3-cleanup

continuwuation:ginger/remove-absolute-action-urls

continuwuation:jade/website

continuwuation:nex/fix/backoff

continuwuation:ginger/fix-mdbook-for-0.5

continuwuation:ginger/no-docker-on-prs

continuwuation:backport/v0.5.0-rc.8-1

continuwuation:nex/fed-improvements

continuwuation:jade/rust-1.90

continuwuation:jade/mirror-dockerhub

continuwuation:jade/clippy-fixes

continuwuation:jade/fix-support

continuwuation:jade/clean-images

continuwuation:jade/wal-compression-type

continuwuation:jade/flake-clone

continuwuation:ginger/upload-rpms-on-schedule

continuwuation:nex/fix/incoming-fetch

continuwuation:nex/fix/upgrade

continuwuation:tom/ci-fedora-rpm

continuwuation:jade/ci-release-fix

continuwuation:jade/rocksdb-10-5

continuwuation:ginger/fix-msc4133-migration

continuwuation:ginger/migrate-busted-tz

continuwuation:hydra/public

continuwuation:nex/feat/manual-extremities

continuwuation:nex/feat/async-media

continuwuation:nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE

continuwuation:nex/feat/better-logging

continuwuation:trigger-ci-so-latest-isnt-on-illegal-car-mods

continuwuation:nex/feat/pins-backfill

continuwuation:jade/tuwunel-2025-06-old

continuwuation:jade/ai-slop-db-docs

continuwuation:nex/fix-create-auth

continuwuation:jade/version-stats

continuwuation:jade/read-receipts

continuwuation:jade/rust-toolchain-no-targets

continuwuation:jade/logging-features

continuwuation:jade/syncv5-typing

continuwuation:jade/msc2815

continuwuation:morguldir/see-eye

continuwuation:jade/css-small-screen

continuwuation:nex/wip-751

continuwuation:tuwunel-rebase

continuwuation:test

continuwuation:oddlid/rename-admin-room-bot

continuwuation:strawberry/nix-ci-stuff

continuwuation:strawberry/valgrind

continuwuation:phonemain

continuwuation:strawberry/morgs-snake-sync-jason-main

continuwuation:newer-media-endpoints

continuwuation:folly-coroutines-async-io

continuwuation:federation-retry-timer-port

continuwuation:bad-attempt-at-extracting-homeserver-signing-key

continuwuation:room-deletion-attempt-do-not-use

Conversation 3 Commits 9 Files changed 11 +200 -318

ginger commented 2026-05-20 19:00:07 +00:00

Owner

Copy link

This pull request adds support for the state_after query parameter introduced on the legacy sync endpoint in Matrix 1.16. Additionally, it adjusts the legacy sync logic to stop adding new entries to the notoriously large roomsynctoken_shortstatehash column, which a future release is expected to drop entirely. Fixes !917.

Pull request checklist:

• This pull request targets the main branch, and the branch is named something other than
  main.
• I have written an appropriate pull request title and my description is clear.
• I understand I am responsible for the contents of this pull request.
• I have followed the contributing guidelines:
  • My contribution follows the code style, if applicable.
  • I ran pre-commit checks before opening/drafting this pull request.
  • I have tested my contribution (or proof-read it for documentation-only changes)
    myself, if applicable. This includes ensuring code compiles.
  • My commit messages follow the commit message format and are descriptive.

<!-- In order to help reviewers know what your pull request does at a glance, you should ensure that 1. Your PR title is a short, single sentence describing what you changed 2. You have described in more detail what you have changed, why you have changed it, what the intended effect is, and why you think this will be beneficial to the project. If you have made any potentially strange/questionable design choices, but didn't feel they'd benefit from code comments, please don't mention them here - after opening your pull request, go to "files changed", and click on the "+" symbol in the line number gutter, and attach comments to the lines that you think would benefit from some clarification. --> This pull request adds support for the `state_after` query parameter introduced on the legacy sync endpoint in Matrix 1.16. Additionally, it adjusts the legacy sync logic to stop adding new entries to the notoriously large `roomsynctoken_shortstatehash` column, which a future release is expected to drop entirely. Fixes !917. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages

👍 1 1 👀 1 ❤️ 1 🎉 1 🚀 2 🥺 1

ginger added the

Bug

Enhancement

Matrix/Client

labels 2026-05-20 19:00:07 +00:00

ginger added 3 commits 2026-05-20 19:00:07 +00:00

feat: Remove all uses of roomsynctoken_shortstatehash 096c9a1768

feat: Add support for state_after 987060dad8

chore: News fragments 3086c24b22

forgejo-actions added the

Changelog

Added

label 2026-05-20 19:00:33 +00:00

Jade commented 2026-05-20 20:33:52 +00:00

Owner

Copy link

Oh hell yeah

Oh hell yeah

ginger added 1 commit 2026-05-21 14:53:28 +00:00

fix: Calculate state at end of last sync correctly b76e0da6e7

ginger added 1 commit 2026-05-21 15:43:45 +00:00

fix: Don't panic on missing SSH 85c44db65c

ginger force-pushed ginger/kill-sync-tokens from 85c44db65c to 47daf80d0f 2026-05-21 16:06:26 +00:00 Compare

ginger added 1 commit 2026-05-21 17:44:16 +00:00

fix: Upgrade warning on room load failures to error 8a47a06baf

Aranjedeath approved these changes 2026-05-22 06:22:35 +00:00

ginger force-pushed ginger/kill-sync-tokens from 8a47a06baf to 9d52582bb1 2026-05-25 17:37:31 +00:00 Compare

nex approved these changes 2026-05-25 18:01:33 +00:00

Dismissed

nex dismissed nex's review 2026-05-25 18:04:28 +00:00

Reason:

erroneous approval

ginger added 1 commit 2026-05-25 21:42:26 +00:00

fix: Additional sync logic fixes 41cb97a11e

ginger added 30 commits 2026-05-26 13:05:15 +00:00

feat: Update policy server implementation to be closer to latest spec ... 9b59c2e676

Untested

chore: Add news fragment

feat: Support stable policy servers

feat: Don't attempt erroneous loopback federation for policy server checks

refactor: Update PS upgrade to use new ruma

fix: Only check loopback via after attempting incoming verification

feat: Support advertising a policy server public key in well-known ... c5d5050f91

# Conflicts:
#	src/api/client/well_known.rs
#	src/core/config/mod.rs

fix: Verify policy server signatures on all events, not just timeline ones ... dd83adc5bf

style: Clarifications

style: Clippy

style: Reformat 6deb1235a9

fix: Treat malformed policy config events as missing c2cc88dbd2

fix: Return Forbidden instead of internal error when PS doesn't sign e127fbd590

feat: Follow spec more closely, code clean up, use ruma request type 0cecf3bbc0

feat: Enable shutdown interrupt in ratelimit handler 7c9cb1937e

fix: Ensure policy server signed with the correct key 88d5cdc5fe

fix: Ensure event_id is removed before policy-checking event f755ea8857

style: Document functions e6dcd5b00f

feat: Add extract_signature helper 460969a94c

perf: Avoid cloning incoming PDUs to check them ... 05aa88d60c

Also allows us to store signatures on PDUs received over federation that we got a fresh signature for

style: Reformat 9023cfb846

fix: Don't return early if the policy server does something stupid ... 48940c06b7

Spec compliance is for nerds I guess

fix: Ensure event_id is correctly stripped before verifying policy server signature e4b8a3da29

style: Fix typo a38e879af2

refactor: Fix several bugs in upgrade endpoint, update MSC4168 impl 4688fb4eef

fix: Adhere to MSC4168 more strongly & in definition order 88e8d646b8

fix: Don't give v12 rooms room IDs f3d03c2ebe

fix: Don't de-power creators when downgrading from v12 to earlier versions 80fd79655b

fix: Include sender in older room versions 566b46eb1a

fix: Correctly update space children on upgrade 27d01907b1

fix: Correctly copy parents and children during upgrade c9c5bfaebd

style: Drop unstable prefix in function definitions dee4266aa6

perf: Skip updating child/parent spaces in upgrade when sender is not joined ceaa806027

style: Combine "unsupported version" checks dc881baf20

chore(deps): update github-actions-digest e081d90b0d

chore(deps): lock file maintenance 42c19e72e0

fix: Don't panic on missing SSH in sliding sync d5bb67b18d

ginger force-pushed ginger/kill-sync-tokens from d5bb67b18d to 624bd3796a 2026-05-26 18:29:33 +00:00 Compare

ginger added 1 commit 2026-05-27 18:14:28 +00:00

fix: Correctly sync newly created rooms 80741eb365

ginger force-pushed ginger/kill-sync-tokens from 80741eb365 to 8182ab3c29 2026-05-27 18:14:41 +00:00 Compare

nex requested review from nex 2026-05-28 12:30:18 +00:00

Some checks failed

Hide all checks

Documentation / Build and Deploy Documentation (pull_request) Failing after 4m16s

Details

Checks / Prek / Check changed files (pull_request) Successful in 7s

Required

Details

Checks / Changelog / Check changelog is added (pull_request_target) Successful in 30s

Required

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 4m57s

Required

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 4m51s

Required

Details

This pull request is blocked because it's outdated.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin ginger/kill-sync-tokens:ginger/kill-sync-tokens

git switch ginger/kill-sync-tokens

Sign in to join this conversation.

Reviewers

No reviewers

Aranjedeath

nex

Labels

Clear labels   

Blocked

This pull request or issue is currently blocked from being merged/closed

  

Bug

Something isn't working as intended

  

Changelog

Added

Changelog entry added

  

Changelog

Missing

No changelog when one is needed

  

Changelog

None

Changelog is unnecesary for this change

  

Cherry-picking

Commits picked from other conduit projects

  

Database

This requires or includes changes to the database

  

Dependencies

Something dependency related

  

Dependencies/Renovate

Automatic dependency upgrades by Renovate

  

Difficulty

Easy

Low difficulty to implement - touches few parts of the codebase, low complexity

  

Difficulty

Hard

High difficulty to implement - touches many parts of the codebase, high complexity

  

Difficulty

Medium

Medium difficulty to implement - touches more parts of the codebase, higher complexity

  

Documentation

Improvements or additions to documentation

  

Enhancement

New feature or request

  

Good first issue

Good for newcomers

  

Help wanted

Additional eyes and keyboards are required for this one

  

Inherited

Issues that have been inhereted from the project pre-fork

  

Matrix/Administration

Features pertaining to homeserver administration

  

Matrix/Appservices

Features pertaining to the appservice API

  

Matrix/Auth

Features pertaining to authentication

  

Matrix/Client

Features pertaining to client-to-server interactions

  

Matrix/Core

Issues relating to core matrix functionality, such as state resolution and PDU formats

  

Matrix/E2EE

Issues related to end to end encryption

  

Matrix/Federation

Features pertaining to server-to-server interactions

  

Matrix/Hydra

Issues related to room version 12 and related changes (temporary label)

  

Matrix/MSC

Features pertaining to unstable matrix features

  

Matrix/Media

Features pertaining to media interactions

  

Matrix/T&S

Changes or issues related to trust & safety tooling

  

Merge

This PR is ready to be merged

  

Merge/Manual

This PR should be manually merged

  

Merge/Squash

This PR should be squashed when it is merged

  

Meta

Related to housekeeping, maintenance, or other repo-meta.

  

Meta/CI

Issues related to CI changes

  

Meta/Packaging

Packaging

  

Priority

Blocking

This issue is blocking the next release

  

Priority

High

This issue is very important

  

Priority

Low

This issue is of a rather low priority

  

Security

This item is related to general security

  

Status

Confirmed

This issue has enough information and is confirmed

  

Status

Duplicate

This issue or pull request already exists

  

Status

Invalid

This issue doesn't seem right

  

Status

Needs Investigation

This issue needs further investigation

  

Support

Questions or support requests

  

Wont fix

This will not be worked on

  

old/ci/cd

Ci/CD

Archived

  

old/rust

Pull requests that update Rust code

Archived

No labels

Blocked

Bug

Changelog

Added

Changelog

Missing

Changelog

None

Cherry-picking

Database

Dependencies

Dependencies/Renovate

Difficulty

Easy

Difficulty

Hard

Difficulty

Medium

Documentation

Enhancement

Good first issue

Help wanted

Inherited

Matrix/Administration

Matrix/Appservices

Matrix/Auth

Matrix/Client

Matrix/Core

Matrix/E2EE

Matrix/Federation

Matrix/Hydra

Matrix/MSC

Matrix/Media

Matrix/T&S

Merge

Merge/Manual

Merge/Squash

Meta

Meta/CI

Meta/Packaging

Priority

Blocking

Priority

High

Priority

Low

Security

Status

Confirmed

Status

Duplicate

Status

Invalid

Status

Needs Investigation

Support

Wont fix

old/ci/cd

old/rust

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

4 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

continuwuation/continuwuity!1795

No description provided.