continuwuation/continuwuity
continuwuation/continuwuity
15
84
Fork 97
Code Issues 127 Pull requests 25 Releases 27 Packages 3 Activity Actions 5

WIP: fix: Strip unsigned over federation, improve PDU size checking performance & accuracy #1586

Draft
nex wants to merge 8 commits from nex/fix/federation-format into main
pull from: nex/fix/federation-format
merge into: continuwuation:main
Conversation 15 Commits 8 Files changed 9 +49 -113
nex commented 2026-03-27 22:15:04 +00:00
Owner
Copy link

This pull request changes the federation event formatter factory to strip the unsigned object entirely, as it is unused over federation and only contains untrusted data.
There's also a performance improvement related to checking the size of incoming events, and a fix that more accurately measures the size of local events taking the above into account.

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
This pull request changes the federation event formatter factory to strip the `unsigned` object entirely, as it is unused over federation and only contains untrusted data. There's also a performance improvement related to checking the size of incoming events, and a fix that more accurately measures the size of local events taking the above into account. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. - [x] I have written a [news fragment][n1] for this PR, if applicable<!--(can be done after hitting open!)-->. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages [n1]: https://towncrier.readthedocs.io/en/stable/tutorial.html#creating-news-fragments
fix: Correctly, explicitly format outgoing events
Some checks failed
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m19s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m3s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
3cec9d0077
Henry-Hiles commented 2026-03-27 22:19:20 +00:00
Contributor
Copy link

So, if synapse did this, or if I run my draupnir on c10y, would I not have the issue with event sizes I do right now?

So, if synapse did this, or if I run my draupnir on c10y, would I not have the issue with event sizes I do right now?
chore: Add news fragment
Some checks failed
Check Changelog / Check for changelog (pull_request_target) Successful in 14s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m2s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 20m52s
Details
d636da06e2
nex commented 2026-03-27 22:21:22 +00:00
Author
Owner
Copy link

It's hard to tell. Preliminary research revealed that Synapse still includes unsigned data over federation for locally-originating events, but it doesn't include as much as we did prior to this pull request. However, as it stands, Synapse will not touch any unsigned data received from remote servers, meaning if a remote server continues to send large amounts of unsigned data over federation to Synapse (such as an outdated continuwuity deployment), Synapse will also parrot that data back (provided it still fits within the PDU size limit).

We're sorta trailblazing here by just dropping non-required fields explicitly rather than just removing parts of them. unsigned is already optional, I was told removing unsigned over federation is probably safe, and there's no reason to include other keys that aren't recognised, so this is more of a two-birds-one-stone fix/enhancement.

It's hard to tell. Preliminary research revealed that Synapse still includes unsigned data over federation *for locally-originating events*, but it doesn't include as much as we did prior to this pull request. However, as it stands, Synapse will not touch any unsigned data received from remote servers, meaning if a remote server continues to send large amounts of unsigned data over federation to Synapse (such as an outdated continuwuity deployment), Synapse will also parrot that data back (provided it still fits within the PDU size limit). We're sorta trailblazing here by just dropping non-required fields explicitly rather than just removing parts of them. `unsigned` is already optional, I was told [removing `unsigned` over federation is *probably* safe](https://matrix.to/#/!nD4Jy1hp0We0VmIM9ubjqWLBX_uV8YlTBBPa3a_v2uk/%24cN2bQmiL1G3L6mtvnCI89_-vq9TnTthr8uGb1S6-GrY?via=nexy7574.co.uk&via=matrix.org&via=element.io), and there's no reason to include other keys that aren't recognised, so this is more of a two-birds-one-stone fix/enhancement.
Henry-Hiles commented 2026-03-27 22:21:36 +00:00
Contributor
Copy link

Actually, i guess id get the same error when running a modbot on c10y because event size is also checked locally

Actually, i guess id get the same error when running a modbot on c10y because event size is also checked locally
nex commented 2026-03-27 22:33:25 +00:00
Author
Owner
Copy link

Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early, since it doesn't evaluate it in quite the same way (i.e. it'll hit a false positive). I'll tackle that at a later date though since I'll need to figure out how to do similar stripped evaluations without potentially harming performance (not something I want to be thinking about at 22:30 on a Friday)

Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early, since it doesn't evaluate it in quite the same way (i.e. it'll hit a false positive). I'll tackle that at a later date though since I'll need to figure out how to do similar stripped evaluations without potentially harming performance (not something I want to be thinking about at 22:30 on a Friday)
Henry-Hiles commented 2026-03-27 22:35:27 +00:00
Contributor
Copy link

@nex wrote in #1586 (comment):

Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early, since it doesn't evaluate it in quite the same way (i.e. it'll hit a false positive). I'll tackle that at a later date though since I'll need to figure out how to do similar stripped evaluations without potentially harming performance (not something I want to be thinking about at 22:30 on a Friday)

Alrighty, no worries, enjoy your friday :)

@nex wrote in https://forgejo.ellis.link/continuwuation/continuwuity/pulls/1586#issuecomment-26937: > Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early, since it doesn't evaluate it in quite the same way (i.e. it'll hit a false positive). I'll tackle that at a later date though since I'll need to figure out how to do similar stripped evaluations without potentially harming performance (not something I want to be thinking about at 22:30 on a Friday) Alrighty, no worries, enjoy your friday :)
Jade approved these changes 2026-03-27 22:41:12 +00:00
Dismissed
style: Fix clippy lint
All checks were successful
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m18s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m1s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 26m6s
Details
b0e3b9eeb5
Aranjedeath approved these changes 2026-03-29 01:49:22 +00:00
Dismissed
fix: Correctly handle size evaluation of local events
All checks were successful
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m21s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m59s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 26m5s
Details
053860e496
 
Also improved the performance of `pdu_fits`
nex commented 2026-03-29 12:44:24 +00:00
Author
Owner
Copy link

Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early

This should now be fixed.

> Technically, creating an event that would brush up against the PDU size limit on continuwuity will currently fail early This should now be fixed.
fix: Only pop unsigned from PDUs
Some checks failed
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m15s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m57s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
127030ac38
nex changed title from fix: Correctly, explicitly format outgoing events to fix: Strip unsigned over federation, improve PDU size checking performance & accuracy 2026-03-29 13:24:24 +00:00
nex commented 2026-03-29 13:25:06 +00:00
Author
Owner
Copy link

PR updated to reflect that only unsigned is popped when fed formatting (stripping anything else would result in content hash verification failures)

PR updated to reflect that only `unsigned` is popped when fed formatting (stripping anything else would result in content hash verification failures)
feat: Reduce verbosity of "remote server couldn't process pdu" warning
Some checks failed
Check Changelog / Check for changelog (pull_request_target) Successful in 19s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m18s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m2s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 22m55s
Details
8b206564aa
style: Run linter
All checks were successful
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m21s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m58s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 26m54s
Details
422db2105c
ginger approved these changes 2026-03-29 14:28:50 +00:00
Jade approved these changes 2026-03-29 14:39:01 +00:00
Aranjedeath reviewed 2026-03-29 15:57:28 +00:00
src/service/rooms/timeline/create.rs
@ -283,1 +265,4 @@
}
// Verify that the *full* PDU isn't over 64KiB.
if !pdu_fits(&pdu_json) {
return Err!(Request(TooLarge("PDU is too long large (exceeds 65535 bytes)")));
Aranjedeath commented 2026-03-29 15:57:28 +00:00
Owner
Copy link

too long large is a great error message

too long large is a great error message
nex commented 2026-03-29 16:04:00 +00:00
Author
Owner
Copy link

I'm going to change this to "PDU is so extravagantly humongous that it doth not fit within its constraints" out of spite /j

I'm going to change this to "PDU is so extravagantly humongous that it doth not fit within its constraints" out of spite /j
Aranjedeath commented 2026-03-29 16:06:43 +00:00
Owner
Copy link

@nex wrote in #1586 (comment):

I'm going to change this to "PDU is so extravagantly humongous that it doth not fit within its constraints" out of spite /j

ok but this is a GREAT and UNIQUE error message that will result in users locating the documentation immediately xD

@nex wrote in https://forgejo.ellis.link/continuwuation/continuwuity/pulls/1586#issuecomment-26988: > I'm going to change this to "PDU is so extravagantly humongous that it doth not fit within its constraints" out of spite /j ok but this is a GREAT and UNIQUE error message that will result in users locating the documentation immediately xD
❤️ 1
Aranjedeath approved these changes 2026-03-29 15:58:04 +00:00
Dismissed
Aranjedeath requested changes 2026-03-29 16:30:56 +00:00
Aranjedeath left a comment
Owner
Copy link

we are pausing on this to take a closer look, I am seeing signature errors as a result of the patch

we are pausing on this to take a closer look, I am seeing signature errors as a result of the patch
nex commented 2026-03-29 21:54:36 +00:00
Author
Owner
Copy link

marking as wip until I fix the thing that blows stuff up

marking as wip until I fix the thing that blows stuff up
nex changed title from fix: Strip unsigned over federation, improve PDU size checking performance & accuracy to WIP: fix: Strip unsigned over federation, improve PDU size checking performance & accuracy 2026-03-29 21:54:40 +00:00
All checks were successful
Check Changelog / Check for changelog (pull_request_target) Successful in 9s
Details
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m21s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m58s
Required
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 26m54s
Required
Details
This pull request has changes conflicting with the target branch.
  • src/service/rooms/timeline/create.rs
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin nex/fix/federation-format:nex/fix/federation-format
git switch nex/fix/federation-format
Sign in to join this conversation.
Reviewers
No reviewers
ginger
Jade
Aranjedeath
Labels
Clear labels   
Blocked

This pull request or issue is currently blocked from being merged/closed

  
Bug

Something isn't working as intended

  
Changelog
Added
Changelog entry added

  
Changelog
Missing
No changelog when one is needed

  
Changelog
None
Changelog is unnecesary for this change

  
Cherry-picking

Commits picked from other conduit projects

  
Database

This requires or includes changes to the database

  
Dependencies

Something dependency related

  
Dependencies/Renovate

Automatic dependency upgrades by Renovate

  
Difficulty
Easy
Low difficulty to implement - touches few parts of the codebase, low complexity

  
Difficulty
Hard
High difficulty to implement - touches many parts of the codebase, high complexity

  
Difficulty
Medium
Medium difficulty to implement - touches more parts of the codebase, higher complexity

  
Documentation

Improvements or additions to documentation

  
Enhancement

New feature or request

  
Good first issue

Good for newcomers

  
Help wanted

Additional eyes and keyboards are required for this one

  
Inherited

Issues that have been inhereted from the project pre-fork

  
Matrix/Administration

Features pertaining to homeserver administration

  
Matrix/Appservices

Features pertaining to the appservice API

  
Matrix/Auth

Features pertaining to authentication

  
Matrix/Client

Features pertaining to client-to-server interactions

  
Matrix/Core

Issues relating to core matrix functionality, such as state resolution and PDU formats

  
Matrix/E2EE

Issues related to end to end encryption

  
Matrix/Federation

Features pertaining to server-to-server interactions

  
Matrix/Hydra

Issues related to room version 12 and related changes (temporary label)

  
Matrix/MSC

Features pertaining to unstable matrix features

  
Matrix/Media

Features pertaining to media interactions

  
Matrix/T&S

Changes or issues related to trust & safety tooling

  
Merge

This PR is ready to be merged

  
Merge/Manual

This PR should be manually merged

  
Merge/Squash

This PR should be squashed when it is merged

  
Meta

Related to housekeeping, maintenance, or other repo-meta.

  
Meta/CI

Issues related to CI changes

  
Meta/Packaging

Packaging

  
Priority
Blocking
This issue is blocking the next release

  
Priority
High
This issue is very important

  
Priority
Low
This issue is of a rather low priority

  
Security

This item is related to general security

  
Status
Confirmed
This issue has enough information and is confirmed

  
Status
Duplicate
This issue or pull request already exists

  
Status
Invalid
This issue doesn't seem right

  
Status
Needs Investigation
This issue needs further investigation

  
Support

Questions or support requests

  
Wont fix

This will not be worked on

  
old/ci/cd

Ci/CD

Archived

  
old/rust

Pull requests that update Rust code

Archived

No labels
Blocked
Bug
Changelog
Added
Changelog
Missing
Changelog
None
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/E2EE
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Matrix/T&S
Merge
Merge/Manual
Merge/Squash
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
Wont fix
old/ci/cd
old/rust
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!1586
No description provided.