refactor: Don't embed unsigned in storage #2270

Merged
nex merged 13 commits from nex/feat/unsigned into main 2026-09-27 18:42:52 +00:00
Owner

Right now our unsigned storage is really inefficient: we store a full copy of the replaced state event's content (which also means we don't apply visibility checks!), we store a full PDU in redacted_because, and only strip transaction_id and age over federation.

This PR refactors unsigned handling, so that:

  1. Unsigned is never sent over federation (saves bandwidth since all servers already remove the incoming object).
  2. prev_content and redacted_because now fetch their content on-demand, rather than embedding it - this also allows for access control to be applied correctly.
  3. membership is finally added to unsigned events in the c2s api.

This will greatly reduce storage costs, improve permissions, and reduce wasted network IO.

Fixes #1103
Closes #569
Supersedes #1586

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
Right now our unsigned storage is really inefficient: we store a full copy of the replaced state event's content (which also means we don't apply visibility checks!), we store a *full* PDU in `redacted_because`, and *only* strip `transaction_id` and `age` over federation. This PR refactors unsigned handling, so that: 1. Unsigned is never sent over federation (saves bandwidth since all servers already remove the incoming object). 2. `prev_content` and `redacted_because` now fetch their content on-demand, rather than embedding it - this also allows for access control to be applied correctly. 3. `membership` is finally added to unsigned events in the c2s api. This will greatly reduce storage costs, improve permissions, and reduce wasted network IO. Fixes #1103 Closes #569 Supersedes #1586 **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages
nex self-assigned this 2026-09-20 23:45:00 +00:00
fix: Record redacted_because_id instead of redacted_because
Some checks failed
Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 3s
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m31s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m44s
30f441af5d
Renamed `redacted_because_id` to `org.continuwuity.redacted_by` for proper namespacing; `get_unsigned_context` now lives in helpers.rs instead of `mod.rs`. The helper function also now explicitly wipes `unsigned` for `redacted_because` to avoid leaks (see comment).
fix: Don't allow set_unsigned to panic
Some checks failed
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m23s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m30s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
224a255994
While an error in this function is probably bad, detonating a metric tonne of TNT on the call tree is probably worse
style: English is a difficult language
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m31s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m43s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m12s
9c43928736
nex changed title from WIP: refactor: Don't embed unsigned in storage to refactor: Don't embed unsigned in storage 2026-09-21 02:20:45 +00:00
nex requested review from Owners 2026-09-21 02:21:24 +00:00
@ -18,0 +28,4 @@
/// This function panics if any value cannot be serialised, which should not
/// happen provided `membership`, `prev_content`, and `redacted_because` are
/// well-formed.
pub fn set_unsigned(
Author
Owner

Note that because this function is purely additive, a migration isn't required. Conflicting fields will be overwritten with contextual data as required. The downside to not having a migration is we've got potentially millions of bloated PDUs laying around with now redundant data. Not sure if we fancy a proper migration for this.

Note that because this function is purely additive, a migration isn't required. Conflicting fields will be overwritten with contextual data as required. The downside to not having a migration is we've got potentially millions of bloated PDUs laying around with now redundant data. Not sure if we fancy a proper migration for this.
nex marked this conversation as resolved
fix: Don't unnecessarily embed prev_content in populate_unsigned
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 12s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m25s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m42s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m4s
ba32fbb9ab
ginger requested changes 2026-09-21 17:05:58 +00:00
Dismissed
@ -18,0 +27,4 @@
///
/// This function panics if any value cannot be serialised, which should not
/// happen provided `membership`, `prev_content`, and `redacted_because` are
/// well-formed.
Owner

This should just take an Option<UnsignedContext> parameter, since you're fetching the context and then destructuring it at almost every callsite.

This should just take an `Option<UnsignedContext>` parameter, since you're fetching the context and then destructuring it at almost every callsite.
nex marked this conversation as resolved
@ -24,1 +54,3 @@
use BTreeMap as Map;
if let Some(prev_content) = prev_content {
unsigned.insert("prev_content".to_owned(), to_raw_value(&prev_content)?);
// TODO(nex): prev_sender is still inserted in append.rs because
Owner

oh look we had the same thought :3

oh look we had the same thought :3
nex marked this conversation as resolved
Contributor

i guess something is missing src/core/matrix/event/redact.rs this still looks for redacted_because, but redact() writes org.continuwuity.redacted_by now.

i guess something is missing src/core/matrix/event/redact.rs this still looks for redacted_because, but redact() writes org.continuwuity.redacted_by now.
@ -25,0 +60,4 @@
// proper solution, especially since every callsite has one
}
if let Some(redacted_because) = redacted_because {
unsigned.remove("org.continuwuity.redacted_by");
Contributor

worth moving this remove outside the if, if the get_pdu fails the client ends up with the internal key and no redacted_because, and won't know the event was redacted

worth moving this remove outside the if, if the get_pdu fails the client ends up with the internal key and no redacted_because, and won't know the event was redacted
Author
Owner

Clients seeing the redacted_by is fine considering it's a namespaced key that nobody reads and is primarily for internal reference, it's not really a problem if it's sent to clients

Clients seeing the redacted_by is fine considering it's a namespaced key that nobody reads and is primarily for internal reference, it's not really a problem if it's sent to clients
nex marked this conversation as resolved
@ -328,0 +372,4 @@
.user_can_see_event(
sender_user,
&event.room_id_or_hash(),
event.event_id(),
Contributor

should be &event_id no?

should be &event_id no?
nex marked this conversation as resolved
style: Take an optional context instead of deconstructed parameters
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m55s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
b89778da7b
nex requested review from ginger 2026-09-27 16:25:03 +00:00
nex force-pushed nex/feat/unsigned from b89778da7b
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m55s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
to 05d6922351
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 40s
Checks / Prek / Check changed files (pull_request) Successful in 10s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m4s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m9s
Update flake hashes / update-flake-hashes (pull_request) Successful in 2m45s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m30s
2026-09-27 16:33:39 +00:00
Compare
feat: Add a migration to fix up unsigned data
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Failing after 6s
Checks / Prek / Check changed files (pull_request) Successful in 8s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m6s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m48s
a61026912c
ginger approved these changes 2026-09-27 18:18:36 +00:00
chore: Add news frag
All checks were successful
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m11s
Documentation / Build and Deploy Documentation (pull_request) Successful in 1m14s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m49s
Checks / Prek / Check changed files (push) Successful in 6s
Documentation / Build and Deploy Documentation (push) Successful in 1m29s
Checks / Prek / Pre-commit & Formatting (push) Successful in 1m43s
Checks / Prek / Clippy and Cargo Tests (push) Successful in 10m14s
Release Docker Image / Build linux-arm64 (release) (push) Successful in 12m23s
Release Docker Image / Build linux-arm64 (max-perf) (push) Has been skipped
Release Docker Image / Build linux-amd64 (release) (push) Successful in 14m15s
Release Docker Image / Create Multi-arch Release Manifest (push) Successful in 15s
Release Docker Image / Build linux-amd64 (max-perf) (push) Successful in 35m25s
Release Docker Image / Create Max-Perf Manifest (push) Has been skipped
Release Docker Image / Mirror Images (push) Has been skipped
Release Docker Image / Release Binaries (push) Has been skipped
451c44800b
nex merged commit 451c44800b into main 2026-09-27 18:42:52 +00:00
nex deleted branch nex/feat/unsigned 2026-09-27 18:42:52 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!2270
No description provided.