refactor: Don't embed unsigned in storage #2270
No reviewers
Labels
No labels
Abandoned
Blocked
Bug
Changelog
Added
Changelog
Missing
Changelog
None
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/E2EE
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Matrix/T&S
Merge
Merge/Manual
Merge/Squash
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
bob the builder
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
continuwuation/continuwuity!2270
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "nex/feat/unsigned"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Right now our unsigned storage is really inefficient: we store a full copy of the replaced state event's content (which also means we don't apply visibility checks!), we store a full PDU in
redacted_because, and only striptransaction_idandageover federation.This PR refactors unsigned handling, so that:
prev_contentandredacted_becausenow fetch their content on-demand, rather than embedding it - this also allows for access control to be applied correctly.membershipis finally added to unsigned events in the c2s api.This will greatly reduce storage costs, improve permissions, and reduce wasted network IO.
Fixes #1103
Closes #569
Supersedes #1586
Pull request checklist:
mainbranch, and the branch is named something other thanmain.myself, if applicable. This includes ensuring code compiles.
redacted_because_idinstead ofredacted_becausePdu.set_unsigned395d7b3564unsignedinPDUafterset_unsigned97266cf933set_unsignedto panicWIP: refactor: Don't embedto refactor: Don't embedunsignedin storageunsignedin storage@ -18,0 +28,4 @@/// This function panics if any value cannot be serialised, which should not/// happen provided `membership`, `prev_content`, and `redacted_because` are/// well-formed.pub fn set_unsigned(Note that because this function is purely additive, a migration isn't required. Conflicting fields will be overwritten with contextual data as required. The downside to not having a migration is we've got potentially millions of bloated PDUs laying around with now redundant data. Not sure if we fancy a proper migration for this.
populate_unsigned@ -18,0 +27,4 @@////// This function panics if any value cannot be serialised, which should not/// happen provided `membership`, `prev_content`, and `redacted_because` are/// well-formed.This should just take an
Option<UnsignedContext>parameter, since you're fetching the context and then destructuring it at almost every callsite.@ -24,1 +54,3 @@use BTreeMap as Map;if let Some(prev_content) = prev_content {unsigned.insert("prev_content".to_owned(), to_raw_value(&prev_content)?);// TODO(nex): prev_sender is still inserted in append.rs becauseoh look we had the same thought :3
i guess something is missing src/core/matrix/event/redact.rs this still looks for redacted_because, but redact() writes org.continuwuity.redacted_by now.
@ -25,0 +60,4 @@// proper solution, especially since every callsite has one}if let Some(redacted_because) = redacted_because {unsigned.remove("org.continuwuity.redacted_by");worth moving this remove outside the if, if the get_pdu fails the client ends up with the internal key and no redacted_because, and won't know the event was redacted
Clients seeing the redacted_by is fine considering it's a namespaced key that nobody reads and is primarily for internal reference, it's not really a problem if it's sent to clients
@ -328,0 +372,4 @@.user_can_see_event(sender_user,&event.room_id_or_hash(),event.event_id(),should be &event_id no?
prev_senderunsigned field 144cc36f29b89778da7b05d6922351