continuwuation/continuwuity
continuwuation/continuwuity
Watch
18
115
Fork
You've already forked continuwuity
 132
Code Issues 137 Pull requests 44 Releases 42 Packages 3 Activity Actions

feat: Add timestamp to event endpoint backed by a timestamp index #2332

Open
eleboucher wants to merge 3 commits from eleboucher/continuwuity:feat/timestamp-to-event-index into main
pull from: eleboucher/continuwuity:feat/timestamp-to-event-index
merge into: continuwuation:main
Conversation 1 Commits 3 Files changed 10 +256 -12
eleboucher commented 2026-10-06 18:21:09 +00:00
Contributor
Copy link

Brings back /timestamp_to_event (MSC3030) using a new per-room timestamp index (roomts_pduid), so each lookup is a single database seek instead of a scan, remote events are indexed by when we received them, local and backfilled events by origin_server_ts, and older events are indexed by a background migration.

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
<!-- In order to help reviewers know what your pull request does at a glance, you should ensure that 1. Your PR title is a short, single sentence describing what you changed 2. You have described in more detail what you have changed, why you have changed it, what the intended effect is, and why you think this will be beneficial to the project. If you have made any potentially strange/questionable design choices, but didn't feel they'd benefit from code comments, please don't mention them here - after opening your pull request, go to "files changed", and click on the "+" symbol in the line number gutter, and attach comments to the lines that you think would benefit from some clarification. --> Brings back /timestamp_to_event (MSC3030) using a new per-room timestamp index (roomts_pduid), so each lookup is a single database seek instead of a scan, remote events are indexed by when we received them, local and backfilled events by origin_server_ts, and older events are indexed by a background migration. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages
🚀 1
feat: Add timestamp to event endpoint backed by a timestamp index
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Auto Labeler / Apply labels based on changed files (pull_request_target) Successful in 3s
Details
Checks / Prek / Check changed files (pull_request) Successful in 16s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 6m18s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 10m7s
Details
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Details
c181184b0a
nex reviewed 2026-10-06 23:43:45 +00:00
nex left a comment
Copy link

conceptually lgtm, just a couple style comments and also that compliance issue I'd like input on

conceptually lgtm, just a couple style comments and also that compliance issue I'd like input on
src/service/migrations.rs
Lines 978-980
@ -942,6 +975,38 @@ async fn unembed_unsigned_info(
info!(elapsed=?start.elapsed(), total_migrated_pdus=total_migrated, "Finished migration.");
}
const ROOMTS_UNTIL: &str = "populate_roomts_pduid_until";
async fn populate_roomts_pduid(db: Arc<Database>, until: u64) {
nex commented 2026-10-06 23:37:52 +00:00
Owner
Copy link

nit: this probably wants to go below drop_server_signingkeys

nit: this probably wants to go below `drop_server_signingkeys`
src/service/rooms/timeline/append.rs Outdated
Lines 401-402
@ -398,9 +398,17 @@ impl super::Service {
let count2 = PduCount::Normal(self.services.globals.next_count().unwrap());
let pdu_id: RawPduId = PduId { shortroomid, shorteventid: count2 }.into();
// Remote timestamps are untrusted, so index remote events by when we
// received them
nex commented 2026-10-06 23:40:30 +00:00
Owner
Copy link

I'd like to get the team's thoughts on this: personally I don't fw the idea of disregarding the spec (which explicitly states that this functionality is based on origin_server_ts) just because that field can be lied about. After some thought, I don't personally see "malicious server lies about its own timestamp" to be a problem that actually matters.

cc @Jade @ginger

I'd like to get the team's thoughts on this: personally I don't fw the idea of disregarding the spec (which explicitly states that this functionality is based on `origin_server_ts`) just because that field can be lied about. After some thought, I don't personally see "malicious server lies about its own timestamp" to be a problem that actually matters. cc @Jade @ginger
Jade commented 2026-10-06 23:56:11 +00:00
Owner
Copy link

In this case it's not because of a security issue, but because our timeline is ordered based on the timestamp that is received and this endpoint seems to be primarily used for jump to date - meaning that jumping to the closest origin_server_ts may jump to a completely different place in the timeline to what was actually intended

In this case it's not because of a security issue, but because our timeline is ordered based on the timestamp that is received and this endpoint seems to be primarily used for jump to date - meaning that jumping to the closest `origin_server_ts` may jump to a completely different place in the timeline to what was actually intended
nex commented 2026-10-06 23:58:55 +00:00
Owner
Copy link

Hmm, I guess that usually makes sense, but what about if a server is offline for a month (e.g. due to a VPS provider outage) and as such it receives that month's events late? The user will have to account for an extra month's offset, which isn't very intuitive, and is contrary to what their client will end up actually displaying (the origin time)

Hmm, I guess that usually makes sense, but what about if a server is offline for a month (e.g. due to a VPS provider outage) and as such it receives that month's events late? The user will have to account for an extra month's offset, which isn't very intuitive, and is contrary to what their client will end up actually displaying (the origin time)
Jade commented 2026-10-07 00:00:52 +00:00
Owner
Copy link

That's a good (annoying) point. I don't have an answer right now, maybe once I've not got a headache

That's a good (annoying) point. I don't have an answer right now, maybe once I've not got a headache
nex commented 2026-10-07 00:01:46 +00:00
Owner
Copy link

ofc, no rush. Maybe we could compromise and trust the origin timestamp if it's within, say, 12 hours of our local time? I originally went to say 1hr, but that might not account for clock skew

ofc, no rush. Maybe we could compromise and trust the origin timestamp if it's within, say, 12 hours of our local time? I originally went to say 1hr, but that might not account for clock skew
nex commented 2026-10-07 00:02:14 +00:00
Owner
Copy link

Actually no, that doesn't solve my problem. Perhaps the inverse? Or maybe I should just think about this after a night's rest lmao

Actually no, that doesn't solve my problem. Perhaps the inverse? Or maybe I should just think about this after a night's rest lmao
Jade commented 2026-10-07 00:03:45 +00:00
Owner
Copy link

Clock skew shouldn't be more than like 30 seconds, but that doesn't fix the issue you raised. I was gonna say something like trusting it if it's not before a previous event timestamp we received? Which would work as long as we get events in order?

Clock skew shouldn't be more than like 30 seconds, but that doesn't fix the issue you raised. I was gonna say something like trusting it if it's not before a previous event timestamp we received? Which would work as long as we get events in order?
src/service/rooms/timeline/data.rs Outdated
@ -269,6 +279,48 @@ impl Data {
.try_flatten_stream()
}
pub(super) fn pdus_by_ts<'a>(
nex commented 2026-10-06 23:41:32 +00:00
Owner
Copy link

nit: pdus_by_timestamp is more intuitive

nit: `pdus_by_timestamp` is more intuitive
src/service/rooms/timeline/data.rs Outdated
@ -272,0 +282,4 @@
pub(super) fn pdus_by_ts<'a>(
&'a self,
room_id: &'a RoomId,
ts: MilliSecondsSinceUnixEpoch,
nex commented 2026-10-06 23:41:47 +00:00
Owner
Copy link

nit: this could also just be timestamp

nit: this could also just be `timestamp`
src/service/rooms/timeline/mod.rs Outdated
@ -255,0 +258,4 @@
pub fn pdus_by_timestamp<'a>(
&'a self,
room_id: &'a RoomId,
ts: MilliSecondsSinceUnixEpoch,
nex commented 2026-10-06 23:43:23 +00:00
Owner
Copy link

nit: could be timestamp

nit: could be `timestamp`
fix: Index remote events by origin_server_ts
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Details
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 56s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 40m25s
Details
8ab0af0bf9
nex requested review from nex 2026-10-17 02:16:12 +00:00
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Check changed files (pull_request) Successful in 6s
Required
Details
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Required
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 56s
Required
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 40m25s
Required
Details
This pull request has changes conflicting with the target branch.
  • src/api/client/room/mod.rs
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u feat/timestamp-to-event-index:eleboucher-feat/timestamp-to-event-index
git switch eleboucher-feat/timestamp-to-event-index
Sign in to join this conversation.
Reviewers
No reviewers
nex
Labels
Clear labels   
Abandoned
This pull request appears to be abandoned by its author.

  
Blocked
This pull request or issue is currently blocked from being merged/closed

  
Bug
Something isn't working as intended

  
Changelog
Added
 Changelog entry added

  
Changelog
Missing
 No changelog when one is needed

  
Changelog
None
 Changelog is unnecesary for this change

  
Cherry-picking
Commits picked from other conduit projects

  
Database
This requires or includes changes to the database

  
Dependencies
Something dependency related

  
Dependencies/Renovate
Automatic dependency upgrades by Renovate

  
Difficulty
Easy
 Low difficulty to implement - touches few parts of the codebase, low complexity

  
Difficulty
Hard
 High difficulty to implement - touches many parts of the codebase, high complexity

  
Difficulty
Medium
 Medium difficulty to implement - touches more parts of the codebase, higher complexity

  
Documentation
Improvements or additions to documentation

  
Enhancement
New feature or request

  
Good first issue
Good for newcomers

  
Help wanted
Additional eyes and keyboards are required for this one

  
Inherited
Issues that have been inhereted from the project pre-fork

  
Matrix/Administration
Features pertaining to homeserver administration

  
Matrix/Appservices
Features pertaining to the appservice API

  
Matrix/Auth
Features pertaining to authentication

  
Matrix/Client
Features pertaining to client-to-server interactions

  
Matrix/Core
Issues relating to core matrix functionality, such as state resolution and PDU formats

  
Matrix/E2EE
Issues related to end to end encryption

  
Matrix/Federation
Features pertaining to server-to-server interactions

  
Matrix/Hydra
Issues related to room version 12 and related changes (temporary label)

  
Matrix/MSC
Features pertaining to unstable matrix features

  
Matrix/Media
Features pertaining to media interactions

  
Matrix/T&S
Changes or issues related to trust & safety tooling

  
Merge
This PR is ready to be merged

  
Merge/Manual
This PR should be manually merged

  
Merge/Squash
This PR should be squashed when it is merged

  
Meta
Related to housekeeping, maintenance, or other repo-meta.

  
Meta/CI
Issues related to CI changes

  
Meta/Packaging
Packaging

  
Priority
Blocking
 This issue is blocking the next release

  
Priority
High
 This issue is very important

  
Priority
Low
 This issue is of a rather low priority

  
Security
This item is related to general security

  
Status
Confirmed
 This issue has enough information and is confirmed

  
Status
Duplicate
 This issue or pull request already exists

  
Status
Invalid
 This issue doesn't seem right

  
Status
Needs Investigation
 This issue needs further investigation

  
Support
Questions or support requests

  
bob the builder
pdu logic rewrite starring nexus nicholson

No labels
Abandoned
Blocked
Bug
Changelog
Added
Changelog
Missing
Changelog
None
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/E2EE
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Matrix/T&S
Merge
Merge/Manual
Merge/Squash
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
bob the builder
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
continuwuation/continuwuity!2332
No description provided.