bug: Receiving a v1 PDU causes transaction handler to panic, failing the transaction but not marking it as done #1693

Open
opened 2026-04-21 15:44:22 +00:00 by nex · 0 comments
Owner

Receiving a v1-formatted PDU causes the following panic in the federation transaction handler:

thread 'conduwuit:worker' panicked at src/api/server/send.rs:293:48: prev_events values must be strings
Backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>
  23: <unknown>
  24: <unknown>
  25: <unknown>
  26: <unknown>
  27: start_thread
             at ./nptl/pthread_create.c:447:8
  28: clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78:0
thread 'conduwuit:worker' (1636726) panicked at src/api/server/send.rs:293:48:
prev_events values must be strings
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  2026-04-21T09:32:23.318184Z ERROR conduwuit_core::error::response: Server error, error: M_UNKNOWN: Transaction processing failed unexpectedly, error_debug: M_UNKNOWN: Transaction processing failed unexpectedly, kind: Unknown, status: 500 Internal Server Error
    at src/core/error/response.rs:19 on conduwuit:worker ThreadId(4)
    in conduwuit_router::request::handle with active=719684 handled=0
    in conduwuit_router::request::request
    in conduwuit_router::layers::router with method=PUT path=/_matrix/federation/v1/send/{transaction_id}
  2026-04-21T09:32:23.374740Z ERROR conduwuit_router::request: 500 Internal Server Error, method: PUT, uri: http://nexy7574.co.uk/_matrix/federation/v1/send/1776362455549
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)
    in conduwuit_router::request::request
    in conduwuit_router::layers::router with method=PUT path=/_matrix/federation/v1/send/{transaction_id}

As a result, the handler is unable to mark the transaction as complete, and it is left perpetually "processing". If the remote then tries to send a different transaction after giving up on the failing one, it will be met with the "another txn is already processing" error:

[26-04-21 09:30:23] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362451357 (0.2KiB): 0.0KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:30:37] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362451989 (0.6KiB): 0.0KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:31:01] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362452667 (0.2KiB): 0.0KiB in 0.10s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:31:09] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362453070 (0.2KiB): 0.0KiB in 0.10s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:31:27] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362453462 (0.2KiB): 0.0KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:32:22] 200 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362454173 (0.3KiB): 0.0KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:32:23] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.53s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:32:28] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:32:48] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:33:50] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:34:46] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:35:52] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:36:52] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:38:12] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:39:17] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.04s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:40:17] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:41:07] 500 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.09s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:51:40] 429 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.12s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:51:45] 429 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.14s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:52:01] 429 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.14s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:53:10] 429 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.15s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
[26-04-21 09:54:19] 429 HTTP/1.1 PUT    /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.17s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158
2026-04-21T09:51:40.300757Z DEBUG router{method=PUT path=/_matrix/federation/v1/send/{transaction_id}}:request:handle{active=724839 handled=0}: conduwuit_core::error::response: Client error error=M_LIMIT_EXCEEDED: Still processing another transaction from this origin kind=LimitExceeded { retry_after: None } status=429 Too Many Requests

Problem area:

OwnedEventId::parse(v.as_str().expect("prev_events values must be strings"))

(invalid events should be dropped instead of expecting the sender to not send invalid events)

Receiving a v1-formatted PDU causes the following panic in the federation transaction handler: ``` thread 'conduwuit:worker' panicked at src/api/server/send.rs:293:48: prev_events values must be strings Backtrace: 0: <unknown> 1: <unknown> 2: <unknown> 3: <unknown> 4: <unknown> 5: <unknown> 6: <unknown> 7: <unknown> 8: <unknown> 9: <unknown> 10: <unknown> 11: <unknown> 12: <unknown> 13: <unknown> 14: <unknown> 15: <unknown> 16: <unknown> 17: <unknown> 18: <unknown> 19: <unknown> 20: <unknown> 21: <unknown> 22: <unknown> 23: <unknown> 24: <unknown> 25: <unknown> 26: <unknown> 27: start_thread at ./nptl/pthread_create.c:447:8 28: clone3 at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78:0 thread 'conduwuit:worker' (1636726) panicked at src/api/server/send.rs:293:48: prev_events values must be strings note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace 2026-04-21T09:32:23.318184Z ERROR conduwuit_core::error::response: Server error, error: M_UNKNOWN: Transaction processing failed unexpectedly, error_debug: M_UNKNOWN: Transaction processing failed unexpectedly, kind: Unknown, status: 500 Internal Server Error at src/core/error/response.rs:19 on conduwuit:worker ThreadId(4) in conduwuit_router::request::handle with active=719684 handled=0 in conduwuit_router::request::request in conduwuit_router::layers::router with method=PUT path=/_matrix/federation/v1/send/{transaction_id} 2026-04-21T09:32:23.374740Z ERROR conduwuit_router::request: 500 Internal Server Error, method: PUT, uri: http://nexy7574.co.uk/_matrix/federation/v1/send/1776362455549 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) in conduwuit_router::request::request in conduwuit_router::layers::router with method=PUT path=/_matrix/federation/v1/send/{transaction_id} ``` As a result, the handler is unable to mark the transaction as complete, and it is left perpetually "processing". If the remote then tries to send a different transaction after giving up on the failing one, it will be met with the "another txn is already processing" error: ``` [26-04-21 09:30:23] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362451357 (0.2KiB): 0.0KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:30:37] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362451989 (0.6KiB): 0.0KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:31:01] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362452667 (0.2KiB): 0.0KiB in 0.10s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:31:09] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362453070 (0.2KiB): 0.0KiB in 0.10s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:31:27] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362453462 (0.2KiB): 0.0KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:32:22] 200 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362454173 (0.3KiB): 0.0KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:32:23] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.53s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:32:28] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.06s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:32:48] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:33:50] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:34:46] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:35:52] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.03s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:36:52] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:38:12] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.07s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:39:17] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.04s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:40:17] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.08s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:41:07] 500 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362455549 (6.5KiB): 0.1KiB in 0.09s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:51:40] 429 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.12s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:51:45] 429 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.14s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:52:01] 429 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.14s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:53:10] 429 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.15s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 [26-04-21 09:54:19] 429 HTTP/1.1 PUT /_matrix/federation/v1/send/1776362476773 (40.7KiB): 0.1KiB in 0.17s from matrix.org (Synapse/1.151.0 (b=matrix-org-hotfixes-priv,8667ae8d12)) @ 176.126.240.158 ``` ``` 2026-04-21T09:51:40.300757Z DEBUG router{method=PUT path=/_matrix/federation/v1/send/{transaction_id}}:request:handle{active=724839 handled=0}: conduwuit_core::error::response: Client error error=M_LIMIT_EXCEEDED: Still processing another transaction from this origin kind=LimitExceeded { retry_after: None } status=429 Too Many Requests ``` Problem area: https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/027f6a4b025e75a76354a1912f6186f941d6147f/src/api/server/send.rs#L293 (invalid events should be dropped instead of expecting the sender to not send invalid events)
nex self-assigned this 2026-04-21 15:44:22 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity#1693
No description provided.