Queued messages for unregisterd appservices cause issues at startup #1204

New issue

Open

opened 2025-11-27 15:44:05 +00:00 by Jade · 2 comments

Jade commented

2025-11-27 15:44:05 +00:00

Owner

@Henry-Hiles reported that continuwuity was failing to startup:

   1 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Consumed 15.265s CPU time, 238.4M memory peak, 642.6M read from disk, 38.2M written to disk, 923.6K incoming IP traffic, 694.9K outgoing IP traffic.
   2 │ Oct 31 14:49:48 quadraticserver systemd[1]: Stopped Continuwuity Matrix Server.
   3 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Failed with result 'timeout'.
   4 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Main process exited, code=killed, status=9/KILL
   5 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Killing process 863340 (conduwuit) with signal SIGKILL.
   6 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: State 'stop-sigterm' timed out. Killing.
   7 │ Oct 31 14:48:17 quadraticserver conduwuit[863340]: 2025-10-31T18:48:17.909767Z  WARN signal: conduwuit::signal: Received SIGTERM
   8 │ Oct 31 14:48:17 quadraticserver systemd[1]: Stopping Continuwuity Matrix Server...
   9 │ Oct 31 14:46:28 quadraticserver conduwuit[863340]: 2025-10-31T18:46:28.913004Z  WARN conduwuit_service::sending::sender: Missing appservice registration id="ooye"
  10 │ Oct 31 14:46:28 quadraticserver conduwuit[863340]: 2025-10-31T18:46:28.352042Z  INFO main:start: conduwuit_service::migrations: Loaded RocksDB database with schema version 18
  11 │ Oct 31 14:46:27 quadraticserver conduwuit[863340]: 2025-10-31T18:46:27.993270Z  INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=92 sequence=636727779 time=2.934187027s
  12 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032843Z  INFO conduwuit::server: 0.5.0-rc.8 server_name=federated.nexus database_path="/var/lib/continuwuity/" log_levels=info
  13 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032831Z  WARN conduwuit_core::config::check: All URLs are allowed for URL previews via setting "url_preview_domain_explicit_allowlist" to "*". This opens up s
     │ ignificant attack surface to your server. You are expected to be aware of the risks by doing this.
  14 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032785Z  WARN conduwuit_core::config::check: No support contact information (support_email or support_mxid) is configured in the well_known section. Users in 
     │ the admin room will be automatically listed as support contacts in the /.well-known/matrix/support endpoint.
  15 │ Oct 31 14:46:24 quadraticserver systemd[1]: Started Continuwuity Matrix Server.

This smells like:

 	async fn send_events_dest_appservice(
 		&self,
 		id: String,
 		events: Vec<SendingEvent>,
 	) -> SendingResult {
 		let Some(appservice) = self.services.appservice.get_registration(&id).await else {
 			return Err((
 				Destination::Appservice(id.clone()),
 				err!(Database(warn!(?id, "Missing appservice registration"))),
 			));
 		};

is getting hit, and that error is causing hangs at startup

Two part fix ideally:

Stop an error in sending from hanging startup?
Clear out messages that no longer have a valid destination on startup or appservice removal?

@Henry-Hiles reported that continuwuity was failing to startup: ``` 1 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Consumed 15.265s CPU time, 238.4M memory peak, 642.6M read from disk, 38.2M written to disk, 923.6K incoming IP traffic, 694.9K outgoing IP traffic. 2 │ Oct 31 14:49:48 quadraticserver systemd[1]: Stopped Continuwuity Matrix Server. 3 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Failed with result 'timeout'. 4 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Main process exited, code=killed, status=9/KILL 5 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: Killing process 863340 (conduwuit) with signal SIGKILL. 6 │ Oct 31 14:49:48 quadraticserver systemd[1]: continuwuity.service: State 'stop-sigterm' timed out. Killing. 7 │ Oct 31 14:48:17 quadraticserver conduwuit[863340]: 2025-10-31T18:48:17.909767Z WARN signal: conduwuit::signal: Received SIGTERM 8 │ Oct 31 14:48:17 quadraticserver systemd[1]: Stopping Continuwuity Matrix Server... 9 │ Oct 31 14:46:28 quadraticserver conduwuit[863340]: 2025-10-31T18:46:28.913004Z WARN conduwuit_service::sending::sender: Missing appservice registration id="ooye" 10 │ Oct 31 14:46:28 quadraticserver conduwuit[863340]: 2025-10-31T18:46:28.352042Z INFO main:start: conduwuit_service::migrations: Loaded RocksDB database with schema version 18 11 │ Oct 31 14:46:27 quadraticserver conduwuit[863340]: 2025-10-31T18:46:27.993270Z INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=92 sequence=636727779 time=2.934187027s 12 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032843Z INFO conduwuit::server: 0.5.0-rc.8 server_name=federated.nexus database_path="/var/lib/continuwuity/" log_levels=info 13 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032831Z WARN conduwuit_core::config::check: All URLs are allowed for URL previews via setting "url_preview_domain_explicit_allowlist" to "*". This opens up s │ ignificant attack surface to your server. You are expected to be aware of the risks by doing this. 14 │ Oct 31 14:46:25 quadraticserver conduwuit[863340]: 2025-10-31T18:46:25.032785Z WARN conduwuit_core::config::check: No support contact information (support_email or support_mxid) is configured in the well_known section. Users in │ the admin room will be automatically listed as support contacts in the /.well-known/matrix/support endpoint. 15 │ Oct 31 14:46:24 quadraticserver systemd[1]: Started Continuwuity Matrix Server. ``` This smells like: https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/7e4071c1178c07cb6c1b6eefbbbc26d7cff8d065/src/service/sending/sender.rs#L672-L682 is getting hit, and that error is causing hangs at startup Two part fix ideally: - Stop an error in sending from hanging startup? - Clear out messages that no longer have a valid destination on startup or appservice removal?