FR: More helpful INFO logging during database open stage to detect corrupted db deadlock #1610

New issue

Closed

opened 2026-04-04 19:06:14 +00:00 by jinnatar · 4 comments

jinnatar commented

2026-04-04 19:06:14 +00:00

The problem

If the rocksdb is corrupted, it's possible that the database open will deadlock and never complete. This is extremely confusing to the admin given that the last log message they see is:

2026-04-04T18:58:42.815404Z INFO conduwuit::server: 0.5.6 (2c72338) server_name=miselli.fi database_path="/var/lib/continuwuity" log_levels=info

The next line should be:

2026-04-04T18:59:35.365336Z INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=95 sequence=13075351 time=52.34851955s

.. And by not receiving that line, you're left in the dark about what's going on.

Impact

Extreme admin confusion. The service is "up" but tcp connect to all endpoints is failing and no errors are emitted in the log. Also your matrix homeserver is now down so you can't go ask the chat what the hell is going on. :-)

Proposed solution

Be slightly more verbose in startup logging by adding a new emitted INFO line, something like:

main:start:open: conduwuit_database::engine::open: Starting database open (be patient, but if this hangs for too long your rocksdb may be corrupted, see https://continuwuity.org/troubleshooting#database-corruption)

Obviously it would be nicer if db corruption didn't just deadlock startup in the first place, but that's a hard problem to fix, and verbosity would at least make it easier to figure out how screwed you are.

## The problem If the rocksdb is corrupted, it's possible that the database open will deadlock and never complete. This is extremely confusing to the admin given that the last log message they see is: > 2026-04-04T18:58:42.815404Z INFO conduwuit::server: 0.5.6 (2c72338) server_name=miselli.fi database_path="/var/lib/continuwuity" log_levels=info The next line should be: > 2026-04-04T18:59:35.365336Z INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=95 sequence=13075351 time=52.34851955s .. And by not receiving that line, you're left in the dark about what's going on. ## Impact Extreme admin confusion. The service is "up" but tcp connect to all endpoints is failing and no errors are emitted in the log. Also your matrix homeserver is now down so you can't go ask the chat what the hell is going on. :-) ## Proposed solution Be slightly more verbose in startup logging by adding a new emitted INFO line, something like: > main:start:open: conduwuit_database::engine::open: Starting database open (be patient, but if this hangs for too long your rocksdb may be corrupted, see https://continuwuity.org/troubleshooting#database-corruption) Obviously it would be nicer if db corruption didn't just deadlock startup in the first place, but that's a hard problem to fix, and verbosity would at least make it easier to figure out how screwed you are.