bug: Work around rocksdb#13333 #1122

New issue

Open

opened 2025-10-15 18:02:04 +00:00 by Jade · 4 comments

Jade commented

2025-10-15 18:02:04 +00:00

Owner

https://github.com/facebook/rocksdb/issues/13333 - seems to have caused issues recovering many people's databases at this point - for example, #1107. Investigate workarounds for this - either changing our WAL config to enable compression, or patching out the mentioned deadlock.

https://github.com/facebook/rocksdb/issues/13333 - seems to have caused issues recovering many people's databases at this point - for example, https://forgejo.ellis.link/continuwuation/continuwuity/issues/1107. Investigate workarounds for this - either changing our WAL config to enable compression, or patching out the mentioned deadlock.

Jade added the

labels

2025-10-15 18:02:04 +00:00

nyanbinary commented

2025-10-16 00:30:50 +00:00

Contributor

Would rather the enabling compression method than carrying another patch coming from the perspective of a nixpkgs maintainer ;p.

mcronce commented

2025-10-26 17:08:42 +00:00

That said, it doesn't seem to be in rc.8; I'm currently attempting repair using that, will report back if it seems to be any better

FWIW, while trying different repair options for #1107, I found this (TY for the mention, that's how I got here), clicked through to https://github.com/facebook/rocksdb/issues/13333, and said "cool, I'll try just reverting https://github.com/facebook/rocksdb/pull/12643" - but it turned out it was [already reverted in v0.5.0-rc.6](https://forgejo.ellis.link/continuwuation/rocksdb/commit/3ce04794bcfbbb0d2e6f81ae35fc4acf688b6986), which is what I was running That said, it doesn't seem to be in rc.8; I'm currently attempting repair using that, will report back if it seems to be any better

Jade referenced this issue

2025-10-26 17:45:15 +00:00

feat: Allow selecting the WAL compression mode #1140

Jade commented

2026-02-13 16:09:18 +00:00

Author

Owner

Copying this from matrix so it doesn't get lost like prior solutions:

FTR for anyone encountering this, the manual fix is to build
https://forgejo.ellis.link/continuwuation/rocksdb/src/branch/10.4.fb
With
continuwuation/rocksdb@3ce04794bc
Cherry picked on topp
And run
ldb repair (iirc)
Against your database

Copying this from matrix so it doesn't get lost like prior solutions: FTR for anyone encountering this, the manual fix is to build https://forgejo.ellis.link/continuwuation/rocksdb/src/branch/10.4.fb With https://forgejo.ellis.link/continuwuation/rocksdb/commit/3ce04794bcfbbb0d2e6f81ae35fc4acf688b6986 Cherry picked on topp And run `ldb repair` (iirc) Against your database

Skalli commented

2026-02-22 17:43:54 +00:00

Since I ran into the problem myself and had some trouble following the steps, here is a bit more detailed description if someone else runs into the same problem.
For me it was caused by running out of disk space recovery did not work.
I checked out the branch mentioned by jade:
git clone git clone https://forgejo.ellis.link/continuwuation/rocksdb.git
then cd rocksdb into the directory and checked out the branch.
git checkout 10.4.fb
the cherry pick was just git cherry-pick 3ce04794bc
Afterwards I needed to build not just RocksDB, but the tool ldb for the repair. Regular make failed because warnings were treated as errors, I circumvented it with a parameter to build:
make ldb DISABLE_WARNING_AS_ERROR=1
This build succeeded and I could call the tool to repair the database. It needed a path to the directory and another parameter to ignore a version mismatch, since my version was newer than the one we just built, therefore the last step was:
sudo ./ldb --db=/var/lib/conduwuit repair --ignore_unknown_options
Sudo so it could read the database directory.
Afterwards continuwuity started normally for me again. Hope it helps someone.

Since I ran into the problem myself and had some trouble following the steps, here is a bit more detailed description if someone else runs into the same problem. For me it was caused by running out of disk space recovery did not work. I checked out the branch mentioned by jade: `git clone git clone https://forgejo.ellis.link/continuwuation/rocksdb.git` then `cd rocksdb` into the directory and checked out the branch. `git checkout 10.4.fb` the cherry pick was just `git cherry-pick 3ce04794bc` Afterwards I needed to build not just RocksDB, but the tool `ldb` for the repair. Regular make failed because warnings were treated as errors, I circumvented it with a parameter to build: `make ldb DISABLE_WARNING_AS_ERROR=1` This build succeeded and I could call the tool to repair the database. It needed a path to the directory and another parameter to ignore a version mismatch, since my version was newer than the one we just built, therefore the last step was: `sudo ./ldb --db=/var/lib/conduwuit repair --ignore_unknown_options` Sudo so it could read the database directory. Afterwards continuwuity started normally for me again. Hope it helps someone.