wip: chore: Update to origin/zaidoon/10.10.1, latest RocksDB, fix a few bugs #5

Draft
gamesguru wants to merge 257 commits from gamesguru/rust-rocksdb-zaidoon1:master into rust-rocksdb-upstream-master
First-time contributor

TODO: Updating commit hash in continuwuity project depends first upon this PR's merge commit being added to history. The submodule, librocksdb-sys/rocksdb, also needs a separate PR opened. The second submodule, snappy, I don't know if we need or use.

As well, we might consider distancing ourselves from this 3rd party wrapper
(in light of the portable ability to compile RocksDB statically as a shared object *.so file).

This rust wrapper potentially just a security vulnerability for those not keen on C++.

Release notes are here:

**TODO:** Updating commit hash in `continuwuity` project depends first upon this PR's merge commit being added to history. The submodule, `librocksdb-sys/rocksdb`, also needs a separate PR opened. The second submodule, `snappy`, I don't know if we need or use. As well, **we might consider distancing** ourselves from this 3rd party wrapper (in light of the portable ability to compile RocksDB statically as a shared object `*.so` file). This rust wrapper potentially just a security vulnerability for those not keen on C++. ### Release notes are here: - https://github.com/zaidoon1/rust-rocksdb/releases/tag/v0.46.0 - https://github.com/facebook/rocksdb/releases/tag/v10.10.1
bump dependencies & upgrade to latest rust version
---------

Co-authored-by: Jordan Danford <jordandanford@gmail.com>
Release 0.22.5
This reverts commit 21ca2ff47f.
This reverts commit 9dc5c32724.
BackupEngine is a simple pointer wrapper, so it's safe to send to another thread since the underlying RocksDB backup engine is thread-safe.
Signed-off-by: Jason Volk <jason@zemos.net>
Also remove unused code in ci workflow
in CI, we specify the rust version as 1.70.0...

```yaml
on: [push, pull_request]
env:
  RUST_VERSION: 1.70.0
```

likewise, the package manifest pins the rust version...

```toml
rust-version = "1.70.0"
```

..but this version is not pinned by cargo, via a `rust-toolchain`. this
means that testing this project using the stable toolchain by default
(_at time of writing, this is 1.78.0_) will see test failures due to
error messages not matching the expected snapshots, due to `rustc`
itself making changes to its diagnostics:

```
test tests/fail/snapshot_outlive_db.rs ... mismatch

EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0597]: `db` does not live long enough
 --> tests/fail/snapshot_outlive_db.rs:6:9
  |
4 |     let _snapshot = {
  |         --------- borrow later stored here
5 |         let db = DB::open_default("foo").unwrap();
  |             -- binding `db` declared here
6 |         db.snapshot()
  |         ^^^^^^^^^^^^^ borrowed value does not live long enough
7 |     };
  |     - `db` dropped here while still borrowed
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0597]: `db` does not live long enough
 --> tests/fail/snapshot_outlive_db.rs:6:9
  |
4 |     let _snapshot = {
  |         --------- borrow later stored here
5 |         let db = DB::open_default("foo").unwrap();
  |             -- binding `db` declared here
6 |         db.snapshot()
  |         ^^ borrowed value does not live long enough
7 |     };
  |     - `db` dropped here while still borrowed
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
      your test with the environment variable TRYBUILD=overwrite
```

this commit introduces a `rust-toolchain.toml` file so that the correct
version of rust is used by those building or contributing to this
project.
This will allow using zstd-static-linking-only  if enabled
* Add support for enabling blob cache

---------

Co-authored-by: Matt Jurik <matt.jurik@hulu.com>
Signed-off-by: strawberry <strawberry@puppygock.gay>
Signed-off-by: strawberry <strawberry@puppygock.gay>
Signed-off-by: strawberry <strawberry@puppygock.gay>
* Improve statistics by auto gen enum Ticker & enum Histogram

1. librocksdb-sys/Makefile: Add gen_statistics and gen_statistics.bash,
     make -C librocksdb-sys gen_statistics will generate 2 files:
       src/statistics_enum_ticker.rs
       src/statistics_enum_histogram.rs

2. In statistics.rs, we include! these 2 files and remove old hand
   writing enum Tickers & enum Histograms.

3. File gen_statistics.bash is used for generating these 2 files
   which is called in Makefile.
Support usages where the caller already has an environment built by an
alternative means, which the Env of this crate should take ownership of.

Co-authored-by: Zaidoon Abd Al Hadi <zaidoon@cloudflare.com>
* Fix unsoundness via impure AsRef

---------

Co-authored-by: Oleksandr Anyshchenko <aanischenko@gmail.com>
Co-authored-by: Zaidoon Abd Al Hadi <43054535+zaidoon1@users.noreply.github.com>
Signed-off-by: Jason Volk <jason@zemos.net>
Signed-off-by: Jason Volk <jason@zemos.net>
To instead write to a temporary directory.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
* implement `with_capacity` for `WriteBatch`

* add tests for WriteBatch::with_capacity

* update changelog
 - remove `clippy.toml` because we already have `msrv` in `Cargo.toml`
* fix(build): add ROCKSDB_SCHED_GETCPU_PRESENT for Linux build config

This change adds a definition for ROCKSDB_SCHED_GETCPU_PRESENT in the Linux-specific section of the build script. This ensures that the RocksDB library is aware of the presence of the sched_getcpu function on Linux systems, potentially optimising CPU scheduling.
This exposes the rocksdb_get_db_identity function, which calls
DB::GetDbIdentity. It also adds a setter and accessor to Options to
change the option.

This function is useful to determine if two RocksDB copies originated
from the same place.
This was added to the C API in RocksDB 9.7.0. The RocksDB wiki
recommends setting this option to true:

"We recommend to set track_and_verify_wals_in_manifest to true for
production, it has been enabled in production for the entire database
cluster serving the social graph for all Meta apps."

See: https://github.com/facebook/rocksdb/wiki/Track-WAL-in-MANIFEST
Signed-off-by: Jason Volk <jason@zemos.net>
* Fix C++ linking

Hard-coding the C++ library linking is bad practice, and instead this
crate needs to follow the practices as described under
https://docs.rs/cc/1.2.14/cc/index.html#c-support.

Without doing so, this crate will not compile on when using libc++
instead of libstdc++.

This change will keep the existing behaviour, while allowing downstream
consumers of this crate to override the C++ library with the `CXXSTDLIB`
which is respected by the cc crate.
This lets RocksDB know that getauxval is supported which is needed to enable hardware accelerated crc32c for ARM on Linux
Edit the rustdoc for both DBRawIteratorWithThreadMode and
DBIteratorWithThreadMode to clarify what they do, and link to each
other.
stolen from https://github.com/rust-rocksdb/rust-rocksdb/pull/1007/commits/7d3889271a6de0afd51fbcfa95184958c9fa9761
running cargo fix --edition for now and keeping
only the backwards compatible changes
RocksDB 10.2.1 is marking this API as deprecated. The upstream
documentation has changed to state the following. Update rustdoc to
match, and mark with #[deprecated]. Currently rust-rocksdb is not
using this version of RocksDB, but presumably it will eventually, so
we might as well mark this as deprecated now.

Upstream RocksDB comment:

// DEPRECATED: This option might be removed in a future release.
// There should be no noticeable performance difference whether this option
// is turned on or off when a DB does not use DeleteRange().

https://github.com/facebook/rocksdb/blob/v10.2.1/include/rocksdb/options.h#L1837
Co-authored-by: Burton Qin <bobbqq@163.com>
RocksDB 10.5.0 has marked this option as deprecated. It will do
nothing. The rust-rocksdb crate is still using RocksDB 9.10.0, so
this deprecation notice is a bit premature. However, I assume we will
eventually upgrade, and this notice will then be correct.
Co-authored-by: strawberry <june@girlboss.ceo>
Signed-off-by: strawberry <strawberry@puppygock.gay>
Signed-off-by: June Clementine Strawberry <june@3.dog>
Signed-off-by: June Clementine Strawberry <june@3.dog>
The previous version allows calling a closure even after it gets
dropped, which causes "safe" Rust code to do illegal memory accesses.
To fix it: Store the log closure in an Arc to reference count it.
This ensures that even if an Options is cloned and used with
different DBs it will work.

Unfortunately this is a breaking API change. I think this is worth it
since the previous version can lead to invalid memory accesses.

Add a test that demonstrates the incorrect behavior.

Example that causes invalid memory accesses:

    let mut rdb_opts = rust_rocksdb::Options::default();
    rdb_opts.create_if_missing(true);
    {
        let obj_is_dropped = ExampleObject { arg: 42 };
        rdb_opts.set_callback_logger(
            rust_rocksdb::LogLevel::Debug,
            &|level: rust_rocksdb::LogLevel, msg: &str| {
                println!("arg={} level={level:?} msg={msg}", obj_is_dropped.arg);
            },
        );
    }
    rust_rocksdb::DB::open(&rdb_opts, "test_db")

Output showing that arg changes:

arg=42 level=Info msg=DB pointer 0x15b80b400
arg=-2999674700805604355 level=Info msg=[db/db_impl/db_impl.cc:1117] ------- DUMPING STATS -------
* feat: expose {enable,disable}_file_deletions

Closes #1034
* Expose APIs to import and export Column Families

Adds Checkpoint::export_column_family and
DB::create_column_family_with_import methods exposing the ability to
export and import checkpoints of column families.
Update the rustdoc on new_hyper_clock_cache to copy from RocksDB's
HyperClockCacheOptions in cache.h:
https://github.com/facebook/rocksdb/blob/main/include/rocksdb/cache.h

As of RocksDB 10.7: "HyperClockCache with no estimated_entry_charge
is now production-ready and is the preferred block cache
implementation vs. LRUCache"

https://github.com/facebook/rocksdb/blob/main/HISTORY.md#1070-09192025
* ci: use actions/checkout@v5l (#1043)

* feat: allow user to set checkpoint log size for flush (#1055)

* upgrade to Rust edition 2024 and MSRV 1.89.0

---------

Co-authored-by: Coder <161350311+MamunC0der@users.noreply.github.com>
Co-authored-by: QuantumExplorer <quantum@dash.org>
Signed-off-by: luohewuyang <luohewuyang@outlook.com>
The result of rocksdb_comparator_create must be passed to
rocksdb_comparator_destroy to free it. Add it to OptionsMustOutliveDB
so it is reference counted. Add a test that this works when shared
between DBs.

This fixes a memory leak found by running the tests with valgrind.
If the rocksdb_get_options_from_string failed, we had 2 leaks:

* We forgot to free the newly allocated new_options value created by
  rocksdb_options_create. To avoid this: immediately wrap the result
  in an Options struct, so it will get dropped on all code paths.
* We forgot to free the errptr RocksDB error message. Use ffi_try!
  to fix this, which is what we do everywhere else.

Additionally: into_c_string() will fail if the Rust string contains
NUL (0x00) bytes. Convert the .expect() into a result and add a test
for this case.

To make error conversions a bit clearer, rename error_message to
convert_rocksdb_error. Add a doc comment that it will free the
argument. Also change the function to return an Error, since it is
always used in that way. Also add a doc comment to ffi_try!.
The result of rocksdb_transactiondb_get_base_db must be freed using
rocksdb_transactiondb_close_base_db after we are done with it. Add
a Vec<*mut ffi::rocksdb_t> to MemoryUsageBuilder to fix a leak.

Co-authored-by: Zaidoon Abd Al Hadi <43054535+zaidoon1@users.noreply.github.com>
When rocksdb_create_column_family returns an error, it also returns
a non-NULL pointer that needs to be freed. I have attempted to submit
a fix for this upstream to RocksDB [1]. Add a check to free it if it
returns a non-NULL pointer, which will work with older versions of
RocksDB.

I believe this is the last of my set of fixes for valgrind reported
memory leaks in the rust-rocksdb tests.

[1] https://github.com/facebook/rocksdb/pull/14214
This function previously called
rocksdb_column_family_metadata_get_name, which returns a malloc-ed C
string, without freeing it. Fix this, and try to make this kind of
mistake less likely by renaming the helper functions. This way, the
caller needs to think about if the string must be freed or not. Most
of the existing uses are actually from_cstr_and_free.

* Rename from_cstr to from_cstr_without_free
* Add from_cstr_and_free, which always frees the C string.
* Change existing uses to call the correct variant.
- Add concurrency control to cancel redundant workflow runs
- Add CARGO_TERM_COLOR=always for colored output in logs
- Fix duplicate workflow runs (only run on push to master, not all pushes)
- Standardize checkout with submodules: recursive
- Remove redundant rustup override and cargo generate-lockfile steps
- Update audit action from rustsec/audit-check@master to actions-rust-lang/audit@v1
- Switch from dtolnay/rust-toolchain + Swatinem/rust-cache to
  actions-rust-lang/setup-rust-toolchain (built-in caching, problem matchers)
- Split test and test-multi-threaded-cf into separate parallel jobs
  (fixes disk space issue, faster CI)
- Add save-if to only save cache on master branch (prevents PR cache pollution)
* feat: add raw-ptr access via feature flag

Closes: #1033

* refactor: add unsafe to 'as_raw_ptr'

* doc: remove redundant comments
This type must not outlive the DBs and caches that are added to it,
otherwise the caller will access invalid memory and likely crash.
Use PhantomData to ensure it does not outlive its arguments. This is
technically a breaking API change, but anything that used this code
in this way was likely broken anyway.

Co-authored-by: Oleksandr Anyshchenko <aanischenko@gmail.com>
Take advantage of the new zero-copy C API exposed in RocksDB PRs #13911 and #14036.
New APIs:
- `get_into_buffer` / `get_into_buffer_cf`: Read values directly into a
  caller-provided buffer with zero allocation when the buffer is large enough.
  Returns `GetIntoBufferResult` enum indicating Found/NotFound/BufferTooSmall.
- `batched_multi_get_cf_slice` / `batched_multi_get_cf_slice_opt`: Optimized
  batch lookup using `rocksdb_slice_t` array API directly, eliminating the
  overhead of converting keys from separate pointer+size arrays.
Performance improvements:
- Iterator `key()` and `value()` now use `rocksdb_iter_key_slice` and
  `rocksdb_iter_value_slice` which return slices by value, avoiding output
  parameter overhead.
- `batched_multi_get_cf_slice` creates one Vec<rocksdb_slice_t> instead of
  two separate vectors, improving cache locality.
Added comprehensive tests covering all new APIs including edge cases like
zero-length buffers, empty values, binary data, and empty databases.
Replace manual usize pointer arithmetic with idiomatic Rust pointer
operations. The previous implementation cast pointers to usize, manually
calculated offsets using mem::size_of, and cast back to pointers. The new
implementation uses ptr.add(index) which is cleaner, more readable, and
lets the compiler handle pointer arithmetic correctly.
Avoid allocating a new ReadOptions on each multi_get call by reusing
the thread-local DEFAULT_READ_OPTS, consistent with multi_get_cf and
other Transaction methods that already use this pattern.
* Add sequence_number method to SnapshotWithThreadMode
Also link the related functions to each other.

Extend the test to validate that calling get_statistics returns None
when not enabled.
gamesguru changed title from chore: Update to origin/zaidoon/10.10.1, latest RocksDB, fix a few bugs to wip: chore: Update to origin/zaidoon/10.10.1, latest RocksDB, fix a few bugs 2026-03-14 14:53:39 +00:00
This pull request has changes conflicting with the target branch.
  • .github/workflows/rust.yml
  • CHANGELOG.md
  • Cargo.toml
  • README.md
  • librocksdb-sys/Cargo.toml
  • librocksdb-sys/build.rs
  • librocksdb-sys/build_version.cc
  • librocksdb-sys/rocksdb
  • librocksdb-sys/snappy
  • rust-toolchain.toml
  • src/compaction_filter_factory.rs
  • src/comparator.rs
  • src/db.rs
  • src/db_iterator.rs
  • src/db_options.rs
  • src/merge_operator.rs
  • src/perf.rs
  • src/prop_name.rs
  • src/statistics.rs
  • src/transactions/optimistic_transaction_db.rs
  • src/transactions/transaction.rs
  • src/transactions/transaction_db.rs
  • src/write_batch.rs
  • tests/fail/open_with_multiple_refs_as_single_threaded.stderr
  • tests/test_comparator.rs
  • tests/test_db.rs
  • tests/test_rocksdb_options.rs
  • tests/test_sst_file_writer.rs
  • tests/test_transaction_db_property.rs
  • tests/util/mod.rs
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u master:gamesguru-master
git switch gamesguru-master

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch rust-rocksdb-upstream-master
git merge --no-ff gamesguru-master
git switch gamesguru-master
git rebase rust-rocksdb-upstream-master
git switch rust-rocksdb-upstream-master
git merge --ff-only gamesguru-master
git switch gamesguru-master
git rebase rust-rocksdb-upstream-master
git switch rust-rocksdb-upstream-master
git merge --no-ff gamesguru-master
git switch rust-rocksdb-upstream-master
git merge --squash gamesguru-master
git switch rust-rocksdb-upstream-master
git merge --ff-only gamesguru-master
git switch rust-rocksdb-upstream-master
git merge gamesguru-master
git push origin rust-rocksdb-upstream-master
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/rust-rocksdb-zaidoon1!5
No description provided.