Avoid allocating a new ReadOptions on each multi_get call by reusing
the thread-local DEFAULT_READ_OPTS, consistent with multi_get_cf and
other Transaction methods that already use this pattern.
Replace manual usize pointer arithmetic with idiomatic Rust pointer
operations. The previous implementation cast pointers to usize, manually
calculated offsets using mem::size_of, and cast back to pointers. The new
implementation uses ptr.add(index) which is cleaner, more readable, and
lets the compiler handle pointer arithmetic correctly.
Take advantage of the new zero-copy C API exposed in RocksDB PRs #13911 and #14036.
New APIs:
- `get_into_buffer` / `get_into_buffer_cf`: Read values directly into a
caller-provided buffer with zero allocation when the buffer is large enough.
Returns `GetIntoBufferResult` enum indicating Found/NotFound/BufferTooSmall.
- `batched_multi_get_cf_slice` / `batched_multi_get_cf_slice_opt`: Optimized
batch lookup using `rocksdb_slice_t` array API directly, eliminating the
overhead of converting keys from separate pointer+size arrays.
Performance improvements:
- Iterator `key()` and `value()` now use `rocksdb_iter_key_slice` and
`rocksdb_iter_value_slice` which return slices by value, avoiding output
parameter overhead.
- `batched_multi_get_cf_slice` creates one Vec<rocksdb_slice_t> instead of
two separate vectors, improving cache locality.
Added comprehensive tests covering all new APIs including edge cases like
zero-length buffers, empty values, binary data, and empty databases.
This type must not outlive the DBs and caches that are added to it,
otherwise the caller will access invalid memory and likely crash.
Use PhantomData to ensure it does not outlive its arguments. This is
technically a breaking API change, but anything that used this code
in this way was likely broken anyway.
Co-authored-by: Oleksandr Anyshchenko <aanischenko@gmail.com>
- Add concurrency control to cancel redundant workflow runs
- Add CARGO_TERM_COLOR=always for colored output in logs
- Fix duplicate workflow runs (only run on push to master, not all pushes)
- Standardize checkout with submodules: recursive
- Remove redundant rustup override and cargo generate-lockfile steps
- Update audit action from rustsec/audit-check@master to actions-rust-lang/audit@v1
- Switch from dtolnay/rust-toolchain + Swatinem/rust-cache to
actions-rust-lang/setup-rust-toolchain (built-in caching, problem matchers)
- Split test and test-multi-threaded-cf into separate parallel jobs
(fixes disk space issue, faster CI)
- Add save-if to only save cache on master branch (prevents PR cache pollution)
This function previously called
rocksdb_column_family_metadata_get_name, which returns a malloc-ed C
string, without freeing it. Fix this, and try to make this kind of
mistake less likely by renaming the helper functions. This way, the
caller needs to think about if the string must be freed or not. Most
of the existing uses are actually from_cstr_and_free.
* Rename from_cstr to from_cstr_without_free
* Add from_cstr_and_free, which always frees the C string.
* Change existing uses to call the correct variant.
When rocksdb_create_column_family returns an error, it also returns
a non-NULL pointer that needs to be freed. I have attempted to submit
a fix for this upstream to RocksDB [1]. Add a check to free it if it
returns a non-NULL pointer, which will work with older versions of
RocksDB.
I believe this is the last of my set of fixes for valgrind reported
memory leaks in the rust-rocksdb tests.
[1] https://github.com/facebook/rocksdb/pull/14214
The result of rocksdb_transactiondb_get_base_db must be freed using
rocksdb_transactiondb_close_base_db after we are done with it. Add
a Vec<*mut ffi::rocksdb_t> to MemoryUsageBuilder to fix a leak.
Co-authored-by: Zaidoon Abd Al Hadi <43054535+zaidoon1@users.noreply.github.com>
If the rocksdb_get_options_from_string failed, we had 2 leaks:
* We forgot to free the newly allocated new_options value created by
rocksdb_options_create. To avoid this: immediately wrap the result
in an Options struct, so it will get dropped on all code paths.
* We forgot to free the errptr RocksDB error message. Use ffi_try!
to fix this, which is what we do everywhere else.
Additionally: into_c_string() will fail if the Rust string contains
NUL (0x00) bytes. Convert the .expect() into a result and add a test
for this case.
To make error conversions a bit clearer, rename error_message to
convert_rocksdb_error. Add a doc comment that it will free the
argument. Also change the function to return an Error, since it is
always used in that way. Also add a doc comment to ffi_try!.
The result of rocksdb_comparator_create must be passed to
rocksdb_comparator_destroy to free it. Add it to OptionsMustOutliveDB
so it is reference counted. Add a test that this works when shared
between DBs.
This fixes a memory leak found by running the tests with valgrind.
* ci: use actions/checkout@v5l (#1043)
* feat: allow user to set checkpoint log size for flush (#1055)
* upgrade to Rust edition 2024 and MSRV 1.89.0
---------
Co-authored-by: Coder <161350311+MamunC0der@users.noreply.github.com>
Co-authored-by: QuantumExplorer <quantum@dash.org>
* Expose APIs to import and export Column Families
Adds Checkpoint::export_column_family and
DB::create_column_family_with_import methods exposing the ability to
export and import checkpoints of column families.
The previous version allows calling a closure even after it gets
dropped, which causes "safe" Rust code to do illegal memory accesses.
To fix it: Store the log closure in an Arc to reference count it.
This ensures that even if an Options is cloned and used with
different DBs it will work.
Unfortunately this is a breaking API change. I think this is worth it
since the previous version can lead to invalid memory accesses.
Add a test that demonstrates the incorrect behavior.
Example that causes invalid memory accesses:
let mut rdb_opts = rust_rocksdb::Options::default();
rdb_opts.create_if_missing(true);
{
let obj_is_dropped = ExampleObject { arg: 42 };
rdb_opts.set_callback_logger(
rust_rocksdb::LogLevel::Debug,
&|level: rust_rocksdb::LogLevel, msg: &str| {
println!("arg={} level={level:?} msg={msg}", obj_is_dropped.arg);
},
);
}
rust_rocksdb::DB::open(&rdb_opts, "test_db")
Output showing that arg changes:
arg=42 level=Info msg=DB pointer 0x15b80b400
arg=-2999674700805604355 level=Info msg=[db/db_impl/db_impl.cc:1117] ------- DUMPING STATS -------
RocksDB 10.5.0 has marked this option as deprecated. It will do
nothing. The rust-rocksdb crate is still using RocksDB 9.10.0, so
this deprecation notice is a bit premature. However, I assume we will
eventually upgrade, and this notice will then be correct.
RocksDB 10.2.1 is marking this API as deprecated. The upstream
documentation has changed to state the following. Update rustdoc to
match, and mark with #[deprecated]. Currently rust-rocksdb is not
using this version of RocksDB, but presumably it will eventually, so
we might as well mark this as deprecated now.
Upstream RocksDB comment:
// DEPRECATED: This option might be removed in a future release.
// There should be no noticeable performance difference whether this option
// is turned on or off when a DB does not use DeleteRange().
https://github.com/facebook/rocksdb/blob/v10.2.1/include/rocksdb/options.h#L1837
* Fix C++ linking
Hard-coding the C++ library linking is bad practice, and instead this
crate needs to follow the practices as described under
https://docs.rs/cc/1.2.14/cc/index.html#c-support.
Without doing so, this crate will not compile on when using libc++
instead of libstdc++.
This change will keep the existing behaviour, while allowing downstream
consumers of this crate to override the C++ library with the `CXXSTDLIB`
which is respected by the cc crate.
This was added to the C API in RocksDB 9.7.0. The RocksDB wiki
recommends setting this option to true:
"We recommend to set track_and_verify_wals_in_manifest to true for
production, it has been enabled in production for the entire database
cluster serving the social graph for all Meta apps."
See: https://github.com/facebook/rocksdb/wiki/Track-WAL-in-MANIFEST
This exposes the rocksdb_get_db_identity function, which calls
DB::GetDbIdentity. It also adds a setter and accessor to Options to
change the option.
This function is useful to determine if two RocksDB copies originated
from the same place.
* fix(build): add ROCKSDB_SCHED_GETCPU_PRESENT for Linux build config
This change adds a definition for ROCKSDB_SCHED_GETCPU_PRESENT in the Linux-specific section of the build script. This ensures that the RocksDB library is aware of the presence of the sched_getcpu function on Linux systems, potentially optimising CPU scheduling.
* implement `with_capacity` for `WriteBatch`
* add tests for WriteBatch::with_capacity
* update changelog
- remove `clippy.toml` because we already have `msrv` in `Cargo.toml`
* Fix unsoundness via impure AsRef
---------
Co-authored-by: Oleksandr Anyshchenko <aanischenko@gmail.com>
Co-authored-by: Zaidoon Abd Al Hadi <43054535+zaidoon1@users.noreply.github.com>
Support usages where the caller already has an environment built by an
alternative means, which the Env of this crate should take ownership of.
Co-authored-by: Zaidoon Abd Al Hadi <zaidoon@cloudflare.com>
* Improve statistics by auto gen enum Ticker & enum Histogram
1. librocksdb-sys/Makefile: Add gen_statistics and gen_statistics.bash,
make -C librocksdb-sys gen_statistics will generate 2 files:
src/statistics_enum_ticker.rs
src/statistics_enum_histogram.rs
2. In statistics.rs, we include! these 2 files and remove old hand
writing enum Tickers & enum Histograms.
3. File gen_statistics.bash is used for generating these 2 files
which is called in Makefile.
in CI, we specify the rust version as 1.70.0...
```yaml
on: [push, pull_request]
env:
RUST_VERSION: 1.70.0
```
likewise, the package manifest pins the rust version...
```toml
rust-version = "1.70.0"
```
..but this version is not pinned by cargo, via a `rust-toolchain`. this
means that testing this project using the stable toolchain by default
(_at time of writing, this is 1.78.0_) will see test failures due to
error messages not matching the expected snapshots, due to `rustc`
itself making changes to its diagnostics:
```
test tests/fail/snapshot_outlive_db.rs ... mismatch
EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0597]: `db` does not live long enough
--> tests/fail/snapshot_outlive_db.rs:6:9
|
4 | let _snapshot = {
| --------- borrow later stored here
5 | let db = DB::open_default("foo").unwrap();
| -- binding `db` declared here
6 | db.snapshot()
| ^^^^^^^^^^^^^ borrowed value does not live long enough
7 | };
| - `db` dropped here while still borrowed
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0597]: `db` does not live long enough
--> tests/fail/snapshot_outlive_db.rs:6:9
|
4 | let _snapshot = {
| --------- borrow later stored here
5 | let db = DB::open_default("foo").unwrap();
| -- binding `db` declared here
6 | db.snapshot()
| ^^ borrowed value does not live long enough
7 | };
| - `db` dropped here while still borrowed
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
your test with the environment variable TRYBUILD=overwrite
```
this commit introduces a `rust-toolchain.toml` file so that the correct
version of rust is used by those building or contributing to this
project.
