continuwuation/rocksdb
12
0
Fork 1
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity
rocksdb/options/db_options.h
Danny Chen b6f498b2c9 Add verify_manifest_content_on_close option (#14451) 
Summary:
Add a new mutable DB option `verify_manifest_content_on_close` (default: false).
When enabled, on DB close the MANIFEST file is read back and all records are
validated (CRC checksums via log::Reader and logical content via
VersionEdit::DecodeFrom). If corruption is detected, a fresh MANIFEST is written
from in-memory state using the existing LogAndApply recovery path.

This complements the existing size validation in VersionSet::Close() with content
validation, reusing the same manifest reading pattern as VersionSet::Recover().

Implementation plan:

## Part 1: New DB Option — verify_manifest_content_on_close
- A new mutable bool DB option (default: false) that can be dynamically toggled
  via SetDBOptions() at runtime, following the pattern of other mutable manifest
  options like max_manifest_file_size.
- Propagation: SetDBOptions() -> DBImpl::mutable_db_options_ ->
  versions_->UpdatedMutableDbOptions() -> VersionSet::verify_manifest_content_on_close_

## Part 2: Core Implementation — Content Validation in VersionSet::Close()
- Inserted after existing size check, before closed_ = true
- Opens manifest as SequentialFileReader, creates log::Reader with checksum=true
- Loops ReadRecord with WALRecoveryMode::kAbsoluteConsistency, decodes each
  record as VersionEdit
- On corruption: fires OnIOError listeners, logs error, calls LogAndApply with
  empty edit to trigger manifest rewrite from in-memory state
- If manifest can't be opened for reading: logs warning, doesn't fail close

## Part 3: Unit Tests (in version_set_test.cc)
- ManifestContentValidationOnClose_Clean: enable option, normal close, verify
  no manifest rotation
- ManifestContentValidationOnClose_CorruptRecord: enable option, corrupt manifest
  via SyncPoint, verify rotation occurs and DB reopens cleanly
- ManifestContentValidationOnClose_Disabled: default off, verify content
  validation does not run
- ManifestContentValidationOnClose_SizeCheckFails: truncate manifest so size
  check fails first, verify recovery via size-check path

## What Happens If a Corruption is Detected
If corruption was detected, four things happen:
1. **Notify listeners** — Fires `OnIOError` on all registered event listeners
   (from db_options_->listeners) so monitoring/alerting systems can observe
   the corruption event. Uses `FileOperationType::kVerify` to categorize it.
2. **Permit unchecked errors** — `PermitUncheckedError()` silences RocksDB's
   debug-mode assertion that every `IOStatus` must be inspected. These statuses
   are informational-only here; the real recovery is via `LogAndApply`.
3. **Log the error** — Writes a `ROCKS_LOG_ERROR` message with the filename
   for operational visibility (grep-able in production logs).
4. **Rewrite the manifest via `LogAndApply`** — This is the actual recovery.
   `LogAndApply` is called with an empty `VersionEdit` (no changes). Internally,
   `LogAndApply` detects that the current `descriptor_log_` is null (it was
   reset at line 5551, or by the previous `LogAndApply` in the size-check
   path) and creates a brand-new MANIFEST file. It serializes the entire
   current in-memory LSM state — all column families, all levels, all file
   metadata, sequence numbers, etc. — into this new file. It then atomically
   updates the `CURRENT` file pointer to reference the new MANIFEST.
   This works because the in-memory state was built from the original manifest
   during `DB::Open()` and has been kept fully up to date through all
   subsequent operations (flushes, compactions, etc.) during the DB's lifetime.
   The on-disk manifest is essentially a journal of changes; `LogAndApply`
   with an empty edit produces a fresh, compacted snapshot of that state.

## Flow Diagram of Manifest Content Validation

VersionSet::Close()
│
├─ Close descriptor_log_ and check size
│  └─ Size mismatch? → LogAndApply (rewrite manifest)
│
├─ Content validation (if s.ok() && option enabled)
│  ├─ Open manifest for sequential reading
│  │  └─ Can't open? → WARN log, continue
│  │
│  ├─ For each record:
│  │  ├─ ReadRecord (CRC32 check, kAbsoluteConsistency)
│  │  └─ DecodeFrom (VersionEdit logical check)
│  │
│  └─ Corruption detected?
│     ├─ Notify OnIOError listeners
│     ├─ LOG_ERROR
│     └─ LogAndApply (rewrite manifest from in-memory state)
│
└─ closed_ = true; return s;

## How This Relates to the Existing Size Check
The existing size check (lines 5556-5582) and the new content validation are
complementary:
| Check          | What it catches                         | How it checks              |
|----------------|-----------------------------------------|----------------------------|
| Size check     | Truncation, partial writes, extra bytes | Compare expected vs actual file size |
| Content check  | Bit-rot, silent corruption, bad records | CRC32 + VersionEdit decode |
The size check catches gross corruption (file too short or too long). The
content check catches subtle corruption where the file is the right size but
individual bytes have been flipped (e.g., storage media bit-rot, buggy
filesystem, incomplete block write).
Both recovery paths use the same mechanism: `LogAndApply` with an empty
`VersionEdit` to rewrite the manifest from in-memory state.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14451

Reviewed By: xingbowang

Differential Revision: D96004906

Pulled By: dannyhchen

fbshipit-source-id: 0b0ecdada3a74e97d2cadbba2091b8b577f1d684
2026-03-19 12:01:23 -07:00

167 lines
5.5 KiB
C++
Raw Permalink Blame History

// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#include <string>
#include <vector>
#include "rocksdb/options.h"
namespace ROCKSDB_NAMESPACE {
class SystemClock;
struct ImmutableDBOptions {
static const char* kName() { return "ImmutableDBOptions"; }
ImmutableDBOptions();
explicit ImmutableDBOptions(const DBOptions& options);
void Dump(Logger* log) const;
bool create_if_missing;
bool create_missing_column_families;
bool error_if_exists;
bool paranoid_checks;
bool open_files_async;
bool flush_verify_memtable_count;
bool compaction_verify_record_count;
bool track_and_verify_wals_in_manifest;
bool track_and_verify_wals;
bool verify_sst_unique_id_in_manifest;
Env* env;
std::shared_ptr<RateLimiter> rate_limiter;
std::shared_ptr<SstFileManager> sst_file_manager;
std::shared_ptr<Logger> info_log;
InfoLogLevel info_log_level;
int max_file_opening_threads;
std::shared_ptr<Statistics> statistics;
bool use_fsync;
std::vector<DbPath> db_paths;
std::string db_log_dir;
// The wal_dir option from the file. To determine the
// directory in use, the GetWalDir or IsWalDirSameAsDBPath
// methods should be used instead of accessing this variable directly.
std::string wal_dir;
size_t max_log_file_size;
size_t log_file_time_to_roll;
size_t keep_log_file_num;
size_t recycle_log_file_num;
int table_cache_numshardbits;
uint64_t WAL_ttl_seconds;
uint64_t WAL_size_limit_MB;
uint64_t max_write_batch_group_size_bytes;
bool allow_mmap_reads;
bool allow_mmap_writes;
bool use_direct_reads;
bool use_direct_io_for_flush_and_compaction;
bool allow_fallocate;
bool is_fd_close_on_exec;
bool advise_random_on_open;
size_t db_write_buffer_size;
std::shared_ptr<WriteBufferManager> write_buffer_manager;
bool use_adaptive_mutex;
std::vector<std::shared_ptr<EventListener>> listeners;
bool enable_thread_tracking;
bool enable_pipelined_write;
bool unordered_write;
bool allow_concurrent_memtable_write;
bool enable_write_thread_adaptive_yield;
uint64_t write_thread_max_yield_usec;
uint64_t write_thread_slow_yield_usec;
bool skip_stats_update_on_db_open;
WALRecoveryMode wal_recovery_mode;
bool allow_2pc;
std::shared_ptr<Cache> row_cache;
WalFilter* wal_filter;
bool dump_malloc_stats;
bool avoid_flush_during_recovery;
bool enforce_write_buffer_manager_during_recovery;
bool allow_ingest_behind;
bool two_write_queues;
bool manual_wal_flush;
CompressionType wal_compression;
bool background_close_inactive_wals;
bool atomic_flush;
bool avoid_unnecessary_blocking_io;
bool prefix_seek_opt_in_only;
bool persist_stats_to_disk;
bool write_dbid_to_manifest;
bool write_identity_file;
size_t log_readahead_size;
std::shared_ptr<FileChecksumGenFactory> file_checksum_gen_factory;
bool best_efforts_recovery;
int max_bgerror_resume_count;
uint64_t bgerror_resume_retry_interval;
bool allow_data_in_errors;
std::string db_host_id;
FileTypeSet checksum_handoff_file_types;
CacheTier lowest_used_cache_tier;
std::shared_ptr<CompactionService> compaction_service;
bool enforce_single_del_contracts;
uint64_t follower_refresh_catchup_period_ms;
uint64_t follower_catchup_retry_count;
uint64_t follower_catchup_retry_wait_ms;
Temperature metadata_write_temperature;
Temperature wal_write_temperature;
CompactionStyleSet calculate_sst_write_lifetime_hint_set;
// Beginning convenience/helper objects that are not part of the base
// DBOptions
std::shared_ptr<FileSystem> fs;
SystemClock* clock;
Statistics* stats;
Logger* logger;
// End of convenience/helper objects.
bool IsWalDirSameAsDBPath() const;
bool IsWalDirSameAsDBPath(const std::string& path) const;
const std::string& GetWalDir() const;
const std::string& GetWalDir(const std::string& path) const;
};
struct MutableDBOptions {
static const char* kName() { return "MutableDBOptions"; }
MutableDBOptions();
explicit MutableDBOptions(const DBOptions& options);
void Dump(Logger* log) const;
int max_background_jobs;
int max_background_compactions;
uint32_t max_subcompactions;
bool avoid_flush_during_shutdown;
size_t writable_file_max_buffer_size;
uint64_t delayed_write_rate;
uint64_t max_total_wal_size;
uint64_t delete_obsolete_files_period_micros;
unsigned int stats_dump_period_sec;
unsigned int stats_persist_period_sec;
size_t stats_history_buffer_size;
int max_open_files;
uint64_t bytes_per_sync;
uint64_t wal_bytes_per_sync;
bool strict_bytes_per_sync;
size_t compaction_readahead_size;
int max_background_flushes;
uint64_t max_manifest_file_size;
int max_manifest_space_amp_pct;
size_t manifest_preallocation_size;
bool verify_manifest_content_on_close;
std::string daily_offpeak_time_utc;
};
Status GetStringFromMutableDBOptions(const ConfigOptions& config_options,
const MutableDBOptions& mutable_opts,
std::string* opt_string);
Status GetMutableDBOptionsFromStrings(
const MutableDBOptions& base_options,
const std::unordered_map<std::string, std::string>& options_map,
MutableDBOptions* new_options);
bool MutableDBOptionsAreEqual(const MutableDBOptions& this_options,
const MutableDBOptions& that_options);
} // namespace ROCKSDB_NAMESPACE
Reference in a new issue View git blame Copy permalink