02bce9b1af
Summary: **Context:** RocksDB currently selects files for long-running compaction outputs to the bottommost level, preventing these selected files files from being selected, but does not execute the compaction immediately like other compactions. Instead, this compaction is forwarded to another Env::Priority::bottom thread pool, where it waits (potentially for a long time) until its thread is ready to execute. This extended L0 lock time in universal compaction caused our users write stall and read performance regression. **Summary:** This PR is to eliminate L0 lock time during bottom priority compaction waiting to execute by the following - Create and forward an intended compaction only consists of last input file (or sorted run if non-L0) instead of all the input files. This eliminate the locking for non-bottommost level input files while waiting for bottom priority thread is up to run. - Re-pick compaction that outputs to max output level when bottom priority thread is up to run - Refactor universal compaction picking logic to make it cleaner and easier to force picking compaction with max output level when bottom priority thread is up to run - Guard feature behind a temporary option as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/13633 Test Plan: - New unit test to cover the case that's not covered by existing tests - bottom priority thread re-picks compaction ends up picking nothing due to LSM shape changes - Adapted existing unit tests to verify various bottom priority compaction behavior with this new option - Stress test `python3 tools/db_crashtest.py --simple blackbox --compaction_style=1 --target_file_size_base=1000 --write_buffer_size=1000 --compact_range_one_in=10000 --compact_files_one_in=10000 ` Reviewed By: cbi42 Differential Revision: D76005505 Pulled By: hx235 fbshipit-source-id: 9688f22d4a84f619452820f12f15b765c17301fd
64 lines
3 KiB
C++
64 lines
3 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#pragma once
|
|
|
|
#include "db/compaction/compaction_picker.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
class FIFOCompactionPicker : public CompactionPicker {
|
|
public:
|
|
FIFOCompactionPicker(const ImmutableOptions& ioptions,
|
|
const InternalKeyComparator* icmp)
|
|
: CompactionPicker(ioptions, icmp) {}
|
|
|
|
Compaction* PickCompaction(
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
const MutableDBOptions& mutable_db_options,
|
|
const std::vector<SequenceNumber>& /* existing_snapshots */,
|
|
const SnapshotChecker* /* snapshot_checker */,
|
|
VersionStorageInfo* version, LogBuffer* log_buffer,
|
|
bool /* require_max_output_level*/ = false) override;
|
|
|
|
Compaction* CompactRange(const std::string& cf_name,
|
|
const MutableCFOptions& mutable_cf_options,
|
|
const MutableDBOptions& mutable_db_options,
|
|
VersionStorageInfo* vstorage, int input_level,
|
|
int output_level,
|
|
const CompactRangeOptions& compact_range_options,
|
|
const InternalKey* begin, const InternalKey* end,
|
|
InternalKey** compaction_end, bool* manual_conflict,
|
|
uint64_t max_file_num_to_ignore,
|
|
const std::string& trim_ts) override;
|
|
|
|
// The maximum allowed output level. Always returns 0.
|
|
int MaxOutputLevel() const override { return 0; }
|
|
|
|
bool NeedsCompaction(const VersionStorageInfo* vstorage) const override;
|
|
|
|
private:
|
|
Compaction* PickTTLCompaction(const std::string& cf_name,
|
|
const MutableCFOptions& mutable_cf_options,
|
|
const MutableDBOptions& mutable_db_options,
|
|
VersionStorageInfo* version,
|
|
LogBuffer* log_buffer);
|
|
|
|
Compaction* PickSizeCompaction(const std::string& cf_name,
|
|
const MutableCFOptions& mutable_cf_options,
|
|
const MutableDBOptions& mutable_db_options,
|
|
VersionStorageInfo* version,
|
|
LogBuffer* log_buffer);
|
|
|
|
// Will pick one file to compact at a time, starting from the oldest file.
|
|
Compaction* PickTemperatureChangeCompaction(
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
|
|
LogBuffer* log_buffer) const;
|
|
};
|
|
} // namespace ROCKSDB_NAMESPACE
|