cc487ba367
Summary: ## Background Compaction statistics are collected at various levels across different classes and structs. * `InternalStats::CompactionStats`: Per-level Compaction Stats within a job (can be at subcompaction level which later get aggregated to the compaction level) * `InternalStats::CompactionStatsFull`: Contains two per-level compaction stats - `output_level_stats` for primary output level stats and `proximal_level_stats` for proximal level stats. Proximal level statistics are only relevant when using Tiered Storage with the per-key placement feature enabled. * `InternalStats::CompactionOutputsStats`: Simplified version of `InternalStats::CompactionStats`. Only has a subset of fields from `InternalStats::CompactionStats` * `CompactionJobStats`: Job-level Compaction Stats. (can be at subcompaction level which later get aggregated to the compaction level) Please note that some fields in Job-level stats are not in Per-level stats and they don't map 1-to-1 today. ## Issues * In non-remote compactions, proximal level compaction statistics were not being aggregated into job-level statistics. Job level statistics were missing stats for proximal level for tiered storage compactions with per-key-replacement feature enabled. * During remote compactions, proximal level compaction statistics were pre-aggregated into job-level statistics on the remote side. However, per-level compaction statistics were not part of the serialized compaction result, so that primary host lost that information and weren't able to populate `per_key_placement_comp_stats_` and `internal_stats_.proximal_level_stats` properly during the installation. * `TieredCompactionTest` was only checking if (expected stats > 0 && actual stats > 0) instead actual value comparison ## Fixes * Renamed `compaction_stats_` to `internal_stats_` for `InternalStats::CompactionStatsFull` in `CompactionJob` for better readability * Removed the usage of `InternalStats::CompactionOutputsStats` and consolidated them to `InternalStats::CompactionStats`. * Remote Compactions now include the internal stats in the serialized `CompactionServiceResult`. `output_level_stats` and `proximal_level_stats` get later propagated in sub_compact output stats accordingly. * `CompactionJob::UpdateCompactionJobStats()` now takes `CompactionStatsFull` and aggregates the `proximal_level_stats` as well * `TieredCompactionTest` is now doing the actual value comparisons for input/output file counts and record counts. Follow up is needed to do the same for the bytes read / written. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13464 Test Plan: Unit Tests updated to verify stats ``` ./compaction_service_test ``` ``` ./tiered_compaction_test ``` Reviewed By: pdillinger Differential Revision: D71220393 Pulled By: jaykorean fbshipit-source-id: ad70bffd9614ced683f90c7570a17def9b5c8f3f
46 lines
1.5 KiB
C++
46 lines
1.5 KiB
C++
// Copyright (c) Meta Platforms, Inc. and affiliates.
|
|
//
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#include "db/compaction/compaction_state.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
Slice CompactionState::SmallestUserKey() {
|
|
for (const auto& sub_compact_state : sub_compact_states) {
|
|
Slice smallest = sub_compact_state.SmallestUserKey();
|
|
if (!smallest.empty()) {
|
|
return smallest;
|
|
}
|
|
}
|
|
// If there is no finished output, return an empty slice.
|
|
return Slice{nullptr, 0};
|
|
}
|
|
|
|
Slice CompactionState::LargestUserKey() {
|
|
for (auto it = sub_compact_states.rbegin(); it < sub_compact_states.rend();
|
|
++it) {
|
|
Slice largest = it->LargestUserKey();
|
|
if (!largest.empty()) {
|
|
return largest;
|
|
}
|
|
}
|
|
// If there is no finished output, return an empty slice.
|
|
return Slice{nullptr, 0};
|
|
}
|
|
|
|
void CompactionState::AggregateCompactionStats(
|
|
InternalStats::CompactionStatsFull& internal_stats,
|
|
CompactionJobStats& job_stats) {
|
|
for (const auto& sc : sub_compact_states) {
|
|
sc.AggregateCompactionOutputStats(internal_stats);
|
|
job_stats.Add(sc.compaction_job_stats);
|
|
}
|
|
}
|
|
} // namespace ROCKSDB_NAMESPACE
|