* Check that NewWritableFile succeeded when copying over backup files (#13734)
Summary:
I am seeing crashes during backups. The stack trace points back to `WritableFileWriter` creation inside `BackupEngineImpl::CopyOrCreateFile`. I believe the issue is that we are calling `writable_file_->GetRequiredBufferAlignment()` with a `null` `writable_file`.
https://github.com/facebook/rocksdb/blob/v10.2.1/utilities/backup/backup_engine.cc#L2396-L2397https://github.com/facebook/rocksdb/blob/v10.2.1/file/writable_file_writer.h#L210
Here's how I think the flow is:
```cpp
io_s = dst_env->GetFileSystem()->NewWritableFile(dst, dst_file_options,
&dst_file, nullptr);
// say there was some issue and dst_file is nullptr
// evaluates to false
if (io_s.ok() && !src.empty()) {
// we don't go down this branch
auto src_file_options = FileOptions(src_env_options);
src_file_options.temperature = *src_temperature;
io_s = src_env->GetFileSystem()->NewSequentialFile(src, src_file_options,
&src_file, nullptr);
}
// say this evaluates to true
if (io_s.IsPathNotFound() && *src_temperature != Temperature::kUnknown) {
// Retry without temperature hint in case the FileSystem is strict with
// non-kUnknown temperature option
io_s = src_env->GetFileSystem()->NewSequentialFile(
src, FileOptions(src_env_options), &src_file, nullptr);
}
// this is now from the NewSequentialFile call, not NewWritableFile
if (!io_s.ok()) {
return io_s;
}
// dst_file is still nullptr
```
If the first `NewWritableFile` fails and `IsPathNotFound
Tests: existing unit tests
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13734
Reviewed By: pdillinger
Differential Revision: D77390694
Pulled By: archang19
fbshipit-source-id: 865a3a646079ae2349a3b6f25e53ae85df8e4985
* Add a new periodic task to trigger compactions (#13736)
Summary:
address an existing limitation on compaction triggering mechanism that relies on events like flush/compaction/SetOptions. This is important for periodic compactions where files can become eligible without any of these events. The periodic task now runs every 12 hours and check CFs that enables `periodic_compaction_second` (TBD if we want to expand to all CFs) for eligible compactions.
Some of the periodic tasks probably don't need to run immediately after Register(). I'm keeping the existing behavior for now for patch release and to makes tests happy.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13736
Test Plan:
- new unit test that fails before this change.
- ran crash test for hours with the periodic task running every 5 seconds: `python3 ./tools/db_crashtest.py blackbox --test_batches_snapshot=0 --periodic_compaction_seconds=10`
Reviewed By: pdillinger
Differential Revision: D77460715
Pulled By: cbi42
fbshipit-source-id: 00f61502753185e76830c9ed44c5ccc4f4f16bfa
* update history and version for 10.4.1
* Update HISTORY.md
Co-authored-by: Andrew Chang <39173193+archang19@users.noreply.github.com>
---------
Co-authored-by: Andrew Chang <andrewrchang@meta.com>
Co-authored-by: Andrew Chang <39173193+archang19@users.noreply.github.com>
3df9173a8c