Summary:
Add file size validation in ReadFooterFromFile function.
Deprecate skip_checking_sst_file_sizes_on_db_open option.
This change is used to address this issue
https://github.com/facebook/rocksdb/issues/13619
It supports file size validation in ReadFooterFromFile. In favor of this
change, CheckConsistency function and
skip_checking_sst_file_sizes_on_db_open flag are deprecated.
The CheckConsistency function checks each file size matches what was
recorded in manifest during DB open. Meantime, ReadFooterFromFile was
called for each file in LoadTables function. Since ReadFooterFromFile
always validates file size, the CheckConsistency is redundant.
In addtion, CheckConsistency is executed in a single thread. This could
slow down DB open when a network file system is used. Therefore, the
flag skip_checking_sst_file_sizes_on_db_open was added to skip this
check. After this change, ReadFooterFromFile was executed in parallel
through multiple threads. Therefore, the concern of DB open slowness is
eliminated, and the flag could be deprecated.
When paranoid check flag is set to true, corrupted file will fail to open the DB.
When paranoid check flag is set to false, DB will still be able to open, the
healthy ones can be accessed, while the corrupted ones not.
There is 2 slight concerns of this change.
*If max_open_files is set with smaller value, engine will not open all
the files during DB open. This means if there is a corruption on file
size, it will not be detected during DB open, but rather at a later
time. Since the default is -1, which means open all the files, and it is
rarely overridden and a lot of new features rely on it to be -1, the
risk is very low.
*If FIFO compaction is used, engine could fail to open DB unnecessarily
on the corrupted files that would never be used again. However, this is
a very rare case as well. The error could still be ignored by setting
paranoid_checks operationally. The risk is very low.
To remain backward compatibility. The public facing flag was kept and
marked as no-op internally. Another change is required to fully remove
the flag.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13676
Test Plan:
make check
A new unit test was added to validate file size check API works as
expected.
Reviewed By: pdillinger
Differential Revision: D76168033
Pulled By: xingbowang
fbshipit-source-id: 8ceacf39bcfe02ff7aa289868c341366ee9f3a8e
11a259a5f0