91f227d9be
Summary:
Add a circular ring buffer (InjectedErrorLog) to FaultInjectionTestFS that records the last 1000 injected errors. Each entry captures the timestamp, thread ID, FS API name with all arguments (file path, offset, buffer size, first 8 bytes of data in hex), and the injected error status. For example:
Append("/path/035354.sst", size=4096, head=[1a 2b 3c ...]) -> IOError (Retryable): injected write error
RenameFile("/path/tmp.sst", "/path/035355.sst") -> IOError: injected metadata write error
Read(offset=16384, size=4096) -> IOError (Retryable): injected read error
The ring buffer is printed to a file automatically:
- On any fatal signal (SIGABRT, SIGSEGV, SIGTERM, SIGINT, SIGHUP, SIGFPE, SIGBUS, SIGILL, SIGQUIT, SIGXCPU, SIGXFSZ, SIGSYS) via a registered crash callback
- At the end of db_stress main(), for diagnostic visibility even when the test completes normally
This addresses a key debugging gap: when write fault injection causes secondary failures (e.g., the builder error propagation issue in T257612259), the injected errors were previously completely silent with no logging trail. The ring buffer provides the missing diagnostic context to correlate fault injection with downstream failures.
Changes:
- port/stack_trace.h/.cc: Add RegisterCrashCallback() API; extend InstallStackTraceHandler() to catch all catchable termination signals
- utilities/fault_injection_fs.h: Add InjectedErrorLog class with printf-style Record(), HexHead() for data bytes, and signal-safe PrintAll()
- utilities/fault_injection_fs.cc: Record full API arguments and error status at all 31 fault injection call sites
- db_stress_tool/db_stress_tool.cc: Register crash callback and print ring buffer at end of main()
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14431
Reviewed By: hx235
Differential Revision: D95435430
Pulled By: xingbowang
fbshipit-source-id: 6c18e1b072044575d6c8c3f198070127b0f80608
|
||
|---|---|---|
| .. | ||
| win | ||
| jemalloc_helper.h | ||
| lang.h | ||
| likely.h | ||
| malloc.h | ||
| mmap.cc | ||
| mmap.h | ||
| port.h | ||
| port_dirent.h | ||
| port_example.h | ||
| port_posix.cc | ||
| port_posix.h | ||
| README | ||
| stack_trace.cc | ||
| stack_trace.h | ||
| sys_time.h | ||
| util_logger.h | ||
| xpress.h | ||
This directory contains interfaces and implementations that isolate the rest of the package from platform details. Code in the rest of the package includes "port.h" from this directory. "port.h" in turn includes a platform specific "port_<platform>.h" file that provides the platform specific implementation. See port_posix.h for an example of what must be provided in a platform specific header file.