Fault Injection: A New Perspective

Dmitry Vostokov, 15th June 2009
http://www.dumpanalysis.org

Why should we be interested in constructing (or modeling) software defects? Besides fun and developing debugging exercises there is a scientific side to it: we can construct software bugs to check hypotheses when troubleshooting and debugging complex and hard to reproduce issues. The closest term is called “fault injection”. When working on this issue I ac­cidentally discovered this now out-of print book published more than 10 years ago:

SoftwareFaultInjection_book.jpg

Software Fault Injection:
Inoculating Programs Against Errors
ISBN: 978-0471183815

Software defect construction is a more general term than fault injection. The latter is used for testing but we want to simulate bugs and abnormal system conditions to study debugging and memory dump analysis techniques or to build reproduction environments. In this issue, Kapildev Ramlal, a senior Citrix escalation engineer, recollects his experience with DLL injection tools he wrote to effec­tively and efficiently debug numerous problems in ter­minal services environments. Then Konstantin Chebotarev, a Citrix development analysis engineer, shows us how to construct a CPU spike and finally I discuss how to model a runaway exception handling.

There are many tools that inject faults, for example NotMyFault sysinternals tool[1]. It is very useful for studying crash dump analysis of kernel memory dumps and minidumps. Another use for these tools is to check certain debugging configurations like a postmortem debugger or WER settings are set up correctly. Typical example for user space is TestDefaultDebugger package[2]. Generalizing above, we can say that injecting bugs helps to test troubleshooting methods and tools. This is what I call metatroubleshooting, troubleshooting and debugging troubleshooting and debugging tools (DebugWare).

Creating software defects deliberately is entertaining and has great educational value for understanding internals and doing deep down debugging. I’m currently working on the following title:

Software Defect Construction: Simulation and Modeling of Software Bugs ISBN: 978-1906717759

The book will cover all spectrum of software faults including kernel space bugs, implementation and design defects, provide debugging suggestions and compilations of resources.

[1] http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx
[2] http://support.citrix.com/article/ctx111901