While an incident response is in progress, the CERT team should be investigating while working toward data and system recovery. Unfortunately, it is the nature of the breach which dictates the course of recovery. Having backups or offline, redundant systems during this time is invaluable.
To recover systems, the response team must bring any downed systems or applications back online, such as authentication servers, database servers, and any other production resources.
Having production backup hardware ready for use is highly recommended, such as extra hard drives, hot-spare servers, and the like. Ready-made systems should have all production software loaded and ready for immediate use. Only the most recent and pertinent data needs to be imported. This ready-made system should be kept isolated from the rest of the network. If a compromise occurs and the backup system is a part of the network, then the purpose of having a backup system is defeated.
System recovery can be a tedious process. In many instances there are two courses of action from which to choose. Administrators can perform a clean re-installation of the operating system on each affected system followed by restoration of all applications and data. Alternatively, administrators can patch the offending vulnerabilities and bring the affected system back into production.
Performing a clean re-installation ensures that the affected system is cleansed of any trojans, backdoors, or malicious processes. Re-installation also ensures that any data (if restored from a trusted backup source) is cleared of any malicious modifications. The drawback to total system recovery is the time involved in rebuilding systems from scratch. However, if there is a hot backup system available for use where the only action to take is to dump the most recent data, system downtime is greatly reduced.
Patching affected systems is a more dangerous course of action and should be undertaken with great caution. The problem with patching a system instead of reinstalling is determining whether or not a given system is cleansed of trojans, security holes, and corrupted data. Most rootkits (programs or packages that a cracker uses to gain root access to a system), trojan system commands, and shell environments are designed to hide malicious activities from cursory audits. If the patch approach is taken, only trusted binaries should be used (for example, from a mounted, read-only CD-ROM).