With network security threats and vulnerabilities increasing, approaches based on online detection remain attractive. A complete, permanent record of all activity on a subnet can be used to evaluate and train intrusion detection algorithms, assist in responding to an intrusion in progress, and, if properly constructed, serve as forensic evidence in legal proceedings.
CITI has built a prototype of a cryptographically secured archiver of network packet data. This prototype Packet Vault writes captured network packets to long-term CD-ROM storage using strong encryption for later analysis and for evidentiary purposes. The cryptographic organization of the Vault permits selected traffic to be made available without revealing other traffic, by encrypting each packet with a key dependent on its source and destination IP addresses. Using commodity hardware, the prototype operates with a 10 Mbps network but requires excessive manual supervision. See Antonelli et al [AUH99] for details.
In the first year, we propose to build a robust 10 Mbps Vault. Our goal for the second and third years is to extend the capacity of the Vault as far as technology permits. Both implementations will be open-source, based either on Linux or OpenBSD.
Imagine a Packet Vault capturing and storing the traffic on a heavily loaded 100 Mbps network. The challenge is to capture, process, and store about a terabyte each day. We have to be very sensitive to the recurring cost of operation, which includes personnel costs for system operation and maintenance, storage costs for media, and the cost of the media itself. Our target is the ability to store a year's worth of Vault data in a cubic meter, at a cost of $50,000 for physical media. These targets will be realized in only a few years.
Our targets translate into $0.135 and 2.7 cc per GB. According to Gray and Shenoy [GS00], storage cost is improving by a factor of four every three years. Today's digital tape technology costs $1.5/GB. If the cost for storage media falls by a factor of four in the next three years, this will result in an annual cost of almost $140,000, a dominating and forbidding price tag. Under certain assumptions, though, such as compressibility of the raw network traffic, volume discounts for tape cartridges purchased by the thousands, and the emergence of unconventional storage media such as optical tape, we might anticipate this cost to fall by an additional factor of two to four. This would fall within our cost target.
The picture is brighter for the physical size of storage. Today's DLT tapes already achieve the physical target, storing 40 GB in about 10 cc of space, or 0.25 cc/GB.
In the first phase of work, we will develop a production 10 Mbps Vault, leveraging our experiences with the Vault prototype. Improvements in performance and reliability will be achieved by porting the prototype to high-capacity hardware, including available hardware encryption devices and mass storage output devices, and by collapsing the Vault architecture onto a single host. The goal is to permit the Vault to archive all traffic found on a fully-loaded 10 Mbps network segment, and to permit the Vault to run for extended periods of time without supervision.
In the second phase, we propose to extend the design of the Vault to allow operation in 100 Mbps environments. At such speeds, simply scaling up the current Vault architecture may not be feasible, and will require use of robust mass storage technologies to deal with data volumes of about a terabyte per day.
In the third phase, we will extend the Vault design beyond 100 Mbps as far as current technology permits. At such speeds, scaling the current architecture will not be feasible, requiring instead the investigation of: parallelism in the data pipeline, multiple mass storage devices, a parallel architecture in which groups of Vault engines cooperate to cover a high-speed network, and distributing packets equitably among the available engines.
In addition to the above system engineering tasks, additional issues to be investigated include:
Phase 1 - 10 Mbps APVSource code for the first phase of our project is available as a compressed tar file apv10.tar.gz.
An operations document assisting with installing and operating the 10 Mbps APV is available here.
Phase 2 - 100 Mbps APVSource code for the second phase of our project is available as a compressed tar file apv100.tar.gz.
An operations document assisting with installing and operating the 100 Mbps APV is available here.
Project Sponsor (Phase 1)
Institute for Security Technology Studies, Dartmouth College