Pcap data is a wonderful resource; indispensible when doing certain kinds of analyses. Sometimes, however, you want to be able to use network flow data instead. Flow is much more compact, and allows you to more easily explore the relationships among hosts on a network before digging into the nuts and bolts with pcap. It also allows you to archive months or years of data in a much more efficient manner than you can with pcap.
But what if your environment doesn’t collect flow data by default? Don’t worry. There’s still a way to take advantage of what flow has to offer.
First, we’ll need the SiLK Security Suite from CERT. Installation is fairly simple. Start by installing the necesary libraries and tools:
sudo apt-get -y install libglib2.0-dev
sudo apt-get -y install libpcap-dev
sudo apt-get -y install python-dev
Next, download the software (be sure to download the current versions, which will be listed on the CERT website):
cd ~mkdir tmpcd tmpwget http://tools.netsa.cert.org/releases/silk-3.11.0.tar.gz
tar -zxvf libfixbuf-1.7.0.tar.gz
./configure && make
sudo make install
Next, install YAF:
tar -zxvf yaf-2.7.1.tar.gz
sudo make install
Finally, install SiLK:
tar -xvzf silk-3.11.0.tar.gz
sudo make install
cat <<EOF >>silk.conf
sudo mv silk.conf /etc/ld.so.conf.d/
Now that SiLK has been installed, it’s time to convert your pcap data into a SiLK repository that can be queried by the various tools in the SiLK suite.
In this example, assume we have a pcap file from an edge router. To convert it, we need to use YAF and
rwflowpack. In order to do that, we need to create both a
sensor.conf file and a
sensor.conf file describes how the data is being collected (or, in this case, has already been collected). It’s needed by
rwflowpack. Let’s call our pcap file
sensor.conf file would look something like this:
probe S0 ipfix
rwflowpack where to look for its data (in this case,
/home/tmp, which is where you should copy the pcap file). It defines the internal network as being anything in 10.0.0.0/8, and defines external traffic as anything else.
# The syntactic format of this file
# version 2 supports sensor descriptions, but
# otherwise identical to 1
sensor 0 S0 “Internal"
# Editing above this line is sufficient for sensor
# Be sure you understand the workings of the
# packing system before editing the class and
# type definitions below. In particular, if you
# change or add-to the following, the C code in
# packlogic-twoway.c will need to change as well.
type 0 in in
type 1 out out
type 2 inweb iw
type 3 outweb ow
type 4 innull innull
type 5 outnull outnull
type 6 int2int int2int
type 7 ext2ext ext2ext
type 8 inicmp inicmp
type 9 outicmp outicmp
type 10 other other
default-types in inweb inicmp
# The layout of the tree below SILK_DATA_ROOTDIR.
# Use the default, which assumes a single class.
# The plug-in to load to get the packing logic
# to use in rwflowpack.
# The --packing-logic switch to rwflowpack will
# override this value.
# If SiLK was configured with hard-coded packing
# logic, this value is ignored.
Now that these two files have been created, let’s create a directory structure to work in. Make a directory for the repository we’re creating, and a tmp directory for the intermediate stages. I’ll assume you’re in
cp example.pcap /home/tmp
cp *.conf /home/repository
We’re ready to convert the pcap data to flow:
yaf --silk --noerror --in=/home/tmp/example.pcap \
I recommend tailing the log file that’s created by
rwflowpack to know when it’s done, so you can kill the process once it finishes. You’ll know when that is because you’ll start seeing entries like:
Flushing files after 120 seconds.
Once it’s finished, you can test your new repository by trying various SiLK tools:
rwfilter --sensors=S0 --start=2015/01/10 --proto=0- --type=in --pass=test.rw
This, of course, assumes that the converted pcap data included data from Jan 10, 2015.