Pcap data is a wonderful resource; indispensible when doing certain kinds of analyses. Sometimes, however, you want to be able to use network flow data instead. Flow is much more compact, and allows you to more easily explore the relationships among hosts on a network before digging into the nuts and bolts with pcap. It also allows you to archive months or years of data in a much more efficient manner than you can with pcap.
But what if your environment doesn’t collect flow data by default? Don’t worry. There’s still a way to take advantage of what flow has to offer.
First, we’ll need the SiLK Security Suite from CERT. Installation is fairly simple. Start by installing the necesary libraries and tools:
sudo apt-get -y install libglib2.0-dev sudo apt-get -y install libpcap-dev sudo apt-get -y install python-dev
Next, download the software (be sure to download the current versions, which will be listed on the CERT website):
cd ~mkdir tmpcd tmpwget http://tools.netsa.cert.org/releases/silk-3.11.0.tar.gz wget http://tools.netsa.cert.org/releases/libfixbuf-1.7.0.tar.gz wget http://tools.netsa.cert.org/releases/yaf-2.7.1.tar.gz
cd ~/tmp tar -zxvf libfixbuf-1.7.0.tar.gz cd libfixbuf-1.7.0 ./configure && make sudo make install
Next, install YAF:
cd ~/tmp tar -zxvf yaf-2.7.1.tar.gz cd yaf-2.7.1 export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure --enable-applabel make sudo make install
Finally, install SiLK:
cd ~/tmp tar -xvzf silk-3.11.0.tar.gz cd silk-3.11.0 ./configure \ --with-libfixbuf=/usr/local/lib/pkgconfig/ --with-python make sudo make install cat <<EOF >>silk.conf /usr/local/lib /usr/local/lib/silk EOF sudo mv silk.conf /etc/ld.so.conf.d/
Now that SiLK has been installed, it’s time to convert your pcap data into a SiLK repository that can be queried by the various tools in the SiLK suite.
In this example, assume we have a pcap file from an edge router. To convert it, we need to use YAF and
rwflowpack. In order to do that, we need to create both a
sensor.conf file and a
sensor.conf file describes how the data is being collected (or, in this case, has already been collected). It’s needed by
rwflowpack. Let’s call our pcap file
sensor.conf file would look something like this:
probe S0 ipfix poll-directory /home/tmp end probe group my-networkS0 ipblocks 10.0.0.0/8 end group sensor S0 ipfix-probes S0 internal-ipblocks @my-networkS0 external-ipblocks remainder end sensor
rwflowpack where to look for its data (in this case,
/home/tmp, which is where you should copy the pcap file). It defines the internal network as being anything in 10.0.0.0/8, and defines external traffic as anything else.
# silk.conf # The syntactic format of this file # version 2 supports sensor descriptions, but # otherwise identical to 1 version 2 sensor 0 S0 “Internal" class all sensors S0 end class # Editing above this line is sufficient for sensor # definition. # Be sure you understand the workings of the # packing system before editing the class and # type definitions below. In particular, if you # change or add-to the following, the C code in # packlogic-twoway.c will need to change as well. class all type 0 in in type 1 out out type 2 inweb iw type 3 outweb ow type 4 innull innull type 5 outnull outnull type 6 int2int int2int type 7 ext2ext ext2ext type 8 inicmp inicmp type 9 outicmp outicmp type 10 other other default-types in inweb inicmp end class default-class all # The layout of the tree below SILK_DATA_ROOTDIR. # Use the default, which assumes a single class. path-format "%N/%T/%Y/%m/%d/%x" # The plug-in to load to get the packing logic # to use in rwflowpack. # The --packing-logic switch to rwflowpack will # override this value. # If SiLK was configured with hard-coded packing # logic, this value is ignored. packing-logic "packlogic-twoway.so"
Now that these two files have been created, let’s create a directory structure to work in. Make a directory for the repository we’re creating, and a tmp directory for the intermediate stages. I’ll assume you’re in
mkdir /home/tmp mkdir /home/repository cp example.pcap /home/tmp cp *.conf /home/repository
We’re ready to convert the pcap data to flow:
yaf --silk --noerror --in=/home/tmp/example.pcap \ --out=/home/tmp/example.yaf rwflowpack \ --sensor-conf=/home/repository/sensor.conf \ --site-config-file=/home/repository/silk.conf \ --root-directory=/home/repository \ --log-directory=/home/repository
I recommend tailing the log file that’s created by
rwflowpack to know when it’s done, so you can kill the process once it finishes. You’ll know when that is because you’ll start seeing entries like:
Flushing files after 120 seconds.
Once it’s finished, you can test your new repository by trying various SiLK tools:
export SILK_DATA_ROOTDIR=/home/repository/ export SILK_CONFIG_FILE=/home/repository/silk.conf rwfilter --sensors=S0 --start=2015/01/10 --proto=0- --type=in --pass=test.rw rwcut test.rw
This, of course, assumes that the converted pcap data included data from Jan 10, 2015.