Converting pcap files to network flow data

Pcap data is a wonderful resource; indispensible when doing certain kinds of analyses. Sometimes, however, you want to be able to use network flow data instead. Flow is much more compact, and allows you to more easily explore the relationships among hosts on a network before digging into the nuts and bolts with pcap. It also allows you to archive months or years of data in a much more efficient manner than you can with pcap.

But what if your environment doesn’t collect flow data by default? Don’t worry. There’s still a way to take advantage of what flow has to offer.

First, we’ll need the SiLK Security Suite from CERT. Installation is fairly simple. Start by installing the necesary libraries and tools:

sudo apt-get -y install libglib2.0-dev
sudo apt-get -y install libpcap-dev
sudo apt-get -y install python-dev

Next, download the software (be sure to download the current versions, which will be listed on the CERT website):

cd ~mkdir tmpcd tmpwget http://tools.netsa.cert.org/releases/silk-3.11.0.tar.gz

wget http://tools.netsa.cert.org/releases/libfixbuf-1.7.0.tar.gz

wget http://tools.netsa.cert.org/releases/yaf-2.7.1.tar.gz

Now, install fixbuf:

cd ~/tmp
tar -zxvf libfixbuf-1.7.0.tar.gz
cd libfixbuf-1.7.0
./configure && make
sudo make install

Next, install YAF:

cd ~/tmp
tar -zxvf yaf-2.7.1.tar.gz
cd yaf-2.7.1
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
./configure --enable-applabel
make
sudo make install

Finally, install SiLK:

cd ~/tmp
tar -xvzf silk-3.11.0.tar.gz
cd silk-3.11.0
./configure \
--with-libfixbuf=/usr/local/lib/pkgconfig/ --with-python
make
sudo make install

cat <<EOF >>silk.conf
/usr/local/lib
/usr/local/lib/silk
EOF

sudo mv silk.conf /etc/ld.so.conf.d/

Now that SiLK has been installed, it’s time to convert your pcap data into a SiLK repository that can be queried by the various tools in the SiLK suite.

In this example, assume we have a pcap file from an edge router. To convert it, we need to use YAF and rwflowpack. In order to do that, we need to create both a sensor.conf file and a silk.conf file.

The sensor.conf file describes how the data is being collected (or, in this case, has already been collected). It’s needed by rwflowpack. Let’s call our pcap file example.pcap. The sensor.conf file would look something like this:

probe S0 ipfix
    poll-directory /home/tmp
end probe

group my-networkS0
    ipblocks 10.0.0.0/8
end group

sensor S0
    ipfix-probes S0
    internal-ipblocks @my-networkS0
    external-ipblocks remainder
end sensor

This tells rwflowpack where to look for its data (in this case, /home/tmp, which is where you should copy the pcap file). It defines the internal network as being anything in 10.0.0.0/8, and defines external traffic as anything else.

Now, the silk.conf file:

# silk.conf 

# The syntactic format of this file
#    version 2 supports sensor descriptions, but
# otherwise identical to 1
version 2

sensor 0 S0    “Internal"

class all
    sensors S0
end class

# Editing above this line is sufficient for sensor
# definition.

# Be sure you understand the workings of the 
# packing system before editing the class and 
# type definitions below. In particular, if you
# change or add-to the following, the C code in
# packlogic-twoway.c will need to change as well.

class all
    type  0 in      in
    type  1 out     out
    type  2 inweb   iw
    type  3 outweb  ow
    type  4 innull  innull
    type  5 outnull outnull
    type  6 int2int int2int
    type  7 ext2ext ext2ext
    type  8 inicmp  inicmp
    type  9 outicmp outicmp
    type 10 other   other

    default-types in inweb inicmp
end class

default-class all

# The layout of the tree below SILK_DATA_ROOTDIR.
# Use the default, which assumes a single class.
path-format "%N/%T/%Y/%m/%d/%x"

# The plug-in to load to get the packing logic 
# to use in rwflowpack.
# The --packing-logic switch to rwflowpack will
# override this value.
# If SiLK was configured with hard-coded packing
# logic, this value is ignored. 
packing-logic "packlogic-twoway.so"

Now that these two files have been created, let’s create a directory structure to work in. Make a directory for the repository we’re creating, and a tmp directory for the intermediate stages. I’ll assume you’re in /home currently:

mkdir /home/tmp

mkdir /home/repository

cp example.pcap /home/tmp

cp *.conf /home/repository

We’re ready to convert the pcap data to flow:

yaf --silk --noerror --in=/home/tmp/example.pcap \
--out=/home/tmp/example.yaf

rwflowpack \
--sensor-conf=/home/repository/sensor.conf \
--site-config-file=/home/repository/silk.conf \
--root-directory=/home/repository \
--log-directory=/home/repository

I recommend tailing the log file that’s created by rwflowpack to know when it’s done, so you can kill the process once it finishes. You’ll know when that is because you’ll start seeing entries like:

Flushing files after 120 seconds.

Once it’s finished, you can test your new repository by trying various SiLK tools:

export SILK_DATA_ROOTDIR=/home/repository/

export SILK_CONFIG_FILE=/home/repository/silk.conf 

rwfilter --sensors=S0 --start=2015/01/10 --proto=0- --type=in --pass=test.rw

rwcut test.rw

This, of course, assumes that the converted pcap data included data from Jan 10, 2015.

HowToPcapNetflowSiLK