BjOG – Bjou’s Blog, that is!

Working on my student research project, I have to monitor a quite large network. Therefore, I have configured one of the main switches to mirror the traffic to my hi-end sniffing machine. Trying to capture the traffic with software depending on libpcap, I encountered massive packet loss almost immediately using libpcap-0.8 on Debian (which is version 0.9.5-1). Now I have commited a whole day in trying to decrease the amount of dropped packets. There are some promising solutions, namely mmap’ed libpcap, NAPI (polling-enabled Network Driver) and PF_RING, the latter being the most promising, after having read “Improving Passive Packet Capture: Beyond Device Polling“. Now theory sounds great and is one thing, but getting it to run without any useful documentation almost killed me. Now here is how I finally did it…

• Remove libpcap* via apt-get or aptitude and all other software that depends on it (you gotta rebuild it later with the new pfring-enabled libpcap)
• Rebuild your kernel:
  cd /usr/src
  export CVSROOT=:pserver:anonymous@cvs.ntop.org:/export/home/ntop
  mkdir pf_ring && cd pf_ring
  cvs login

  which should produce the following output:

  Logging in to :pserver:anonymous@cvs.ntop.org:2401/export/home/ntop
  CVS password:

  At the prompt, type “ntop” (no quotes), and hit Enter. (Note: ntop will not appear on the screen.) Next type the following:

  cvs checkout PF_RING

  which will download the needed source.

• Now etner the directory and edit the file mkpatch.sh. Therefore, check which kernel you currently have installed or which kernel you want to have after the update. Check by
  uname -a

  which should give you something like Linux rz-sniff 2.6.17.11 #1 SMP Wed Dec 20 13:06:04 CET 2006 i686 GNU/Linux. Open mkpatch.sh, scroll to the first variable declarations and change into your wishes:

  VERSION=${VERSION:-2}
  PATCHLEVEL=${PATCHLEVEL:-6}
  SUBLEVEL=${SUBLEVEL:-17.11}

  and run the script via sh ./mkpatch.sh, which should give you some output and stop with the location of your patchfile.

• Apply the patchfile:
  cd /usr/src
  zcat /usr/src/pf_ring/PF_RING/workspace/linux-2.6.*patch.gz | patch --dry-run -p0

  If it runs without any error, run again without the --dry-run option

• Now you gotta build your custom kernel. This may vary on different systems, a 2.6 kernel under Debain Sarge can be build as followed:
  cd /usr/src/linux
  apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 build-essential
  make menuconfig

  It is normally a good idea to take the configuration of your existing (working!) kernel 2.6 as a starting point for the configuration of your new kernel. Usually the current kernel configuration is saved in a file under /boot, e.g. /boot/config-2.6.17.11. Load it using the arrow keys and selection the option at the bottom.
  Now for Intel Network Cards, select Device Driver – Network Service Support – Ethernet (1000Mbit) and enable NAPI-Support by selecting ‘y’ on Use Rx Polling. Go back (3x ESC) and select Networking and Networking Options. Make sure that PF_RING sockets are enabled. Leave and save the settings. Now build the kernel and install it:

  make-kpkg clean
  fakeroot make-kpkg –initrd –revision=pfring.1.0 kernel_image
  cd ../
  dpkg -i linux-image-2.6.17.11_pfring.1.0_i386.deb

  This will install your new kernel (including a ramdisk) and also update grub. You can now reboot your system, and you should then have a new kernel. You can check that by running uname -r.

• Compile libpfring, and test that it works.
  First off, start with doing the following:
  cp /usr/src/linux/include/linux/ring.h /usr/include/linux

  which will add the neccessary header file, ring.h, to the standard include directory.

  Next, run the following:

  cd /usr/src/pf_ring/PF_RING/userland/libpfring
  make

  which should give you a short output.
  Now, we are going to generate a .so from these files, by running the following:

  gcc -shared -Wl,-soname -Wl,libpfring.so.0.9.4 -o libpfring.so.0.9.4 *.o -lc

  We copy the files we just created to a common system directory:

  cp libpfring.a libpfring.so.0.9.4 /usr/local/lib
  cp pfring.h /usr/local/include
  ln -s /usr/local/lib/libpfring.so.0.9.4 /usr/local/lib/libpfring.so

  Next, we need to add /usr/local/lib to the list of directories the dynamic loader will search:

  echo "/usr/local/lib" >> /etc/ld.so.conf
  ldconfig

  To check that the dynamic loader sees the libraries, run the following:

  ldconfig -v |grep pfring

  which should produce the following output:

  libpfring.so.0.9.4 -> libpfring.so.0.9.4

  You can test the installation with

  ./pfcount -v -i ethx

  with x being your interface number. Be carefull, the thing can hang on a very busy interface (e.g. > 100Mbit)

  Now, run the following:

  dmesg

  and the last few lines of output should look similar to the following:

  RING: succesfully allocated 128 KB [tot_mem=26509372][order=5]
  RING: allocated 80 slots [slot_len=1618][tot_mem=131072]
  device eth1 entered promiscuous mode
• Next step is to build libpcap to use the PF_RING interface to the kernel.
  First, run the following:
  cd /usr/src/pf_ring/PF_RING/userland/
  ls |grep pcap

  The output should be something similar to:

  libpcap-0.9.4-ring

  which indicates that this version of PF_RING was built to work with a patched version of libpcap-0.9.4, so we need to download that version.

  The simplest way to download that version is to run the following:

  wget http://www.tcpdump.org/release/libpcap-0.9.4.tar.gz

  Now do:

  tar -zxvf libpcap-0.9.4.tar.gz
  cd libpcap-0.9.4
  mv pcap-int.h pcap-int.h.orig
  mv pcap-linux.c pcap-linux.c.orig
  cp ../libpcap-0.9.4-ring/pcap* .
  ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"

  If the ./configure command completes without any errors, run the following:

  make && gcc -shared -Wl,-soname -Wl,libpcap.so.`cat VERSION` -o libpcap.so.`cat VERSION` *.o -lc

  Now, run the following:

  make install && cp libpcap.so.0.9.4 /usr/local/lib

  Next, make sure the dynamic loader sees this new library:

  ldconfig -v |grep pcap

  the output should look similar to:

  libpcap.so.0.9.4 -> libpcap.so.0.9.4

• Compile your software using the new libpcap and pfring, e.g. ntop. If you are getting ntop from CVS also (like PF_RING), change into the ntop directory and compile it, using the following commands. These can be used with other software, too (i.e. the compile options)
  ./autogen.sh --noconfig
  ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib -lpfring -lpcap"
  make && make install

  Now check if your setup was successful:

  ldd /usr/local/bin/ntop

  should produce output similar to:

  linux-gate.so.1 => (0xffffe000)
  libpfring.so.0.9.4 => /usr/local/lib/libpfring.so.0.9.4 (0xb7f8c000)
  libntopreport-3.2.4.so => /usr/local/lib/libntopreport-3.2.4.so (0xb7ee6000)
  libntop-3.2.4.so => /usr/local/lib/libntop-3.2.4.so (0xb78f9000)
  libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb78e7000)
  libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb78e3000)
  libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb78b5000)
  libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7784000)
  libresolv.so.2 => /lib/tls/i686/cmov/libresolv.so.2 (0xb7771000)
  libnsl.so.1 => /lib/tls/i686/cmov/libnsl.so.1 (0xb775a000)
  libpcap.so.0.9.4 => /usr/local/lib/libpcap.so.0.9.4 (0xb772f000)
  libgdbm.so.3 => /usr/lib/libgdbm.so.3 (0xb7729000)
  libgd.so.1 => /usr/lib/libgd.so.1 (0xb76f7000)
  libpng12.so.0 => /usr/lib/libpng12.so.0 (0xb76d4000)
  libz.so.1 => /usr/lib/libz.so.1 (0xb76c0000)
  librrd_th.so.2 => /usr/lib/librrd_th.so.2 (0xb7679000)
  /lib/ld-linux.so.2 (0xb7f99000)
  libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7654000)
  libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0xb7634000)
  libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xb75ca000)
  libart_lgpl_2.so.2 => /usr/lib/libart_lgpl_2.so.2 (0xb75b4000)

  where the line containing libpfring and libpcap are of particular importance.

  At this point, you have a version of ntop that will use the PF_RING ring module in the kernel. Congrats.

• Some Statistics:
  cat /proc/net/pf_ring/info

  will give you some output like this:

  Version : 3.2.1
  Bucket length : 128 bytes
  Ring slots : 4096
  Sample rate : 1 [1=no sampling]
  Capture TX : No [RX only]
  Total rings : 0

  Now there is one drawback: libpcap will not report accurate drop statistics when linked with pfring, so the 0 dropped packets, that ntop reports, are definately wrong.

  cat /proc/net/pf_ring/20

  (where 20 is some random number) shows the actual stats:

  Bound Device : eth0
  Version : 6
  Sampling Rate : 0
  Cluster Id : 0
  Tot Slots : 7181
  Slot Len : 146
  Data Len : 128
  Tot Memory : 1048576
  Tot Packets : 62206634
  Tot Pkt Lost : 14292126
  Tot Insert : 47914508
  Tot Read : 47914434

  Do the math (and wisely distinguish between capture- and drop-rate): 14292126 out of 62206634 is a drop-rate of about 23%. Sounds high? Well, it isn’t, regarding the fact, that my network has peaks up to and over 600Mbps and a current average of about 150Mbps after a 30min runtime. Viewing table 3 in the PDF mentioned in the introduction, you can see, that the performance highly depends on the packet size.
  Now 55% of my traffic has sizes between 60 and 256 bytes (i.e. small packets), for which the table promises a capture-rate of 75%. Another 35% of my traffic is above 1025 bytes, where a Linux 2.6 with NAPI + PF_RING and extended libpcap performs best (93% capture-rate). It is quite poor in between (47% capture-rate), but still the best of its competitors. Anyhow, regarding the fact, that most of my packets are big or small, it was a wise choice to use this specific polling strategy, and a 23% drop-rate is about the result I have expected. Tests with standard kernel and libpcap had drop-rates of up to 40%.
  In my study thesis “Intrusion Detection with heterogenous Sensors” you can find detailed long-term statistics in the Evaluation Chapter (Chapter 5), “Sniffing Performance” featuring nice graphs that illustrate the work of the ringbuffer.

  /var/log/messages should have some stats, too, upon every application using pf_ring

  Welcome to PF_RING 3.2.1
  (C) 2004-06 L.Deri
  NET: Registered protocol family 27
  PF_RING: bucket length 128 bytes
  PF_RING: ring slots 4096
  PF_RING: sample rate 1 [1=no sampling]
  PF_RING: capture TX No [RX only]
  PF_RING: transparent mode Yes
  PF_RING initialized correctly.
  PF_RING: registered /proc/net/pf_ring/
  RING: succesfully allocated 1024 KB [tot_mem=598076][order=8]
  RING: allocated 7181 slots [slot_len=146][tot_mem=1048576]

  Now it is possible to manually remove and insert the kernelmodule into the kernel.
  lsmod shows if ring.ko is loaded and used, rmmod removes it and it can be loaded with configurable parameters via

  insmod /lib/modules/2.6.17.11/kernel/net/ring/ring.ko bucket_len=64 num_slots=4096 sample_rate=1 transparent_mode=0

  bucket_len: Specifies the maximum packet length captured by PF_RING. This is equivalent to snaplen of libpcap. If you are doing something like ntop where you only want to look at the packet headers, then a bucket_len of 64 works. But if you want to inspect the entire packet, then you will have to make sure that the bucket_len is at least as big as the MTU
  of the network.
  num_slots: Number of slots in the ring. The bigger the better is the performance the more memory you use. 4096 should be fine, you can tweak it though.
  sample_rate: 1 means regard every packet (no sampling), 2 means every second and so on…
  transparent_mode: By default a packet that is handled by at least a ring is not forwarded to the upper Linux layers. This will result in faster capture speeds but will prevent legacy applications (not recompiled with the new libpcap-ring) from operating. If you set it to 1 it reverts the ring to the old behaviour (i.e. packets are forwarded to upper layers) but this will decrease the benefits of the ring as it will result in worse results.

  Tweak it according to your purpsose.

  To conclude: I don’t know if it makes any difference, but by checking your NIC via

  ethtool -g eth0

  you can see the internal ringbuffer settings. You can change it to a greater value using

  ethtool -G eth0 rx 4096