Suricata incompatibility with pcap utils? (tcpdump / tshark)

Tantamount

I needed to look at the network traffic today and noticed all kinds of wacky info from tcpdump like this:

tcpdump -i igb0 -nn -s 0 -X not tcp and not udp

13:18:35.753390 3a:40:20:01:04:70 > 60:00:00:00:00:08, ethertype Unknown (0x1f04), length 48:
        0x0000:  0ca0 0000 0000 0000 0001 2001 0470 1f04  .............p..
        0x0010:  0ca0 0000 0000 0000 0002 8100 5a67 6b90  ............Zgk.
        0x0020:  1898                                     ..

After staring at this for a while I realized that what I'm looking at is ipv6 traffic, however part of the header had been truncated. 2001:0470:1f04:0ca0::2 is the ipv6 destination address, but you can see that the source address has been cut off, and the frame info before that is also gone.

If I turn off suricata (in inline mode), this stops.

Oddly, this doesn't affect the ipv6 traffic – everything works okay -- it just seems to be an issue with pcap retrieving all of the traffic.

bmeeks

@Tantamount:

I needed to look at the network traffic today and noticed all kinds of wacky info from tcpdump like this:
tcpdump -i igb0 -nn -s 0 -X not tcp and not udp
13:18:35.753390 3a:40:20:01:04:70 > 60:00:00:00:00:08, ethertype Unknown (0x1f04), length 48:
        0x0000:  0ca0 0000 0000 0000 0001 2001 0470 1f04  .............p..
        0x0010:  0ca0 0000 0000 0000 0002 8100 5a67 6b90  ............Zgk.
        0x0020:  1898                                     ..
After staring at this for a while I realized that what I'm looking at is ipv6 traffic, however part of the header had been truncated. 2001:0470:1f04:0ca0::2 is the ipv6 destination address, but you can see that the source address has been cut off, and the frame info before that is also gone.

If I turn off suricata (in inline mode), this stops.

Oddly, this doesn't affect the ipv6 traffic – everything works okay -- it just seems to be an issue with pcap retrieving all of the traffic.

That is strange. My first suspicion would be an interaction with the new Netmap driver. Inline mode turns on the Netmap driver (for supported NICs). Turning off Suricata unloads the Netmap driver. I don't think this is a straight up Suricata problem, but instead may be something either with tcpdump or Netmap itself. How about posting a Redmine bug at https://redmine.pfsense.org/ and give all the details to reproduce.

EDIT: Just had another idea – there is a Netmap-enabled version of pcap. I don't believe it is part of the pfSense system, though. Might be that with Netmap running you would need to use a version of pcap that can speak Netmap.

Bill

Tantamount

I'm been looking more into this, and apparently, the problem isn't with pcap.

This odd traffic is being broadcasted on the network from pfsense – I was able to pick it up from a Windows machine using Wireshark.

Those packets are mangled icmp v6 replies that are being tunneled through ipv4. (I'm using hurricane electric's ipv6) My guess is this traffic is being generated for the rrd "quality" graph.

Here's the same type of packet, but with suricata off:

19:50:42.858359 IP 72.52.104.74 > my.public.ip.address: IP6 2001:470:1f04:ca0::1 > 2001:470:1f04:ca0::2: ICMP6, echo reply, seq 981, length 8
        0x0000:  4500 0044 ab0f 4000 f729 55cf 4834 684a  E..D..@..)U.H4hJ
        0x0010:  6cd6 655d 6000 0000 0008 3a40 2001 0470  l.e]`.....:@...p
        0x0020:  1f04 0ca0 0000 0000 0000 0001 2001 0470  ...............p
        0x0030:  1f04 0ca0 0000 0000 0000 0002 8100 1207  ................
        0x0040:  c8b3 03d5

So somehow, 34 bytes are being stripped from the frame and then dumped onto the network with Suricata is on. Since the first 34 bytes contain the mac addresses, the switch is broadcasting the traffic since it doesn't know what to do with it.

Okay, looks like I was able to make it stop doing this by disabling Suricata on the interface and re-enabling instead of clicking the little "stop" icon for the interface… Btw, clicking that "stop" icon only temporarily stops Suricata -- it fired back up again all on its own -- the only way I could force it to stop for good was to disable it on the interface.

I'm going to wait to see if I start to see this strange traffic again -- it's possible it only happens when Suricata has been running for a while.

Tantamount

Okay, the traffic started again. I've completely disabled Suricata for now. I'm going to virtualize pfsense instead to see if this is somehow a driver issue.

7:41pm was when I restarted Suricata.
9:00pm was when the malformed traffic started up again.

Tantamount

Just wanted to confirm that this happens even in a vm (VMware Workstation 12 Pro), so it's not a hardware/driver issue.