Suricata causes strange "ghost" bandwidth measured to various external IP's, but it's not real.

eric.marshall

We have Suricata running on 4 virtual (virtio) interfaces on our pfsense VM in inline mode. I have nmap running in forced emulation mode (does not work with virtio interfaces in native mode), and a few other little tunables.

3 of them work fine, but any time I enable it on this interface, we get a steady stream of "fake" data flowing "out" of our environment (into the interface). Hovers around 100Mbit/s continuous, forever.

The upstream firewall appliance does not see/measure any of this traffic. It's not actually happening. The traffic measurement drops to 0 the moment suricata is disabled on this interface.

The "origin" of this data that appears to be flowing into this interface, is external IP's, even though this is a private network.

Thoughts?

bmeeks

This is probably an artifact of the netmap device used by Suricata when running with Inline IPS Mode. The netmap kernel device does not always interoperate with other kernel features (namely shapers/limiters and, to some degree, VLANs).

My guess is some internal kernel traffic counters are getting falsely incremented. While it is certainly weird, my suspicion is it is harmless (as in not performance impacting).

Edit: did you mean nmap or maybe ntop instead?

eric.marshall

@bmeeks Sorry that was a froidian typo on my part. Indeed I'm running Netmap in emulation mode. Not nmap, wrong map ;)

Your explanation for what is going on here sounds highly reasonable to me. I have been operating on the assumption that the anomalous bandwidth readings here were likely harmless.

Thanks!

bmeeks

@eric-marshall said in Suricata causes strange "ghost" bandwidth measured to various external IP's, but it's not real.:

@bmeeks Sorry that was a froidian typo on my part. Indeed I'm running Netmap in emulation mode. Not nmap, wrong map ;)

Your explanation for what is going on here sounds highly reasonable to me. I have been operating on the assumption that the anomalous bandwidth readings here were likely harmless.

Thanks!

Okay, netmap in emulation mode is going to be very slow compared to native mode. But emulation mode by itself should not result in that ghost traffic. Or at least you are the first to report it. What hypervisor are you running?

Are you running something else like ntop or another traffic accounting system?

eric.marshall

@bmeeks

Yes, there's a performance hit here, but unfortunately the virtio/netmap situation in pfsense seems to be too old to work in native mode on KVM. We run Proxmox.

When run in native mode, it fails. Works for a few minutes then stalls/crashes.

Luckily, even in emulation mode we can still pass ~700-900Mbit/s through the gateways with IPS enabled, so the performance is acceptable.

I don't think there's any other traffic monitoring going on... I don't have any additional traffic monitoring packages installed.

bmeeks

Pure speculation here, but there was a patch submitted to FreeBSD upstream that fixed an issue where the traffic counters would not register at all when a netmap device was in use. That caused the counters to always show zero traffic.

A fix for that was submitted to FreeBSD upstream by, I believe, the OPNSense team. That fix then made its way into pfSense with a recent base OS update. Maybe the fix has a side-effect for emulation mode operation ??

Might be something you want to report upstream in FreeBSD. But it may be specific to your particular setup with Proxmox. Maybe other Proxmox users with a pfSense VM can chime in here.