Configuring pfSense/netmap for Suricata Inline IPS mode on em/igb interfaces



  • The following instructions apply to cards using the igb or em drivers. If your interface is not named eg "em0" or "igb0" or similar, these instructions may be of limited use to you.

    You can get output requested below from ssh, console access, or in the pfSense ui under Diagnostics->Command Prompt

    All example commands will show "igb0" -- substitute your netmap interface eg "igb4" or "em1" where necessary.

    1. Disable all hardware offloading in the ui (System / Advanced / Networking)
    0_1544248233701_checksumoffloading.png
    Notice: you must reboot.

    2. Verify that all hardware offloading is disabled (also note your mtu) by running ifconfig igb0

    Partial example output:

    igb0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
            options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP>
    

    Options which should not appear in the output: TXCSUM RXCSUM TSO4 TSO6 LRO TXCSUM6 RXCSUM6

    If you have all hardware offloading disabled in the ui (System / Advanced / Networking), but you still any of those options in the output, then you will need to make manual adjustments to your config.xml. Backup your configuration first.

    You will need to add the following to your config.xml in the <system> node (see the link above for details, and remember to substitute your interface name):

    <shellcmd>ifconfig igb0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 -vlanhwtso</shellcmd>
    

    3. Add System Tunables
    Add the following system tunables in System / Advanced / System Tunables:

    3a. disable all flow control by adding dev.igb.0.fc with a value:0 -- netmap is most performant with flow control disabled (if your interface is em1 then this would be dev.em.1.fc = 0, etc)

    3b. dev.netmap.buf_size
    The value of this tunable will depend on your mtu. If your mtu is set to the default of 1500, then the default buf_size of 2048 is likely sufficient. If your MTU is larger, then dev.netmap.buf_size needs to be at least as large as your MTU. Changing this value effectively increases the amount of RAM reserved for netmap. If you need to increase this do it in increments or make sure you have plenty of RAM (the value is multiplied by other settings).


    Troubleshooting Errors

    "bad pkt" errors If cat /var/log/system.log | grep netmap shows errors similar to the one below, then netmap is trying to process a packet larger than the buffers we allocated with dev.netmap.buf_size:

    Dec  6 23:25:38 hostname kernel: 338.512666 [1071] netmap_grab_packets       bad pkt at 1054 len 4939
    

    The final number represents the size of the packet. In this example the packet is 4939 bytes long, over twice the default buf_size of 2048. This packet is considered mal-formed and is dropped as a result. The message is an indication that a packet was dropped. This is not the end of the world and is only an issue if the error is filling your system.log -- packets have been dropping silently since you started using pfSense. Periodically there are failures in the negotiation of packet size and MTUs see this link for more information. You can raise dev.netmap.buf_size to reduce these errors at the expense of memory (and perhaps additional CPU usage - see section 5.2 of this paper for additional information -- note they are using 12MB of cache with a 40gpbs connection, this is unlikely to effect you).

    If the number at the end of the error is quite small, that is an indication that netmap ran into an invalid ethernet packet that was too short to hold the basic required information. If you're seeing many of these, that could be a sign of an incompatible driver.

    "netmap_transmit" errors:

    Dec  7 23:45:05 kernel: 105.845217 [2925] netmap_transmit           igb0 full hwcur 210 hwtail 44 qlen 165 len 66 m 0xfffff8010d4a2d00
    

    If cat /var/log/system.log | grep netmap shows errors similar to the one above, then netmap is trying to put a packet into a buffer that is already full. The error is indicating that the packet has been dropped as a result. As with the "bad pkt" error, this is not the end of the world if it happens periodically. However, if you see this filling your log, then it's an indication that Suricata (or whatever application is using netmap) cannot process the packets fast enough. In other words: you need to get a faster CPU or disable some of your rules. You can buy yourself a tiny bit of burst by modifying buffers, but all you are doing is buying a few seconds of breathing room if you are saturating your link.

    Tips
    On i5/i7 chips you may consider compiling a version of Hyperscan that supports avx2. That process is beyond the scope here, but it improves Suricata processing speed.

    On i7s you probably want to disable hyperthreading if you are going for maximum transmission speeds (search for hyperthreading at the link - there is a tradeoff here for heavy Suricata use).

    Asking for help? Provide the following:
    If you run into an issue not discussed above and would like help, please provide the output from the following commands (excluding any sensitive IPs/hostnames, and remembering to substitute your interface(s) where I have igb0):

    ifconfig igb0
    sysctl -a | grep netmap
    sysctl -a | grep msi
    sysctl -a | grep igb
    sysctl -a | grep rss
    cat /var/log/system.log | grep netmap
    cat /var/log/system.log | grep sig
    cat /var/log/suricata/suricata_*/suricata.log | grep -m 1 "signatures processed"
    

    Please also provide your cpu type/model, total RAM, and average available RAM.


    FAQ

    Why can Suricata process my rules in Legacy mode but not in netmap mode? See this post.

    How can I reduce CPU usage or increase Suricata throughput?

    • Start by disabling unnecessary rules as the Emerging Threats Users Guide suggests. There's little point in running content matching for traffic that will never pass your firewall because it has no place to go.

    • Set "encrypt-handling" to "bypass" in the suricata.yaml. This may be possible using the "Advanced Configuration Pass-Through" option in the ui. Suricata still inspects the encryption handshake, but ignores all encrypted traffic after the handshake as it cannot be analyzed.

    • Create bypass rules that exclude known traffic hogs that are unlikely to be threats (eg Netflix, Spotify, Youtube, etc)

    • Enable local bypass (under "stream" in suricata.yaml -- research this before enabling so you understand the risks).

    • If your cpu is being under used but traffic is bottlenecked, check out these sections of the manual: Runmodes and Threading -- Suricata ships with autofp mode enabled by default, but the developers suggest "workers" mode for increased performance (they ship with autofp enabled for packet capture usage).

    Further Reading
    Slide presentation by the creator of netmap
    A general talk on high-speed packet inspection in linux-style operating systems - short bit on netmap
    PF_RING FT is a potential future alternative
    DPDK is another potential future alternative
    https://blog.cloudflare.com/kernel-bypass/