Porting SYN cookies implementation to PF module of FreeBSD
Having bought two pfSense XG-1540 appliances, they seem to be doing good under normal traffic. But during TCP SYN floods it suffers a lot. I want the SYNPROXY feature to get enabled dynamically as the traffic increases for that particular rule (based on some threshold), this would save the servers behind it. Is it possible?
And also why do the PF states get bloated up when SYNPROXY is enabled? It stores some information like "PROXY:SRC", which apparently looks to be trivial, can this be avoided?
Can we implement something similar to SYN cookies for the PF module which can offer some sort of respite for it? And thereby adding a PF state entry only upon establishing a legitimate connection.
I would be happy to implement this myself. But can someone help with where to start? How to interpret the FreeBSD kernel code?
While this doesn't answer your question, one thing to be careful about is syn-proxy breaks TCP window sizing, limiting to only 64KiB, which is not enough for high bandwidth high latency links of today.
As for syn-floods. In my limited experience, my PFSense box with stock configuration handled a standard syn flood quite well. What did it in was a distributed syn flood. Watching a BSDCAN 2016 presentation about FreeBSD and DDOS, it turns out FreeBSD has an O(NM) algorithm that makes it break under DDOS. Where N is number of source IP addresses and M is the total number of states. With a DDOS of syn flood, that O(NM) effectively turns into O(N^2), which is very very very bad. One of the first recommendation the presented had was to lower the TCP open timeout from 90sec to 16sec. 90sec is way overkill since modern TCP drops the connection in 24 seconds if it cannot be established. Letting a state suck up memory for another 66 seconds just makes SYN attacks that much worse.
The other issue is FreeBSD likes to purge all at once, which pretty much stops everything. He had two recommendations. 1) reduce the interval between purges to reduce the window of new states that need to be purged 2) Make FreeBSD have a limit on how many states to purge in a single pass
FreeBSD handles existing states very well O(1), the issue is some poorly chosen algorithms with O(N^2) scaling intermixed with some naturally O(N) problems like purging. Limiting the window and the rate of how many states can be created will help reduce the resource depletion DOS attacks. If you actually get hit with a volumetric DDOS, there's nothing you can do, but for these asymmetric resource attacks, there are a few tweaks that can be made.
edit: Firewall rules let you limit the rate new states can be created per rule. And the "window" I keep talking about is the TCP open timeout since DDOS syn attacks are almost always spoofed/Raw-socket and don't actually establish the connection.