Traffic being dropped for being out of state isn't out of state
-
We discovered today that a pfsense firewall we have standing in front of a cloud backup solution is dropping packets that should pass based on our rule set. The Protocol listed in the drops is TCP:A or TCP:PA, which a little bit of forum searching told me the packets were out of state, therefore dropped. Except the packets AREN'T out of state - there is a valid TCP state in the state table for this conenction. I happen to have access to the pfsense firewall the traffic originates behind - and it too have a valid state table entry, but it isn't dropping the traffic.
The network looks ilike this
<server backing="" up="">–-------<pfsense2.1.5>--internet-----<cloud pfsense2.2="">----- <cloud 2="" backup="" server1="" and="">The cloud firewall in this scenario is running load balancing with two hosts behind it, that is working fine.
The "server backing up sources" a TCP packet from 53372 destined to the cloud backup's loadbalancer IP on tcp port 44445. the 2.1.5 firewall NAT's the connection to it's overload IP, changing source port to 19123. You can see this in the state table:
10.10.10.102:53372 -> NATpublicip:19123 -> loadbalancerip:44445 ``` That is the only state conenction in this firewall for this backup connection. Fair enough. On the pfsense 2.2 firewall, I see an entry on the WAN interface with
realIpOfBackupNode:44445 (loadbalancerip:44445) <-NATpublicip:19123
and on the LAN I see
NATpublicip:19123 -> realIpOfBackupNode:44445
so, state, all the way through. All show ESTABLISHED:ESTABLISHED. I am getting 1514 byte packets all the way through, i've done packet captures on all sides of both firewalls. There IS traffic passing through the 2.2 firewall on this TCP connection, and I was able to find the ONE entry in the Firewall log that showed this session establish, just as you'd expect. But my logs are geting hammered with drops on the Default Drop Rule for traffic that should go over this established TCP session. Every now and then I see retransmits - but I see those packets go through the 2.2 firewall, because they show up on the LAN capture. What would be causing these packets to be "out of state" when there is a valid state entry on the firewall? Is it possible that I'm getting packets outside the TCP window? I don't see window size re-negotiation, and lots of traffic IS making it through, but we are concerned that something is not right here. I am having the same problem with another server that is backing up, but I don't have access to their firewall, nor do I know what kind of firewall it is. I mention this only because I don't THINK the issue is with the pfsense 2.1.5 firewall in my little diagram above. The 2.2 firewall is where I see the drops, and after 5 hours of looking I just can't say for sure why it is happening. Any help would be greatly appreciated.</cloud></cloud></pfsense2.1.5></server>
-
I have opened a ticket with pfsense, but of course the behavior has stopped :)
The host that was receving the backup traffic had it's storage undergoing a re-balancing activity at the time the "out of state" packets were being recorded. The only explanation I currently have for why packets were being dropped as out-of-state despite having an established state in the firewlal is that the hsot must have been responding too slowly because of this disk re-balance. The connection never dropped - it was active an never re-negotiated for at least 8 hours. I'm still confused as to EXACTLY what happened, but now that the local storage on the server that receives the backups has finished, everything curretnly looks good.