Upgrade to 2.01 has many broken connections

lfreeman

After upgrading to 2.01 I now see hundreds of blocked packets for the protocols: TCP:FA, TCP:PA, TCP:FPA, TCP:RA, TCP:SA, even TCP:lo.

May 8 09:00:08 LAN 10.10.16.46:35396 199.59.150.41:443 TCP:RA
May 8 08:56:54 WAN 69.171.229.70:443 xx.xx.xx.130:62118 TCP:RA
May 8 08:52:04 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:FPA
May 8 08:51:32 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:FPA
May 8 08:51:16 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:FPA
May 8 08:51:08 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:FPA
May 8 08:51:05 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:FA
May 8 08:51:05 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:PA
May 8 08:51:05 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:PA
May 8 08:51:05 LAN 10.10.16.46:41253 74.125.225.131:443 TCP:PA
May 8 08:48:29 WAN 69.171.228.70:443 xx.xx.xx.130:43965 TCP:lo
May 8 08:44:17 WAN 69.171.228.70:443 xx.xx.xx.130:63809 TCP:lo
May 8 08:44:16 LAN 10.10.16.46:43132 199.59.150.41:443 TCP:RA
May 8 08:43:13 WAN 69.171.229.76:443 xx.xx.xx.130:2392 TCP:RA
May 8 08:41:20 LAN 26.218.224.180:53255 xxx.7.xxx.140:443 TCP:RA
May 8 08:39:00 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:38:21 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:38:21 WAN 69.171.229.72:443 xx.xx.xx.130:58015 TCP:RA
May 8 08:38:02 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:37:53 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:37:48 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:37:45 LAN 10.10.16.39:4123 74.125.227.136:80 TCP:FA
May 8 08:37:29 LAN xx.218.xxx.192:59037 xxx.7.xxx.140:443 TCP:RA
May 8 08:35:46 WAN 66.220.149.88:443 xx.xx.xx.130:62470 TCP:RA

Most of these appear to be callbacks to Google apps or Facebook, etc, and we're seeing problems with Google search and some other web apps timing out or freezing.

I have four ipsec tunnels open to other pfsense firewalls to connect my various server and client sites. All of these sites have a private ip address as the remote network.

My connection out is a 10Mb fiber which doesn't seem to have any packet loss. I have no static routes. Pf is configured to only allow in the ipsec traffic between the other locations. The standard outbound NAT is enabled and nothing else.

I read that asymmetrical routing could cause this problem but I don't see how I could have that problem. What am I missing? These packets weren't being logged before.

Thanks.

jimp

Those look like either asymmetric routing, or out-of-state traffic (where the connection state was removed, and then that packet arrived later).

The latter is especially common if you're using squid or another proxy.

Not sure what the TCP:lo one is, I'd have to see the raw filter log associated with that line.

lfreeman

I would agree but how do I detect the asymmetrical routing and fix it? My connection is Verizon FIOS with a static IP. I have a single gateway defined in pfSense and attached that to the WAN interface. The ipsec connections are all standard settings that are working on my other pfsense firewalls. No static routes. So where does the asymmetrical routing occur?

lfreeman

No proxies are used either.

jimp

If you have no proxy, static routes, etc, then it's harder to say.

The raw log might give some better insight.

Also under System > Advanced, on the Firewall/NAT tab, check that your optimization mode is not set to Aggressive, and make sure your state table size isn't set too low (the default is ~10% of RAM, at 1KB per state so with 1GB of RAM it would use 100,000 states.

cmb

The raw log would be helpful, we're mis-parsing something on the "TCP:lo" lines.

Everything except those is probably no different than it's ever been, those are all FINs or RSTs and just normal out of state traffic that happens on occasion with every firewall. Just retransmits on connections that are already closed trying to close them again. Those cannot be the cause of any connectivity problems.

lfreeman

I'm using default values:
State table size 1207/97000
MBUF Usage 1716/25600

Optimization is set to Conservative.

I'll try to get a packet capture of the TCP:lo lines

jimp

No need for a packet capture, if you see it in the GUI log, go to Diag > Command and run:

clog /var/log/filter.log

Then find the relevant line in the raw log, and copy/paste it here.

lfreeman

Ok.

Firewall log:

May 9 10:35:46 WAN 69.171.228.70:443 50.XX.XX.130:36046 TCP:lo

packet:

May 9 10:35:46 lascolinas pf: 69.171.228.70.443 > 50.XX.XX.130.36046: Flags [R.], cksum 0x76a3 (correct), seq 4021180300:4021180346, ack 165425407, win 0, length 46 [RST+ BIG-IP: [0x116b7f6:165] Flow e]
May 9 10:36:38 lascolinas pf: 00:00:52.481053 rule 1/0(match): block in on bge0: (tos 0x0, ttl 64, id 2239, offset 0, flags [DF], proto TCP (6), length 788)

jimp

those don't appear to be connected lines, the timestamp should match on both.

lfreeman

May 9 10:35:46 lascolinas pf: 00:01:06.694431 rule 1/0(match): block in on bge1: (tos 0x0, ttl 231, id 59683, offset 0, flags [DF], proto TCP (6), length 86)
May 9 10:35:46 lascolinas pf: 69.171.228.70.443 > 50.84.43.130.36046: Flags [R.], cksum 0x76a3 (correct), seq 4021180300:4021180346, ack 165425407, win 0, length 46 [RST+ BIG-IP: [0x116b7f6:165] Flow e]

jimp

Looks like that BIG-IP: bit is throwing off the regex. Might need to try to tighten it up a little.

lfreeman

Ok maybe this is a clue. How do these packets even show up on my LAN interface with the subnet ip of 10.10.16.0/24?

May 9 14:32:02 LAN 22.42.215.213:40849 209.85.145.188:5228 TCP:FPA
May 9 14:31:41 LAN 22.42.215.213:40849 209.85.145.188:5228 TCP:FPA

May 9 14:31:41 lascolinas pf: 00:07:13.689474 rule 1/0(match): block in on bge0: (tos 0x0, ttl 64, id 14828, offset 0, flags [none], proto TCP (6), length 75)
May 9 14:31:41 lascolinas pf: 22.42.215.213.40849 > 209.85.145.188.5228: Flags [FP.], cksum 0xaf5e (correct), seq 3655068907:3655068930, ack 2331443737, win 8011, options [nop,nop,TS val 6610240 ecr 4266176071], length 23
May 9 14:32:02 lascolinas pf: 00:00:20.554728 rule 1/0(match): block in on bge0: (tos 0x0, ttl 64, id 14829, offset 0, flags [none], proto TCP (6), length 75)
May 9 14:32:02 lascolinas pf: 22.42.215.213.40849 > 209.85.145.188.5228: Flags [FP.], cksum 0xa73e (correct), seq 0:23, ack 1, win 8011, options [nop,nop,TS val 6612320 ecr 4266176071], length 23

cmb

which NIC is bge0?

lfreeman

bge0 is the LAN interface. bge1 is the WAN interface.

bge0 is wired to our internal switch and bge1 is directly connected to our TWC fiber router. There is no other route out of the internal network except via bge1.

lfreeman

I turned off our wireless access point and I haven't seen any more strange packets since. The only devices attached wirelessly were smartphones. Maybe a trojan on a phone was trying to DoS someone's server? The source ip is a DoD address.

cmb

In order for logs like you're seeing on the LAN interface to show up, you'd have to somehow have gotten LAN and WAN mixed onto the same subnet. The logs such as:

May 9 14:31:41 lascolinas pf: 00:07:13.689474 rule 1/0(match): block in on bge0: (tos 0x0, ttl 64, id 14828, offset 0, flags [none], proto TCP (6), length 75)
May 9 14:31:41 lascolinas pf: 22.42.215.213.40849 > 209.85.145.188.5228: Flags [FP.], cksum 0xaf5e (correct), seq 3655068907:3655068930, ack 2331443737, win 8011, options [nop,nop,TS val 6610240 ecr 4266176071], length 23

showing blocking of traffic with both source and destination of public IPs indicates that, you'll never see such traffic unless WAN and LAN are somehow interconnected (or a few other possible but much less likely scenarios like some host on LAN with a public IP manually configured). Given it stopped when the wireless AP was turned off, I suspect maybe somehow that was combining your LAN and WAN networks?