Excessive TCP: PA FA RA
-
Hi All,
Lately I am seeing an overly excessive amount of TCP: PA, FA, FPA dropped packets to legit traffic in the firewall logs.
I have the Optimization set to Conservative.
I cannot find anywhere with suggestions to take to reduce this packet loss.
Could the Traffic Shaper help if I throttle my bandwidth down? I am not using the shaper now.
I have a single WAN, LAN and OPT1 network. The only package is PFblockerNG.
Any help or suggestion is highly appreciated.
Dan -
Comes up in search.
Are you using vpn or 2 wans?
-
No VPN and one WAN.
Only suggestion in a search I could find was the Conservative setting.
I also increased the nmbclusters to 1000000. I am using all igb interfaces.
All the offloading is checked/disabled.PA, RA, FA happens on all the interfaces and happens 20-30 times a minute. The good traffic does eventually get through but I have a lot of broken packets.
-
Here is an example in my firewall log. There has to be something wrong here!
-
https://doc.pfsense.org/index.php/Why_do_my_logs_show_%22blocked%22_for_traffic_from_a_legitimate_connection
Any time there are excessive ACKs being dropped, it's almost always out of state traffic. Weird that it's logging a block between 20.1 and 20.2. Any wireless clients on the network? They are notorious for leaving connections hanging and switching networks on a whim.
-
No there are not any Wireless networks, just one WAN and 2 subnets (192.168.1.0/24, 192.168.20.0/24)
I am also not doing any Clustering or Load Balancing.My issue is that this is a new problem.
So what else could be causing this issue?
-
First off, are you actually seeing any problems on your network or are you just reacting to log spam?
-
I haven't noticed any issues, but these logs alarmed me since I have never seen this before. I suspect my internal network speed must be slower if there are all these bad packets but not sure how to test that.
-
Some of those are to loopback, 127.0.0.1
What are you running on pfsense on port 19006??
-
As far as I know nothing is running on port 19006. But those eventually disappeared.
The real issue is that every connection to PFsense gets a PA, RA, or FA before it goes through. Even local to local connections. Every HTTP access gets one too, but then goes through.
There is something wrong here when every incoming or outgoing packet drops before it goes through.Any suggestions?
-
FA and RA are just FIN and Reset packets. They're trying to close the connection. No point worrying about blocking packets that are meant to kill a connection if the connection is already dead.
-
Post screens of your LAN configuration and firewall rules. Something screwy is going on. Local traffic doesn't hit the firewall at all, and I noted this when I said it was funny that it's logging a block between 192.168.20.1 and 192.168.20.2. No VLANs configured?
-
Are you running some sort of port forward or weird nat reflection setups… I agree with I can think of nothing that would be running on pfsense that listens on that port.. Post up a output of sockstat -4 -l
example of mine
[2.3.2-RELEASE][root@pfsense.local.lan]/root: sockstat -4 -l USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root php-fpm 4623 5 udp4 *:* *:* root radiusd 76653 13 udp4 192.168.2.253:1812 *:* root radiusd 76653 14 udp4 192.168.2.253:1814 *:* proxy ftp-proxy 68730 3 tcp4 127.0.0.1:8021 *:* ladvd ladvd 38973 9 udp4 *:* *:* root ladvd 38787 5 udp4 *:* *:* dhcpd dhcpd 12642 16 udp4 *:67 *:* dhcpd dhcpd 12642 20 udp4 *:64384 *:* unbound unbound 93283 4 udp4 192.168.9.253:53 *:* unbound unbound 93283 5 tcp4 192.168.9.253:53 *:* unbound unbound 93283 8 udp4 192.168.2.253:53 *:* unbound unbound 93283 9 tcp4 192.168.2.253:53 *:* unbound unbound 93283 15 udp4 192.168.3.253:53 *:* unbound unbound 93283 16 tcp4 192.168.3.253:53 *:* unbound unbound 93283 19 udp4 192.168.4.253:53 *:* unbound unbound 93283 20 tcp4 192.168.4.253:53 *:* unbound unbound 93283 21 udp4 192.168.6.253:53 *:* unbound unbound 93283 22 tcp4 192.168.6.253:53 *:* unbound unbound 93283 25 udp4 192.168.7.253:53 *:* unbound unbound 93283 26 tcp4 192.168.7.253:53 *:* unbound unbound 93283 27 udp4 127.0.0.1:53 *:* unbound unbound 93283 28 tcp4 127.0.0.1:53 *:* unbound unbound 93283 31 tcp4 127.0.0.1:953 *:* root openvpn 31208 6 udp4 24.13.snipped:4142 *:* root openvpn 18443 6 udp4 24.13.snipped:1194 *:* root openvpn 14412 6 tcp4 24.13.snipped:443 *:* root syslogd 49062 7 udp4 192.168.9.253:514 *:* root ntpd 44109 21 udp4 *:123 *:* root ntpd 44109 23 udp4 192.168.9.253:123 *:* root ntpd 44109 26 udp4 192.168.2.253:123 *:* root ntpd 44109 29 udp4 192.168.3.253:123 *:* root ntpd 44109 31 udp4 127.0.0.1:123 *:* root ntpd 44109 34 udp4 192.168.4.253:123 *:* root ntpd 44109 36 udp4 192.168.5.253:123 *:* root ntpd 44109 38 udp4 192.168.6.253:123 *:* root ntpd 44109 41 udp4 192.168.7.253:123 *:* root nginx 40900 6 tcp4 *:443 *:* root nginx 40900 8 tcp4 *:80 *:* root nginx 40737 6 tcp4 *:443 *:* root nginx 40737 8 tcp4 *:80 *:* root nginx 40531 6 tcp4 *:443 *:* root nginx 40531 8 tcp4 *:80 *:* root xinetd 28003 0 udp4 127.0.0.1:6969 *:* root sshd 13927 5 tcp4 *:22 *:* root php-fpm 264 5 udp4 *:* *:* [2.3.2-RELEASE][root@pfsense.local.lan]/root:
-
KOM - See attached configs, I do use a Virtual IP that redirects to an assigned IP, No VLAN's
Johnpoz- No port forwards to port 19006. I use a few rules to direct email traffic to the correct IP
As far as the port 19006 hits, saw more this morning, so I setup a TCP port monitor to capture what process is causing it. But it looks like PFsense is generating the packets by looking at the sockstat
Here are the Firewall rules and sockstat
I noticed the extended Internet daemon port in the sockstat list. What does this mean?
-
Your rules are a bit of a mess. Rules are applied to traffic entering an interface. You can delete almost all of your LAN/OPT1 rules and replace them with a single Allow Any rule on each. Generally, you don't specify a Source since the network the traffic is coming from is the source. For example, on your LAN rules you don't need to specify Source as LAN Net since no other traffic is going to be coming into the LAN interface other than LAN Net traffic. Those 2 Allow All rules at the bottom of your WAN rules needs to go, pronto.
Clean your rules up and this might help eliminate any weirdness going on.
-
Yeah with KOM here those rules are complete mess..
Your lan rule is any any at the top, for udp/tcp why is it called allow email?? What email runs on udp?
Anyhow - all the rules below that are just pointless.
Rules are evaluated top down, first rule wins, rest are not even looked at. As the packets enter the interface.
Those rules on your wan are BAD!!!
What do you have in your xinetd conf?
cat /var/etc/xinetd.conf
What packages do you have installed other than pfblocker?
-
Thanks for taking the time to help with these rules.
As for the LAN rules, see change below. I simplified it to any-any
As for the WAN rules, I have trimmed them down to the ones needed for the NAT port forwarding, see change below.
So in actuality, I should have only the WAN direct ports and have all the other interfaces pass any to any. I have no need for any restrictions on any of my internal networks. The only blocking I care about is from the WAN. But I do know that you also have to use the rules to direct traffic, as in the WAN rules below.
Don't I need a rule to allow internet traffic to my LAN. It does seem to work without one.
-
Not sure why you are not using protocol any on your LANX rules if you really want no restrictions between LAN interfaces.
-
"I do know that you also have to use the rules to direct traffic"
Huh? Sure if you want to do policy routing out a specific gateway or vpn connection, etc. But lan to opt etc.. Or just out the default gateway no there is no need to "direct" anything.
If you really want no restrictions than that rule should be any not tcp/udp. So you won't be able to ping stuff with that setup. Even though you would be able to hit http..
So what is the content of your xinetd.conf ??
cat /var/etc/xinetd.confYou clearly had something listening on that 19006 port..
-
Ehm, no. Directly connected networks are known to the system by their routing table entries that do not need a gateway entry in the table. Gateways are only needed for "foreign" destinations, i.e. networks that are not directly connected to the system. The best example is of course the default gateway which is the 0.0.0.0/0 entry (often marked as "default" as it is in pfSense also) in the routing table, it's not a directly connected network so in order to reach it a gateway has to be configured. Like so in my pfSense system (public IPs censored):
$ netstat -nr -f inet Routing tables Internet: Destination Gateway Flags Netif Expire default 88.195.aaa.1 UGS em1 10.0.0.0/8 127.0.0.1 UGS lo0 10.71.14.0/24 link#2 U em0 10.71.14.1 link#2 UHS lo0 88.195.aaa.0/19 link#3 U em1 88.195.bbb.ccc link#3 UHS lo0 127.0.0.1 link#7 UH lo0 172.16.0.0/12 127.0.0.1 UGS lo0 192.168.0.0/16 127.0.0.1 UGS lo0 192.168.1.0/24 link#3 U em1 192.168.1.200 link#3 UHS lo0
Here the 88.195.aaa.0/19 is a directly connected network (the WAN network), so is my LAN network of 10.71.14.0/24. The system knows how to reach hosts on those networks without a need to send the traffic to a gateway by issuing an ARP query on the connected network to figure out which MAC address the traffic should be sent to on the ethernet level.
The default gateway, the one that is needed to reach the "world out there, the internet" is the first line that says "default 88.195.aaa.1 UGS em1". This says that in order to connect to any IP address/network that does not match an entry in the routing table of this system forward the traffic to address 88.195.aaa.1 and not very surprisingly the routing table also has instructions on how to reach that address, the "88.195.aaa.0/19 link#3 U em1" line.
-
I did change the LAN tcp/udp to any right after I posted the rules.
See attached the xinetd file. Why are those 19001-19008 ports in there? Is this normal?
Here is my routing table. The first 2 entries are DNS. Does this look ok?
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default xx.xx.129.113 UGS igb2
68.105.28.16 xx.xx.129.113 UGHS igb2
68.105.29.16 xx.xx.129.113 UGHS igb2
xx.xx.129.112/28 link#3 U igb2
xx.xx.129.114 link#3 UHS lo0
xx.xx.129.117 link#3 UHS lo0
xx.xx.129.117/32 link#3 U igb2
xx.xx.129.124 link#3 UHS lo0
xx.xx.129.124/32 link#3 U igb2
127.0.0.1 link#7 UH lo0
192.168.1.0/24 link#4 U igb3
192.168.1.1 link#4 UHS lo0
192.168.3.0/24 link#5 U igb4
192.168.3.1 link#5 UHS lo0
192.168.10.0/24 link#2 U igb1
192.168.10.1 link#2 UHS lo0
192.168.20.0/24 link#1 U igb0
192.168.20.1 link#1 UHS lo0xx.xx.129.113 is the default gateway assigned by ISP
xx.xx.129.114-125 is my assigned IP block. Currently only using 114,117,124
192.168.3.0/24 is the subnet used for the wireless router - OPT3
114-WAN/igb2, 117-OPT1/igb0, 124-OPT2/igb1, LAN/igb0, OPT3/igb4 -
The reason I said "I do know that you also have to use the rules to direct traffic" is because there was a time in the past where the LAN any-any rule would not work for some devices on the same LAN subnet unless I gave it a specific rule. That does not seem to be the case now, so any-any is working for all devices on the LAN subnet as it should.
Also, that statement does seem to be true for the WAN where there is no any-any rule. Or any interface which does not have an any-any rule.
So, does my posted new WAN rules look ok?Dan
-
Here we go again with the port 19006, see below.
192.168.1.2 is my main desktop that I use. I had a TCP monitor running and it did not capture this.
-
Then you weren't capturing correctly or something else on your network is sending those packets from that IP address.
-
Maybe I missed it. can't predict when it happens, but will leave the capture tool running on 192.168.1.2. I have it filtered for incoming and outgoing port 19006.
Is the xinetd.conf I posted earlier correct? It shows 192.168.20.2 tied to port 19006. Why does it do this? -
Probably a package. What have you installed and why?
there was a time in the past where the LAN any-any rule would not work for some devices on the same LAN subnet unless I gave it a specific rule.
Poppycock. The firewall NEVER gets in the way there unless you are bridging interfaces or some other edge case.
-
Diagnostics > Command Prompt
Execute: cat /var/etc/xinetd.conf
-
how exactly would the firewall even see that traffic to loopback.. Your pc if wanting to talk to a 127.0.0.1 address wouldn't even put it on the wire, that is localhost. That traffic doesn't go out on the wire.
So that has to be coming from your firewall, or some sort of port forward that you send to loopback?
service 19006-tcp
{
type = unlisted
bind = 127.0.0.1
port = 19006
socket_type = stream
protocol = tcp
wait = no
user = nobody
server = /usr/bin/nc
server_args = -w 2000 192.168.20.2 993
}Your running something with NC… netcat, not sure of what package or config settings would create those.. I sure and the hell do not have them that is for sure.
-
.. I sure and the hell do not have them that is for sure.
Hmmm, scary stuff for a firewall… 8)
Netcat is often referred to as a "Swiss Army knife" utility, and for a good reason. Just like the multi-function usefulness of the venerable Swiss Army pocket knife, netcat's functionality is as helpful. Some of its features include port scanning, transferring files, port listening and it can be used a backdoor.
[http://www.catonmat.net/blog/unix-utilities-netcat/]
imaps 993 udp imap4 protocol over TLS/SSL
-
yeah not sure what he is doing, or what would of done that..
Yeah NC is very powerful tool.. Why there would be stuff like that setup in his xinetd I have no idea.. The only thing that was in mine the tftp proxy, and I rem them out because not using it and was just causing log spam.
I am not a nc guru by any means, but looks like to me if sees traffic on loopback to port 19006 send it over to that 192.168.20.2 IP on port 993.
I do believe that if you setup nat reflection that pfsense starts with ports 19000, so you had prob setup some sort of nat reflection. Or if he has port forwards and has it using nat reflection these sorts of entries would be put in.
Maybe this is caused by having auto nat reflection enabled?? I personal see nat reflection as an abomination that should be killed with greek fire whenever possible.. Looking back at his firewall rules he does have some port forwards to that 20.2 IP.. But he has the ports all hidden in an alias. So he has some sort of nat reflection going on and then some weirdness is causing out of state..
Again going to state this for the record that nat reflection is an abomination… Turn it off and your problems will go away would be my guess..
-
…I rem them out because...
... Turn it off and your problems will go away would be my guess..To disable a service config, add "disable =yes", (then exec the usual 'killall HUP xinetd'), like in:
service 6969-udp { disable = yes type = unlisted bind = 127.0.0.1 port = 6969 socket_type = dgram protocol = udp wait = yes user = root server = /usr/libexec/tftp-proxy server_args = -v }
-
So I take it that these xinetd.conf entries are for NAT reflection?
I am using NAT reflection on all my port forwards NAT+Proxy. I did that so the LAN can communicate with other interfaces
So, if I should not be using NAT Reflection, should I setup rules instead?
I know for a fact if I turn off NR I cannot open my websites from 192.168.1.2 to the web server @ 192.168.20.2 or load my config page on the NAS @ 192.168.10.2. NR solved all the local communication.I want the LAN subnet to be able to connect to OPT1/2/3. NAT Reflection does this for me.
Here are my Port Forward rules. NAT Reflection Enabled (NAT+Proxy) on first three, system default (Pure NAT) on the last rule
-
…So, if I should not be using NAT Reflection, should I setup rules instead?
No.
Split DNS. Tell your DNS server to point to your local servers, case a LAN-host requests that global server address of yours. -
I use my ISP's DNS and DNS Resolver in PFsense. Do not have a local DNS Server setup.
Are you saying no to using NAT reflection altogether and find a different method. Or just to using rules.
So johnpoz, why the boo.. against NAT Reflection?
-
Have a look see at Services / DNS Forwarder / Host Overrides
-
I do not use DNS Forwarder, I use Resolver, but I do see the host override in there.
So if I put / Host-www / Domain-mydomain.com / IP-192.168.20.2 / in there, 192.168.1.0/24 and 192.168.3.0/24 and 192.168.10.0/24 will be able to get to www.mydomain.com on 192.168.20.2
I host a lot of domains, so I guess I would have to have a list of all of them including all the sub domains. NAT reflection seems easier to me.
-
I'm going to start a new thread on the DNS Resolver host override issue and lock this one. This thread has too many issues that are just compounding.