Excessive TCP: PA FA RA


  • Banned

    Hi All,
    Lately I am seeing an overly excessive amount of TCP: PA, FA, FPA dropped packets to legit traffic in the firewall logs.
    I have the Optimization set to Conservative.
    I cannot find anywhere with suggestions to take to reduce this packet loss.
    Could the Traffic Shaper help if I throttle my bandwidth down? I am not using the shaper now.
    I have a single WAN, LAN and OPT1 network. The only package is PFblockerNG.
    Any help or suggestion is highly appreciated.
    Dan



  • Comes up in search.

    Are you using vpn or 2 wans?


  • Banned

    No VPN and one WAN.

    Only suggestion in a search I could find was the Conservative setting.
    I also increased the nmbclusters to 1000000. I am using all igb interfaces.
    All the offloading is checked/disabled.

    PA, RA, FA happens on all the interfaces and happens 20-30 times a minute. The good traffic does eventually get through but I have a lot of broken packets.


  • Banned

    Here is an example in my firewall log. There has to be something wrong here!




  • https://doc.pfsense.org/index.php/Why_do_my_logs_show_"blocked"_for_traffic_from_a_legitimate_connection

    Any time there are excessive ACKs being dropped, it's almost always out of state traffic.  Weird that it's logging a block between 20.1 and 20.2.  Any wireless clients on the network?  They are notorious for leaving connections hanging and switching networks on a whim.


  • Banned

    No there are not any Wireless networks, just one WAN and 2 subnets (192.168.1.0/24, 192.168.20.0/24)
    I am also not doing any Clustering or Load Balancing.

    My issue is that this is a new problem.

    So what else could be causing this issue?



  • First off, are you actually seeing any problems on your network or are you just reacting to log spam?


  • Banned

    I haven't noticed any issues, but these logs alarmed me since I have never seen this before. I suspect my internal network speed must be slower if there are all these bad packets but not sure how to test that.


  • Rebel Alliance Global Moderator

    Some of those are to loopback, 127.0.0.1

    What are you running on pfsense on port 19006??


  • Banned

    As far as I know nothing is running on port 19006. But those eventually disappeared.

    The real issue is that every connection to PFsense gets a PA, RA, or FA before it goes through. Even local to local connections. Every HTTP access gets one too, but then goes through.
    There is something wrong here when every incoming or outgoing packet drops before it goes through.

    Any suggestions?



  • FA and RA are just FIN and Reset packets. They're trying to close the connection. No point worrying about blocking packets that are meant to kill a connection if the connection is already dead.



  • Post screens of your LAN configuration and firewall rules.  Something screwy is going on.  Local traffic doesn't hit the firewall at all, and I noted this when I said it was funny that it's logging a block between 192.168.20.1 and 192.168.20.2.  No VLANs configured?


  • Rebel Alliance Global Moderator

    Are you running some sort of port forward or weird nat reflection setups… I agree with I can think of nothing that would be running on pfsense that listens on that port.. Post up a output of sockstat -4 -l

    example of mine

    
    [2.3.2-RELEASE][root@pfsense.local.lan]/root: sockstat -4 -l
    USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
    root     php-fpm    4623  5  udp4   *:*                   *:*
    root     radiusd    76653 13 udp4   192.168.2.253:1812    *:*
    root     radiusd    76653 14 udp4   192.168.2.253:1814    *:*
    proxy    ftp-proxy  68730 3  tcp4   127.0.0.1:8021        *:*
    ladvd    ladvd      38973 9  udp4   *:*                   *:*
    root     ladvd      38787 5  udp4   *:*                   *:*
    dhcpd    dhcpd      12642 16 udp4   *:67                  *:*
    dhcpd    dhcpd      12642 20 udp4   *:64384               *:*
    unbound  unbound    93283 4  udp4   192.168.9.253:53      *:*
    unbound  unbound    93283 5  tcp4   192.168.9.253:53      *:*
    unbound  unbound    93283 8  udp4   192.168.2.253:53      *:*
    unbound  unbound    93283 9  tcp4   192.168.2.253:53      *:*
    unbound  unbound    93283 15 udp4   192.168.3.253:53      *:*
    unbound  unbound    93283 16 tcp4   192.168.3.253:53      *:*
    unbound  unbound    93283 19 udp4   192.168.4.253:53      *:*
    unbound  unbound    93283 20 tcp4   192.168.4.253:53      *:*
    unbound  unbound    93283 21 udp4   192.168.6.253:53      *:*
    unbound  unbound    93283 22 tcp4   192.168.6.253:53      *:*
    unbound  unbound    93283 25 udp4   192.168.7.253:53      *:*
    unbound  unbound    93283 26 tcp4   192.168.7.253:53      *:*
    unbound  unbound    93283 27 udp4   127.0.0.1:53          *:*
    unbound  unbound    93283 28 tcp4   127.0.0.1:53          *:*
    unbound  unbound    93283 31 tcp4   127.0.0.1:953         *:*
    root     openvpn    31208 6  udp4   24.13.snipped:4142     *:*
    root     openvpn    18443 6  udp4   24.13.snipped:1194     *:*
    root     openvpn    14412 6  tcp4   24.13.snipped:443      *:*
    root     syslogd    49062 7  udp4   192.168.9.253:514     *:*
    root     ntpd       44109 21 udp4   *:123                 *:*
    root     ntpd       44109 23 udp4   192.168.9.253:123     *:*
    root     ntpd       44109 26 udp4   192.168.2.253:123     *:*
    root     ntpd       44109 29 udp4   192.168.3.253:123     *:*
    root     ntpd       44109 31 udp4   127.0.0.1:123         *:*
    root     ntpd       44109 34 udp4   192.168.4.253:123     *:*
    root     ntpd       44109 36 udp4   192.168.5.253:123     *:*
    root     ntpd       44109 38 udp4   192.168.6.253:123     *:*
    root     ntpd       44109 41 udp4   192.168.7.253:123     *:*
    root     nginx      40900 6  tcp4   *:443                 *:*
    root     nginx      40900 8  tcp4   *:80                  *:*
    root     nginx      40737 6  tcp4   *:443                 *:*
    root     nginx      40737 8  tcp4   *:80                  *:*
    root     nginx      40531 6  tcp4   *:443                 *:*
    root     nginx      40531 8  tcp4   *:80                  *:*
    root     xinetd     28003 0  udp4   127.0.0.1:6969        *:*
    root     sshd       13927 5  tcp4   *:22                  *:*
    root     php-fpm    264   5  udp4   *:*                   *:*
    [2.3.2-RELEASE][root@pfsense.local.lan]/root:
    
    

  • Banned

    KOM - See attached configs, I do use a Virtual IP that redirects to an assigned IP, No VLAN's

    Johnpoz- No port forwards to port 19006. I use a few rules to direct email traffic to the correct IP

    As far as the port 19006 hits, saw more this morning, so I setup a TCP port monitor to capture what process is causing it. But it looks like PFsense is generating the packets by looking at the sockstat

    Here are the Firewall rules and sockstat

    I noticed the extended Internet daemon port in the sockstat list. What does this mean?










  • Your rules are a bit of a mess.  Rules are applied to traffic entering an interface.  You can delete almost all of your LAN/OPT1 rules and replace them with a single Allow Any rule on each.  Generally, you don't specify a Source since the network the traffic is coming from is the source.  For example, on your LAN rules you don't need to specify Source as LAN Net since no other traffic is going to be coming into the LAN interface other than LAN Net traffic.  Those 2 Allow All rules at the bottom of your WAN rules needs to go, pronto.

    Clean your rules up and this might help eliminate any weirdness going on.


  • Rebel Alliance Global Moderator

    Yeah with KOM here those rules are complete mess..

    Your lan rule is any any at the top, for udp/tcp why is it called allow email??  What email runs on udp?

    Anyhow - all the rules below that are just pointless.

    Rules are evaluated top down, first rule wins, rest are not even looked at.  As the packets enter the interface.

    Those rules on your wan are BAD!!!

    What do you have in your xinetd conf?

    cat /var/etc/xinetd.conf

    What packages do you have installed other than pfblocker?


  • Banned

    Thanks for taking the time to help with these rules.

    As for the LAN rules, see change below. I simplified it to any-any

    As for the WAN rules, I have trimmed them down to the ones needed for the NAT port forwarding, see change below.

    So in actuality, I should have only the WAN direct ports and have all the other interfaces pass any to any. I have no need for any restrictions on any of my internal networks. The only blocking I care about is from the WAN. But I do know that you also have to use the rules to direct traffic, as in the WAN rules below.

    Don't I need a rule to allow internet traffic to my LAN. It does seem to work without one.





  • Netgate

    Not sure why you are not using protocol any on your LANX rules if you really want no restrictions between LAN interfaces.


  • Rebel Alliance Global Moderator

    "I do know that you also have to use the rules to direct traffic"

    Huh?  Sure if you want to do policy routing out a specific gateway or vpn connection, etc.  But lan to opt etc.. Or just out the default gateway no there is no need to "direct" anything.

    If you really want no restrictions than that rule should be any not tcp/udp.  So you won't be able to ping stuff with that setup.  Even though you would be able to hit http..

    So what is the content of your xinetd.conf ??
    cat /var/etc/xinetd.conf

    You clearly had something listening on that 19006 port..



  • Ehm, no. Directly connected networks are known to the system by their routing table entries that do not need a gateway entry in the table. Gateways are only needed for "foreign" destinations, i.e. networks that are not directly connected to the system. The best example is of course the default gateway which is the 0.0.0.0/0 entry (often marked as "default" as it is in pfSense also) in the routing table, it's not a directly connected network so in order to reach it a gateway has to be configured. Like so in my pfSense system (public IPs censored):

    
    $ netstat -nr -f inet
    Routing tables
    
    Internet:
    Destination        Gateway            Flags      Netif Expire
    default            88.195.aaa.1       UGS         em1
    10.0.0.0/8         127.0.0.1          UGS         lo0
    10.71.14.0/24      link#2             U           em0
    10.71.14.1         link#2             UHS         lo0
    88.195.aaa.0/19    link#3             U           em1
    88.195.bbb.ccc     link#3             UHS         lo0
    127.0.0.1          link#7             UH          lo0
    172.16.0.0/12      127.0.0.1          UGS         lo0
    192.168.0.0/16     127.0.0.1          UGS         lo0
    192.168.1.0/24     link#3             U           em1
    192.168.1.200      link#3             UHS         lo0
    
    

    Here the 88.195.aaa.0/19 is a directly connected network (the WAN network), so is my LAN network of 10.71.14.0/24. The system knows how to reach hosts on those networks without a need to send the traffic to a gateway by issuing an ARP query on the connected network to figure out which MAC address the traffic should be sent to on the ethernet level.

    The default gateway, the one that is needed to reach the "world out there, the internet" is the first line that says "default            88.195.aaa.1      UGS        em1". This says that in order to connect to any IP address/network that does not match an entry in the routing table of this system forward the traffic to address 88.195.aaa.1 and not very surprisingly the routing table also has instructions on how to reach that address, the "88.195.aaa.0/19    link#3            U          em1" line.


  • Banned

    I did change the LAN tcp/udp to any right after I posted the rules.

    See attached the xinetd file. Why are those 19001-19008 ports in there? Is this normal?

    Here is my routing table. The first 2 entries are DNS. Does this look ok?

    Routing tables

    Internet:
    Destination          Gateway            Flags    Netif Expire
    default                  xx.xx.129.113    UGS      igb2
    68.105.28.16        xx.xx.129.113    UGHS    igb2
    68.105.29.16        xx.xx.129.113    UGHS    igb2
    xx.xx.129.112/28  link#3                U            igb2
    xx.xx.129.114      link#3                UHS        lo0
    xx.xx.129.117      link#3                UHS        lo0
    xx.xx.129.117/32  link#3                U            igb2
    xx.xx.129.124      link#3                UHS        lo0
    xx.xx.129.124/32  link#3                U            igb2
    127.0.0.1              link#7                UH          lo0
    192.168.1.0/24    link#4                U            igb3
    192.168.1.1          link#4                UHS        lo0
    192.168.3.0/24    link#5                U            igb4
    192.168.3.1          link#5                UHS        lo0
    192.168.10.0/24  link#2                U            igb1
    192.168.10.1        link#2                UHS        lo0
    192.168.20.0/24  link#1                U            igb0
    192.168.20.1        link#1                UHS        lo0

    xx.xx.129.113 is the default gateway assigned by ISP
    xx.xx.129.114-125 is my assigned IP block. Currently only using 114,117,124
    192.168.3.0/24 is the subnet used for the wireless router - OPT3
    114-WAN/igb2, 117-OPT1/igb0, 124-OPT2/igb1, LAN/igb0, OPT3/igb4

    xinetd.txt


  • Banned

    The reason I said "I do know that you also have to use the rules to direct traffic" is because there was a time in the past where the LAN any-any rule would not work for some devices on the same LAN subnet unless I gave it a specific rule. That does not seem to be the case now, so any-any is working for all devices on the LAN subnet as it should.

    Also, that statement does seem to be true for the WAN where there is no any-any rule. Or any interface which does not have an any-any rule.
    So, does my posted new WAN rules look ok?

    Dan


  • Banned

    Here we go again with the port 19006, see below.

    192.168.1.2 is my main desktop that I use. I had a TCP monitor running and it did not capture this.



  • Netgate

    Then you weren't capturing correctly or something else on your network is sending those packets from that IP address.


  • Banned

    Maybe I missed it. can't predict when it happens, but will leave the capture tool running on 192.168.1.2. I have it filtered for incoming and outgoing port 19006.
    Is the xinetd.conf I posted earlier correct? It shows 192.168.20.2 tied to port 19006. Why does it do this?


  • Netgate

    Probably a package. What have you installed and why?

    there was a time in the past where the LAN any-any rule would not work for some devices on the same LAN subnet unless I gave it a specific rule.

    Poppycock. The firewall NEVER gets in the way there unless you are bridging interfaces or some other edge case.


  • Netgate

    Diagnostics > Command Prompt

    Execute: cat /var/etc/xinetd.conf


  • Rebel Alliance Global Moderator

    how exactly would the firewall even see that traffic to loopback.. Your pc if wanting to talk to a 127.0.0.1 address wouldn't even put it on the wire, that is localhost.  That traffic doesn't go out on the wire.

    So that has to be coming from your firewall, or some sort of port forward that you send to loopback?

    service 19006-tcp
    {
    type = unlisted
    bind = 127.0.0.1
    port = 19006
    socket_type = stream
    protocol = tcp
    wait = no
    user = nobody
    server = /usr/bin/nc
    server_args = -w 2000 192.168.20.2 993
    }

    Your running something with NC… netcat, not sure of what package or config settings would create those.. I sure and the hell do not have them that is for sure.



  • @johnpoz:

    .. I sure and the hell do not have them that is for sure.

    Hmmm, scary stuff for a firewall… 8)

    Netcat is often referred to as a "Swiss Army knife" utility, and for a good reason. Just like the multi-function usefulness of the venerable Swiss Army pocket knife, netcat's functionality is as helpful. Some of its features include port scanning, transferring files, port listening and it can be used a backdoor.

    [http://www.catonmat.net/blog/unix-utilities-netcat/]

    imaps 993 udp imap4 protocol over TLS/SSL


  • Rebel Alliance Global Moderator

    yeah not sure what he is doing, or what would of done that..

    Yeah NC is very powerful tool.. Why there would be stuff like that setup in his xinetd I have no idea.. The only thing that was in mine the tftp proxy, and I rem them out because not using it and was just causing log spam.

    I am not a nc guru by any means, but looks like to me if sees traffic on loopback to port 19006 send it over to that 192.168.20.2 IP on port 993.

    I do believe that if you setup nat reflection that pfsense starts with ports 19000, so you had prob setup some sort of nat reflection.  Or if he has port forwards and has it using nat reflection these sorts of entries would be put in.

    Maybe this is caused by having auto nat reflection enabled??  I personal see nat reflection as an abomination that should be killed with greek fire whenever possible.. Looking back at his firewall rules he does have some port forwards to that 20.2 IP..  But he has the ports all hidden in an alias.  So he has some sort of nat reflection going on and then some weirdness is causing out of state..

    Again going to state this for the record that nat reflection is an abomination… Turn it off and your problems will go away would be my guess..



  • @johnpoz:

    …I rem them out because...
    ... Turn it off and your problems will go away would be my guess..

    To disable a service config, add "disable =yes", (then exec the usual 'killall HUP xinetd'), like in:

    
    service 6969-udp
    {
            disable = yes
            type = unlisted
            bind = 127.0.0.1
            port = 6969
            socket_type = dgram
            protocol = udp
            wait = yes
            user = root
            server = /usr/libexec/tftp-proxy
            server_args = -v
    }
    
    

  • Banned

    So I take it that these xinetd.conf entries are for NAT reflection?

    I am using NAT reflection on all my port forwards NAT+Proxy. I did that so the LAN can communicate with other interfaces

    So, if I should not be using NAT Reflection, should I setup rules instead?
    I know for a fact if I turn off NR I cannot open my websites from 192.168.1.2 to the web server @ 192.168.20.2 or load my config page on the NAS @ 192.168.10.2. NR solved all the local communication.

    I want the LAN subnet to be able to connect to OPT1/2/3. NAT Reflection does this for me.

    Here are my Port Forward rules. NAT Reflection Enabled (NAT+Proxy) on first three, system default (Pure NAT) on the last rule




  • @dcol:

    …So, if I should not be using NAT Reflection, should I setup rules instead?

    No.
    Split DNS. Tell your DNS server to point to your local servers, case a LAN-host requests that global server address of yours.


  • Banned

    I use my ISP's DNS and DNS Resolver in PFsense. Do not have a local DNS Server setup.

    Are you saying no to using NAT reflection altogether and find a different method. Or just to using rules.

    So johnpoz, why the boo.. against NAT Reflection?



  • Have a look see at Services / DNS Forwarder / Host Overrides


  • Banned

    I do not use DNS Forwarder, I use Resolver, but I do see the host override in there.

    So if I put / Host-www / Domain-mydomain.com / IP-192.168.20.2 / in there, 192.168.1.0/24 and 192.168.3.0/24 and 192.168.10.0/24 will be able to get to www.mydomain.com on 192.168.20.2

    I host a lot of domains, so I guess I would have to have a list of all of them including all the sub domains. NAT reflection seems easier to me.


  • Banned

    I'm going to start a new thread on the DNS Resolver host override issue and lock this one. This thread has too many issues that are just compounding.


Locked