Firewall Blocks as TCP:SA and it is resolved by TCP retransmission number. Why?
-
Hello professionals,
From few weeks ago, when clients talks to server it keeps freezing. I am trying to figure out why.
so whenever, network transaction(client file upload to server) freezing happened, I got 'TCP retransmissions' from Wireshark.
when multiple files upload one by one, client tries to open different port (source port).
- File #1: Source port 10795 - succeed
- File #2: Source port 10796 - succeed
. . - File #n: Source port 10797 - Fail
In that case, client keeps sending TCP retransmissions to server because firewall blocks server's response.
I wonder why firewall suddenly blocks the server's SYN,ACK packets. Is that because of Asymmetric routing? Server's response packets could be routing different path?
One more curious thing is, when I change TCP retransmission Max number from default (4) to 8 from the client PC (Windows), no more freezing happened. It looks like problem has been resolved.
Why changing transmission number could resolve this issue? why it makes firewall no longer blocking?
I appreciate your comments.
-
@eeebbune Since you mention asymetric routing, what is your setup? Anything fancy?
-
@eeebbune blocks because its late? So that 10.110.18.88 doesn't send syn,ack back for like what 27 some seconds? I see a syn an 574.925, but no syn,ac back until 602.825 = almost 28 seconds..
See how the retrans are longer and longer apart.. If your device is taking that long to respond, setting more retrans, ie 8 vs 4 would keep the box ending syns, and could keep the state open on the firewall..
Off the top of my head I don't know how long pf would keep the state open waiting for the syn,ack back - but its not going to be forever. I have never run into such a thing in all my years in the biz.. a syn,ack normal comes back in like ms, not 27 seconds after the first syn.. Even after your last syn on retrans your what 13 seconds after? That is an insane amount of time to wait for a syn,ack.
edit
I thought the timeout was 30 seconds.. for tcp opening.. looks like your around 28 seconds, have you adjusted the tcp timings in the advanced firewall? or adjusted https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html the optimization? I think if your set to aggressive the tcp opening timeout is only 5 seconds.
-
@netblues Hello, Oh I forgot to explain.
PFSense firewall has 3 active ports(interfaces), and those are bridged.
I named for each interface,-
LAN: Core switch - Firewall connection
-
EC1: Firewall Active SD WAN connection
-
EC2: Firewall Standby SD WAN connection
-
SDWAN BRG: LAN+EC1+EC2
EC1 and EC2 has connection in order to check heartbeat (VRRP).
From the firewall block logs, incoming packets from server to client (10.110.18.88 -> a.b.c.6) are blocked in LAN interface.
This architecture has been set up for last 7 months, but suddenly happened. Couldn't find any reason.
-
-
@eeebbune said in Firewall Blocks as TCP:SA and it is resolved by TCP retransmission number. Why?:
server to client (10.110.18.88 -> a.b.c.6) are blocked in LAN interface.
And yeah if your device at 18.88 takes forever to answer the syn, the firewall can close the state it opened when something tried to talk to it..
If talking thru a firewall device A talks to B, A sends a syn.. Firewall says oh I allow that traffic - let me create a state and send on your syn.. If that box B doesn't answer that state will be closed.. If B then say oh hey here is your syn,ack like 30 seconds later firewall will say oh sorry buddy I have not state for your syn,ack
From what you posted your 18.88 box is not answering the syn in timely manner.. Which is why they are sending retrans.. because they never got an answer..
Did you adjust the optimization time in pfsense? Or manually adjust the timers in advanced? I show you right at the border of the 30 second tcp open timer.. From what I see as the first syn sent at 574.714899, and don't see a syn,ack back until 602.835140
If I send a syn to something, it should answer in milliseconds, not seconds.. If something is taking that long to respond - you have something wrong on that thing.. Or its overloaded, your running it on a potato?
-
@johnpoz Hi,
My firewall option is Normal.
However, yesterday I changed 'TCP opening' default to 60 and 'TCP first' 3600 and tested.
No progress I found.
Also, on the wireshark result, MSS value has been changed from 1342 to 1460. Could it be a matter?
Thank you,
-
@eeebbune not for the handshake.. Those are tiny syn and syn,ack
Where exactly is that sniff from, and I am not sure what you show blocked is actually the same traffic in your sniff.. Without timestamps to reference.. For all I know that is some other connection on same ports that were blocked..
Here is the thing - if what your talking to takes seconds to respond to a syn vs milliseconds - your going to have a bad day no matter how you look at it.
edit: here is an answer from a syn.. It is less than 1 ms.. Not 28 seconds.. If the device your sending a syn too doesn't answer in 20 some seconds you have something wrong with that box.. Or its just never seeing the syn at all.. Even if we take the time of the last retrans which it got??? It took 13 seconds to send a syn,ack?
here is normal sort of response time talking to something - here talking to my nas.. You can see the time sent the syn, and when got back the syn,ack
Its less than 1 ms..
here is talking to something out on the internet.. 11ms not 27 seconds..
edit2: So if this is the same session.. The ports match up, once you sent syn,ack that .6 box answered with ack in like less than 1 ms.. So why is 18.88 taking 27 seconds to respond? Maybe the hamster is tired running around the wheel powering it?
