Problems with Linux Clients
-
I have a wierd issue that I am not sure how to fix.
Any linux client that I deploy onto any of the LAN segments on my pfsense 1.2 release firewall will not accept return connections from outbound initiated transactions on TCP.
So, I can do an nslookup on my dns server that is on a different segment, but I cannot use HTTP, HTTPS or SSH to get to a remote server. I can even ping out from the client.
In terms of the HTTP connections: I can see the traffic pass out of the LAN interface and return from the remote host. It does the SYN-ACK sequence ok. then the client sends an HTTP get and the remote server gets it and then responds, but then a retransmission is sent from the client and it stops listening. After a while the server sends a reset. Then I close the browser and the client sends a bunch of fins, but never gets a response. The Server side shows the fins, and it sends a bunch of resets.
I have tried multiple distro's of linux and all show the same symptoms. I have tried physical and vmware (esx) based virtual machines and the same happens.
The rub it that If I use a Windows client, everything works fine. No hiccups at all. If I use the linux client to connect to other machines on the same subnet, all works fine with the linux client.
I am not sure what to do.
I have created rules that allow any protocol from any machine to any machine on the LAN and remote network sides. I have made specific rules too. Nothing seems to work for Linux. There is not NATing involved.
Please help. What could be wrong?
Bellow are dumps from the firewall interfaces:
Linux Client : 2.2.2.2
Remote Server: 1.1.1.1Linux client initiating the connection:
12:00:50.883156 IP (tos 0x0, ttl 64, id 43740, offset 0, flags [DF], proto: TCP (6), length: 60) 2.2.2.2.52755 > 1.1.1.1.80: S, cksum 0x3690 (correct), 3937195275:3937195275(0) win 5840 <mss 6="" 557015="" 1460,sackok,timestamp="" 0,nop,wscale="">12:00:50.883750 IP (tos 0x0, ttl 126, id 43685, offset 0, flags [none], proto: TCP (6), length: 64) 1.1.1.1.80 > 2.2.2.2.52755: S, cksum 0xcca8 (correct), 2695844581:2695844581(0) ack 3937195276 win 64240 <mss 0="" 1460,nop,wscale="" 0,nop,nop,timestamp="" 0,nop,nop,sackok="">12:00:50.885282 IP (tos 0x0, ttl 64, id 43741, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: ., cksum 0x8826 (correct), ack 1 win 92 <nop,nop,timestamp 0="" 557018="">12:00:50.887153 IP (tos 0x0, ttl 64, id 43742, offset 0, flags [DF], proto: TCP (6), length: 439) 2.2.2.2.52755 > 1.1.1.1.80: P 1:388(387) ack 1 win 92 <nop,nop,timestamp 0="" 557018="">12:00:51.054342 IP (tos 0x0, ttl 126, id 44528, offset 0, flags [DF], proto: TCP (6), length: 52) 1.1.1.1.80 > 2.2.2.2.52755: ., cksum 0x72c2 (correct), ack 388 win 63853 <nop,nop,timestamp 557018="" 121050008="">Server Side responding to linux client:
12:00:50.883194 IP (tos 0x0, ttl 63, id 37274, offset 0, flags [DF], proto: TCP (6), length: 60) 2.2.2.2.52755 > 1.1.1.1.80: S, cksum 0x3690 (correct), 3937195275:3937195275(0) win 5840 <mss 6="" 557015="" 1460,sackok,timestamp="" 0,nop,wscale="">12:00:50.883726 IP (tos 0x0, ttl 127, id 48585, offset 0, flags [none], proto: TCP (6), length: 64) 1.1.1.1.80 > 2.2.2.2.52755: S, cksum 0xcca8 (correct), 2695844581:2695844581(0) ack 3937195276 win 64240 <mss 0="" 1460,nop,wscale="" 0,nop,nop,timestamp="" 0,nop,nop,sackok="">12:00:50.885296 IP (tos 0x0, ttl 63, id 6598, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: ., cksum 0x8826 (correct), ack 1 win 92 <nop,nop,timestamp 0="" 557018="">12:00:50.887163 IP (tos 0x0, ttl 63, id 6364, offset 0, flags [DF], proto: TCP (6), length: 439) 2.2.2.2.52755 > 1.1.1.1.80: P 1:388(387) ack 1 win 92 <nop,nop,timestamp 0="" 557018="">12:00:51.054328 IP (tos 0x0, ttl 127, id 48586, offset 0, flags [DF], proto: TCP (6), length: 52) 1.1.1.1.80 > 2.2.2.2.52755: ., cksum 0x72c2 (correct), ack 388 win 63853 <nop,nop,timestamp 557018="" 121050008="">12:00:51.160952 arp who-has 172.16.253.214 tell 1.1.1.1
12:00:55.667807 IP (tos 0x0, ttl 127, id 48600, offset 0, flags [DF], proto: TCP (6), length: 419) 1.1.1.1.80 > 2.2.2.2.52755: P 1:368(367) ack 388 win 63853 <nop,nop,timestamp 557018="" 121050054="">12:00:56.408492 arp who-has 172.16.253.25 tell 1.1.1.1
12:00:58.596275 IP (tos 0x0, ttl 127, id 48602, offset 0, flags [DF], proto: TCP (6), length: 419) 1.1.1.1.80 > 2.2.2.2.52755: P 1:368(367) ack 388 win 63853 <nop,nop,timestamp 557018="" 121050084="">12:01:04.610856 IP (tos 0x0, ttl 127, id 48603, offset 0, flags [DF], proto: TCP (6), length: 419) 1.1.1.1.80 > 2.2.2.2.52755: P 1:368(367) ack 388 win 63853 <nop,nop,timestamp 557018="" 121050144="">12:01:11.036586 IP (tos 0x0, ttl 127, id 48604, offset 0, flags [DF], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x3e86 (correct), 368:368(0) ack 388 win 0Linux client attempting to close the connection after killing the browser:
12:03:50.627766 IP (tos 0x0, ttl 64, id 43743, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xadb5 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 736757="" 121050008="">12:03:50.850007 IP (tos 0x0, ttl 64, id 43744, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xacd7 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 736979="" 121050008="">12:03:51.294488 IP (tos 0x0, ttl 64, id 43745, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xab1b (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 737423="" 121050008="">12:03:52.182324 IP (tos 0x0, ttl 64, id 43746, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xa7a3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 738311="" 121050008="">12:03:53.957874 IP (tos 0x0, ttl 64, id 43747, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xa0b3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 740087="" 121050008="">12:03:57.509722 IP (tos 0x0, ttl 64, id 43748, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x92d3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 743639="" 121050008="">12:04:04.614547 IP (tos 0x0, ttl 64, id 43749, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x7713 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 750743="" 121050008="">12:04:18.823811 IP (tos 0x0, ttl 64, id 43750, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x3f92 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 764952="" 121050008="">12:04:47.239846 IP (tos 0x0, ttl 64, id 43751, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xd091 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 793368="" 121050008="">Server attempting to close the connection after killing the browser:
12:03:50.627803 IP (tos 0x0, ttl 63, id 33551, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xadb5 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 736757="" 121050008="">12:03:50.628460 IP (tos 0x0, ttl 127, id 48834, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:03:50.850022 IP (tos 0x0, ttl 63, id 9342, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xacd7 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 736979="" 121050008="">12:03:50.850380 IP (tos 0x0, ttl 127, id 48835, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:03:51.294501 IP (tos 0x0, ttl 63, id 15217, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xab1b (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 737423="" 121050008="">12:03:51.294908 IP (tos 0x0, ttl 127, id 48836, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:03:52.182344 IP (tos 0x0, ttl 63, id 36890, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xa7a3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 738311="" 121050008="">12:03:52.183199 IP (tos 0x0, ttl 127, id 48837, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:03:53.957893 IP (tos 0x0, ttl 63, id 50233, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xa0b3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 740087="" 121050008="">12:03:53.958400 IP (tos 0x0, ttl 127, id 48838, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:03:57.509739 IP (tos 0x0, ttl 63, id 2377, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x92d3 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 743639="" 121050008="">12:03:57.510169 IP (tos 0x0, ttl 127, id 48839, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:04:04.614570 IP (tos 0x0, ttl 63, id 23654, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x7713 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 750743="" 121050008="">12:04:04.614894 IP (tos 0x0, ttl 127, id 48840, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:04:18.823833 IP (tos 0x0, ttl 63, id 13383, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0x3f92 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 764952="" 121050008="">12:04:18.824387 IP (tos 0x0, ttl 127, id 48841, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0
12:04:47.239862 IP (tos 0x0, ttl 63, id 36216, offset 0, flags [DF], proto: TCP (6), length: 52) 2.2.2.2.52755 > 1.1.1.1.80: F, cksum 0xd091 (correct), 388:388(0) ack 1 win 92 <nop,nop,timestamp 793368="" 121050008="">12:04:47.240132 IP (tos 0x0, ttl 127, id 48842, offset 0, flags [none], proto: TCP (6), length: 40) 1.1.1.1.80 > 2.2.2.2.52755: R, cksum 0x0dac (correct), 2695844582:2695844582(0) win 0</nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></mss></mss></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></mss></mss> -
Ok, update:
If I go to the advanced setup and disable all packet filtering, then the linux clients will work. Duh.
Problem of course then it is no longer a firewall.
I am not sure if it is the auto natting or something wrong in a rule.
Suggestions?
-
I haven't gone through your logs in detail.
Did you enable OS Fingerprinting in any firewall rules? You would do this by setting a specific OS type in the selection box (instead of leaving it as 'any') when editing the firewall rule.
In 1.2.2, I found that the OS Fingerprinting technique does not correctly detect my Linux clients, causing the default block rule to apply to TCP connections from Linux clients. UDP connections are unaffected because OS Fingerprinting apples only to TCP.
-
These are traces not from pfSense, right? Please run simultaneous traces on both interfaces of pfSense and post them here.
This is weird part:
12:00:51.160952 arp who-has 172.16.253.214 tell 1.1.1.1Why would your server needed to have MAC for 172.16.253.214? especially considering this is theoretically impossible.
-
This is weird part:
12:00:51.160952 arp who-has 172.16.253.214 tell 1.1.1.1Why would your server needed to have MAC for 172.16.253.214? especially considering this is theoretically impossible.
Unless you assume that the OP was obfuscating their IP addresses using search-and-replace, when that would make perfect sense ;)
-
@Cry:
This is weird part:
12:00:51.160952 arp who-has 172.16.253.214 tell 1.1.1.1Why would your server needed to have MAC for 172.16.253.214? especially considering this is theoretically impossible.
Unless you assume that the OP was obfuscating their IP addresses using search-and-replace, when that would make perfect sense ;)
Oops -))) Then why would you hide 172.16.0.0/16 waisting you fingers muscules searching and replacing???
-
I haven't gone through your logs in detail.
Did you enable OS Fingerprinting in any firewall rules? You would do this by setting a specific OS type in the selection box (instead of leaving it as 'any') when editing the firewall rule.
In 1.2.2, I found that the OS Fingerprinting technique does not correctly detect my Linux clients, causing the default block rule to apply to TCP connections from Linux clients. UDP connections are unaffected because OS Fingerprinting apples only to TCP.
I Tried it with OS Finger printing on and off. Still blocked it. sounds like you have the same problem. What did you do to fix this?
Also, in terms of the arps listed in another reply, that was my mistake for not editing them out. The packet traces are from the pfsense firewall. Assume the 172.16.x.x addresses you see are actually 1.1.1.x addresses. DOH!
-
I Tried it with OS Finger printing on and off. Still blocked it. sounds like you have the same problem. What did you do to fix this?
Similar, but not quite the same, I think. I left OS fingerprinting turned off and re-wrote a couple of the firewall rules. It wasn't essential for me, anyway.
I had an afterthought …. I have Untangle (in transparent bridge mode) in between my LAN clients and pfSense, so that may be affecting OS fingerprint detection since Untangle is supposed to work at Layer 7. If/when I get the time, I'll check it without.
Sorry that doesn't help you, though.
-
Thanks,
I upgraded to 1.2.2. Still no luck. I have all OS finger printing disabled.
It seems like the return packets from the server is not coming back in through the WAN interface.
I just dont get why windows is ok and Linux is not.
-
Thanks,
I upgraded to 1.2.2. Still no luck. I have all OS finger printing disabled.
It seems like the return packets from the server is not coming back in through the WAN interface.
I just dont get why windows is ok and Linux is not.
Correction. I found that all my rules were having return traffic blocked by the default rule. Changed the rule state setting to Keep State. Now all is working with all clients.
Thanks for getting my brain moving.