Able to ping, nslookup and curl in pfSense box but curl failed in clients

rcoleman-netgate · Oct 10, 2022, 2:21 AM

@mltobing What's between your pfSense and the world?
This suggests the issue does not lie in your pfSense but on the next step out - it goes in the pfSense and out the WAN port but nothing is coming back.

mltobing · Oct 10, 2022, 2:29 AM

@rcoleman-netgate That really strange. There is ISP modem after pfSense box. I removed pfSense box and change it with OpenWRT AP all working fine.

If you think like that, now I am not sure if this issue related to Intel Atom box or the NICs

rcoleman-netgate · Oct 10, 2022, 2:41 AM

@mltobing Are you running any type of VPN?

mltobing · Oct 10, 2022, 2:58 AM

@rcoleman-netgate I am not running any type of VPN

I forgot if we cannot curl, because of that we didn't get any reply. But we have no issue with ping, so I tried to capture and we got reply.

Strange. Why pfSense box allowed ICMP and nslookup but blocked other traffic. On my laptop I checked the network status "internet access" but I cannot browsing the internet.

Thanks for your fast response. I will go out and reply you later

mltobing · Oct 10, 2022, 2:56 PM

@rcoleman-netgate I tried this scenario to get packets from OpenWRT. I ran curl on my laptop first then pfsense box

pfSense WAN captured packets from both of them but didn't forward requests from my laptop to OpenWRT. We can see OpenWRT captured packets after 20:55:48 only. Do you know why pfSense WAN didn't forward packets from my laptop?

stephenw10 · Oct 10, 2022, 3:44 PM

It's not that pfSense is not forwarding the responses it's that it never gets any responses to forward. For some reason.

There must be some difference between the packets from the client and those from pfSense. The TTL would be different for example.

The pcap on openwrt doesn't show any of the traffic from the laptop behind pfSense. Was is started after that had failed?

Steve

stephenw10 · Oct 10, 2022, 3:50 PM

My number one suspect here would be the USB NIC you're using except you have that as LAN and it appears to be passing inbound there.
What is the WAN NIC in that device? What hardware off-loading do you have enabled?

Steve

mltobing · Oct 10, 2022, 6:07 PM

@stephenw10 The NIC assignment were correct. If I set them wrong, WAN interface didn't get any IP Address.
Because this is initial setup I didn't change any hardware offloading setting.

I tested again and this time I captured OpenWRT LAN interface. pfSense WAN directly connected to OpenWRT LAN.
Traffics forwarded from my laptop don't have TS val, but from pfSense have.

stephenw10 · Oct 10, 2022, 6:27 PM

Ok so pfSense is sending that traffic as expected and OpenWRT sees it on the LAN but no replies.
Are there any replies on the OpenWRT WAN?

You're probably going to need to open that in Wireshark and look at it more closely.

You might also connect the laptop to OpenWRT directly and pcap the same traffic there when it succeeds.

https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps

TCP timestamps are enabled by default In Linux kernel.,[27] and disabled by default in Windows Server 2008, 2012 and 2016.[28]

Steve

mltobing · Oct 10, 2022, 7:23 PM

@stephenw10 I am not familiar with packet inspection. Is there something I need to check ?

This is the packets capture result on OpenWRT LAN interface

Top = my laptop => pfSense => OpenWRT (unable to connect to internet, no reply from OpenWRT WAN)
Bottom = my laptop => OpenWRT (able to connect to internet)

stephenw10 · Oct 10, 2022, 7:30 PM

OK, it looks like you may have some asymmetric routing somewhere.

In the working pcap you can see there is traffic in that TCP session that did not pass the interface.

By far the most likely is that the laptop you're testing from has some other connection. Like maybe it has wifi directly to the ISP "modem".

pfSense will block out of state TCP traffic like that. Lesser firewalls may not.

Do you see any blocked TCP traffic in the pfSense firewall log?

It's still hard to see how that could happen though because each device is source NATing on the way out...

But that fact the ping works also points to that.

Steve

mltobing · Oct 10, 2022, 8:19 PM

@stephenw10 I did so many tests but traffics filtered by destination only these

stephenw10 · Oct 10, 2022, 9:11 PM

Hmm 192.168.1.101 should not be blocked inbound on LAN like that. Assuming that's still the LAN subnet?

But if it was that traffic would not appear on the pfSense WAN

Are you able to upload those pcaps so I can review them?
https://nc.netgate.com/nextcloud/s/9jFb69WaCHzfSRB

Steve

rcoleman-netgate · Oct 10, 2022, 9:16 PM

@stephenw10 That is a SYN packet - it could be because of a closed state or a bad state reference (like you commented about asymmetry).

stephenw10 · Oct 10, 2022, 9:40 PM

SYN should be passed. Any other flag combo might be dropped. Hard to see where another route might exist though since the laptop is already connected via wifi.

mltobing · Oct 10, 2022, 10:01 PM

Files uploaded. I thought I disabled the wifi during the test

mltobing · Oct 10, 2022, 11:27 PM

@rcoleman-netgate @stephenw10 @viragomann and the others on this thread. Thank you very much for your help.
This issue resolved after I checked "Disable hardware checksum offload" on System / Advanced / Networking (default uncheck).
I can access pfsense.org and other websites now

stephenw10 · Oct 10, 2022, 11:31 PM

Ah, that will do it. I should have pressed that question when I asked it earlier. Lesson for today.

Good result.

Steve