PfSense blocks traffic to a single HTTPS website

maxxer

I've a weird situation in a recently installed pfSense: they cannot load an https website. With the former firewall (Endian) they had no issues, now that we upgraded to pfSense the site is not loading anymore. We tried connecting a laptop directly to the upstream router and the site loads instantly, os it must be something on the pfSense side.

We're running 2.3.4 (but was happening even with 2.3.3) installed as a XenServer7.1 VM. We've run these two commands on the LAN interface of pfSense's VM to fix tcp offloading:


xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

While most of the https sites work (at least all the ones I tried) and http has no problem at all, this specific site (www.ubibanca.com) won't load from any browser behind the firewall. I managed to capture a dump on pfSense's wan interface, here's a Wireshark screenshot. I'm not familiar with TCP protocol, but from what I can see pfSense is apparently not replying to the webserver's ACK packets.

What can cause this?
thanks

pfsense-packetcapture.jpg_thumb

JKnott

Well, it looks like the packets need to be fragmented, because there's a hop with a smaller MTU along the path, but the router can't fragment. That would be because the do not fragment bit is set. What is the MTU set for? What type of Internet connection do you have? What operating system? Where are those ICMP packets coming from? You have the address hidden.

BTW, if you're providing networking support, knowing TCP/IP is part of the job.

johnpoz

"pfSense is apparently not replying to the webserver's ACK packets."

It wouldn't be pfsense that would be responding or not responding - it would be the client. Check on the client with a sniff..

Looks like frag needed keeps being sent, so you yeah you got something going on there.. JKnott is correct on all counts, what is your MTU set for - do you have do not fragment set? What sort of connection do you have?

Notice when I go to that same site www.ubibanca.com, get the redirect to https - but then only 1514, not 1526 - Are you doing any sort of QinQ maybe?

redirect.png_thumb

wansniff.png_thumb

maxxer

@JKnott:

Well, it looks like the packets need to be fragmented, because there's a hop with a smaller MTU along the path, but the router can't fragment. That would be because the do not fragment bit is set. What is the MTU set for? What type of Internet connection do you have? What operating system? Where are those ICMP packets coming from? You have the address hidden.

MTU is at default value, 1500. pfSense is attached to a Microtik with an Hyperlan (wireless) connection.

All the hidden addresses are pfSense's WAN address (which is a static ip address), so the ICMPs are generated by the firewall itself. Sadly the remote endpoint doesn't reply to those packets.

Setting MTU/MSS to 1492/1452, and checking Clear invalid DF bits instead of dropping the packets (as suggested here) improved the situation, since now the site loads (even if not at a great speed)…

Thanks for your help, I will investigate more on the best size for MTU/MSS

JKnott

@johnpoz:

JKnott is correct on all counts

Of course! :D