pfSense mangling packets?

johnpoz

@Tueem not sure exactly what that is - but loads this up here

user@UC:/tmp$ curl 13.107.213.67:443
<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>
<html xmlns='http://www.w3.org/1999/xhtml'>

<head>
    <meta content='text/html; charset=utf-8' http-equiv='content-type' />
    <style type='text/css'>
        body {
            font-family: Arial;
            margin-left: 40px;
        }

        img {
            border: 0 none;
        }

        #content {
            margin-left: auto;
            margin-right: auto
        }

        #message h2 {
            font-size: 20px;
            font-weight: normal;
            color: #000000;
            margin: 34px 0px 0px 0px
        }

        #message p {
            font-size: 13px;
            color: #000000;
            margin: 7px 0px 0px0px
        }

        #errorref {
            font-size: 11px;
            color: #737373;
            margin-top: 41px
        }
    </style>
    <title>Service unavailable</title>
</head>

<body>
    <div id='content'>
        <div id='message'>
            <h2>Our services aren't available right now</h2>
            <p>We're working to restore all services as soon as possible. Please check back soon.</p>
        </div>
        <div id='errorref'>
            <span>            </span>
        </div>
    </div>
</body>
</html>
user@UC:/tmp$

Notice this part

Our services aren't available right now
We're working to restore all services as soon as possible. Please check back soon.

Tueem

@johnpoz Yeah that appears to be normal (its the minecraft authentication servers).
Minecraft probably accesses these using another protocol.
But when loading it in a Browser or curl it you can see that behind the firewall not even that "not available" page is available

Tueem

This post is deleted!

johnpoz

@Tueem I sure can - that was from a client behind pfsense.

Tueem

@johnpoz Yeah but for me it isn't and I'm trying to figure out why and how to fix that.
Any Client behind my pfsense errors out which can be seen in the images I sent.

johnpoz

@Tueem your packet capture you see them send a fin.. There is your syn, then their syn,ack and and they send a fin.. Now sure how you think that points to "managled" packets. Maybe they just don't like your IP? You say it works outside pfsense - but is that IP you talk to them from the same?

Tueem

@johnpoz Yeah the WAN is just my ISPs Router. When I connect directly to the ISP router I can access the site no problem and even doing the curl in pfsense itself works fine.

johnpoz

@Tueem maybe they are looking for a specific source port in your traffic? Pfsense will change the source port when it nats, some soho/isp routers do not do this.. If your double natting and you see this when only when pfsense is behind your isp nat router.. you would get this.

client ip:X --> server IP:Y pfsense wan IP:A --> server IP:Y isp router public IP:A --> server IP:Y

Set pfsense outbound nat for a client to static nat..

You would then end up with this

client ip:X --> server IP:Y pfsense wan IP:X --> server IP:Y isp router public IP:X --> server IP:Y

stephenw10

I note in your pcap there don't seem to be any significantly sized packets. You could have an MTU issue.

Tueem

@johnpoz That did not fix it unfortunately.

Tueem

@stephenw10 Can you elaborate? I've changed to MTU settings and left all the boxes empty.

johnpoz

@Tueem well pfsense doesn't "mangle" traffic.. And if it was then you would be having way more issues than just this one thing that runs something other than actual https over the standard 443 port.

Not sure what you were doing when you sniffed that - was that during your curl test? But the server sent you a fin, so he told you he was done..

I don't play minecraft, but I guess I could try fire it up.. Do I need to create an account at some specific site so it will talk to this 13.x address? That 13.x address is owned by MS.. Prob some service hosted in azure..

What is the exact error you get when you try and play minecraft? I know my grandkids have played it on their phones while they have been here over my wifi, and they had no issues.

stephenw10

@Tueem said in pfSense mangling packets?:

Can you elaborate?

Well it's hard to see from a picture of the pcap text but it looks like it's only seeing small packets. In an established TCP connection I'd expect to see large packets at the path MTU size. So at least 1400 for most connections.

If you do not see that in the full pcap then you might have an MTU issue somewhere.

johnpoz

@stephenw10 while I agree when your moving data your packets should get bigger.. But that 13.x box sends a fin right way, all we are seeing there is the handshake.

Tueem

@johnpoz Yeah the tcpdump output was during the curl.
After further investigation I figured out that that 13.x address is the one that minecraft.net resolves to.
I host a paper-1.8 server that runs through a Wireguard VPN and when I try to connect it just tells me that the authentication servers are down. I got the IP by sniffing the firewall logs and looking for IPs my server tried to connect to.
On my PC which doesn't go through the VPN it connects fine. (I can confirm its not an issue of the VPN because if I use the official wireguard client directly it connects just fine.)

stephenw10

@Tueem said in pfSense mangling packets?:

I can confirm its not an issue of the VPN because if I use the official wireguard client directly it connects just fine.

Like from the server directly?

Tueem

@stephenw10 From my PC directly yes

stephenw10

Hmm, same server I assume?

Where did you run the pcap from above?

Does it show a similar failure when actually looking for the traffic to the minecraft auth server?

Tueem

@stephenw10 Yes the same Wireguard endpoint and same IP all the same.

The test above is from tcpdump on the pfsense box using the lan nic.

I've attached the tcpdump on the same interface which happens when i try to connect to the server (not curling)

NOTE: The IP of minecraft.net has changed but the behaviour stays the same

johnpoz

@Tueem here he is sending you a reset.. Which is not hey I am done with this conversation (fin).. He is saying GO AWAY I am done talking to you, I don't care if you still want to continue the conversation or not.