VMware Workstation VMs Web Traffic Being Blocked

dfinjr

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

https://www.imperva.com/blog/mtu-mss-explained/

Just because I wanted to see what would happen. I turned on MSS clamping for the WAN and traffic started passing just fine to the VMs.

Cisco had to of been doing this. For giggles I just started with 1434 for the MSS setting on the wan and the traffic is passing. Doing some other tests to see what that did...

dfinjr

Get this! Remember that firewall fail I was seeing out of my DMZ over to my VMs (port 52311)? I was still seeing failures there so I figured "why not" and applied 1434 everywhere else and what was looking like a failing firewall rule disappeared. The DMZ is able to register to the hosting server now no problems.

I mean I am happy with the result and all but I can't say I have a solid handle entirely around why Cisco was doing this without a single mention of it... kind of blows my mind really.

Would you guys like to see anything from any of the systems to perhaps see if things are good now?

Does 1434 make sense to use? I sort of pulled it out of the air from the Cisco packet captures...

stephenw10

An mss like 1380 to give a packet size if 1434 is commonly seen for VPN connections. Azure recommend 1350 for example.

I would run some test pings to determine what the actual packet size is you can pass on each route. I would not expect to need that between the internal interfaces at all but certainly not as low as that.

Steve

johnpoz

@dfinjr are you running through some odd ball internet connection or a vpn? Some sort of tunnel? If not you should really be using the standard 1500, and you shouldn't have to do anything. If there is something odd out there on the internet where it or its path to it has some lower than 1500 mtu, then PMTUD (Path MTU Discovery) should figure this out and use a lower mtu.

https://en.wikipedia.org/wiki/Path_MTU_Discovery

This is also why in a syn, the size is sent.. Hey I can talk at this mtu, can you? And the server your talking to answers, etc.

dfinjr

Sorry for the delay in response everyone. Been traveling the last few days and haven't been in a position to run further things or keep tabs on the forum but I'm back now.

@stephenw10 I'll happily do some tests but for those tests to be meaningful do you want me to wipe out the MSS I put in there? Also, are you able to supply me with an example line you'd like for me to do around the ping tests? (I am new with modified pings and not 100% certain about the construction of the line and what size to set it to and so forth and I don't want to do it wrong and give you bad data back)

@johnpoz I am running normal/non-special consumer level Xfinity/comcast internet. No VPNs involved. No Tunnels. I totally agree, I do have the MTUs set to 1500 but then the traffic wasn't arriving correctly only after the MSS setting did the system start getting love. I have never had to do anything like this in the past and cannot answer why we have to do it now. I never tried lowering the MTU specifically because I didn't see anything in Cisco to make me think that anything but 1500 MTU was running prior. I don't recall if things changed between the syn before/after the mss change. Do you want me to run a capture to try to pull it in with the new mss setting and would that be valuable?

johnpoz

@dfinjr you should remove whatever clamping mss you set.

And then need to make sure you capture the syn and syn,ack of your connection. I looked back at old pcap you posted, but the one with the large 1753 mtu, the capture did not have the syn and syn,ack of the start of that conversation.

On a normal 1500 mtu, when you do a ping with df bit set and size... you should be able to ping at 1472, while 1473 would fail... This has to do with overhead, etc We could have a class on frame size and what determines what in mtu, etc.. if need be..

But why you set 1472 is the 28 bytes overhead, 8 Bytes for icmp, and 20 for IP.

So for example... I can ping google with full 1472...

$ ping -f -l 1472 www.google.com

Pinging www.google.com [172.217.1.100] with 1472 bytes of data:
Reply from 172.217.1.100: bytes=68 (sent 1472) time=15ms TTL=116
Reply from 172.217.1.100: bytes=68 (sent 1472) time=14ms TTL=116
Reply from 172.217.1.100: bytes=68 (sent 1472) time=11ms TTL=116
Reply from 172.217.1.100: bytes=68 (sent 1472) time=12ms TTL=116

Ping statistics for 172.217.1.100:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 11ms, Maximum = 15ms, Average = 13ms

This means I have 1500 mtu all the way from my connection to them.

If I try and send 1473.. It fails.

$ ping -f -l 1473 www.google.com

Pinging www.google.com [172.217.1.100] with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 172.217.1.100:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

So what specific site are you having issues with? For example I was using amazon.com to test because once you click into a category for something loads of pictures and such that generate large packets..

So on one of these sites your having issues with - start ping at 1472 if fails drop it down until it passes..

But from here for example I can ping amazon.com with 1472..

$ ping -f -l 1472 www.amazon.com

Pinging d3ag4hukkh62yn.cloudfront.net [52.84.54.224] with 1472 bytes of data:
Reply from 52.84.54.224: bytes=1472 time=21ms TTL=234
Reply from 52.84.54.224: bytes=1472 time=15ms TTL=234
Reply from 52.84.54.224: bytes=1472 time=15ms TTL=234

Problem is - you don't always know what your talking to - when you talk to www.amazon.com your going to be talking to multiple IPs.. sure images served up from 1 IP, scripts from another IP, etc. etc.. Most sites are no longer just 1 IP that handles everything being loaded from that site, etc.

Same goes for speedtest.net - the website IP(s) is can promise you not the IP you actual IP your downloading or uploading too for the actual speedtest part..

stephenw10

MSS only does anything to TCP so I would expect to be able to run that ping test with it still in place. But your network has shown....unexpected behaviour! So I would remove it to be sure.

Steve

dfinjr

@johnpoz
Understood completely. Thank you for the detailed explanation that helped me a lot and thank you for the example. I am leaning on the system access at the moment but I think I'll be able to do that test here shortly. As for the primary sites I was having issues with originally, they were pfsense.org, speedtest.net and gmail.com.

As soon as my system dependency frees up I'll perform that test and share the results.

dfinjr

Sorry for the delays everyone. I am working on shaking a moment free with work to do the tests. as soon as I do I will post them here.

dfinjr

@johnpoz

Here are the pings you requested. No MSS is enabled and the websites are again failing to browse:

The specific sites that I had trouble with this time was speedtest.net and gmail.com. I used to have problems with pfsense.org but this time it loaded (not sure why, probably cached content or something).

Test with gmail.com: (worked)

Test with speedtest.net: (timed out 1472)

...worked at 1434, going up...

failed at 1454... (chose randomly)

failed at 1444... (truncated till I found what did/didn't work)

continued to work up to 1440 but failed at 1441:

Hope these ping tests tell you something @johnpoz @stephenw10

Uploaded a failed capture again, longer one for 80+443 from the client that has been subject to this conversation most of this forum post (172.16.0.202). Here is the link for that again:
https://drive.google.com/drive/folders/14C1MTTuwjUnvNYgDJfBy0gmiSQmO5HTQ?usp=sharing

Again, sorry it has taken me so long to get you all this info work has been kicking my butt.

Please let me know what you think and if there is any other testing you would like.

Thanks again for the attention to this.

johnpoz

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

Test with speedtest.net: (timed out 1472)

Well you got something wrong with your isp or the path from your isp to them... Works just fine here

What about loading www.speedtest.net - with packet capture so we can see what your syn and the answer syn,ack has in them.

dfinjr

@johnpoz

Sure would be happy to... Without MSS I am guessing?

johnpoz

@dfinjr example

dfinjr

@johnpoz
With MSS on:

MSS=1394

With MSS off:

MSS=1430

johnpoz

@dfinjr where did you sniff that.. What are you looking at exactly for when you see 1430?

Which packet.. Where exactly did you set MSS - what interface?

How could the SYN that has not yet entered pfsense be changed?

Need to see the SYN that you send, and what the syn,ack says in response.

If I change the MSS on pfsense lan, to 1400, and do a sniff on lan, I still see 1460, if sniff on wan.. see 1360 (which is 40 minus what I set of 1400)

stephenw10

That looks like exactly like what he's seeing. The SYN goes out at 1460. The SYN-ACK comes back at 1430 or 1394 depening if MSS is set or not.

Really the only question I have here is why this is necessary. I expect path MTU to discover that if there really is a restriction somewhere.

Steve

johnpoz

@stephenw10 said in VMware Workstation VMs Web Traffic Being Blocked:

The SYN goes out at 1460

Where are you seeing that?

Oh I see it now in the info section.

stephenw10

Mmm, yeah if you look at the latest pcap and then look at, for example, tcp.stream eq 11.
You can see the syn-ack comes back with mss 1430 but it still fails. Whatever packets it's sending are not passed through VMWare.

dfinjr

@johnpoz

Hi there.

I did that sniff at 172.16.0.202. What I was looking for was the matching public IP address right at the beginning of the TCP conversation between me and speedtest.net. Pretty much client hello then looked for the [SYN, ACK] to see the options for that digging out that MSS. At that point though the server has already responded. I think I found what you are after though under the [SYN, ECN, CWR] where the client is outlining the MSS: (tell me if I am wrong)

I just went through the capture from the no MSS setting and I don't even see that packet originating like what I saw in the screen shot directly above...

On the PFSENSE I honestly have set all my interfaces with the MSS setting at 1434 because then everything started playing ball again. So this would be existing on all interfaces.

Am I missing it or is the capture with MSS on not coming up with that packet as what I showed in the screen shot above from when MSS is enabled?

@stephenw10
I have no idea why it is necessary... that's why I am still here to try and figure it out :)

On you last commend I am kind of hanging on that. I am not modifying anything within the VMWare configuration at all, only MSS on the PFsense... But how is that possible? If the packets weren't getting out of VMWare then the changes in firewall wouldn't matter right? So why do they?

I obviously can't make sense of this yet...

stephenw10

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

If the packets weren't getting out of VMWare then the changes in firewall wouldn't matter right? So why do they?

They do get out. But when they come back the large packets are not passed back to the VM by VMWare. The pcaps you ran on the host showed the reply packets arriving there but pcaps on the VM itself did not show them.
I'm assuming that last pcap was on the VM?

For example. Look at this tcp stream from packetcapture-5:
Screenshot from 2022-03-17 22-45-07.png

I wish I knew some better way to show that on the forum. Suggestions welcome!

The client send mss 1460. The server responds 1430.
But when the server uses 1430 (1484 total packet size) the client never sees it. It just sends a duplicate ACK.