VMware Workstation VMs Web Traffic Being Blocked
-
@johnpoz
The only VMs in play outside of testing are the ones that exist off that hosting laptop, no others outside of that. The 172.16.0.201 is the hosting laptop which browses fine but anything it is hosting isn't getting anywhere on the netgate.No issue at the hosting laptop.
.204 is a vm and does have the problem.
I have a total of 6 VMs running off that laptop normally some times up to 10. So far all of them have this issue.
I am starting to hang on the idea of what @DaddyGo brought up earlier about RFC1191. Cisco's can do that amongst a good amount of other network hardware (based on some shooting from the hip research) and it appears that netgate appliance does not. I am hoping to find a away to enable something for it if it is there. I am just running out of ideas.
@johnpoz you seem to be very comfortable with VMware Workstation. Any recommended changes there above what I've already done?
-
@dfinjr I don't normally use workstation - I used esxi for many years, and have supported it at the enterprise level, etc. Its not much different to be honest.. esxi just more bells and whistles, etc.
If the host can talk fine.. and has no issues... Then its something with workstation or workstation on that hardware, or some setting, etc.
If there is talk about some problem with specific nic, that seems to have gone out the window when you are using the usb nic and still having a problem.
You shouldn't have to do anything odd to handle weird mtu - because there shouldn't be any odd mtus on your network. Everything should be set and be using 1500 mtu..
Do you not have another laptop or pc you could fire up workstation on?
I have never seen such an issue in all my years working with workstation, player or esxi or even back in the day when they called it server version 1.. I have been using vmware products since they came out really - and have never run into such an issue... And have never seen sniffs with odd ball mtu sizes like that. I work with sniffs all the time from DCs and all kinds of customers network - Yes I have seen mtu issues, but they are standard jumbo size, where someone thought it would be good idea to turn on jumbo - but the network doesn't support it, or they think they can enable it on 1 host on the network, etc.. A 1767 frame with a 1753 mtu?? Just really nothing comes to mind that could cause that - other then setting it to that. And your other sniffs are non standard as well.. The one that said cisco in the capture had a 1434 frame?? Makes no sense!
-
Another tack here might be to examine VMware's network logs, those on the virtual hosts, and any applicable firewall rules.
-
@johnpoz
Sounds like you have an awesome level of experience to say the least!I did a test just like you described a little earlier today. I spun up a VM (ubuntu) off of a linux mint laptop I had laying around on virtual box. I was able to browse just fine. Because of that I feel that the issue seems to be weirdly localized between how the packets are being generated from VMware Workstation and how pfsense is having to deal with it. Literally my only guess at this point. Keyword being guess :)
Tell me if this is an accurate thought process here. I see one of two things happening.
1 - The VMWare Workstation is always generating crazy packets like this but the Cisco ASA is pulling it down and putting it in check (RFC1191) as the packets traverse making the vms under VMware Workstation work correctly.
2 - The VMware Workstation vms are only generating these crazy packets when the PFSense is in the mix (somehow).
All I know is when the Cisco ASA is in the mix the packets don't go crazy and with the PFSense in the mix they seem to go crazy. But that sentence is specifically tied to VMware Workstation since another system hosting a vm on virtual box seems to be fine.
I feel like my gut idea is that it feels less likely that the packets are changing on the presence of a network appliance, that seems far fetched.
Thinking to an earlier test I did as well I put the VMhosting laptop on google wifi for a bit and that also worked fine. So... that makes me think that it must be something about how google wifi is handling those packets which happens to be the same way that the Cisco ASA is handling those packets.
In summary to not miss anything.
VMware Workstation - works when connected to Cisco ASA - packets normalize
VMware Workstation - works when connected to Google Wifi - assuming packets normalized due to the passing of web traffic
VMware Workstation - does not pass all traffic and client creates odd sized MTUs. Assuming this is the same problem from my test with the TPLink VPN router I tested with earlier. I think perhaps the google wifi and the Cisco ASA are just able to tolerate/fix that traffic on the fly.Call me crazy please...
-
2 makes no sense, the router or the hardware used for your router at the edge of the network would have zero to do with what mtu the host or vm uses..
@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:
Cisco ASA is in the mix the packets don't go crazy
What I saw in the sniff where you said ciscoasa was still ODD ball frame size... These are ODD.. I sorted the sniff by the largest frame - and there are loads and loads of this max frame of
In a normal ethernet network you should see 1514 with a 1500 mtu, wireshark would prob hide the other 8 bits where the frame could be 1522 if you were doing vlan tags..
If you sniff at pfsense for some normal say your host of your vm that is working correctly - do you see normal sized frames?
-
@johnpoz
I don't know if I would call it normal. I attached it so you can see what I mean. packetcapture-8.cap -
@dfinjr yeah that is normal...
1514 would be the frame size with a 1500 mtu..
If your not seeing this, then something is OFF.. while sure you can see smaller sized frames... But when you move large amounts of data, say loading pictures or something off amazon or doing a speedtest, etc. etc.. Your going to at some point get to max out your mtu, and this should be what you just posted. Not 1434 ;)
Unless you were talking to something over the internet, and your path had a lower mtu at some point in the path.. Path discovery was mentioned somewhere in the thread... So unless you got some weird ass internet connection.. But again those connections normally have a standard like PPPoE etc. has slightly lower mtu.. If you were over a vpn, sure again lower mtu... But 1434 is just ODD!! not any standard that I am aware of anywhere.
-
@johnpoz said in VMware Workstation VMs Web Traffic Being Blocked:
But when you move large amounts of data, say loading pictures or something off amazon or doing a speedtest, etc. etc.. Your going to at some point get to max out your mtu, and this should be what you just posted. Not 1434 ;)
I'd expect smaller frame sizes if something (e.g., VMware) is throttling the amount of CPU available for various tasks, or is just saturating the CPU. I presume VMware also throttles I/O traffic to maintain quality of service among several VMs. It might be interesting to turn off all but 1 VM and see whether the problem persists.
-
@bpsdtzpw said in VMware Workstation VMs Web Traffic Being Blocked:
I'd expect smaller frame sizes if something (e.g., VMware) is throttling the amount of CPU available for various tasks
why would you think that - that makes zero sense.. Smaller mtu would mean actually more work to do..
-
I am just wondering. What do you think is going on? I understand a bit clearer now from what you said is normal but you also say something is off and I totally get why you're saying that but what is your gut response on to what is happening in my lab?
@bPsdTZpW
Current testing all VMs are off but one. Same animal of a problem is still showing up sadly. -
I'd expect smaller frame sizes if something (e.g., VMware) is throttling the amount of CPU available for various tasks
why would you think that - that makes zero sense.. Smaller mtu would mean actually more work to do..
I didn't mean a smaller MTU. I meant smaller actual frames being used because there just isn't enough of a required resource (e.g., I/O bandwidth, CPU time) to fill out every frame to the full MTU. This also might have to do with latency management in the virtual environment.
-
@bPsdTZpW
Current testing all VMs are off but one. Same animal of a problem is still showing up sadly.Hmm. Are there any VMware logs, especially performance/QOS logs? If so, it might be interesting to see whether VMware is throwing any errors, or unexpectedly throttling resources.
-
@bpsdtzpw said in VMware Workstation VMs Web Traffic Being Blocked:
I meant smaller actual frames being used because there just isn't enough of a required resource
What? Look up frame and mtu ;)
-
@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:
what is your gut response on to what is happening in my lab?
I have no idea to be honest - never seen such a thing.. I sure can not reproduce it here, I installed workstation 16.2.2 and works just fine, you saw my sniffs, etc.
I showed you were to check if mtu had had been changed.. I am at a loss to why what your seeing would be happening..
-
Thank you for your opinion. I feel essentially the same way, can't nail down what is happening. Please let me know if any other ideas. I am going to continue troubleshooting here and there aiming for resolution to this weird problem but ultimately I'll have to make a decision in the next few weeks as that is what is left in my return policy for the appliance.
I think this goes without saying but I want to keep it if I can figure out how to get it all to play ball :). I'll keep fighting, I got time left.
If you think of anything that I can supply you for a random troubleshooting thought please done hesitate.
I do have one other sub option which would be to purchase the direct support for $400 and see if they can pull off something we didn't. Any reason from your opinion to think that they may be able to get further than we did?
-
@johnpoz said in VMware Workstation VMs Web Traffic Being Blocked:
@bpsdtzpw said in VMware Workstation VMs Web Traffic Being Blocked:
I meant smaller actual frames being used because there just isn't enough of a required resource
What? Look up frame and mtu ;)
If I were writing a VMM, it would be important to maintain reasonable performance within spec, including I/O bandwidth, latency, compute availability, etc.
To do this, the VMM would need to apportion baremetal resources to the VMs, both statically and dynamically, depending on available resources and how VMs (and other things running on the baremetal) are using them. So, to maintain both acceptable bandwidth and acceptable latency, the VMM might send a message to a VM's (virtual) ethernet driver, asking it to send smaller-than-permissible frames. It would still work to send frames as large as the MTU, but the VMM might punish violations of its request by delaying such frames, reducing resource quantums to the offending VM, etc.
This kind of resource management might explain the preponderance of smaller-than-MTU frames that puzzled you above.
BTW, I don't know WMware internals, but I do have experience designing and implementing VMMs.
-
I could believe this is a combination of things. Changing any one of them removes it but you happen to have hit the right combo!
VMWare not playing nicely with the NIC. USB/WIFI NICs never sit well with me but I've never tried that with VMWare&Windows.
Have you actually run any MTU tests yet? Like pinging with larger packets until it fails?
We see the replies coming back from external sites much larger when pfSense is routing but they are not over 1500B. Why is VMWare dropping them?
It seems like something is breaking PMTUD so one thing to do here would be to check for the presence of the ICMP packet-too-large messages when both Cisco and pfSense are routing.
Steve
-
@stephenw10 : Checking VMware logs might help. Presumably it can be configured to shout if it's dropping packets....
Aha!
<p>
Troubleshoot lost connectivity by capturing dropped packets through the pktcap-uw utility.A packet might be dropped at a point in the network stream for many reasons, for example, a firewall rule, filtering in an IOChain and DVfilter, VLAN mismatch, physical adapter malfunction, checksum failure, and so on. You can use the pktcap-uw utility to examine where packets are dropped and the reason for the drop.
</p>https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-84627D49-F449-4F77-B931-3C55E4A8ECA1.html
-
I'll happily perform another test. I don't actually know how to do the MTU tests such as the pinging with larger packets. I googled it earlier and messed with it for a few minutes but then stopped just because my attention got changed.
Do you have an example large ping test that you would like for me to run? If so, let me know what command and where (and if any captures) and I'll get that done ASAP.
I can do some basic watching of both scenarios looking for that ICMP packets too large messages you're talking about.
@bpsdtzpw said in VMware Workstation VMs Web Traffic Being Blocked:
pktcap-uw utility
I looked at this quick and this appears to be tied to vsphere esxi hosts vs my lowly VMware Workstation :)
-
@bpsdtzpw said in VMware Workstation VMs Web Traffic Being Blocked:
would be important to maintain reasonable performance within spec, including I/O bandwidth, latency
That wouldn't make for smaller or larger frames - or change the mtu of the network your connected too.. You understanding creating smaller packets would cause more work.. Trying to use a larger frame then your network was set to allow would again cause more work - And how would the VM software change the mtu of the switch port its connected too, etc..
