VMware Workstation VMs Web Traffic Being Blocked

dfinjr

@stephenw10
No traffic is being ran over VPN.

I did a packet capture from the netgate for the host in question (172.16.0.202) for anything over port 443 and then went to the host and generated some traffic. This is what I got back: (opened on Wireshark on another system)

During this capture same results experienced as mentioned earlier. Blocks on all sites except google and netgate.

Not sure what these retransmissions are.

I've attached the full packet capture. packetcapture-2.cap

dfinjr

@stephenw10
Also now noticing packet out of order and fragment errors. Would like your interpretation on what is going on here.

stephenw10

TCP retransmissions imply traffic is not reaching the destination or the target is choosing not to reply.
Was that taken on the pfSense WAN?

It could be an MTU issue. It reads like the client never receives the large packets from the server.

Though in that screenshot you see the same thing to our IP, but that works?

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

During this capture same results experienced as mentioned earlier.

Hi,

I can confirm (in a similar environment - like yours) that this may be a Win issue, VMw workstation16PRO + Ubuntu 20.04 desktop does not have this phenomenon.

This could be interesting even though ESXi: https://communities.vmware.com/t5/VMware-vSphere-Discussions/TCP-Retransmission-issue-on-a-VM-Machine/td-p/2259652

what do you use as a network adapter on your VMw workstation? ...: NAT, host - only or custom, what is the windows machine adapter type e1000 or vmxnet (?)

dfinjr

@stephenw10
I did this capture from the interface supporting the vlan where the VMs are at.

Yep, I don't understand it and have been digging into this ever since I discovered it with no idea how to troubleshoot it out.

I'd be happy to do some other packet captures for you. I did this capture from the vlan where those VMS and 2 physical machines reside specifically looking at the 172.16.0.202 address for anything happening or attempting to happen over 443, which is just one of the 6 VMS doing this same behavior.

stephenw10

Can you pcap on the client seeing the issues?

On the pfSense VLAN interface it looks like all the traffic is present so it would good to see how much of that makes it to the client.

Given that physical devices on the same VLAN can connect no problem it really looks like the hypervisor is dropping the traffic. Or maybe sending it to the wrong place etc.

Steve

dfinjr

@daddygo
Hello,
The hosting system for the VMs is on Win10 + VMware workstation 16 pro. I would of suspected something possibly like that if I didn't have the same data flying just fine over a 5506x Cisco ASA before attempting to make the jump over to the netgate appliance.

For the network adapter I have the VMs using bridged mode so they're seeing straight through to the intel hardline network adapter: (taken from 172.16.0.202)

From vmware workstation for that host:

I think the screen shot it shows that the type is vmnet0.

Thanks!
David

dfinjr

@stephenw10
I did a short packet capture and want to share it but there is a hard limit on the file size, tried compressing it but still not less than 2mb. Is there another method I could share it with you?

I would think it was something to do with vmware workstation but can't think of a reason why it would work on a Cisco ASA and not this netgate appliance. Don't believe I did anything special with the Cisco at all.

Either case, I definitely want to use the new netgate hardware for a few reasons, one being this hardware isn't 7yrs old :).

dfinjr

@stephenw10
Think I got it:
pcap4.pcapng.gz

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

I think the screen shot it shows that the type is vmnet0.

Nope, that would be part of the "custom" option, you are currently using the E1000e driver from VMwareTools as the default driver Intel 82574

it's definitely the interaction between virtual environment and windows, I wouldn't think pfSense is cool to differentiate between physical machine and VM machine on the same subnet

try playing around with the virtual machine NIC driver, as a high number of retransmission may indicate a stall....

+++edit:
https://kb.vmware.com/s/article/2032184

dfinjr

@daddygo said in VMware Workstation VMs Web Traffic Being Blocked:

https://kb.vmware.com/s/article/2032184

I'll give this a shot here in a few and update the post. Thank you!

dfinjr

@daddygo
Looks like the vmware specific drivers do not want to play ball with the host:

dfinjr

@stephenw10
Hi Steve,
Was wondering if you had any other ideas on what you were seeing on the network packet captures?

Thanks!
David

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

Looks like the vmware specific drivers do not want to play ball with the host:

Ouupss...

Unfortunately I don't use Windows with workstation16, only on ESXi with vmxnet3 and no problems behind pfSense....

just out of curiosity would you try it behind NAT on your workstation16...(?)

BTW:
windows is a strange beast when it comes to type 2 hypervisors, although I haven't seen this on VirtualBox yet either, I'll install one on workstation16 tomorrow.

I'm getting more and more excited, but I can definitely feel that it's not a pfSense question

edit:
Ubuntu behaves perfectly well in this setting, so I suspect to MS

stephenw10

Mmm, I would not think either pfSense or the Cisco router would have any idea which traffic comes from a VM. And it shouldn't matter.
Things I could imagine happening are the Cisco doing some fragmentation or reassembly pfSense is not.

DaddyGo

@stephenw10 said in VMware Workstation VMs Web Traffic Being Blocked:

Cisco doing some fragmentation or reassembly pfSense is not.

Yuppp,... This is also possible, but the problem is that it is transferred to the VM environment in different ways, ergo one of the device drivers (VM) is messing with us (maybe)

BTW:
ens33 has nothing wrong with this on Ubuntu

stephenw10

In the pcap there are some jumbo frames shown between internal hosts. So possibly something is trying to use them externally. I don't see that in the capture though.

dfinjr

@daddygo
Ok so interestingly enough, switching it to NAT and then allowing it to get an "IP" from the VMware hosting pieces with its little internal DHCP server it does restore internet functionality. However, I think with the solutions that I'm hosting (Endpoint management with BigFix) that this may very well stop it from service providing for the rest of the infrastructure. Haven't tested that out yet or anything but wanted to drop the note that it is allowing for browsing under nat settings.

I'll test out the other functionality to see if I am getting roadblocks for its other services. Clients external to that system register to the management area so I am worried that if it was buried behind the hosting laptops identity on the network.

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

it does restore internet functionality.

I felt it was going to be
which is why I don't use a windows on type two hypervisor, hihihihi

but still I understand your concern and I'm sure it can be made to work, now I'm going to watch CNN (damn war , ): ), but tomorrow I'll install a win10 (on ws16) and test it, because it's interesting

I'll let you know what I find

dfinjr

@dfinjr
It makes sense to me why going NAT would work but I did just verify that it makes the hosting services invisible on the network. So it does fix browsing but does break the availability of listening services.