VMware Workstation VMs Web Traffic Being Blocked
-
@stephenw10
Hi Steve,
I do run a pi-hole in the environment and made sure that from the google search example that it wasn't an ad based link and wasn't being blocked for that reason. Also, if I do just direct webpage hits like "speedtest.net" or "bestbuy.com" they're getting blocked the same as mentioned in the post.I am not running pfblocker, honestly didn't know that was a thing till you mentioned it.
Yep, the physical hosts have identical configuration static as the virtual hosts. Same subnet, gateway and DNS Servers. I also for the sake of testing changed the settings to allow it to go 8.8.8.8 direct and skip the pi-hole but no difference in results was found.
Thanks!
David -
If it's resolving and there's a route the next most common thing is some sort of MTU issue and that could apply to traffic doing through the hypervisor only.
Are you routing traffic over a VPN?Try running a packet capture for those connections and see what's actually happening at the pfSense interfaces.
Steve
-
@stephenw10
No traffic is being ran over VPN.I did a packet capture from the netgate for the host in question (172.16.0.202) for anything over port 443 and then went to the host and generated some traffic. This is what I got back: (opened on Wireshark on another system)
During this capture same results experienced as mentioned earlier. Blocks on all sites except google and netgate.
Not sure what these retransmissions are.
I've attached the full packet capture. packetcapture-2.cap
-
@stephenw10
Also now noticing packet out of order and fragment errors. Would like your interpretation on what is going on here.
-
TCP retransmissions imply traffic is not reaching the destination or the target is choosing not to reply.
Was that taken on the pfSense WAN?It could be an MTU issue. It reads like the client never receives the large packets from the server.
Though in that screenshot you see the same thing to our IP, but that works?
-
@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:
During this capture same results experienced as mentioned earlier.
Hi,
I can confirm (in a similar environment - like yours) that this may be a Win issue, VMw workstation16PRO + Ubuntu 20.04 desktop does not have this phenomenon.
This could be interesting even though ESXi: https://communities.vmware.com/t5/VMware-vSphere-Discussions/TCP-Retransmission-issue-on-a-VM-Machine/td-p/2259652
what do you use as a network adapter on your VMw workstation? ...: NAT, host - only or custom, what is the windows machine adapter type e1000 or vmxnet (?)
-
@stephenw10
I did this capture from the interface supporting the vlan where the VMs are at.Yep, I don't understand it and have been digging into this ever since I discovered it with no idea how to troubleshoot it out.
I'd be happy to do some other packet captures for you. I did this capture from the vlan where those VMS and 2 physical machines reside specifically looking at the 172.16.0.202 address for anything happening or attempting to happen over 443, which is just one of the 6 VMS doing this same behavior.
-
Can you pcap on the client seeing the issues?
On the pfSense VLAN interface it looks like all the traffic is present so it would good to see how much of that makes it to the client.
Given that physical devices on the same VLAN can connect no problem it really looks like the hypervisor is dropping the traffic. Or maybe sending it to the wrong place etc.
Steve
-
@daddygo
Hello,
The hosting system for the VMs is on Win10 + VMware workstation 16 pro. I would of suspected something possibly like that if I didn't have the same data flying just fine over a 5506x Cisco ASA before attempting to make the jump over to the netgate appliance.For the network adapter I have the VMs using bridged mode so they're seeing straight through to the intel hardline network adapter: (taken from 172.16.0.202)
From vmware workstation for that host:
I think the screen shot it shows that the type is vmnet0.
Thanks!
David -
@stephenw10
I did a short packet capture and want to share it but there is a hard limit on the file size, tried compressing it but still not less than 2mb. Is there another method I could share it with you?I would think it was something to do with vmware workstation but can't think of a reason why it would work on a Cisco ASA and not this netgate appliance. Don't believe I did anything special with the Cisco at all.
Either case, I definitely want to use the new netgate hardware for a few reasons, one being this hardware isn't 7yrs old :).
-
@stephenw10
Think I got it:
pcap4.pcapng.gz -
@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:
I think the screen shot it shows that the type is vmnet0.
Nope, that would be part of the "custom" option, you are currently using the E1000e driver from VMwareTools as the default driver Intel 82574
it's definitely the interaction between virtual environment and windows, I wouldn't think pfSense is cool to differentiate between physical machine and VM machine on the same subnet
try playing around with the virtual machine NIC driver, as a high number of retransmission may indicate a stall....
+++edit:
https://kb.vmware.com/s/article/2032184 -
@daddygo said in VMware Workstation VMs Web Traffic Being Blocked:
https://kb.vmware.com/s/article/2032184
I'll give this a shot here in a few and update the post. Thank you!
-
@daddygo
Looks like the vmware specific drivers do not want to play ball with the host:
-
@stephenw10
Hi Steve,
Was wondering if you had any other ideas on what you were seeing on the network packet captures?Thanks!
David -
@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:
Looks like the vmware specific drivers do not want to play ball with the host:
Ouupss...
Unfortunately I don't use Windows with workstation16, only on ESXi with vmxnet3 and no problems behind pfSense....
just out of curiosity would you try it behind NAT on your workstation16...(?)
BTW:
windows is a strange beast when it comes to type 2 hypervisors, although I haven't seen this on VirtualBox yet either, I'll install one on workstation16 tomorrow.I'm getting more and more excited, but I can definitely feel that it's not a pfSense question
edit:
Ubuntu behaves perfectly well in this setting, so I suspect to MS
-
Mmm, I would not think either pfSense or the Cisco router would have any idea which traffic comes from a VM. And it shouldn't matter.
Things I could imagine happening are the Cisco doing some fragmentation or reassembly pfSense is not. -
@stephenw10 said in VMware Workstation VMs Web Traffic Being Blocked:
Cisco doing some fragmentation or reassembly pfSense is not.
Yuppp,... This is also possible, but the problem is that it is transferred to the VM environment in different ways, ergo one of the device drivers (VM) is messing with us (maybe)
BTW:
ens33 has nothing wrong with this on Ubuntu -
In the pcap there are some jumbo frames shown between internal hosts. So possibly something is trying to use them externally. I don't see that in the capture though.
-
@daddygo
Ok so interestingly enough, switching it to NAT and then allowing it to get an "IP" from the VMware hosting pieces with its little internal DHCP server it does restore internet functionality. However, I think with the solutions that I'm hosting (Endpoint management with BigFix) that this may very well stop it from service providing for the rest of the infrastructure. Haven't tested that out yet or anything but wanted to drop the note that it is allowing for browsing under nat settings.I'll test out the other functionality to see if I am getting roadblocks for its other services. Clients external to that system register to the management area so I am worried that if it was buried behind the hosting laptops identity on the network.
