VMware Workstation VMs Web Traffic Being Blocked

dfinjr

Hello,
I am a new PFSense user but am very used to Cisco. Looking to make the switch and am running into troubles with part of my environment and I cannot sort what is going on. I am running a Netgate-2100 with ports set to VLAN mode and the configuration seems to be working fine except with VMs hosted by VMware Workstation.

On the VLAN where the VMs reside there is two physical systems as well, one raspberry pi and one windows system hosting the VMware Workstation software. Both the physical systems on the same vlan pass traffic as would be expected with no problem. However, I both see firewall blocks in the logs on pfsense as well as verified that the traffic isn't able to take place on the VMs themselves. It is almost like URL filtering but can't figure out why/where that could be possibly taking place.

If I go to google that page will load and you can search within it but if I go to google for example and search pfsense and then click on the first hit on google for pfsense to pfsense.org it'll "err_timed_out" me. Weirdly netgate.com also loads. So far outside of those two sites it is timing me out.

I thought my problem might be related to this forum article I found: https://forum.netgate.com/topic/95343/firewall-blocking-traffic-between-the-lan-interfaces/4

After trying items in that article I saw no changes.

I do pings/nslookups to all sites that are failing and they all complete without any issue. Again, feels like URL filtering... but why to the VMs and not to the other hosts on the vlan. I could make sense if the physical hosts were struggling too but they don't seem to be facing any restrictions at all.

I have tried floating and interface set firewall rules in any combination in/out/any to any/any. Using automatic nat and bypass rules for traffic on the same interface.

I'm fresh out of ideas.

Any/all help would be appreciated. Happy to contribute any other relevant data to the post to help nail this down.

Thanks!

stephenw10

Often the first site in Google search results will be paid-for advertisement links that go through google ad-services. That can be blocked by browser plugins or by DNS filtering. Though not usually for pfSense as a search term.
Are you running pfBlocker?
Are the VMs configured the same as the physical hosts? Using the same DNS servers?

Steve

dfinjr

@stephenw10
Hi Steve,
I do run a pi-hole in the environment and made sure that from the google search example that it wasn't an ad based link and wasn't being blocked for that reason. Also, if I do just direct webpage hits like "speedtest.net" or "bestbuy.com" they're getting blocked the same as mentioned in the post.

I am not running pfblocker, honestly didn't know that was a thing till you mentioned it.

Yep, the physical hosts have identical configuration static as the virtual hosts. Same subnet, gateway and DNS Servers. I also for the sake of testing changed the settings to allow it to go 8.8.8.8 direct and skip the pi-hole but no difference in results was found.

Thanks!
David

stephenw10

If it's resolving and there's a route the next most common thing is some sort of MTU issue and that could apply to traffic doing through the hypervisor only.
Are you routing traffic over a VPN?

Try running a packet capture for those connections and see what's actually happening at the pfSense interfaces.

Steve

dfinjr

@stephenw10
No traffic is being ran over VPN.

I did a packet capture from the netgate for the host in question (172.16.0.202) for anything over port 443 and then went to the host and generated some traffic. This is what I got back: (opened on Wireshark on another system)

During this capture same results experienced as mentioned earlier. Blocks on all sites except google and netgate.

Not sure what these retransmissions are.

I've attached the full packet capture. packetcapture-2.cap

dfinjr

@stephenw10
Also now noticing packet out of order and fragment errors. Would like your interpretation on what is going on here.

stephenw10

TCP retransmissions imply traffic is not reaching the destination or the target is choosing not to reply.
Was that taken on the pfSense WAN?

It could be an MTU issue. It reads like the client never receives the large packets from the server.

Though in that screenshot you see the same thing to our IP, but that works?

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

During this capture same results experienced as mentioned earlier.

Hi,

I can confirm (in a similar environment - like yours) that this may be a Win issue, VMw workstation16PRO + Ubuntu 20.04 desktop does not have this phenomenon.

This could be interesting even though ESXi: https://communities.vmware.com/t5/VMware-vSphere-Discussions/TCP-Retransmission-issue-on-a-VM-Machine/td-p/2259652

what do you use as a network adapter on your VMw workstation? ...: NAT, host - only or custom, what is the windows machine adapter type e1000 or vmxnet (?)

dfinjr

@stephenw10
I did this capture from the interface supporting the vlan where the VMs are at.

Yep, I don't understand it and have been digging into this ever since I discovered it with no idea how to troubleshoot it out.

I'd be happy to do some other packet captures for you. I did this capture from the vlan where those VMS and 2 physical machines reside specifically looking at the 172.16.0.202 address for anything happening or attempting to happen over 443, which is just one of the 6 VMS doing this same behavior.

stephenw10

Can you pcap on the client seeing the issues?

On the pfSense VLAN interface it looks like all the traffic is present so it would good to see how much of that makes it to the client.

Given that physical devices on the same VLAN can connect no problem it really looks like the hypervisor is dropping the traffic. Or maybe sending it to the wrong place etc.

Steve

dfinjr

@daddygo
Hello,
The hosting system for the VMs is on Win10 + VMware workstation 16 pro. I would of suspected something possibly like that if I didn't have the same data flying just fine over a 5506x Cisco ASA before attempting to make the jump over to the netgate appliance.

For the network adapter I have the VMs using bridged mode so they're seeing straight through to the intel hardline network adapter: (taken from 172.16.0.202)

From vmware workstation for that host:

I think the screen shot it shows that the type is vmnet0.

Thanks!
David

dfinjr

@stephenw10
I did a short packet capture and want to share it but there is a hard limit on the file size, tried compressing it but still not less than 2mb. Is there another method I could share it with you?

I would think it was something to do with vmware workstation but can't think of a reason why it would work on a Cisco ASA and not this netgate appliance. Don't believe I did anything special with the Cisco at all.

Either case, I definitely want to use the new netgate hardware for a few reasons, one being this hardware isn't 7yrs old :).

dfinjr

@stephenw10
Think I got it:
pcap4.pcapng.gz

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

I think the screen shot it shows that the type is vmnet0.

Nope, that would be part of the "custom" option, you are currently using the E1000e driver from VMwareTools as the default driver Intel 82574

it's definitely the interaction between virtual environment and windows, I wouldn't think pfSense is cool to differentiate between physical machine and VM machine on the same subnet

try playing around with the virtual machine NIC driver, as a high number of retransmission may indicate a stall....

+++edit:
https://kb.vmware.com/s/article/2032184

dfinjr

@daddygo said in VMware Workstation VMs Web Traffic Being Blocked:

https://kb.vmware.com/s/article/2032184

I'll give this a shot here in a few and update the post. Thank you!

dfinjr

@daddygo
Looks like the vmware specific drivers do not want to play ball with the host:

dfinjr

@stephenw10
Hi Steve,
Was wondering if you had any other ideas on what you were seeing on the network packet captures?

Thanks!
David

DaddyGo

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

Looks like the vmware specific drivers do not want to play ball with the host:

Ouupss...

Unfortunately I don't use Windows with workstation16, only on ESXi with vmxnet3 and no problems behind pfSense....

just out of curiosity would you try it behind NAT on your workstation16...(?)

BTW:
windows is a strange beast when it comes to type 2 hypervisors, although I haven't seen this on VirtualBox yet either, I'll install one on workstation16 tomorrow.

I'm getting more and more excited, but I can definitely feel that it's not a pfSense question

edit:
Ubuntu behaves perfectly well in this setting, so I suspect to MS

stephenw10

Mmm, I would not think either pfSense or the Cisco router would have any idea which traffic comes from a VM. And it shouldn't matter.
Things I could imagine happening are the Cisco doing some fragmentation or reassembly pfSense is not.

DaddyGo

@stephenw10 said in VMware Workstation VMs Web Traffic Being Blocked:

Cisco doing some fragmentation or reassembly pfSense is not.

Yuppp,... This is also possible, but the problem is that it is transferred to the VM environment in different ways, ergo one of the device drivers (VM) is messing with us (maybe)

BTW:
ens33 has nothing wrong with this on Ubuntu