VMware Workstation VMs Web Traffic Being Blocked

dfinjr

FYI I have disconnected the switch from scenario so it is direct from netgate > vm hosting laptop.

Still seeing the browsing issue but I know we're about to capture based on that, just wanted to let you know.

stephenw10

You should filter by the destination IP in pfSense if you can otherwise we see all the local traffic and make sure you set the packet count to 1000.

On the other side try to capture on the VMWare host not the VM. We already have caps from the VM showing the large packets failing to arrive. We need to see if they arrive at the VM host interface.

Steve

dfinjr

@stephenw10
So some difficulty here.

attempting to capture packets I have this two virtual adapters on the hosting system:

Very interesting. Just sniffing these two interface yield nothing. I go to .202 and browse to netgate.com and 0 packets show up on either of these.

Did you mean the literal physical adapter on the host machine?

johnpoz

@dfinjr Those are like I showed you - those are for use say with natting or just a host network and only vms can only talk to each other.

For someone that has multiple workstations running vm workstation, seem not to know much about how it works ;)

I am really curious where exactly use sniffed and on what - because when I snff on the physical interface of the host that is doing bridging like you said your doing.. I see duplicated packets.. Which is common on such a setup..

https://ask.wireshark.org/question/8712/duplicate-packets-from-vmware-host/

There was actually a some what recent thread around here where user was seeing the dupe packets, and it was this because of his vm setup and what interface sniffing on, etc. But that is not actually what is going on the wire.. If you sniff elsewhere on the network..

If you notice the time stamps - they are nano seconds apart, you are seeing as enters the bridge and when leaves the bridge. You can see the same sort of thing when sniffing on a vlan interface on a higher end switch, you see when it enters the vlan and when it leaves because your sniffing on the vlan and not specific interface. Same sort of thing here when your sniffing on a bridged interface..

stephenw10

@dfinjr said in VMware Workstation VMs Web Traffic Being Blocked:

Did you mean the literal physical adapter on the host machine?

Yes I would start there. Make sure the replies from the external hosts are making it that far first. Since you only changed the Cisco for pfSense it seems something must be different. Though it's hard to see what!

dfinjr

@stephenw10
Sorry got held up with work, attempting the capture pieces now.

dfinjr

Apologize that took me so long everyone. Had some work stuff hit me between the eyes and beyond that I wanted to get a good clean capture, filtering out most of the noise by ports that don't matter to this. I think I have a good capture. 2 files: test2 capture was done from the VM hosting system listening just for the IP address 172.16.0.202 and not capturing anything to do with 3389 and 52311 (BigFix Port) and file packet capture-5 is from pfsense listening on the interface where the system resides but filtering out 3389 and 52311 again. I haven't analyzed this myself yet but think it is the most solid one I've captured yet. With the filters I was able to capture a far longer. Captured till the browser on the VM gave up and spit out an "err_timed_out". Please let me know what you think. packetcapture-5.cap test2.pcapng.gz

dfinjr

@stephenw10 @johnpoz @DaddyGo

Think I finally got a good capture. I've added it to the forum.

@johnpoz

I'm doing my best with the VMware Workstation software. It isn't something I use outside of a means to an end and don't have any real professional experience with it. It hosts my VMs but I am far from an expert with it. Any educational points you can throw my way is of course appreciated.

stephenw10

Ah, it looks like on the test2 cap, on the VMWare host, you filtered by 172.16.0.202 as destination so we don't see any of the traffic 172.16.0.202 is sending.
But that does mean we can see the large packets leaving the pfSense LAN and arriving at the VMWare host. If you filter by, for example, ip.addr == 151.101.2.219 again you can see that.

So given that those packets do not arrive at the actual VM it must be some issue with virtual networking dropping that.

It looks like none of the traffic leaving pfSense toward the VM host is flagged do-not-fragment. And it was previously. You might want to uncheck that setting in pfSense if it's still enabled because that should not be required.

Steve

dfinjr

@stephenw10
Thank you for the information Steve. I'll look at what I can do to alter the vmware configuration to allow the traffic to flow. Not sure what to do there but at least I can focus on it a bit more to see if I can get it to go.

Question is, in your opinion, with knowing that this is the way that the traffic was flowing from the ASA before, do you think that the Cisco appliance was shaping the traffic or something so that it could flow?

dfinjr

@stephenw10
Also, this is the filter I did for the Wireshark. I did have the .202 IP but it was destination only:

stephenw10

If it's an MTU issue try testing that with large pings. You can just try pinging out from a VM to pfSense with do-not-fragment set and see where it fails. Or doesn't fail.

It seems like Cisco was reducing the path-mtu somehow. mss set perhaps? That might show in the cap we have from when it was in place...

stephenw10

Or maybe we don't have a pcap taken when the Cisco was routing?

dfinjr

@stephenw10
packetcapture-6.cap VMhostSpeedTest.pcapng VMSpeedTest.pcapng

Does this by chance show anything different. You're much faster on the analysis than I am. What I did is started a capture generic vlan on the pfsense, did a capture based on the vm host and the vm itself all fixated on speedtest.net.

I started with a ping and then attempted to browse via IP over 443.

stephenw10

Most of this missed in the pfSense capture because 1000 packets only covers 8s.
Still only seeing inbound traffic on the host but I wonder if that's a quirk of the bridge mode.
The VM itself still doesn't see any of the full sized packets.

Is there a pcap from when the Cisco was routing and traffic was working?

dfinjr

@stephenw10
If it would be helpful for me to do a packet capture on the Cisco I would be happy to. Just let me know. Thank you again for all the time you're giving this.

dfinjr

@stephenw10

I'll get one for you now.

Are you happy with the same filters as the capture before this one or anything you'd like for me to change on the capture?

stephenw10

Those filters are fine. Just to test aganst speedtest you might try using, say, 151.101.0.0/16 instead. It would be nice to capture only that traffic.

I'm signing off for tonight though, 2.30am here. I'll check back tomorrow

Steve

dfinjr

@stephenw10
Thank you Steve.
I created a capture but it was too big for the forum. You'll find them here:
https://drive.google.com/drive/folders/14C1MTTuwjUnvNYgDJfBy0gmiSQmO5HTQ?usp=sharing

Thanks for everything today!

dfinjr

@stephenw10
One last offer I was going to make was if you'd like I can also host a gotomeeting if you want to see any of this in real time. Just let me know if that would be interesting to you. Would give me a chance to thank you verbally.