Problem with VMworkstation and Pfsense

grim · Dec 11, 2016, 11:53 PM

Hello guys,

I have the following problem:
Pfsense( last version) VM running on VMworkstarion with 2 physical nics bridged to the vm directly (1 pcie nic for wan and 1 pci nic for lan)
Behind the pfsesne there is a tplink with DDWRT
The setup was running fine till yestarday, 1 month uptime.
I decided to move the VM to another HDD(shutdown->copy->run the copied vm) and then the problem appeared.

From the LAN interface you can only ping external addresses, no DNS, no internet , no nothing.( only accessible web page is webconfigurator )
I thought that the move might be the possible problem so i shutdown the new VM and started the old one. The same problem was still present.
I wasted almost all my day troubleshooting and found out that,

WAN is static ipv4. (WAN IP and WAN gateway pingable from lan)
Restore 4 day old config with no changes didn't work
Snapshot 5 days old didn't work
The physical adapters are not faulty (tested them both)
If i change the Pfsense LAN adapter from the physical nic to a virtual network im VMworkstation (vmnet 19 for example) and i connect some other VM(win7) to the same vmnet 19 i have internet on the connected vm (win7)….
Connected my laptop directly to the pfsense lan interface - still the same problem ( only accessible web page is webconfigurator ) (tested both DHCP and static with custom DNS settings - the same settings like the Win7 VM)
From the webconfigurator via Ping i can ping google.com from both LAN and WAN interface, but from my Laptop i cannot.

Setup:

Support will be appreciated.
Thanks in advance!

bjaffe · Dec 11, 2016, 11:54 PM

From the LAN interface you can only ping external addresses, no DNS, no internet , no nothing.( only accessible web page is webconfigurator )

From the webconfigurator via Ping i can ping google.com from both LAN and WAN interface, but from my Laptop i cannot.

I'm a little confused by these two statements. If you go to Diagnostics > Ping > Source address = LAN.. can you successfully ping google.com? 8.8.8.8? The WAN Gateway?
Are you missing the LAN net to any rule under Firewall > Rules > LAN? If you're able to reach google via ping from the LAN interface of pfSense directly (firewall generated traffic) and not the laptop, usually it would be filtering the traffic. Did you check the firewall logs (Status > System logs > Firewall)?

If your laptop can ping 8.8.8.8 and not google, I would check DNS, are you using the DNS resolver? Is it in forwarding mode? If it is, do you have DNS servers setup in System > General Setup?

WAN is static ipv4. (WAN IP and WAN gateway pingable from lan)

Is the WAN address public?

Behind the pfsesne there is a tplink with DDWRT

So is the laptop that you're testing from behind the tplink? Is this device NATting? If so, it would be hitting a double NAT and your firewall rules under LAN would need to be Source = any rather than "LAN net" because it wouldn't take into account the double NAT.

bjaffe · Dec 11, 2016, 11:57 PM

I just now saw the diagram…. just look at the last thing I said in my previous reply, you'll most likely need to go to Firewall > Rules > LAN and change the Source = LAN net to Source = Any... it'll probably work then.

grim · Dec 12, 2016, 12:01 AM

@bjaffe:

From the LAN interface you can only ping external addresses, no DNS, no internet , no nothing.( only accessible web page is webconfigurator )

From the webconfigurator via Ping i can ping google.com from both LAN and WAN interface, but from my Laptop i cannot.

I'm a little confused by these two statements. If you go to Diagnostics > Ping > Source address = LAN.. can you successfully ping google.com? 8.8.8.8? The WAN Gateway?
Are you missing the LAN net to any rule under Firewall > Rules > LAN? If you're able to reach google via ping from the LAN interface of pfSense directly (firewall generated traffic) and not the laptop, usually it would be filtering the traffic. Did you check the firewall logs (Status > System logs > Firewall)?

If your laptop can ping 8.8.8.8 and not google, I would check DNS, are you using the DNS resolver? Is it in forwarding mode? If it is, do you have DNS servers setup in System > General Setup?

WAN is static ipv4. (WAN IP and WAN gateway pingable from lan)

Is the WAN address public?

Behind the pfsesne there is a tplink with DDWRT

So is the laptop that you're testing from behind the tplink? Is this device NATting? If so, it would be hitting a double NAT and your firewall rules under LAN would need to be Source = any rather than "LAN net" because it wouldn't take into account the double NAT.

I'm a little confused by these two statements. If you go to Diagnostics > Ping > Source address = LAN.. can you successfully ping google.com? 8.8.8.8? The WAN Gateway?
-Yes
-From laptop only 8.8.8.8, from Win7 VM both 8.8.8.8 and google.com (DNS resolver works, tried both resolver and forwarder)

Wan public IP
Firewall rules OK work for the win7 VM

So is the laptop that you're testing from behind the tplink? Is this device NATting? If so, it would be hitting a double NAT and your firewall rules under LAN would need to be Source = any rather than "LAN net" because it wouldn't take into account the double NAT.

Tried withouth the TPlink, laptop directly to LAN port same issue
TPLink was primary in NAT (was working, double nat )but now in bridge mode also worked before

bjaffe · Dec 12, 2016, 12:09 AM

Okay, so DNS isn't working from the laptop? That's the main problem? And also, everything is working from the windows 7 VM right?

If so, what's the DNS server on the laptop pointing to?
Are the laptop and the win7 VM on the same LAN in the same subnet?
Can you try changing the DNS server on the laptop to 8.8.8.8 and see if you can ping google.com?

grim · Dec 12, 2016, 12:10 AM

@bjaffe:

I just now saw the diagram…. just look at the last thing I said in my previous reply, you'll most likely need to go to Firewall > Rules > LAN and change the Source = LAN net to Source = Any... it'll probably work then.

Doesn't work tried :/

If i do this with no changes to the Pfsense the VM have internet and everything, just change from workstation from bridge to VMnet 19, win7 also connected to host only Vmnet 19

grim · Dec 12, 2016, 12:12 AM

@bjaffe:

Okay, so DNS isn't working from the laptop? That's the main problem? And also, everything is working from the windows 7 VM right?

If so, what's the DNS server on the laptop pointing to?
Are the laptop and the win7 VM on the same LAN in the same subnet?
Can you try changing the DNS server on the laptop to 8.8.8.8 and see if you can ping google.com?

DNS on laptops points to pfsense LAN
the same settings are on win7 vm also same DNS
Tried 8.8.8.8 on laptop, problem persists

same IP, gateway,mask, DNS on laptop and on VM when i try

grim · Dec 12, 2016, 12:25 AM

Also on Pfsense there are 4 DNS : 2 ISP dns , 8.8.8.8 and 1 Open dns
When i nslookup from pfsense all 4 DNS respond

Also noticed when i nslookup from laptop

DNS server : LAN IP
DNS server Name : UnKnown

Nslookup From VM

DNS server: LAN IP
DNS server name: Pfsense hostname

bjaffe · Dec 12, 2016, 12:26 AM

If i do this with no changes to the Pfsense the VM have internet and everything, just change from workstation from bridge to VMnet 19, win7 also connected to host only Vmnet 19

This leads be to believe it's some sort of network filtering problem. Can you redraw the diagram with the full IP assignments / netmasks?

grim · Dec 12, 2016, 12:46 AM

like i said same settings :/

The setup was running for more than a month with IPsec VPN, squid , snort ect. configured

bjaffe · Dec 12, 2016, 12:49 AM

This paints a very clear picture. I don't think I can actually help you here… seems like a problem with the virtual environment rather than with pfSense itself.

grim · Dec 12, 2016, 12:57 AM

@bjaffe:

This paints a very clear picture. I don't think I can actually help you here… seems like a problem with the virtual environment rather than with pfSense itself.

When you bridge a physical nic to a VM you assign it to different VMnet networks. in my case WAN - VMnet 17 - PciE nic and LAN - VMnet 18 - Pci Nic
i also tried changing these networks :(

From the laptop i can only ping public ip's not hostnames and i can open webconfigurator
From the Vm full internet access

Thank you for your fast and dedicated support tho

gjaltemba · Dec 12, 2016, 2:04 AM

Sounds like the dns client on the laptop is not working properly.

Did you try
ipconfig /flushdns
tracert 8.8.8.8

Problem started after a reboot. If possible, restart all network devices to get a clean start.

grim · Dec 12, 2016, 2:19 AM

@gjaltemba:

Sounds like the dns client on the laptop is not working properly.

Did you try
ipconfig /flushdns
tracert 8.8.8.8

Problem started after a reboot. If possible, restart all network devices to get a clean start.

That was one of the first things i've tried I flush after every change, laptop directly to wan cable has no problems, nor the vm

I've rebooted everything even the virtualizator + server
The problem is not in the laptop , the problem started on all infrastructure behind pfsense, I use the laptop on the lan port to exclude it so we don't bother troubleshoot there

grim · Dec 12, 2016, 8:04 AM

Another thing i noticed is that when the problem occurs the PFsense FW log is spammed with blocked entries
source WAN ip 192.168.1.1 destination 224.0.0.1 which is a muticast address

Even tho my WAN ip is a public ip not 192.168.1.1

KOM · Dec 12, 2016, 7:36 PM

Just a guess but after you moved the vm, you might have to go and reassign the interfaces again via the console menu. You shouldn't have to do this when just moving it's target folder, but you never know.

grim · Dec 14, 2016, 8:08 AM

@KOM:

Just a guess but after you moved the vm, you might have to go and reassign the interfaces again via the console menu. You shouldn't have to do this when just moving it's target folder, but you never know.

I will try this later , even tho from the webconfigurator both interfaces are with the correct mac :/
And when i powered up the old VM ( the original) the problem was there.

Next thing i will try, when i have time is a clean install with basic settings and restore module by module to try if the fault is there…

Because fine tuning of the proxy, av, ipsec, IPS, user control and other shit took me so much time :<

another thing some services go through, maybe they are not DNS dependent.

MasterX-BKC- · Dec 14, 2016, 7:27 PM

most likely the fault is in vmware workstation, or windows, config wise. they can be tricky to understand.

if you restore a known working config, after that it must be one of those 2, i have numerous pfsense units running under vmware ESXi with zero problems at all.

AndroBourne · Dec 14, 2016, 8:46 PM

It may not be related but aside from all the DNS stuff. I had a similar issue on my LAN when I first configured my VM PFSense box.

All this kinda sounds like a virtual adapter issues. I would try to blow out the original adapter and then recreate it and see if that works. My issue actually ended up being that the article I was reading about configuring the PFSense in a VM stated to use the Legacy VM adapters which actually caused my issue (same one you were having). I changed the virtual adapter from legacy to the standard adapter and issue went away.

Of course that is going to all depend on the version of PFSense you are running and the types of NICs you have. I would recommend you try both ways (legacy and non legacy) in the VM adapter settings and if that changes anything for you.

grim · Dec 26, 2016, 12:01 PM

Problem solved,

Symantec was blocking the traffic on the host :o i don't know how but disabling the AV solved the problem…

Thank you all for the support.