PfSense on esxi 6.7, can get it to work propperly.

marcel1988

Yes, this did the trick :)

marcel1988

so after a few day's working perfectly.
it now randomly stops working.

In the PfSense VM i see this:

When i reboot the VM everything works again.
What can this be?

kiokoman

try to reinstall open-vm-tools
maybe do a fsck also

marcel1988

@kiokoman

i have reinstalled the open-vm-tools and nothing changed
what you mean by fsck?

And what about the new update?
i'm now on:

2.4.4-RELEASE-p3 (amd64)
built on Wed May 15 18:53:44 EDT 2019
FreeBSD 11.2-RELEASE-p10

kiokoman

fsck: from console is option 5 and F key ( F: Reboot and run a filesystem check )
and yes, upgrade to 2.4.5

marcel1988

ok now i understand what you mean. i just did that a couple of hours ago and it seems to work again.
What about the update?

after the update i need to reinstall vm-tools again? and another fsck?

kiokoman

nope, no need, make a backup of your config just in case and do the upgrade

marcel1988

Still this is not working fine. My internet disconnects every 4-5 hour.
So can you please check my settings and tell me if the are right:

First off all, the network settings:

ESXI WAN settings: + VLAN ID 4095
ESXI LAN settings:
PfSense settings:

Cable managment:

WAN T-mobile > WAN Port intel NIC ESXI
LAN Port intel NIC > port 1 into standard network switch. ( not managed )
port 2 network switch > Onboard LAN port of ESXI server for managment.
The rest of the network ports of the switch are 2 Ubiquiti WIFI accespoint, and some computers.

As you can see, i also installed OpnSense just to find our of PfSense was the problem.
But also OpnSense has the same problems.

kiokoman

it's ok, was the cable removed during this screenshot i suppose? physical adapters are not green
if you are using igbn (native esxi driver) try to install net-igb (intel driver for esxi)

also install open-vm-tools from pfsense packages if you didn't already

marcel1988

Yes, the cable is removed otherwise i dont have good internet ofcourse :)

Both the drivers are installed:

Also the package in PfSense for open-vm-tools is installed.

kiokoman

idk, settings are ok, maybe do a backup of your configuration ad reinstall a new clean vm with pfsense 2.4.5 or try 2.5.0

Cool_Corona

Can I get a Teamviewer into the box... then I will have your ESXi/pfsense running in 5 mins

marcel1988

@kiokoman
I have tried freshinstall of 2.4.5 and 2.5.0
I have tried backing up and restoring.

Maybe te problem is the fact i dont enter a MAC-ADDRESS into the wan port of the T-mobile fiber?
and i leave the MTU also clear?

@Cool_Corona
if you can do it in 5 minutes. you can also tell me how ;)

kiokoman

that would be something specific to your isp

marcel1988

So i tried adding a MAC-ADDRESS but then there is no internet at all.
Where in PfSense can i find a log? so maybe i can see where the problem is occurring.

kiokoman

all logs are under status / system log
what kind of connection is it? pppoe ? dhcp? static or dynamic ip?

gcu_greyarea

I have a cable modem and for testing purposes I have been switching between a pfSense VM and physical appliances.

Not sure why - but my connection is stable with the mac address used when the connection was first setup. This may be a provider thing...

Either way - if you want your pfSense VM to use a custom mac address on the WAN interface you'll need to allow forged transmits.

I'd also suggest you enable promiscuous mode.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-7DC6486F-5400-44DF-8A62-6273798A2F80.html

and

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-92F3AB1F-B4C5-4F25-A010-8820D7250350.html

and

https://wahlnetwork.com/2013/04/29/how-the-vmware-forged-transmits-security-policy-works/

For good measure also allow mac changes.

Once its all working... you can revert these settings until you know exactly which setting breaks your environment.

To sum this up: on the WAN vSwitch and Portgroup you need to:

Allow Mac Changes
Allow Forged Transmits
Allow Promiscuous Mode
In pfsense - on the WAN Interface use a Mac Address that has previously worked with your provider

marcel1988

@kiokoman

These where my settings when i was using the Ubiquiti USG router:

marcel1988

ok, again the internet stopt working and i pull this out the logfile:

Apr 6 14:32:33 check_reload_status updating dyndns wan
Apr 6 14:32:33 check_reload_status Syncing firewall
Apr 6 14:32:33 php-fpm 2504 /interfaces_assign.php: Creating rrd update script
Apr 6 14:32:45 check_reload_status Syncing firewall
Apr 6 14:32:49 check_reload_status Syncing firewall
Apr 6 14:32:52 login login on ttyv0 as root
Apr 6 14:33:06 login login on ttyv0 as root
Apr 6 14:33:52 php-fpm 2504 /interfaces.php: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 6 14:33:52 php-fpm 2504 /interfaces.php: Gateway, none 'available' for inet6, use the first one configured. ''
Apr 6 14:33:52 check_reload_status Restarting ipsec tunnels
Apr 6 14:33:54 php-fpm 2504 /interfaces.php: Unbound /var/unbound/root.key file is corrupt, removing and recreating.
Apr 6 14:33:56 check_reload_status updating dyndns wan
Apr 6 14:33:56 kernel vlan0: changing name to 'vmx0.300'
Apr 6 14:33:58 check_reload_status Reloading filter
Apr 6 14:33:58 php-fpm 2504 /interfaces.php: Creating rrd update script
Apr 6 14:34:24 check_reload_status rc.newwanip starting vmx0.300
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Default gateway setting Interface WAN_DHCP Gateway as default.
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Gateway, none 'available' for inet6, use the first one configured. ''
Apr 6 14:34:24 check_reload_status Restarting ipsec tunnels
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip: Info: starting on vmx0.300.
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip: on (IP address: REMOVED FOR PRIVACY!!!!!!) (interface: []) (real interface: vmx0.300).
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip called with empty interface.
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> REMOVED FOR PRIVACY!!!!!! - Restarting packages.
Apr 6 14:34:25 check_reload_status Reloading filter
Apr 6 14:34:25 check_reload_status Starting packages
Apr 6 14:34:26 php-fpm 341 /rc.start_packages: Restarting/Starting all packages.
Apr 6 14:34:26 php-fpm 340 /interfaces_assign.php: Unbound /var/unbound/root.key file is corrupt, removing and recreating.
Apr 6 14:34:26 login login on ttyv0 as root
Apr 6 14:34:29 check_reload_status updating dyndns wan
Apr 6 14:34:29 check_reload_status Syncing firewall
Apr 6 14:34:29 php-fpm 340 /interfaces_assign.php: Creating rrd update script
Apr 6 14:35:01 pkg-static pfSense-repo upgraded: 2.4.5 -> 2.4.5_2
Apr 6 14:35:09 check_reload_status Syncing firewall
Apr 6 14:35:14 pkg-static fusefs-libs-2.9.9_1 installed
Apr 6 14:35:14 pkg-static libdnet-1.13_3 installed
Apr 6 14:35:14 pkg-static libmspack-0.10.1 installed
Apr 6 14:35:14 pkg-static open-vm-tools-nox11-11.0.1_2,2 installed
Apr 6 14:35:14 php /etc/rc.packages: Beginning package installation for Open-VM-Tools .
Apr 6 14:35:14 check_reload_status Syncing firewall
Apr 6 14:35:14 check_reload_status Syncing firewall
Apr 6 14:35:14 php /etc/rc.packages: Successfully installed package: Open-VM-Tools.
Apr 6 14:35:14 pkg-static pfSense-pkg-Open-VM-Tools-10.1.0_2,1 installed
Apr 6 14:35:16 check_reload_status Reloading filter
Apr 6 14:35:16 check_reload_status Starting packages
Apr 6 13:34:57 php-fpm 340 /rc.start_packages: Restarting/Starting all packages.
Apr 6 14:35:18 kernel VMware memory control driver initialized

After a reboot of the ESXI host, everything is working again.

kiokoman

uhm check the dhcp log and the gateway log also
REMOVED FOR PRIVACY!!!!!! your ip is 2 lines below