PfSense on esxi 6.7, can get it to work propperly.

marcel1988

ok now i understand what you mean. i just did that a couple of hours ago and it seems to work again.
What about the update?

after the update i need to reinstall vm-tools again? and another fsck?

kiokoman

nope, no need, make a backup of your config just in case and do the upgrade

marcel1988

Still this is not working fine. My internet disconnects every 4-5 hour.
So can you please check my settings and tell me if the are right:

First off all, the network settings:

ESXI WAN settings: + VLAN ID 4095
ESXI LAN settings:
PfSense settings:

Cable managment:

WAN T-mobile > WAN Port intel NIC ESXI
LAN Port intel NIC > port 1 into standard network switch. ( not managed )
port 2 network switch > Onboard LAN port of ESXI server for managment.
The rest of the network ports of the switch are 2 Ubiquiti WIFI accespoint, and some computers.

As you can see, i also installed OpnSense just to find our of PfSense was the problem.
But also OpnSense has the same problems.

kiokoman

it's ok, was the cable removed during this screenshot i suppose? physical adapters are not green
if you are using igbn (native esxi driver) try to install net-igb (intel driver for esxi)

also install open-vm-tools from pfsense packages if you didn't already

marcel1988

Yes, the cable is removed otherwise i dont have good internet ofcourse :)

Both the drivers are installed:

Also the package in PfSense for open-vm-tools is installed.

kiokoman

idk, settings are ok, maybe do a backup of your configuration ad reinstall a new clean vm with pfsense 2.4.5 or try 2.5.0

Cool_Corona

Can I get a Teamviewer into the box... then I will have your ESXi/pfsense running in 5 mins

marcel1988

@kiokoman
I have tried freshinstall of 2.4.5 and 2.5.0
I have tried backing up and restoring.

Maybe te problem is the fact i dont enter a MAC-ADDRESS into the wan port of the T-mobile fiber?
and i leave the MTU also clear?

@Cool_Corona
if you can do it in 5 minutes. you can also tell me how ;)

kiokoman

that would be something specific to your isp

marcel1988

So i tried adding a MAC-ADDRESS but then there is no internet at all.
Where in PfSense can i find a log? so maybe i can see where the problem is occurring.

kiokoman

all logs are under status / system log
what kind of connection is it? pppoe ? dhcp? static or dynamic ip?

gcu_greyarea

I have a cable modem and for testing purposes I have been switching between a pfSense VM and physical appliances.

Not sure why - but my connection is stable with the mac address used when the connection was first setup. This may be a provider thing...

Either way - if you want your pfSense VM to use a custom mac address on the WAN interface you'll need to allow forged transmits.

I'd also suggest you enable promiscuous mode.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-7DC6486F-5400-44DF-8A62-6273798A2F80.html

and

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-92F3AB1F-B4C5-4F25-A010-8820D7250350.html

and

https://wahlnetwork.com/2013/04/29/how-the-vmware-forged-transmits-security-policy-works/

For good measure also allow mac changes.

Once its all working... you can revert these settings until you know exactly which setting breaks your environment.

To sum this up: on the WAN vSwitch and Portgroup you need to:

Allow Mac Changes
Allow Forged Transmits
Allow Promiscuous Mode
In pfsense - on the WAN Interface use a Mac Address that has previously worked with your provider

marcel1988

@kiokoman

These where my settings when i was using the Ubiquiti USG router:

marcel1988

ok, again the internet stopt working and i pull this out the logfile:

Apr 6 14:32:33 check_reload_status updating dyndns wan
Apr 6 14:32:33 check_reload_status Syncing firewall
Apr 6 14:32:33 php-fpm 2504 /interfaces_assign.php: Creating rrd update script
Apr 6 14:32:45 check_reload_status Syncing firewall
Apr 6 14:32:49 check_reload_status Syncing firewall
Apr 6 14:32:52 login login on ttyv0 as root
Apr 6 14:33:06 login login on ttyv0 as root
Apr 6 14:33:52 php-fpm 2504 /interfaces.php: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 6 14:33:52 php-fpm 2504 /interfaces.php: Gateway, none 'available' for inet6, use the first one configured. ''
Apr 6 14:33:52 check_reload_status Restarting ipsec tunnels
Apr 6 14:33:54 php-fpm 2504 /interfaces.php: Unbound /var/unbound/root.key file is corrupt, removing and recreating.
Apr 6 14:33:56 check_reload_status updating dyndns wan
Apr 6 14:33:56 kernel vlan0: changing name to 'vmx0.300'
Apr 6 14:33:58 check_reload_status Reloading filter
Apr 6 14:33:58 php-fpm 2504 /interfaces.php: Creating rrd update script
Apr 6 14:34:24 check_reload_status rc.newwanip starting vmx0.300
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Default gateway setting Interface WAN_DHCP Gateway as default.
Apr 6 14:34:24 php-fpm 340 /interfaces_assign.php: Gateway, none 'available' for inet6, use the first one configured. ''
Apr 6 14:34:24 check_reload_status Restarting ipsec tunnels
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip: Info: starting on vmx0.300.
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip: on (IP address: REMOVED FOR PRIVACY!!!!!!) (interface: []) (real interface: vmx0.300).
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: rc.newwanip called with empty interface.
Apr 6 14:34:25 php-fpm 2504 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> REMOVED FOR PRIVACY!!!!!! - Restarting packages.
Apr 6 14:34:25 check_reload_status Reloading filter
Apr 6 14:34:25 check_reload_status Starting packages
Apr 6 14:34:26 php-fpm 341 /rc.start_packages: Restarting/Starting all packages.
Apr 6 14:34:26 php-fpm 340 /interfaces_assign.php: Unbound /var/unbound/root.key file is corrupt, removing and recreating.
Apr 6 14:34:26 login login on ttyv0 as root
Apr 6 14:34:29 check_reload_status updating dyndns wan
Apr 6 14:34:29 check_reload_status Syncing firewall
Apr 6 14:34:29 php-fpm 340 /interfaces_assign.php: Creating rrd update script
Apr 6 14:35:01 pkg-static pfSense-repo upgraded: 2.4.5 -> 2.4.5_2
Apr 6 14:35:09 check_reload_status Syncing firewall
Apr 6 14:35:14 pkg-static fusefs-libs-2.9.9_1 installed
Apr 6 14:35:14 pkg-static libdnet-1.13_3 installed
Apr 6 14:35:14 pkg-static libmspack-0.10.1 installed
Apr 6 14:35:14 pkg-static open-vm-tools-nox11-11.0.1_2,2 installed
Apr 6 14:35:14 php /etc/rc.packages: Beginning package installation for Open-VM-Tools .
Apr 6 14:35:14 check_reload_status Syncing firewall
Apr 6 14:35:14 check_reload_status Syncing firewall
Apr 6 14:35:14 php /etc/rc.packages: Successfully installed package: Open-VM-Tools.
Apr 6 14:35:14 pkg-static pfSense-pkg-Open-VM-Tools-10.1.0_2,1 installed
Apr 6 14:35:16 check_reload_status Reloading filter
Apr 6 14:35:16 check_reload_status Starting packages
Apr 6 13:34:57 php-fpm 340 /rc.start_packages: Restarting/Starting all packages.
Apr 6 14:35:18 kernel VMware memory control driver initialized

After a reboot of the ESXI host, everything is working again.

kiokoman

uhm check the dhcp log and the gateway log also
REMOVED FOR PRIVACY!!!!!! your ip is 2 lines below

marcel1988

GENERAL LOG.txt DHCP LOG.txt

All files attached. :)
Hope this helps.

kiokoman

No DHCPOFFERS received.

maybe try with e1000 driver instead of vmnet3

marcel1988

@kiokoman

So i need to reset the Pfsense ESXI to default.
inside the esxi change both adapters from vmnet3 to e1000.
fire up PfSense and do it all over?

kiokoman

no just stop pfsense, edit the vm change from vmxnet3 to e1000 restart the vm, do it only for wan.
when you start pfsense it will noticed the change and ask you for the new wan interface in the console

marcel1988

ok perfect, i have done this now and i also cleared all the logs.