Pfsense no internet access on Esxi server



  • Good afternoon,

    First post on here and i apologise if i have the wrong category etc.

    I have built a Esxi server and just want to setup standard routing so that all my virtual machine on the local LAN port group can access external (Internet).

    Now i have setup WAN and LAN interface using :
    route add -net 51.x.x.x/32 -iface em0
    route add default 51.x.x.x

    **The 51.x.x.x address is my Esxi server ip.

    From the shell of the pfSense i can ping 8.8.8.8 and ping google.com

    My issue lies in the other virtuals
    The DHCP is working correctly and allocation ip address
    I can ping the pfSense - obviously as the DHCP is working

    The DNS also seems to be working as i can resolve google.com via a ping but i dont get any responses.

    I have done a windows tracert to 216.59.209.238 google.com
    1 <1ms <1m <1ms pfsense.localdomain [192.168.1.1]
    2 * * * *

    To me the router is not directing the traffic to the WAN - well thats my guess ( i am a software developer and not a network guy).

    For info - my virtual machine MAC and IP have been set 51.89.x.x and done the as per OVH specify.

    Any help would be really appreciated.



  • It's pretty simple really. Here is what you do:

    1. Connect your modem to physical server NIC port 1. This will be the WAN.

    2. Connect your LAN switch to physical server NIC port 2. This will be your LAN.

    3. In ESXi, create a vswitch called WAN and assign the WAN NIC to it.

    4. In ESXi, create another vswitch called LAN and assign LAN NIC to it.

    5. Create pfSense VM with two vNICs, first connected to WAN vswitch, other connected to LAN vswitch.

    6. Connect any physical LAN clients to the LAN switch that is connected to ESXi LAN NIC.

    7. Create additional VMs and connect their NIC to LAN vswitch.

    You're done. It should all just work.

    https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-vmware-vsphere-esxi.html



  • Hi KOM, that is exactly my setup the only difference is that i dont have NIC port 2. The server on OVH has one external WAN NIC.

    My other machines that are connected on the LAN (Port group LAN) can communicate with each other via IP or name.



  • @craig121 said in Pfsense no internet access on Esxi server:

    The server on OVH has one external WAN NIC.

    Sorry, I guess I didn't deduce that from your original post. I've never had to work with a router-on-a-stick config, but I believe it involves creating various VLANs to separate your networks.

    https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/router-on-a-stick.html

    https://docs.netgate.com/pfsense/en/latest/solutions/azure-appliance/launching-an-instance-single-nic.html

    Search for 'pfsense router stick' and you should get a lot of tutorials to work with.



  • Sorry i should have been clearer, in ESXi we have virtual switches to create the isolation.

    Therefore the LAN is on one switch where all the virtuals connect to, The pfSense virtual LAN port is connected to this switch.

    The WAN port of the pfSense virtual is connected to another switch that is connected to the NIC with external access.

    Therefore this is what i cannot understand how i can see the pfSense and obtain IP and DNS from it, but it will not route the external traffic.

    I will have a read of router on a stick as well.

    Apologies i am not very good with networks.

    Thanks



  • https://doc.pfsense.org/index.php/Connectivity_Troubleshooting

    Start at the beginning. What are your pfSense LAN config details re address, mask, DNS etc. What are your DHCP server config details?



  • Thanks Kom this should help me identify.

    I have just notice that it states

    Check that the WAN IP address has the correct subnet mask (Interfaces > WAN)

    An improper subnet mask such as /1 could cause connectivity issues to large portions of the Internet, using /32 for a mask can prevent the gateway from being found/used

    I am using a 32 mask

    route add -net 51.x.x.x/32 -iface em0

    Should i use another Mask i am unsure why the tutorial i followed specified a 32 mask?

    Or how do i calculate the Mask, i really appreciate your help Kom

    Thanks



  • I have check the logs and i have this in the routing log:

    Sep 19 17:35:24 radvd 37048 invalid all-zeros prefix in /var/etc/radvd.conf, line 9

    Line 9 - prefix ::/64 {



  • The mask you should use are details from your ISP or colo. Nobody but them can tell you what it is since it's their network you're on.

    That radvd error is to do with IPv6.



  • Ok i finally got it working, the tutuorial that i originally followed actually caused me all the headache.

    I remove the

    route add -net 51.x.x.x/32 -iface em0
    route add default 51.x.x.x

    On startup

    Went into the graphically interface and added a gateway to the following ip address. - Hey presto it all worked

    Thanks Kom for pointing me to the https://docs.netgate.com/pfsense/en/latest/routing/connectivity-troubleshooting.html

    Cheers



  • Yeah, I should have mentioned that you don't need to manually fiddle with the routing table or anything, and that those entries you added should be removed. Glad its working for you now.



  • Thanks again Kom, I have noticed one that if i dont continually ping the router stops i cant ping or do anything afterwards.

    Strange i guess it could be to do with VMware or something. I have checked the logs and nothing in there.

    Is there any kind of hibernation or anything that could cause this?



  • No, and it shouldn't be doing that. Is it hung or can you access the console?



  • Yeah can still access the console, i just cant ping any longer.



  • If you can still get to WebGUI, check the System log for anything unusual.



  • @craig121 So the netmask on the WAN interface is it /32 now? The default gateway is what?
    If /32, it would mean they are not on the same subnet and Layer 2 is going to behave strangely.

    I have noticed one that if i dont continually ping the router stops i cant ping or do anything afterwards.

    How did you get it to started working again?



  • Hi Awebster, i removed the

    route add -net 51.x.x.x/32 -iface em0
    route add default 51.x.x.x

    from the start up scripts.

    For me to get it working - i logged in via the GUI (web interface) and changed the WAN interface.

    So my current setup is that i have a static ip on the IPv4 address 51.89.243.x my original mask was set to 32. This is ip is a virtual ip from the hosting provider OVH which is has a virtual MAC address, this is set on the ESXI virtual machine NIC adapter (virtual).

    I added the IPv4 Upstream gateway - which is set to the ESXi host ip address. In my case 51.89.227.x. So when i added the gateway i had to change the mask to 16 on the IPv4 address as it was out of the subnet.

    I may have set this wrong as i believe this give full scope of the 3rd mask as well.

    Once i did that hey presto everything works but only for a period of time (around 10 -15mins).

    My new problem is that after that period of time it stop being able to access in the WAN and thats even within the Shell of pfSense. I cant even ping google.com.

    If i navigate into the Web UI and re save the WAN interface it comes back to life for 10 -15mins.

    I am going to have a play around with the ESXi settings in case its anything in there. Overall very impressed with pfSense and will be using it going forward.



  • Ok i have check the system logs (General) and found an entry that seems to be causing the issue:

    Sep 20 07:27:03 kernel arp.51.89.227.x moved from 00:00:0c:9c:f0:04 to d0:40:d3:01:23

    It looks like the MAC is being updated by the host for some strange reason, i will post an update when i have fixed and finalised.



  • I have no idea why this address is changing and i am not sure how to resolve or fix. Is there anything in pfSense that can detect and update accordingly.



  • I have looked at the Diagnostics / ARP table

    Interface IP address MAC address . Hostname
    WAN 51.89.227.x 00:00:0c:9f:f3:04 ns34322.ip-xx-xx-x . Expires in 645 seconds

    Once this has run out this is when the WAN connection stops working. How do i stop the expiry



  • This is an OVH thing. Basically they give you a single /32 and the gateway is on a different subnet.

    You might try changing the WAN interface mask back to /32, then going into System > Routing > Gateways, add your default gateway and click on the Advanced settings button and scroll down to the bottom.
    Check on Use non-local gateway.
    This will allow use of a gateway outside of this interface's subnet.



  • I have set as you have suggested and it worked after a reboot.

    I suppose now its a question of time to see if it works :)



  • Unfortunately the same error - the gateway that i set is to the IP of ESxi server - which is what a tutorial specified.



  • @craig121 said in Pfsense no internet access on Esxi server:

    Unfortunately the same error - the gateway that i set is to the IP of ESxi server - which is what a tutorial specified.

    I'm not sure that setting the gateway to the IP of the ESXi server is correct. Which tutorial did you follow?



  • The tutorial that i followed which is very long winded and you have skip a lot is the playlist:

    Youtube Video



  • Also its very similar to

    https://support.us.ovhcloud.com/hc/en-us/articles/360002175944-How-to-Connect-a-VM-to-the-Internet-Using-VMware-ESXi-6-5

    https://support.us.ovhcloud.com/hc/en-us/articles/360000096990

    I have just tried changing my final gateway ip address octlet to 254. So the first three octlets are the same as the Esxi server and the final octlet is 254.



  • Hallelujah - its finally working and sorted - follow the OVH links for a server on OVH.

    I wish i had looked at the articles rather than following a youtube video where i shouldn't have configured pfSense by adding static routes etc. Plus the gateway was set incorrectly.

    Kom and Awebster - really appreciate your help.


Log in to reply