Pfsense no internet access on Esxi server

craig121

Hi KOM, that is exactly my setup the only difference is that i dont have NIC port 2. The server on OVH has one external WAN NIC.

My other machines that are connected on the LAN (Port group LAN) can communicate with each other via IP or name.

KOM

@craig121 said in Pfsense no internet access on Esxi server:

The server on OVH has one external WAN NIC.

Sorry, I guess I didn't deduce that from your original post. I've never had to work with a router-on-a-stick config, but I believe it involves creating various VLANs to separate your networks.

https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/router-on-a-stick.html

https://docs.netgate.com/pfsense/en/latest/solutions/azure-appliance/launching-an-instance-single-nic.html

Search for 'pfsense router stick' and you should get a lot of tutorials to work with.

craig121

Sorry i should have been clearer, in ESXi we have virtual switches to create the isolation.

Therefore the LAN is on one switch where all the virtuals connect to, The pfSense virtual LAN port is connected to this switch.

The WAN port of the pfSense virtual is connected to another switch that is connected to the NIC with external access.

Therefore this is what i cannot understand how i can see the pfSense and obtain IP and DNS from it, but it will not route the external traffic.

I will have a read of router on a stick as well.

Apologies i am not very good with networks.

Thanks

KOM

https://doc.pfsense.org/index.php/Connectivity_Troubleshooting

Start at the beginning. What are your pfSense LAN config details re address, mask, DNS etc. What are your DHCP server config details?

craig121

Thanks Kom this should help me identify.

I have just notice that it states

Check that the WAN IP address has the correct subnet mask (Interfaces > WAN)

An improper subnet mask such as /1 could cause connectivity issues to large portions of the Internet, using /32 for a mask can prevent the gateway from being found/used

I am using a 32 mask

route add -net 51.x.x.x/32 -iface em0

Should i use another Mask i am unsure why the tutorial i followed specified a 32 mask?

Or how do i calculate the Mask, i really appreciate your help Kom

Thanks

craig121

I have check the logs and i have this in the routing log:

Sep 19 17:35:24 radvd 37048 invalid all-zeros prefix in /var/etc/radvd.conf, line 9

Line 9 - prefix ::/64 {

KOM

The mask you should use are details from your ISP or colo. Nobody but them can tell you what it is since it's their network you're on.

That radvd error is to do with IPv6.

craig121

Ok i finally got it working, the tutuorial that i originally followed actually caused me all the headache.

I remove the

route add -net 51.x.x.x/32 -iface em0
route add default 51.x.x.x

On startup

Went into the graphically interface and added a gateway to the following ip address. - Hey presto it all worked

Thanks Kom for pointing me to the https://docs.netgate.com/pfsense/en/latest/routing/connectivity-troubleshooting.html

Cheers

KOM

Yeah, I should have mentioned that you don't need to manually fiddle with the routing table or anything, and that those entries you added should be removed. Glad its working for you now.

craig121

Thanks again Kom, I have noticed one that if i dont continually ping the router stops i cant ping or do anything afterwards.

Strange i guess it could be to do with VMware or something. I have checked the logs and nothing in there.

Is there any kind of hibernation or anything that could cause this?

KOM

No, and it shouldn't be doing that. Is it hung or can you access the console?

craig121

Yeah can still access the console, i just cant ping any longer.

KOM

If you can still get to WebGUI, check the System log for anything unusual.

awebster

@craig121 So the netmask on the WAN interface is it /32 now? The default gateway is what?
If /32, it would mean they are not on the same subnet and Layer 2 is going to behave strangely.

I have noticed one that if i dont continually ping the router stops i cant ping or do anything afterwards.

How did you get it to started working again?

craig121

Hi Awebster, i removed the

route add -net 51.x.x.x/32 -iface em0
route add default 51.x.x.x

from the start up scripts.

For me to get it working - i logged in via the GUI (web interface) and changed the WAN interface.

So my current setup is that i have a static ip on the IPv4 address 51.89.243.x my original mask was set to 32. This is ip is a virtual ip from the hosting provider OVH which is has a virtual MAC address, this is set on the ESXI virtual machine NIC adapter (virtual).

I added the IPv4 Upstream gateway - which is set to the ESXi host ip address. In my case 51.89.227.x. So when i added the gateway i had to change the mask to 16 on the IPv4 address as it was out of the subnet.

I may have set this wrong as i believe this give full scope of the 3rd mask as well.

Once i did that hey presto everything works but only for a period of time (around 10 -15mins).

My new problem is that after that period of time it stop being able to access in the WAN and thats even within the Shell of pfSense. I cant even ping google.com.

If i navigate into the Web UI and re save the WAN interface it comes back to life for 10 -15mins.

I am going to have a play around with the ESXi settings in case its anything in there. Overall very impressed with pfSense and will be using it going forward.

craig121

Ok i have check the system logs (General) and found an entry that seems to be causing the issue:

Sep 20 07:27:03 kernel arp.51.89.227.x moved from 00:00:0c:9c:f0:04 to d0:40:d3:01:23

It looks like the MAC is being updated by the host for some strange reason, i will post an update when i have fixed and finalised.

craig121

I have no idea why this address is changing and i am not sure how to resolve or fix. Is there anything in pfSense that can detect and update accordingly.

craig121

I have looked at the Diagnostics / ARP table

Interface IP address MAC address . Hostname
WAN 51.89.227.x 00:00:0c:9f:f3:04 ns34322.ip-xx-xx-x . Expires in 645 seconds

Once this has run out this is when the WAN connection stops working. How do i stop the expiry

awebster

This is an OVH thing. Basically they give you a single /32 and the gateway is on a different subnet.

You might try changing the WAN interface mask back to /32, then going into System > Routing > Gateways, add your default gateway and click on the Advanced settings button and scroll down to the bottom.
Check on Use non-local gateway.
This will allow use of a gateway outside of this interface's subnet.

craig121

I have set as you have suggested and it worked after a reboot.

I suppose now its a question of time to see if it works :)