Firewall itself has no internet connection

SteveITS

@coolsaet What does Diagnostics/Routes show for the default routes?

Are you using a Wireguard VPN for Internet access and if so does it work if you disable that?

coolsaet

@steveits
I am using WireGuard only to access the router and LANs behind the router from external.
The WAN is a direct internet connection with static ip.

The routes look normal as far as I can see.
I did notice that the DNS servers I configured also have a route. I can ping them from the router without specifying the source address. Seems that the pfsens has trouble determining what the default route should be when no source is specified. I would think that it would be the default route?

IPv4 Routes
Destination	Gateway	Flags	Uses	MTU	Interface	Expire
default	85.18.8.254	UGS	1023910	1500	vtnet0	
1.0.0.1	85.18.8.254	UGHS	112317	1500	vtnet0	
1.1.1.1	85.18.8.254	UGHS	112323	1500	vtnet0	
8.8.8.8	85.18.8.254	UGHS	112317	1500	vtnet0	
10.66.66.0/24	link#10	U	147598	1420	tun_wg0	
10.66.66.1	link#10	UHS	0	16384	lo0	
10.99.0.0/16	link#5	U	4329865	1500	vtnet4	
10.99.0.1	link#5	UHS	0	16384	lo0	
85.18.8.224/27	link#1	U	0	1500	vtnet0	
85.18.8.229	link#1	UHS	0	16384	lo0	
85.18.8.233	link#1	UHS	0	16384	lo0	
85.18.8.241	link#1	UHS	0	16384	lo0	
85.18.8.244	link#1	UHS	0	16384	lo0	
85.18.8.254	26:3c:94:fe:3c:b9	UHS	482593	1500	vtnet0	
127.0.0.1	link#7	UH	8340	16384	lo0	
172.17.0.0/16	172.42.42.10	UGS	0	1500	vtnet3	
172.21.0.0/16	link#2	U	173661195	1420	vtnet1	
172.21.0.1	link#2	UHS	0	16384	lo0	
172.42.42.0/24	link#4	U	563695	1500	vtnet3	
172.42.42.1	link#4	UHS	0	16384	lo0

viragomann

@coolsaet
Did you check "Hardware Checksum Offloading" in System > Advanced > Networking?

How did you configure the host network?

Run a packet capture on WAN, while you try to ping out without a source stated. I suspect, that the packets don't go out there.

coolsaet

@viragomann said in Firewall itself has no internet connection:

@coolsaet
Did you check "Hardware Checksum Offloading" in System > Advanced > Networking?

Yes, because it is a VM, I disabled Hardware Checksum Offloading. I also tried it with it on, just to be sure.

How did you configure the host network?

Do you mean the WAN connection? It is a static IP with a /27 subnet. I added the gateway in the gateway section and set it as the default gateway (not automatic). I checked the block Bogon an d private/loopback addresses.

For the LAN, I have 3 interfaces with static IP/subnets:

LAN_DATA: 172.21.0.1/16
LAN_VPN: 172.42.42.1/24
LAN_CLIENT: 10.99.0.1/16

I do have another gateway(172.42.42.15) added on the LAN_VPN interface. This is a host (172.42.42.15) in the 172.42.42.1/24 which is a VPN server.
A static route is linked to this gateway:
172.17.0.0/16 --> 172.42.42.15

Run a packet capture on WAN, while you try to ping out without a source stated. I suspect, that the packets don't go out there.

You are correct. I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.

Not sure what this means.

viragomann

@coolsaet said in Firewall itself has no internet connection:

How did you configure the host network?

Do you mean the WAN connection?

I meant the hypervisor network.

I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.

? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all.

coolsaet

@viragomann said in Firewall itself has no internet connection:

I meant the hypervisor network.

It is a proxmox installation with default bridge (vmbr0) with VLANs enabled.
The internet connection (fiber switch) has its own VLAN. The WAN interface on the pfsense is a virtual adapter in proxmox (VIRTIO) with only the internet VLAN provided.
There is no L3 routing in between the fiber switch and the pfsense. Only L2.

? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all.

Yes I did it on the WAN interface. I just choose my words incorrect. What I mean is, the ping doesn't show in the packet capture when I don't specify a source address. It does show up when I provide a source address but the soruce address gets translated tot he WAN IP.

coolsaet

Haven't found a solution to the problem, but I mannaged to find a workaround for the updates.

I added static routes for 208.123.73.207/32 (pkg00-atx.netgate.com) and 208.123.73.209/32 (pkg01-atx.netgate.com) to use the default gateway.
This at least seems to allow the update process to work.

viragomann

@coolsaet
That's quite strange. I cannot see anything in the routing table that should it direct to another gateway without a static route.

So you can confirm that Proxmox has an IP in no more than in one of the pfSense networks?

Did you possibly state a proxy in System > Advanced > Miscellaneous?

coolsaet

@viragomann
No proxy.

And proxmox (hypervisors) have no ip in the pfsense. They are on a different network/router.
Proxmox only provides the L2 connection (VLAN). The VLAN IDs are assigned by proxmox, not the pfsense, but that shouldn't make a difference.

The VMs on the proxmox do use the pfsense.

But the packets don't even leave the pfsense when I don't specify a source address.
It seems as if those packets just disappear instead of using the default route.

coolsaet

@coolsaet

Found out something new, not sure what it means though.

When performing a traceroute to google.com from the shell, I have the same problem. When specifying a source address it works, without it I get a timeout.

Now here is the funny thing, when specifying the interface it also works.

traceroute -i vtnet6 google.com

It even doesn't matter which interface I use (as long as the interface is enabled). So it seems as if the OS doesn't know which interface is its default.

Not sure how I could check this assumption as I am not that familiar with freebsd.

coolsaet

@viragomann
Found something else strange. In the Outbound NAT (auto created), I have 2 times the same network, but only one interface with that network. Is that normal?

viragomann

@coolsaet
Never seen that.

Are these the only automatic outbound NAT rules?
Is the outbound NAT in automatic mode?

It seems as if those packets just disappear instead of using the default route.

Or they are going to another gateway, for instance the VPN GW.
You can investigate with packet capture.

Did you state the WAN gateway also in the WAN interface settings?
Normally this is the correct way to add the WAN gateway. But if you added it in the gateway section you have to state it in the interface settings then additionally.

BTW: I found that you use a public address space for the LAN_VPN: 172.42.42.0/24. That's not recommended.

coolsaet

@viragomann

The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.

I did state the WAN Gateway in the WAN interface settings and also put it as the default gateway in the gateway section.

You are right about the 172.42.42.0/24. Did that on a late night. I already fixed it by changing it to 172.31.42.0/24.

I did packet captures on the WAN interface, but not yet on the VPN gateways. Will try that next.

viragomann

@coolsaet said in Firewall itself has no internet connection:

The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.

There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).

For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.

coolsaet

@viragomann said in Firewall itself has no internet connection:

There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).

I didn't add them on the interfaces.

For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.

Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.

I also did the packet capture on all interfaces and now found it on the WAN interface:

But it seems to have a source IP of 0.0.0.0.

viragomann

@coolsaet said in Firewall itself has no internet connection:

Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.

On WAN, I guess. This wouldn't matter, since the packets are routed out on another interface.

I also did the packet capture on all interfaces and now found it on the WAN interface:
But it seems to have a source IP of 0.0.0.0.

So the packets go out on the correct interface according to the routing table, but they are not natted.
What if you switch the outbound NAT in automatic mode?

coolsaet

@viragomann

I set outbound nat to automatic, but it still show 0.0.0.0 as the source IP.

viragomann

@coolsaet
Pretty strange. I've sadly no idea, where the 0.0.0.0 comes from in this case. If it doesn't nat I'd expect to see 127.0.0.1 as source.
Maybe you can override it with a custom outbound NAT rule. But this shouldn't be necessary in normal circumstances.

What's also weird to me is the MAC 26:3c:94:fe:3c:b9 address at the public IP in the routing table.
I don't expect this to be assigned from Proxmox. Did you spoof it?
Otherwise I can't think of what this should tell us.

Maybe @stephenw10 has an idea, what's going on here and can help.

coolsaet

@viragomann said in Firewall itself has no internet connection:

@coolsaet
Pretty strange. I've sadly no idea, where the 0.0.0.0 comes from in this case. If it doesn't nat I'd expect to see 127.0.0.1 as source.
Maybe you can override it with a custom outbound NAT rule. But this shouldn't be necessary in normal circumstances.

Should I than put a rule in the outbound NAT with source 0.0.0.0? Wouldn't that create other problems?

What's also weird to me is the MAC 26:3c:94:fe:3c:b9 address at the public IP in the routing table.
I don't expect this to be assigned from Proxmox. Did you spoof it?

It is assigned by Proxmox. Apparently, it gives completely random MAC addresses when no prefix is specified.

viragomann

@coolsaet said in Firewall itself has no internet connection:

Should I than put a rule in the outbound NAT with source 0.0.0.0? Wouldn't that create other problems?

That's a very good question. It brings me to another idea.
Normally it should be 127.0.0.1, the loopback address. But in your case it seems to be 0.0.0.0, which may indicate, that there is no IP assigned to the loopback interface for whatever reason.

Run

ifconfig -a

on pfSense to show the interface IP settings. Especially the lo0, but possibly there is some more odd with the interfaces, so best to see the whole output.