Firewall itself has no internet connection
-
My firewall/router is a single instance pfsense on a VM with 6 interfaces:
- WAN
- 4x LAN (Different subnets)
- Wireguard tunnel interface
Everything seems to be working, except for 1 thing. I can not install updates/packages. I have narrowed the problem to a routing issue on the device itself.
When I SSH to the device and try to ping any extrernal IP, I get a timeout. However when including the source address in the ping command as one of the addresses assigned to the firewall (WAN, IP aliases, one of the addresses of the interfaces or 127.0.0.1) I get a reply.ping google.com PING google.com (142.251.39.110): 56 data bytes --- google.com ping statistics --- 7 packets transmitted, 0 packets received, 100.0% packet lossping -S 127.0.0.1 google.com PING google.com (142.251.39.110) from 127.0.0.1: 56 data bytes 64 bytes from 142.251.39.110: icmp_seq=0 ttl=57 time=1.190 ms 64 bytes from 142.251.39.110: icmp_seq=1 ttl=57 time=1.179 ms 64 bytes from 142.251.39.110: icmp_seq=2 ttl=57 time=1.150 msI observe the same behaviour from the gui ping tool. When selecting the automatic option for the source address it fails.
I am at a loss what could be the cause of this.
-
If you "traceroute google.com" can you tell what path it's trying to use? (it's also in the GUI under Diagnostics)
-
@steveits
I doesn't find any path.traceroute google.com traceroute to google.com (142.251.36.14), 64 hops max, 40 byte packets 1 * * * 2 * * * 3 * * *If I put in the source parameter, it works fine.
-
@coolsaet What does Diagnostics/Routes show for the default routes?
Are you using a Wireguard VPN for Internet access and if so does it work if you disable that?
-
@steveits
I am using WireGuard only to access the router and LANs behind the router from external.
The WAN is a direct internet connection with static ip.The routes look normal as far as I can see.
I did notice that the DNS servers I configured also have a route. I can ping them from the router without specifying the source address. Seems that the pfsens has trouble determining what the default route should be when no source is specified. I would think that it would be the default route?IPv4 Routes Destination Gateway Flags Uses MTU Interface Expire default 85.18.8.254 UGS 1023910 1500 vtnet0 1.0.0.1 85.18.8.254 UGHS 112317 1500 vtnet0 1.1.1.1 85.18.8.254 UGHS 112323 1500 vtnet0 8.8.8.8 85.18.8.254 UGHS 112317 1500 vtnet0 10.66.66.0/24 link#10 U 147598 1420 tun_wg0 10.66.66.1 link#10 UHS 0 16384 lo0 10.99.0.0/16 link#5 U 4329865 1500 vtnet4 10.99.0.1 link#5 UHS 0 16384 lo0 85.18.8.224/27 link#1 U 0 1500 vtnet0 85.18.8.229 link#1 UHS 0 16384 lo0 85.18.8.233 link#1 UHS 0 16384 lo0 85.18.8.241 link#1 UHS 0 16384 lo0 85.18.8.244 link#1 UHS 0 16384 lo0 85.18.8.254 26:3c:94:fe:3c:b9 UHS 482593 1500 vtnet0 127.0.0.1 link#7 UH 8340 16384 lo0 172.17.0.0/16 172.42.42.10 UGS 0 1500 vtnet3 172.21.0.0/16 link#2 U 173661195 1420 vtnet1 172.21.0.1 link#2 UHS 0 16384 lo0 172.42.42.0/24 link#4 U 563695 1500 vtnet3 172.42.42.1 link#4 UHS 0 16384 lo0 -
@coolsaet
Did you check "Hardware Checksum Offloading" in System > Advanced > Networking?How did you configure the host network?
Run a packet capture on WAN, while you try to ping out without a source stated. I suspect, that the packets don't go out there.
-
@viragomann said in Firewall itself has no internet connection:
@coolsaet
Did you check "Hardware Checksum Offloading" in System > Advanced > Networking?Yes, because it is a VM, I disabled Hardware Checksum Offloading. I also tried it with it on, just to be sure.
How did you configure the host network?
Do you mean the WAN connection? It is a static IP with a /27 subnet. I added the gateway in the gateway section and set it as the default gateway (not automatic). I checked the block Bogon an d private/loopback addresses.
For the LAN, I have 3 interfaces with static IP/subnets:
- LAN_DATA: 172.21.0.1/16
- LAN_VPN: 172.42.42.1/24
- LAN_CLIENT: 10.99.0.1/16
I do have another gateway(172.42.42.15) added on the LAN_VPN interface. This is a host (172.42.42.15) in the 172.42.42.1/24 which is a VPN server.
A static route is linked to this gateway:
172.17.0.0/16 --> 172.42.42.15Run a packet capture on WAN, while you try to ping out without a source stated. I suspect, that the packets don't go out there.
You are correct. I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.
Not sure what this means.
-
@coolsaet said in Firewall itself has no internet connection:
How did you configure the host network?
Do you mean the WAN connection?
I meant the hypervisor network.
I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.
? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all. -
@viragomann said in Firewall itself has no internet connection:
I meant the hypervisor network.
It is a proxmox installation with default bridge (vmbr0) with VLANs enabled.
The internet connection (fiber switch) has its own VLAN. The WAN interface on the pfsense is a virtual adapter in proxmox (VIRTIO) with only the internet VLAN provided.
There is no L3 routing in between the fiber switch and the pfsense. Only L2.? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all.Yes I did it on the WAN interface. I just choose my words incorrect. What I mean is, the ping doesn't show in the packet capture when I don't specify a source address. It does show up when I provide a source address but the soruce address gets translated tot he WAN IP.
-
Haven't found a solution to the problem, but I mannaged to find a workaround for the updates.
I added static routes for 208.123.73.207/32 (pkg00-atx.netgate.com) and 208.123.73.209/32 (pkg01-atx.netgate.com) to use the default gateway.
This at least seems to allow the update process to work. -
@coolsaet
That's quite strange. I cannot see anything in the routing table that should it direct to another gateway without a static route.So you can confirm that Proxmox has an IP in no more than in one of the pfSense networks?
Did you possibly state a proxy in System > Advanced > Miscellaneous?
-
@viragomann
No proxy.And proxmox (hypervisors) have no ip in the pfsense. They are on a different network/router.
Proxmox only provides the L2 connection (VLAN). The VLAN IDs are assigned by proxmox, not the pfsense, but that shouldn't make a difference.The VMs on the proxmox do use the pfsense.
But the packets don't even leave the pfsense when I don't specify a source address.
It seems as if those packets just disappear instead of using the default route. -
Found out something new, not sure what it means though.
When performing a traceroute to google.com from the shell, I have the same problem. When specifying a source address it works, without it I get a timeout.
Now here is the funny thing, when specifying the interface it also works.
traceroute -i vtnet6 google.comIt even doesn't matter which interface I use (as long as the interface is enabled). So it seems as if the OS doesn't know which interface is its default.
Not sure how I could check this assumption as I am not that familiar with freebsd.
-
@viragomann
Found something else strange. In the Outbound NAT (auto created), I have 2 times the same network, but only one interface with that network. Is that normal?
-
@coolsaet
Never seen that.Are these the only automatic outbound NAT rules?
Is the outbound NAT in automatic mode?It seems as if those packets just disappear instead of using the default route.
Or they are going to another gateway, for instance the VPN GW.
You can investigate with packet capture.Did you state the WAN gateway also in the WAN interface settings?
Normally this is the correct way to add the WAN gateway. But if you added it in the gateway section you have to state it in the interface settings then additionally.BTW: I found that you use a public address space for the LAN_VPN: 172.42.42.0/24. That's not recommended.
-
The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.I did state the WAN Gateway in the WAN interface settings and also put it as the default gateway in the gateway section.
You are right about the 172.42.42.0/24. Did that on a late night.
I already fixed it by changing it to 172.31.42.0/24.I did packet captures on the WAN interface, but not yet on the VPN gateways. Will try that next.
-
@coolsaet said in Firewall itself has no internet connection:
The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.
-
@viragomann said in Firewall itself has no internet connection:
There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).I didn't add them on the interfaces.
For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.
Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.
I also did the packet capture on all interfaces and now found it on the WAN interface:
But it seems to have a source IP of 0.0.0.0.
-
@coolsaet said in Firewall itself has no internet connection:
Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.
On WAN, I guess. This wouldn't matter, since the packets are routed out on another interface.
I also did the packet capture on all interfaces and now found it on the WAN interface:
But it seems to have a source IP of 0.0.0.0.So the packets go out on the correct interface according to the routing table, but they are not natted.
What if you switch the outbound NAT in automatic mode? -
I set outbound nat to automatic, but it still show 0.0.0.0 as the source IP.