Firewall itself has no internet connection

coolsaet

@viragomann said in Firewall itself has no internet connection:

@coolsaet
Did you check "Hardware Checksum Offloading" in System > Advanced > Networking?

Yes, because it is a VM, I disabled Hardware Checksum Offloading. I also tried it with it on, just to be sure.

How did you configure the host network?

Do you mean the WAN connection? It is a static IP with a /27 subnet. I added the gateway in the gateway section and set it as the default gateway (not automatic). I checked the block Bogon an d private/loopback addresses.

For the LAN, I have 3 interfaces with static IP/subnets:

LAN_DATA: 172.21.0.1/16
LAN_VPN: 172.42.42.1/24
LAN_CLIENT: 10.99.0.1/16

I do have another gateway(172.42.42.15) added on the LAN_VPN interface. This is a host (172.42.42.15) in the 172.42.42.1/24 which is a VPN server.
A static route is linked to this gateway:
172.17.0.0/16 --> 172.42.42.15

Run a packet capture on WAN, while you try to ping out without a source stated. I suspect, that the packets don't go out there.

You are correct. I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.

Not sure what this means.

viragomann

@coolsaet said in Firewall itself has no internet connection:

How did you configure the host network?

Do you mean the WAN connection?

I meant the hypervisor network.

I did the packet capture and didn't see the ping. I also did the packet capture with a source address in the ping and then it shows up.

? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all.

coolsaet

@viragomann said in Firewall itself has no internet connection:

I meant the hypervisor network.

It is a proxmox installation with default bridge (vmbr0) with VLANs enabled.
The internet connection (fiber switch) has its own VLAN. The WAN interface on the pfsense is a virtual adapter in proxmox (VIRTIO) with only the internet VLAN provided.
There is no L3 routing in between the fiber switch and the pfsense. Only L2.

? On WAN?
pfSense should do address translation on outgoing packets, so that you should sie the WAN IP a source in the packets.
But the destination must not be changed at all.

Yes I did it on the WAN interface. I just choose my words incorrect. What I mean is, the ping doesn't show in the packet capture when I don't specify a source address. It does show up when I provide a source address but the soruce address gets translated tot he WAN IP.

coolsaet

Haven't found a solution to the problem, but I mannaged to find a workaround for the updates.

I added static routes for 208.123.73.207/32 (pkg00-atx.netgate.com) and 208.123.73.209/32 (pkg01-atx.netgate.com) to use the default gateway.
This at least seems to allow the update process to work.

viragomann

@coolsaet
That's quite strange. I cannot see anything in the routing table that should it direct to another gateway without a static route.

So you can confirm that Proxmox has an IP in no more than in one of the pfSense networks?

Did you possibly state a proxy in System > Advanced > Miscellaneous?

coolsaet

@viragomann
No proxy.

And proxmox (hypervisors) have no ip in the pfsense. They are on a different network/router.
Proxmox only provides the L2 connection (VLAN). The VLAN IDs are assigned by proxmox, not the pfsense, but that shouldn't make a difference.

The VMs on the proxmox do use the pfsense.

But the packets don't even leave the pfsense when I don't specify a source address.
It seems as if those packets just disappear instead of using the default route.

coolsaet

@coolsaet

Found out something new, not sure what it means though.

When performing a traceroute to google.com from the shell, I have the same problem. When specifying a source address it works, without it I get a timeout.

Now here is the funny thing, when specifying the interface it also works.

traceroute -i vtnet6 google.com

It even doesn't matter which interface I use (as long as the interface is enabled). So it seems as if the OS doesn't know which interface is its default.

Not sure how I could check this assumption as I am not that familiar with freebsd.

coolsaet

@viragomann
Found something else strange. In the Outbound NAT (auto created), I have 2 times the same network, but only one interface with that network. Is that normal?

viragomann

@coolsaet
Never seen that.

Are these the only automatic outbound NAT rules?
Is the outbound NAT in automatic mode?

It seems as if those packets just disappear instead of using the default route.

Or they are going to another gateway, for instance the VPN GW.
You can investigate with packet capture.

Did you state the WAN gateway also in the WAN interface settings?
Normally this is the correct way to add the WAN gateway. But if you added it in the gateway section you have to state it in the interface settings then additionally.

BTW: I found that you use a public address space for the LAN_VPN: 172.42.42.0/24. That's not recommended.

coolsaet

@viragomann

The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.

I did state the WAN Gateway in the WAN interface settings and also put it as the default gateway in the gateway section.

You are right about the 172.42.42.0/24. Did that on a late night. I already fixed it by changing it to 172.31.42.0/24.

I did packet captures on the WAN interface, but not yet on the VPN gateways. Will try that next.

viragomann

@coolsaet said in Firewall itself has no internet connection:

The outbound NAT is in Hybrid mode.
I have setup 2 rules with NO NAT for 172.17.0.0/16 and 172.18.0.0/17 as this are the VPN clients subnets which are only used for internal access.

There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).

For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.

coolsaet

@viragomann said in Firewall itself has no internet connection:

There should not be an outbound NAT rule generated automatically for these networks.
The only case I know, where pfSense is doing this is, when you state a gateway in the interface settings. But this should only be done with upstream gateways (WAN).

I didn't add them on the interfaces.

For the VPN you have to add the gateway only in System > Routing > Gateways and add static routes.

Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.

I also did the packet capture on all interfaces and now found it on the WAN interface:

But it seems to have a source IP of 0.0.0.0.

viragomann

@coolsaet said in Firewall itself has no internet connection:

Only added them in the static routing and gateway section, but they are still showing up in the automatic generated Outboun NAT section.

On WAN, I guess. This wouldn't matter, since the packets are routed out on another interface.

I also did the packet capture on all interfaces and now found it on the WAN interface:
But it seems to have a source IP of 0.0.0.0.

So the packets go out on the correct interface according to the routing table, but they are not natted.
What if you switch the outbound NAT in automatic mode?

coolsaet

@viragomann

I set outbound nat to automatic, but it still show 0.0.0.0 as the source IP.

viragomann

@coolsaet
Pretty strange. I've sadly no idea, where the 0.0.0.0 comes from in this case. If it doesn't nat I'd expect to see 127.0.0.1 as source.
Maybe you can override it with a custom outbound NAT rule. But this shouldn't be necessary in normal circumstances.

What's also weird to me is the MAC 26:3c:94:fe:3c:b9 address at the public IP in the routing table.
I don't expect this to be assigned from Proxmox. Did you spoof it?
Otherwise I can't think of what this should tell us.

Maybe @stephenw10 has an idea, what's going on here and can help.

coolsaet

@viragomann said in Firewall itself has no internet connection:

@coolsaet
Pretty strange. I've sadly no idea, where the 0.0.0.0 comes from in this case. If it doesn't nat I'd expect to see 127.0.0.1 as source.
Maybe you can override it with a custom outbound NAT rule. But this shouldn't be necessary in normal circumstances.

Should I than put a rule in the outbound NAT with source 0.0.0.0? Wouldn't that create other problems?

What's also weird to me is the MAC 26:3c:94:fe:3c:b9 address at the public IP in the routing table.
I don't expect this to be assigned from Proxmox. Did you spoof it?

It is assigned by Proxmox. Apparently, it gives completely random MAC addresses when no prefix is specified.

viragomann

@coolsaet said in Firewall itself has no internet connection:

Should I than put a rule in the outbound NAT with source 0.0.0.0? Wouldn't that create other problems?

That's a very good question. It brings me to another idea.
Normally it should be 127.0.0.1, the loopback address. But in your case it seems to be 0.0.0.0, which may indicate, that there is no IP assigned to the loopback interface for whatever reason.

Run

ifconfig -a

on pfSense to show the interface IP settings. Especially the lo0, but possibly there is some more odd with the interfaces, so best to see the whole output.

coolsaet

@viragomann
That was my idea too. But sadly everything looks normal. Or maybe I am missing something?

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 26:3c:94:fe:3c:b9
	inet6 fe80::243c:94ff:fefe:3cb9%vtnet0 prefixlen 64 scopeid 0x1
	inet 85.27.8.241 netmask 0xffffffe0 broadcast 85.27.8.255
	inet 85.27.8.229 netmask 0xffffffe0 broadcast 85.27.8.255
	inet 85.27.8.233 netmask 0xffffffe0 broadcast 85.27.8.255
	inet 85.27.8.234 netmask 0xffffffe0 broadcast 85.27.8.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1420
	description: LAN_DATA
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 5e:3d:cd:15:e1:00
	inet6 fe80::5c3d:cdff:fe15:e100%vtnet1 prefixlen 64 scopeid 0x2
	inet 172.21.0.1 netmask 0xffff0000 broadcast 172.21.255.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether ee:9c:d3:80:80:6a
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: CUSTOMER_VPN
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 26:e2:70:22:2d:37
	inet6 fe80::24e2:70ff:fe22:2d37%vtnet3 prefixlen 64 scopeid 0x4
	inet 172.31.42.1 netmask 0xffffff00 broadcast 172.31.42.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN_SERVICES
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether c6:a6:54:fa:4f:06
	inet6 fe80::c4a6:54ff:fefa:4f06%vtnet4 prefixlen 64 scopeid 0x5
	inet 10.99.0.1 netmask 0xffff0000 broadcast 10.99.255.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: VAULT_VPN
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 42:cd:41:82:9d:1e
	inet6 fe80::40cd:41ff:fe82:9d1e%vtnet5 prefixlen 64 scopeid 0x6
	inet 172.31.43.1 netmask 0xffffff00 broadcast 172.31.43.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: VAULT_SERVERS
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 4e:64:ab:25:a9:ae
	inet6 fe80::4c64:abff:fe25:a9ae%vtnet6 prefixlen 64 scopeid 0x7
	inet 172.30.0.1 netmask 0xffff0000 broadcast 172.30.255.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync
tun_wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	description: VIRT_VPN
	options=80000<LINKSTATE>
	inet 10.66.66.1 netmask 0xffffff00
	groups: wg WireGuard
	nd6 options=101<PERFORMNUD,NO_DAD>

vnet2 is disabled in the interfaces section.

viragomann

@coolsaet
No, it seems all well.

You can try it with a custom outbound NAT rule for 0.0.0.0/32, translating it to the WAN IP.
But to be honest, I would rather tear the VM down and start from scratch with a new installation.

coolsaet

@viragomann said in Firewall itself has no internet connection:

You can try it with a custom outbound NAT rule for 0.0.0.0/32, translating it to the WAN IP.

That does seem to work.

But to be honest, I would rather tear the VM down and start from scratch with a new installation.

I agree. NATting 0.0.0.0 doesn't feel right. I already tried reinstalling pfsense and restoring a backup of the config, but without success. I will be starting over from scratch. It will also give me a chance to improve some things.