Ping not working on CLI or with "default" interface on GUI

llavecita · Oct 26, 2023, 3:27 PM

hello everyone! I have a pfSense installation in a dedicated server with 10Gbit ports, ix0 and ix1. I configured ix0 as WAN, with a static IPv4 and its relevant gateway. If I go to Diagnostics > Ping, and I select the relevant WAN interface, it pings the selected host correctly. If I leave it as default, ping does not go through. If I ping from the CLI, ping does not go through (unless I specify a source address, ie. ping -S 123.123.123.1 8.8.8.8, in which case it works). In System > Routing this system only has one Gateway, which is set as default. WireGuard VPN tunnels have no problem reaching the Internet using this machine as egress point.

netstat -rn shows as follows (after redacting public IPv4s with 123.123.123.0 stuff):

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            123.123.123.123    UGS         ix0
10.95.32.0/24      link#9             U       tun_wg0
10.95.32.99        link#6             UHS         lo0
127.0.0.1          link#6             UH          lo0
123.123.123.0/25   link#3             U           ix0
123.123.123.1      link#6             UHS         lo0

So, for example, if I ping 8.8.8.8, it should go via default, and therefore egress to the Internet. But it does not happen on CLI, unless I add a specific route for that 8.8.8.8 to hit the gateway, in which case ping works. From both CLI and GUI, the gateway can be pinged with no issues.

Any idea? Given the symptoms I am inclined to think that the system is trying to use a different network card by default, ie. igb0/igb1, which are also present in the system, instead of ix0, which is the configured one. But looking at the routing table, it should be selecting ix0 interface.

Thanks a lot!

stephenw10 · Oct 26, 2023, 4:10 PM

Is the gateway actually set on the interface itself? In Interfaces > WAN?

That's what pfSense uses to identify it as a wan and triggers route-to on firewall rules.

If you ping out without setting a source and check the state table do you see a state created? On which interface?

Steve

llavecita · Oct 26, 2023, 4:33 PM

Hello @stephenw10 ,

Thank you for your answer.

I confirm the gateway is set on WAN interface, and it is also set as default gateway.

When pinging out to ie. 8.8.8.8, I confirm there is a state in WAN interface, and it seems it is trying to be sent as 0.0.0.0. That is odd.

WAN	icmp	0.0.0.0:1836 -> 8.8.8.8:1836	0:0	84 / 0	7 KiB / 0 B

Now the question would be why WAN is using 0.0.0.0 for egress traffic if no source IP is selected. WAN currently has a few IP addresses, which are (redacted to random IPs):
123.123.123.12/25, which is the one assigned by the datacenter
80.80.80.1/32, which is from a prefix I am announcing from this server, so it is pingable. This is a Virtual IP.
90.90.90.1/32, which is from another prefix I am announcing from this server, so it is pingable. This is a Virtual IP.

I remember there was a bug where if a WAN interface had Virtual IPs assigned, it could break things. But I think that was fixed?

Thanks a lot!

stephenw10 · Oct 26, 2023, 4:39 PM

Yes, that should be fixed but I agree that's exactly what came to mind when seeing that.

I assume you can ping out from any of VIPs on WAN if you set them as source?

Do you have Outbound NAT set to automatic still? That should prevent it NATing it's own traffic but if it's using manual more something might be over-matching.

Steve

llavecita · Oct 26, 2023, 4:48 PM

@stephenw10

Thank you for your reply.

NAT Outbound is set as manual, and it has nothing, other than the rules that come in pfSense by default, which are:

WAN	127.0.0.0/8	*	*	500 (ISAKMP)	WAN address	*			  
WAN	127.0.0.0/8	*	*	*	WAN address	*			  
WAN	::1/128	*	*	500 (ISAKMP)	WAN address	*			  
WAN	::1/128	*	*	*	WAN address	*

I confirm I can ping from both VIPs (as well as WAN, when WAN is selected) from both GUI (selecting the relevant virtual IP or interface) or CLI (specifying the source with ping -S <SRC_IP> <DST_IP>)

Thanks a lot!

llavecita · Oct 26, 2023, 4:53 PM

I have now deleted both VIPs, so WAN only has one allocated IP (in case it was the Virtual IPs bug I was referring to), but the problem still persists. I have also set NAT Outbound to Automatic to no avail, so I just rolled it back to Manual.

stephenw10 · Oct 26, 2023, 5:28 PM

Hmm, if you run a pcap on WAN is it actually sending traffic with source 0.0.0.0?

llavecita · Oct 26, 2023, 5:37 PM

@stephenw10

That is correct. This is the excerpt of the Packet Capture:

17:29:56.826861 IP 0.0.0.0 > 8.8.8.8: ICMP echo request, id 1836, seq 4001, length 64

I am attaching the exported pcap for your reference, with the corresponding ping packets to 8.8.8.8. capture.pcap

stephenw10 · Oct 26, 2023, 5:41 PM

If you resave the WAN config does it stop?

Was the WAN ever configured as DHCP? Is there a dhclient process running? Anything in the dhcp logs?
That's the only time seeing 0.0.0.0 might be expected.

Steve

llavecita · Oct 26, 2023, 5:53 PM

@stephenw10

I have re-saved WAN interface configuration, and applied it. Problem still persists, unfortunately.

DHCP logs show one entry from yesterday (2023-10-25 18:18:20.595596+00:00 dhclient 5829 Cannot open or create pidfile: No such file or directory), and a few entries for the date when the server was deployed, on 20th of October.

There seems to be a dhclient process running, which keeps changing the PID (second column) incrementally. Maybe it is restarting all the time?

$ ps aux | grep dhclient
root    40010    0.0  0.0  12768   2424  0  S+   17:50       0:00.00 grep dhclient

$ ps aux | grep dhclient
root    40711    0.0  0.0  12768   2432  0  S+   17:50       0:00.00 grep dhclient

$ ps aux | grep dhclient
root    41154    0.0  0.0  12768   2432  0  S+   17:50       0:00.00 grep dhclient

$ ps aux | grep dhclient
root    41588    0.0  0.0  12768   2432  0  S+   17:50       0:00.00 grep dhclient

Thanks a lot!

stephenw10 · Oct 26, 2023, 6:09 PM

Hmm, that's not right! There shouldn;t be any dhclient there if it's all static IPs.

If you run ps auxwwd | grep dhclient you should be able to see which interface it's running on.

I would first try just killing that process.

llavecita · Oct 26, 2023, 6:20 PM

Oh sorry, my brain stopped working after chasing this for too long today. The command above shows grep dhclient process which I am spinning up when running ps aux | grep dhclient, not dhclient process itself. That is why the PID kept incrementing, because I was invoking it all the time. Facepalm.

So yes, there is no dhclient process running, and therefore we cannot kill it.

My apologies. I will continue the research tomorrow and try to figure out why WAN is using 0.0.0.0 as SRC when no address is specified, as it is clear brain fog is kicking in.

stephenw10 · Oct 26, 2023, 6:21 PM

Ah, yes, I should have spotted that!

I guess I'd try disabling wireguard as a test. There's not much else it could be....

stephenw10 · Oct 26, 2023, 7:06 PM

Hmm, do you have a Wireguard gateway defined?

Is your default IPv4 gateway still set to automatic in System > Routing > Gateways?
If it is try setting it to the WAN gateway.

llavecita · Oct 27, 2023, 6:21 AM

Hello again @stephenw10 . New day, new time to chase this mystery.

I have stopped WireGuard service, but issue still persists.

In regards to IPv4 gateway, it is not automatic - it is statically defined. It is the only Gateway in this system (no WireGuard gateways or anything extra), and it is also set as default. Puzzling.

I have tried to add a NAT outbound rule on top of the list, selecting "any" as source on WAN interface, and translating it to the "interface address", and ping works now when not specifying an interface. But this is a dirty patch, as I assume WAN is still sending the packets with 0.0.0.0 as source, just that we are translating that address locally. On top of that, nslookup still does not work even with that workaround, ie:

$ nslookup dns.google
;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable.
;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable.
;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable.
;; UDP setup with 9.9.9.9#53(9.9.9.9) for dns.google failed: host unreachable.

And pfsense updates also don't work. So I am reverting that NAT outbound rule.

I will continue to chase it for a few hours more and, if no luck, I will try reinstalling the system and observe behaviour when freshly installed and no config.xml restores.

llavecita · Oct 27, 2023, 6:49 AM

Here is some additional info, in case it helps

$ ifconfig -a

igb0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
	ether 3c:ec:ef:a7:22:e8
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
	ether 3c:ec:ef:a7:22:e9
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
	ether 9c:69:b4:64:96:7a
	inet6 fe80::9e69:b4ff:fe64:967a%ix0 prefixlen 64 scopeid 0x3
	inet 123.123.123.123(redacted) netmask 0xffffff80 broadcast 123.123.123.1(redacted)
	media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT1
	options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
	ether 9c:69:b4:64:96:7b
	inet6 fe80::9e69:b4ff:fe64:967b%ix1 prefixlen 64 scopeid 0x4
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pflog0: flags=100<PROMISC> metric 0 mtu 33152
	groups: pflog

pfsync0: flags=0<> metric 0 mtu 1500
	maxupd: 128 defer: off
	syncok: 1
	groups: pfsync

tun_wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
	description: WG_EG
	options=80000<LINKSTATE>
	inet 80.80.80.1(redacted) netmask 0xffffff00
	inet 90.90.90.1(redacted) netmask 0xffffff00
	inet 10.95.32.99 netmask 0xffffff00
	groups: wg WireGuard
	nd6 options=101<PERFORMNUD,NO_DAD>

I also verified that in the uploaded .pcap file, the MAC address of the source matches the one in ix0 interface.

stephenw10 · Oct 27, 2023, 11:33 AM

@llavecita said in Ping not working on CLI or with "default" interface on GUI:

inet 123.123.123.123(redacted) netmask 0xffffff80 broadcast 123.123.123.1(redacted)

Is that the other way around? I expect the broadcast address to be .127 on a /25.

llavecita · Oct 27, 2023, 11:53 AM

@stephenw10 I redacted the real values, but you are indeed correct - shown broadcast address in ifconfig for interface ix0 is a .127 (and gateway is a .126)

stephenw10 · Oct 27, 2023, 12:05 PM

Hmm, OK. We discussed this internally last night and there's nothing immediately that presents like that.

Can I assume this behaviour survives a reboot?

llavecita · Oct 27, 2023, 12:22 PM

@stephenw10 Correct, the server has been rebooted several times to no avail. I will find some time today to reinstall it from scratch, without dragging any previous config.xml, to see if behaviour persists on a fresh install. I will report back later today.

Thanks again for your time and dedication!