Ping not working on CLI or with "default" interface on GUI
-
hello everyone! I have a pfSense installation in a dedicated server with 10Gbit ports, ix0 and ix1. I configured ix0 as WAN, with a static IPv4 and its relevant gateway. If I go to
Diagnostics > Ping, and I select the relevant WAN interface, it pings the selected host correctly. If I leave it as default, ping does not go through. If I ping from the CLI, ping does not go through (unless I specify a source address, ie.ping -S 123.123.123.1 8.8.8.8, in which case it works). InSystem > Routingthis system only has one Gateway, which is set as default. WireGuard VPN tunnels have no problem reaching the Internet using this machine as egress point.netstat -rn shows as follows (after redacting public IPv4s with 123.123.123.0 stuff):
Routing tables Internet: Destination Gateway Flags Netif Expire default 123.123.123.123 UGS ix0 10.95.32.0/24 link#9 U tun_wg0 10.95.32.99 link#6 UHS lo0 127.0.0.1 link#6 UH lo0 123.123.123.0/25 link#3 U ix0 123.123.123.1 link#6 UHS lo0So, for example, if I ping 8.8.8.8, it should go via default, and therefore egress to the Internet. But it does not happen on CLI, unless I add a specific route for that 8.8.8.8 to hit the gateway, in which case ping works. From both CLI and GUI, the gateway can be pinged with no issues.
Any idea? Given the symptoms I am inclined to think that the system is trying to use a different network card by default, ie. igb0/igb1, which are also present in the system, instead of ix0, which is the configured one. But looking at the routing table, it should be selecting ix0 interface.
Thanks a lot!
-
Is the gateway actually set on the interface itself? In Interfaces > WAN?
That's what pfSense uses to identify it as a wan and triggers route-to on firewall rules.
If you ping out without setting a source and check the state table do you see a state created? On which interface?
Steve
-
Hello @stephenw10 ,
Thank you for your answer.
I confirm the gateway is set on WAN interface, and it is also set as default gateway.
When pinging out to ie.
8.8.8.8, I confirm there is a state in WAN interface, and it seems it is trying to be sent as0.0.0.0. That is odd.WAN icmp 0.0.0.0:1836 -> 8.8.8.8:1836 0:0 84 / 0 7 KiB / 0 BNow the question would be why WAN is using
0.0.0.0for egress traffic if no source IP is selected. WAN currently has a few IP addresses, which are (redacted to random IPs):
123.123.123.12/25, which is the one assigned by the datacenter
80.80.80.1/32, which is from a prefix I am announcing from this server, so it is pingable. This is a Virtual IP.
90.90.90.1/32, which is from another prefix I am announcing from this server, so it is pingable. This is a Virtual IP.I remember there was a bug where if a WAN interface had Virtual IPs assigned, it could break things. But I think that was fixed?
Thanks a lot!
-
Yes, that should be fixed but I agree that's exactly what came to mind when seeing that.
I assume you can ping out from any of VIPs on WAN if you set them as source?
Do you have Outbound NAT set to automatic still? That should prevent it NATing it's own traffic but if it's using manual more something might be over-matching.
Steve
-
Thank you for your reply.
NAT Outbound is set as manual, and it has nothing, other than the rules that come in pfSense by default, which are:
WAN 127.0.0.0/8 * * 500 (ISAKMP) WAN address * WAN 127.0.0.0/8 * * * WAN address * WAN ::1/128 * * 500 (ISAKMP) WAN address * WAN ::1/128 * * * WAN address *I confirm I can ping from both VIPs (as well as WAN, when WAN is selected) from both GUI (selecting the relevant virtual IP or interface) or CLI (specifying the source with
ping -S <SRC_IP> <DST_IP>)Thanks a lot!
-
I have now deleted both VIPs, so WAN only has one allocated IP (in case it was the Virtual IPs bug I was referring to), but the problem still persists. I have also set NAT Outbound to Automatic to no avail, so I just rolled it back to Manual.
-
Hmm, if you run a pcap on WAN is it actually sending traffic with source 0.0.0.0?
-
That is correct. This is the excerpt of the Packet Capture:
17:29:56.826861 IP 0.0.0.0 > 8.8.8.8: ICMP echo request, id 1836, seq 4001, length 64I am attaching the exported pcap for your reference, with the corresponding ping packets to 8.8.8.8. capture.pcap
-
If you resave the WAN config does it stop?
Was the WAN ever configured as DHCP? Is there a dhclient process running? Anything in the dhcp logs?
That's the only time seeing 0.0.0.0 might be expected.Steve
-
I have re-saved WAN interface configuration, and applied it. Problem still persists, unfortunately.
DHCP logs show one entry from yesterday (
2023-10-25 18:18:20.595596+00:00 dhclient 5829 Cannot open or create pidfile: No such file or directory), and a few entries for the date when the server was deployed, on 20th of October.There seems to be a dhclient process running, which keeps changing the PID (second column) incrementally. Maybe it is restarting all the time?
$ ps aux | grep dhclient root 40010 0.0 0.0 12768 2424 0 S+ 17:50 0:00.00 grep dhclient $ ps aux | grep dhclient root 40711 0.0 0.0 12768 2432 0 S+ 17:50 0:00.00 grep dhclient $ ps aux | grep dhclient root 41154 0.0 0.0 12768 2432 0 S+ 17:50 0:00.00 grep dhclient $ ps aux | grep dhclient root 41588 0.0 0.0 12768 2432 0 S+ 17:50 0:00.00 grep dhclientThanks a lot!
-
Hmm, that's not right! There shouldn;t be any dhclient there if it's all static IPs.
If you run
ps auxwwd | grep dhclientyou should be able to see which interface it's running on.I would first try just killing that process.
-
Oh sorry, my brain stopped working after chasing this for too long today. The command above shows
grep dhclientprocess which I am spinning up when runningps aux | grep dhclient, notdhclientprocess itself. That is why the PID kept incrementing, because I was invoking it all the time. Facepalm.So yes, there is no
dhclientprocess running, and therefore we cannot kill it.My apologies. I will continue the research tomorrow and try to figure out why WAN is using
0.0.0.0as SRC when no address is specified, as it is clear brain fog is kicking in. -
Ah, yes, I should have spotted that!
I guess I'd try disabling wireguard as a test. There's not much else it could be....
-
Hmm, do you have a Wireguard gateway defined?
Is your default IPv4 gateway still set to automatic in System > Routing > Gateways?
If it is try setting it to the WAN gateway. -
Hello again @stephenw10 . New day, new time to chase this mystery.
I have stopped WireGuard service, but issue still persists.
In regards to IPv4 gateway, it is not automatic - it is statically defined. It is the only Gateway in this system (no WireGuard gateways or anything extra), and it is also set as default. Puzzling.
I have tried to add a NAT outbound rule on top of the list, selecting "any" as source on WAN interface, and translating it to the "interface address", and ping works now when not specifying an interface. But this is a dirty patch, as I assume WAN is still sending the packets with 0.0.0.0 as source, just that we are translating that address locally. On top of that, nslookup still does not work even with that workaround, ie:
$ nslookup dns.google ;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable. ;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable. ;; UDP setup with 1.1.1.2#53(1.1.1.2) for dns.google failed: host unreachable. ;; UDP setup with 9.9.9.9#53(9.9.9.9) for dns.google failed: host unreachable.And pfsense updates also don't work. So I am reverting that NAT outbound rule.
I will continue to chase it for a few hours more and, if no luck, I will try reinstalling the system and observe behaviour when freshly installed and no config.xml restores.
-
Here is some additional info, in case it helps
$ ifconfig -a igb0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 3c:ec:ef:a7:22:e8 media: Ethernet autoselect status: no carrier nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 3c:ec:ef:a7:22:e9 media: Ethernet autoselect status: no carrier nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NOMAP> ether 9c:69:b4:64:96:7a inet6 fe80::9e69:b4ff:fe64:967a%ix0 prefixlen 64 scopeid 0x3 inet 123.123.123.123(redacted) netmask 0xffffff80 broadcast 123.123.123.1(redacted) media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: OPT1 options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NOMAP> ether 9c:69:b4:64:96:7b inet6 fe80::9e69:b4ff:fe64:967b%ix1 prefixlen 64 scopeid 0x4 media: Ethernet autoselect status: no carrier nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> enc0: flags=0<> metric 0 mtu 1536 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=100<PROMISC> metric 0 mtu 33152 groups: pflog pfsync0: flags=0<> metric 0 mtu 1500 maxupd: 128 defer: off syncok: 1 groups: pfsync tun_wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420 description: WG_EG options=80000<LINKSTATE> inet 80.80.80.1(redacted) netmask 0xffffff00 inet 90.90.90.1(redacted) netmask 0xffffff00 inet 10.95.32.99 netmask 0xffffff00 groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD>I also verified that in the uploaded .pcap file, the MAC address of the source matches the one in ix0 interface.
-
@llavecita said in Ping not working on CLI or with "default" interface on GUI:
inet 123.123.123.123(redacted) netmask 0xffffff80 broadcast 123.123.123.1(redacted)
Is that the other way around? I expect the broadcast address to be .127 on a /25.
-
@stephenw10 I redacted the real values, but you are indeed correct - shown broadcast address in ifconfig for interface ix0 is a .127 (and gateway is a .126)
-
Hmm, OK. We discussed this internally last night and there's nothing immediately that presents like that.
Can I assume this behaviour survives a reboot?
-
@stephenw10 Correct, the server has been rebooted several times to no avail. I will find some time today to reinstall it from scratch, without dragging any previous config.xml, to see if behaviour persists on a fresh install. I will report back later today.
Thanks again for your time and dedication!
