[Solved] Mysterious firewall rule or feature blocking outgoing LAN traffic?

[lifeboy]

I have a Virtual Machine on a proxmox cluster that was working perfectly for quite some time, until a restart recently. The machine is NATted via pfSense.

The rules are simple. Outgoing traffic from all the LAN addresses / VM Guests is allowed. The LAN is 192.168.121.0/24.
I see the traffic existing the firewall when I inspect the firewall rules log, but the guest is not able to reach any ip address past the firewall default gateway.
For example: I should be able to ping 8.8.8.8 and I actually can from other guests. From this particular guest, which runs Debian 12, I cannot. The guest is on 192.168.121.201.

# ping 197.214.117.194
PING 197.214.117.194 (197.214.117.194) 56(84) bytes of data.
64 bytes from 197.214.117.194: icmp_seq=1 ttl=64 time=0.228 ms
64 bytes from 197.214.117.194: icmp_seq=2 ttl=64 time=0.295 ms
^C
--- 197.214.117.194 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2044ms
rtt min/avg/max/mdev = 0.228/0.284/0.330/0.042 ms
# ping 197.214.117.193
PING 197.214.117.193 (197.214.117.193) 56(84) bytes of data.
^C
--- 197.214.117.193 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3059ms

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3067ms

So the traffic gets to the gateway, but not to the next hop.

I have searched high and low in the settings to see what it may be that's blocking the traffic (which was being passed fine not too long ago), but can't find it.

Anyone have an idea of where to look?

[viragomann]

@lifeboy said in Mysterious firewall rule or feature blocking outgoing LAN traffic?:

The LAN is 192.168.121.0/24.
For example: I should be able to ping 8.8.8.8 and I actually can from other guests. From this particular guest, which runs Debian 12, I cannot. The guest is on 192.168.121.201.

Ensure that the outbound NAT rule covers the whole LAN subnet.

To further investigate sniff the traffic on the WAN interface.
For instance, when you ping 8.8.8.8 from the concerned VM, set 8.8.8.8 as host filter and ICMP as protocol filter.
Ensure that 8.8.8.8 is not used for gateway monitoring. If so, disable the monitoring for the tests.

[lifeboy]

@viragomann I have discovered that as soon as I remove the 1:1 NAT mapping, it all works. So the specific public ip address is probably being blocked for some reason by the upstream router.