Cannot ping client booted after Pfsense

[GXR41455]

Heya, I have a bit of troubles.

I installed pfsense on proxmox to isolate all the vm:

I followed the guide there:
https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html

For some reason I can't find, pfsense is unable to ping client on the internal side if they booted after itself.

The client can ping pfsense internal side without trouble.

If I reboot pfsense from the console, it can ping the client.
Of course when pfsense cannot ping the client, they have no internet/can't ping the gateway/can't ping pfsense's external.

Tried:
Reinstalled pfsense a couple of time.
Tried different clients (VM centos, VM kali, raspberry), they all exhibit the same problem.
Tried OVS bridge, same problem. Pfense cannot ping client booted after itself.
Proxmox firewall is disabled at VM side / node side.
Tried different network setup on proxmox.
It's with the default rules.
Client firewall is down.

Hardware checksum offload is disabled as recommended by netgate.

Proxmox settings:

VM settings:

Pfsense settings:

Packet sniffer (promiscuous mode) from pfsense webui:
Same method for both. ping client from pfsense, ping pfsense from client

Working:
11:52:02.701412 ARP, Request who-has 192.168.3.16 tell 192.168.3.1, length 28
11:52:02.702078 ARP, Reply 192.168.3.16 is-at 96:e2:04:09:eb:e6, length 28
11:52:02.702116 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 21384, seq 0, length 64
11:52:02.702752 IP 192.168.3.16 > 192.168.3.1: ICMP echo reply, id 21384, seq 0, length 64
11:52:03.705230 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 21384, seq 1, length 64
11:52:03.705854 IP 192.168.3.16 > 192.168.3.1: ICMP echo reply, id 21384, seq 1, length 64
11:52:04.707248 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 21384, seq 2, length 64
11:52:04.707890 IP 192.168.3.16 > 192.168.3.1: ICMP echo reply, id 21384, seq 2, length 64
11:52:07.872480 ARP, Request who-has 192.168.3.1 tell 192.168.3.16, length 28
11:52:07.872521 ARP, Reply 192.168.3.1 is-at e2:04:94:55:9d:3f, length 28
11:52:23.043570 IP 192.168.3.16 > 192.168.3.1: ICMP echo request, id 1989, seq 1, length 64
11:52:23.043658 IP 192.168.3.1 > 192.168.3.16: ICMP echo reply, id 1989, seq 1, length 64
11:52:24.064738 IP 192.168.3.16 > 192.168.3.1: ICMP echo request, id 1989, seq 2, length 64
11:52:24.064790 IP 192.168.3.1 > 192.168.3.16: ICMP echo reply, id 1989, seq 2, length 64
11:52:25.088751 IP 192.168.3.16 > 192.168.3.1: ICMP echo request, id 1989, seq 3, length 64
11:52:25.088802 IP 192.168.3.1 > 192.168.3.16: ICMP echo reply, id 1989, seq 3, length 64

Non-working:
11:53:14.857722 IP6 fe80::94e2:4ff:fe09:ebe6 > ff02::2: ICMP6, router solicitation, length 8
11:53:25.276746 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 2106, seq 0, length 64
11:53:26.302230 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 2106, seq 1, length 64
11:53:27.324854 IP 192.168.3.1 > 192.168.3.16: ICMP echo request, id 2106, seq 2, length 64

Interestingly, on the non-working one, the client > pfense successful ping does not appear in the packet sniffer.
The main difference I see is the non-working one doing a ICMP6 router solicitation versus the working one doing an ARP request.

WAN/LAN having no ipv6 could be a problem but shouldn't Pfsense having none would make it default to ipv4?