pFsense not responding to multicast arp whohas

Ofloo

I'm using pfsense 2.7.0 latest version, today I noticed strange behavior. I have a FreeBSD bhyve VM which, static IPv6 on the pFsense interface. The IPv6 addresses are assigned through SLAAC. It gets an IPv6 other clients on the same bridge can ping the gateway/static IPv6 on the interface on pFsense.

When I tcpdump the traffic I see that the client is asking through multicast who has the IPv6 this also arrives on the pFsense interface, however no responds as the ping timeouts keep continuing. But for some reason whohas gets no answer. It does work for all the other clients.

When I in turn then ping from pFsense too that very same clients I also get some ping timeouts but eventually after loosing a few packets everything works. This to me doesn't make any sense. Can anyone explain. The issue is clear it's pFsense that isn't doing what it is supposed to be doing it's not responding to the ARP requests ! Once the ARP requests got through network traffic worked.

Clearly pFsense does not respond to whohas client does and once arp table is filled on the client side traffic flows ! So the client does respond while pFsense does not.

It is vlan traffic though. But shouldn't matter one bit especially that it does work for all the other clients on that same bridge.

No bridge filters are disabled as well, checked tcpdump on the pFsense side and only who-has arrived nothing left the interface to respond.

net.link.bridge.pfil_local_phys: 0 -> 0
net.link.bridge.pfil_member: 0 -> 0
net.link.bridge.pfil_bridge: 0 -> 0
net.link.bridge.pfil_onlyip: 0 -> 0

But why is this? Or what could the cause be !?

PS: it wasn't being filtered, tried adding a allow any rule to the top did not matter also all my filter deny rules have log flag enabled none where triggered.

JKnott

@Ofloo

There's no such thing as arp on IPv6. When a device needs the MAC address, it sends a Neighbor Solicit and expects a reply. Do you see the reply? Is the device being pinged capable of responding?

BTW, pfSense has nothing to do with communications between devices on the same LAN.

Ofloo

no there's no reply

ifconfig lagg0.100
lagg0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: PUBLIC
	options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
	ether 0c:c4:7a:bd:29:84
	inet6 fe80::ec4:7aff:febd:2984%lagg0.100 prefixlen 64 scopeid 0xe
	inet6 fc03:dead:1eaf:cafe::1 prefixlen 64
	inet 10.13.17.1 netmask 0xffffff00 broadcast 10.13.17.255
	inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1
	groups: vlan
	vlan: 100 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

tcpdump -ni lagg0.100 host fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0.100, link-type EN10MB (Ethernet), capture size 262144 bytes
17:40:27.837061 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:28.832493 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:29.918182 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:30.921650 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:31.912520 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:32.950058 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
17:40:33.971711 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32

JKnott

@Ofloo said in pFsense not responding to multicast arp whohas:

no there's no reply

Again, if the devices are on the same LAN, as they appears to be, pfSense has nothing to do with it. Things like neighbor solicitation, just like arp on IPv4 work only on the local subnet and do not pass through pfSense.

Ofloo

@JKnott it's not because it doesn't pass through pfsense that pfsense can't have like a bad network driver ! Then the issue is still a pfsense issue. This might seem hard to grasp but pfsense is the package not just the pf firewall !

And if even that would be the case then pf wouldn't really be a pfsense thing they if you talk development they just package a gui around FreeBSD with pf firewall ! But still they package the os and drivers and so forth. So pFsense isn't just a firewall because that's not even what they develop. They develop a gui, yet on their site they claim to develop a firewall. Which is my point exactly it's the package.

JKnott

@Ofloo

You could completely remove pfSense from the LAN and your problem would very likely still be there. For traffic entirely on the local LAN, the entire responsibility of pfSense is the DNS & DHCP servers, along with SLAAC. It will not block arp or neighbor solicit.

johnpoz

@Ofloo said in pFsense not responding to multicast arp whohas:

who has fc03:dead:1eaf:cafe::1

So this is pfsense address? And your saying its not answering?

Or are you saying the traffic is not flowing through a bridge on pfsense to some other client that has that address?

Ofloo

@johnpoz the client is bhyve bridged, however it reaches the interface on pFsense, which is a vlan on a lagg0 and that isn't answering. Maybe shouldn't of mentioned the bridge as it only confuses things. pFsense has no bridge it's LACP lagg0.100 with vlan. And it has the IPv6 on that interface but it's not answering. The call of who has.

johnpoz

@Ofloo said in pFsense not responding to multicast arp whohas:

Maybe shouldn't of mentioned the bridge as it only confuses

No you clearly should and need to "mention" it..So pfsense has a leg in this bridge you setup on bhyve? I have no direct experience with that hypervisor - but plenty of other ones, proxmox, esxi, hyper-v, virtualbox, etc.

But have seen in the past things on setup in a virtual switch/port/bridge having issues talking to each other, etc.

A drawing exactly have setup and you sniffed on pfsense and saw the discovery? To a multicast address - its possible rules on pfsense are not letting that through?

Ofloo

@johnpoz

Pfsense receives the traffic but doesn't respond to who-has ofcourse it also has a pseudo vlan interface lagg0.100 but is not bridged ! That's what I thought as well but pfctl -d doesn't improve replying, the only thing that did work at some point but not always is that I ping from pfsense to lets say server 1 once the ndp table is filled all traffic flows. reboot the vm ndp table is empty again and same store all over.

johnpoz

@Ofloo is pfsense physical - from that drawing can't tell if physical or virtual.

When pfsense pings something.. And then you can ping pfsense back, what source IP is it pinging from?

I tired to duplicate - I don't have a lagg setup.. Set same ULA address as you, and not seeing any problems it answering neighbor solicitation

If me I would remove the lagg from the equation.. Does it work then?

Ofloo

@johnpoz network interfaces lagg0 are real cards no virtual cards

jimp

Could be this:

https://redmine.pfsense.org/issues/13423

In which case there won't be a fix until the next release since it requires a change in the kernel.

Ofloo

@jimp it could be that issue not sure

21:03:12.076167 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:13.100224 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:14.096234 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:15.121082 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:16.116240 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:17.117124 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:18.198316 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32
21:03:19.196136 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fc03:dead:1eaf:cafe::1, length 32

This is what i see so no responds from pfsense, this looks exactly the same as the bug you referring too

now when i ping from pfsense to the server/client

ping6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae
PING6(56=40+8+8 bytes) fc03:dead:1eaf:cafe::1 --> fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=0 hlim=64 time=0.565 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=1 hlim=64 time=0.198 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=2 hlim=64 time=0.222 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=3 hlim=64 time=0.186 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=4 hlim=64 time=0.227 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=5 hlim=64 time=0.198 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=6 hlim=64 time=0.196 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=7 hlim=64 time=0.212 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=8 hlim=64 time=0.213 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=9 hlim=64 time=0.220 ms
16 bytes from fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae, icmp_seq=10 hlim=64 time=0.222 ms

and after that !!! it works visa versa

21:06:38.446559 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > fc03:dead:1eaf:cafe::1: ICMP6, echo request, seq 0, length 16
21:06:38.446628 IP6 fc03:dead:1eaf:cafe::1 > fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae: ICMP6, echo reply, seq 0, length 16
21:06:39.455884 IP6 fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae > fc03:dead:1eaf:cafe::1: ICMP6, echo request, seq 1, length 16
21:06:39.455913 IP6 fc03:dead:1eaf:cafe::1 > fc03:dead:1eaf:cafe:5a9c:fcff:fe07:6cae: ICMP6, echo reply, seq 1, length 16

jimp

Sounds like that issue, then.

We'll have a fix for that in Plus 23.09. It's working well there now for us on lab systems that hit the problem before.

No ETA on whenever the next CE release might be.

Ofloo

@jimp I understand, that this is the case however I think it's sad that CE is neglected in this way.