IPV6 Neighbor Solicitation Not answered !?? => No IPV6 :(

louis2

Hello,

I have VLAN's hosted by pfSense (2.6) towards jails form another FreeBSD system (TrueNas-core).

And I noticed that the communication was terribly slow, what I did not at all understood. So I started packet capture on pfSense for the involved VLAN.

The capture shows some thing quite unexpected. The Jail application is trying to use IPV6 a couple of times before it switches to IPV4.
However before the IPV6-communication can start, the sender needs to know the physical frame address of the gateway. So the Jail-communication layers sends a Neighbor Solicitation (135).

Message ^Neighbor Solicitation for "a;b;c:100;;1" from "6e;b3;11;42;2f;91", where "a;b;c;100;;1" is the pfSense gateway address for that VLAN ("a;b;c;100;;1"/64)

And the problem is that the pfSense gateway is not answering that request !! Without that pfSense answer .... there is no IPV6 communication ....

Note that:

the jail IP-addresses are statically defined in the jail setup and that I am not using DHCP for this vlan, so that functionality is disabled on pfSense.
the vlan starts at a lagg at pfSense and is arriving via two managed switches on a vlan-trunk at the TrueNas-core side.

Is there an explanation for this / what am I doing wrong?
Or is there perhaps? a bug somewhere?

JKnott

@louis2

If that jail is trying to reach a valid address on pfsense, then a response should be sent. Are the addresses correct?

louis2

@jknott

As far as I can see, and I double checked, the address send and the adres of pfSense are identical.

A bit more info

For this testsetup I am using vlan100. vlan100 is on truenas-core member of bridge100.
Neither the bridge nor the vlan have IP-addresses assigned.

vnet0;bridge100
jail-ip4: vnet0 ipv4-address / 24
jail-ip6: vnet0 ipv6-address / 64

<my slash48 range>;100;;1/64 pfSense GW
<my slash48 range>;100;;1/64 Jail default router
<my slash48 range>;100;;1;66/64 Jail IPV6

Yesterday, one of the things I did is trying to send pings from the jail shell ^ping6 www.google.com".^
That did not work, up to the moment I did start a ping ^the other way around^
from my windows pc ^ping -6 <the jail ipv6>^

I could repeat that today ;)

As stated in my previous post, the package capture on pfSense vlan100 interface clearly shows that the incoming Neighbor Solicitation
and not the answer ......

louis2

I still do not understand why this is not working. Below two pieces of wireshark traces:

the first piece contains part of the solicitation message as started from the TrueNas jail (not answered by pfSense
the second piece part of a solicitation messages started by pFsense. (answered by the windows PC).

Does NOT work Neighbor Solicitation TrueNas (Jail) => pfSense
Ethernet II, Src; 6e;b3;11;42;2f;91 (6e;b3;11;42;2f;91), Dst; IPv6mcast_ff;00;00;01 (33;33;ff;00;00;01)
Destination; IPv6mcast_ff;00;00;01 (33;33;ff;00;00;01)
Source; 6e;b3;11;42;2f;91 (6e;b3;11;42;2f;91)
Type; IPv6 (0x86dd)
Internet Protocol Version 6, Src; A;B;C;100;;1;66, Dst; ff02;;1;ff00;1

Does work Neighbor Solicitation pfSense => Windows PC
Ethernet II, Src; Shenzhen_09;07;48 (6c;b3;11;09;07;48), Dst; Shenzhen_13;29;76 (6c;b3;11;13;29;76)
Destination; Shenzhen_13;29;76 (6c;b3;11;13;29;76)
Source; Shenzhen_09;07;48 (6c;b3;11;09;07;48)
Type; IPv6 (0x86dd)
Internet Protocol Version 6, Src; 2001;984;a874;17;;1, Dst; A;B;C;17;1421;6200;99c7;fb55

I also noticed that there are a couple of earlier issues raised related to Neighbor Solicitation, so I wonder if there is perhaps a bug ......

Not sure of cause since I simply do not understand why this is not working. I just regard it "very strange"

louis2

Hello,

I am still fighting this problem. Still NO ipv6 possible.

Let me explain the problem again.

I have a TrueNAS (core) system connected to pfSense via multiple VLAN's related to multiple jails on TrueNAS core. TrueNAS "host" is also connected via a VLAN.

On TrueNAS I use fixed IPV4 and IPV6 addresses for every thing. No problems with IPV4, however IPV6 does not work at all, at least not if it is initiated from the TrueNAS side.

The reason for that is a failing Neighbor Solicitation with as consequence that TrueNAS does not know how to forward the L2-package to pfSense.

What I would expect is that

pfSense would respond on the NS — Neighbor Solicitation (ICMPv6 type 135) as send by TrueNAS
and next to that pfSens would periodically send RA — Router Advertisement (ICMPv6 type 134)

Both do NOT happen !!!

not if DHCP6 is off and RA is off (my default for a server vlan) but also not
when enabling DHCP6 and/or RA

I am really completely lost! So please help if you have the required knowledge, to solve the problem or to identity the bug

Louis

More info related to the Neighbor Solicitation can e.g. be found on
https://blogs.infoblox.com/ipv6-coe/disabling-ipv6-router-advertisements-in-the-data-center/
https://blog.apnic.net/2019/10/18/how-to-ipv6-neighbor-discovery/
RS — Router Solicitation (ICMPv6 type 133)
RA — Router Advertisement (ICMPv6 type 134)
NS — Neighbor Solicitation (ICMPv6 type 135)
NA — Neighbor Advertisement (ICMPv6 type 136)

Some info related to pfSense behavoir on
https://docs.netgate.com/pfsense/en/latest/services/dhcp/ipv6-ra.html

johnpoz

@louis2 said in IPV6 Neighbor Solicitation Not answered !?? => No IPV6 :(:

pfSense would respond on the NS — Neighbor Solicitation (ICMPv6 type 135) as send by TrueNAS

And does pfsense see this solicitation? Your sniffing on pfsense and see this NS and your saying pfsense is just not answering?

simple sniff on one of pfsense vlan interfaces where I have ipv6 enabled - and looks normal to me.

louis2

@johnpoz

Yep I am using pfSense its capture function, since it is the simple way to do that.

And yep I absolutely see no reaction form pfSense and also not a RA.
{At least not as long I do not try to access TrueNAS via the network (the opposite direction).}

The simple test I use to check is to start a ^ping6 www.google.com^ on the command line of either TrueNAS-host or one of the jails

Here a screenshot of the pfSense screen and also more detail from a downloaded capture using wireshark

NeighborSolicitationProblem.txt

johnpoz

@louis2 what are the firewall rules on this interface?

louis2

@johnpoz

It all ready was a very open rule-set (just for testing at the moment), above which I just added a line ^permit all^. Which did not change any thing

For further info, initially I was testing with DHCP and RA off. However because I could not and can not get things working, today I did expiriment with dhcp on and of and all kind of RA settings ....... nothing worked .... so far

Also note that I am using pfSense 2.6.0 development release (already for months, in fact without any recent issues.

johnpoz

@louis2 said in IPV6 Neighbor Solicitation Not answered !?? => No IPV6 :(:

I am using pfSense 2.6.0 development release (already for months, in fact without any recent issues.

This really should be brought up in the dev section then with specific version your running - quite possible some sort of issue was introduced. I will move this to the dev section. If you could post the snapshot your running prob be helpful

louis2

@johnpoz

John two things:

I did create a jira ticket yesterday expressing my concern, asking jim to think about it (I could just express, my doubts without a hard proof that it is a pfSense issue). Jim closed it intermediately. Please have a look Issue #12663
I do not understand what I should additionally post if you could clarify

johnpoz

@louis2 said in IPV6 Neighbor Solicitation Not answered !?? => No IPV6 :(:

12663

While as Jim mentioned there - redmine is for reporting actual issue, and they should be able to be duplicated or multiple users seeing the issue.. Redmine is not for discussion or troubleshooting a problem.

Now if there is something discovered, and other users are seeing the same problem, etc. Then you could report what the issue is in redmine for correction, etc.

what I should additionally post if you could clarify

What specific snapshot are you running? Did this work on snapshot X, and stop working on Y.. etc.. Normally snapshots come out every day, what day version are you running. Should show those details right on the systeminfo widget on pfsense gui.

louis2

@johnpoz

John, I do understand jim ... to a certain extend ..... however IMHO it is serious enough to check .. What ever

Related to the build:

I am refreshing my system to the latest build every couple of days (lets say a week).
I am currently running
2.6.0-BETA (amd64)
built on Mon Jan 03 06:18:19 UTC 2022
FreeBSD 12.3-STABLE
the problem was already there when I did my first new truenas system trails with IPV6 and I do not now the exact date, however I was already desperate when I started this "post" a month ago. So the first time I became aware of the problem should be late November early December or so

johnpoz

@louis2 I haven't noticed any other reports of such an issue.

If you take truenas and your jails out of the equation are you seeing the same problem.. So your saying IPv6 isn't workable on pfsense 2.6.. I find that hard to believe, or you would think there would be multiple threads/posts about it not working.

IPv6 is pretty hard to work if RAs don't function, or client can not discover pfsense via NDP..

louis2

@johnpoz

John, In first instance I mainly verdict TrueNas-core which is based on FreeBSD 12.3 just like pfSense. However I was not sure as well, so I opened this blog.

However since I had severe issues with TrueNAS-core, I did decide to try TrueNAS scale with is their linux based release candidate (I think it is not jet RC but that apart). And up to my surprise the IPV6 behavoir or that linux version was identical to the FreeBSD version. And that did my doubts related to pfSense increase ! .... and I started again testing with all kind of dhcp and RA settings on pfSense.
When that did not solve any thing, I asked for help here.

Do note that:

the problem only occurs when the server in this case TrueNAS initiates the communication. (The other way around works!)
I ones did a test connect my windows PC on the truenas host vlan, which seems to work (I can redo that test), however it is for sure that windows behaves different
I can not go back to pfsense 2.5 to test that, since the config files are not compatible, that would be a too big effort (and error sensitive)

louis2

@louis2

I forgot to add that

if I start a command like "ping6 www.google.com" on truenas, that only results in a lot of the unanswered NS — Neighbor Solicitation (ICMPv6 type 135) messages
however ... surprise surprise (not) ...... if I start some form of communication from the other side (ping -6 truenas from my pc) then

As expected I get ping results on my pc AND
as by magic the pings starting at truenas towards google start working as wel :)

johnpoz

@louis2 well first thing I would do is get current on your snapshot.. You are a few days behind..

louis2

@johnpoz

Done! No change

johnpoz

@louis2 I don't run dev, and for sure can not duplicate what your seeing on my network - ipv6 working just fine..

As to rolling back to 2.5 - where did you see you could not do that because of issues with xml? I was not aware that was a thing..

This seems odd?

Src: A:B:C:110::1:10, Dst: ff02::1:ff00:1

Did you edit that to obfuscate something.. A:B:C ??

louis2

@johnpoz

John the ABC is edited just to hide my address. Since I try not to share too much.

By the way, I do not mind to share full wireshark traces etc, wich you, however not via the public forum