Solved!

My ISP digged deep into this and like I thought it was a routing issue on their side!
I moved to another city last year they didn't changed my public fixed IP addresses. Once they changed my IPv6 /56 it all worked.

TL:DR IPv6 routing issue on ISP side.

Tks for your support

@wishyou

Good. When I started with pfSense, that option wasn't available, so my prefix changed on occasion.

@IsaacFL said in Multiple IPv6 capable connections:

/etc/inc/interfaces.inc

It looks as if fe80::1:1 gets statically enforced. So changing the 2nd box might work to see whether there are other problems. The OPNsense code is different here, but I haven't read all relevant interface files so far.

@netblues

Sorry for the heavy handed smudging, wanted to be sure I was t posting unnecessary details re MAC or private addresses, I've tried to be more selective in this response.

Heres the diagnostics that led me to think its something to do with the Ipv6 tunnel to AirVPN.

From my local subnet my local PC gets a IPv4 and IPv6 address

With the egress gateway set to default I can a IP test site ping over both IPv4 and IPv6

% ping -c 3 ifconfig.co PING ifconfig.co (104.28.18.94): 56 data bytes 64 bytes from 104.28.18.94: icmp_seq=0 ttl=54 time=508.991 ms 64 bytes from 104.28.18.94: icmp_seq=1 ttl=54 time=47.812 ms 64 bytes from 104.28.18.94: icmp_seq=2 ttl=54 time=77.452 ms % ping6 -c 3 ifconfig.co PING6(56=40+8+8 bytes) 2605:e000:xxxx:xxxx:9051:ad0b:d360:b654 --> 2606:4700:3032::681c:125e 16 bytes from 2606:4700:3032::681c:125e, icmp_seq=0 hlim=56 time=88.167 ms 16 bytes from 2606:4700:3032::681c:125e, icmp_seq=1 hlim=56 time=92.328 ms 16 bytes from 2606:4700:3032::681c:125e, icmp_seq=2 hlim=56 time=127.620 ms

I can also get an IP address back from curl'ing the site over both IPv4 and IPv6 so I think can correctly conclude my basic DNS, routing and transport is working correctly over the default non VPN gateway.

% curl ifconfig.co 199.249.223.130 % curl -6 ifconfig.co 2605:e000:xxxx:xxxx:9051:ad0b:d360:b654

If I change my gateway to VPN_WAN_V6 for ICMP and TCP/UDP both pings and curl stop functioning. They just hang.

ping6 ifconfig.co PING6(56=40+8+8 bytes) 2605:e000:xxx:xxx:9051:ad0b:d360:b654 --> 2606:4700:3034::681c:135e ^C % curl -6 ifconfig.co ^C

I'm not sure this is useful, but heres the ifconfig of the openvpn interface

ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet6 fe80::ae1f:6bff:fe73:87e0%ovpnc1 prefixlen 64 scopeid 0x1c inet6 fde6:7a:7d20:5a2::1001 prefixlen 64 inet 10.9.162.3 --> 10.9.162.1 netmask 0xffffff00 groups: tun openvpn nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 84260

I'm sure this is a newbie IPv6 user error, theres something I'm not understanding clearly like a possible need to do some address translation for IPv6 traffic egressing over a IPv6 link established in a IPv4 tunnel?

thanks for reading and any suggestions.

@JKnott Yes we have build an instance of pfSense in AWS and firewall been working well with IPv4. Recently we have a project where the equipment we need to remotely manage are IPv6 only. Have turned IPv6 and all seem in place but I can see Firewall blocking the traffic from these devices. WAN and LAN have IPv6 assigned and I can see them in pfSense. Even LAN traffic deosn't seem to be working on IPv6, for example I can ping IPv4 address but not IPv6

I think the solution is true: need more than a /64

Reading about NDP on Wikipedia made some sense and I managed to find a document on RIPE.NET that explained about the importance of being a /64 or more..

I consider my question answered :(

@anthonys

Glad to hear it. Yeah, ISPs sometimes cause their own problems, as the staff doesn't fully understand the differences between IPv4 and IPv6. When I had that problem last year, I had to educate both tier 2 support and the senior tech about what was actually happening. They knew the basics of IPv6, but not some of the finer points. At least you got relatively quick response from your ISP on this. It took me 3 months and a lot of work to get the people who should have fixed the problem to do anything. Since I had my own router, they refused to do anything, even though both the tier 2 guy and senior tech told them it was an ISP problem. What finally did the trick is the senior tech brought his own modem to my home and saw the problem. He then went to the head end and tried with 4 different CMTS. The failure only occurred with the one I was connected to. This was weeks after I provided the error (see above) to them.

BTW, I have decades of experience with telecommunications, computers and networks and so had the ability to work through this problem. A regular customer wouldn't have a hope of getting it resolved.

@IsaacFL

Yes, I know the RA has both. People have to get away from the IPv4 way of thinking. There are essentially unlimited addresses available. You can have multiple addresses on an interface. In my case, I have link local, GUA and ULA. I could even have multiple GUA & ULA if I wished. Sometimes you just want a local network for some devices that share the same network as the devices that connect to the Internet. As mentioned, there is an issue with pfSense where it forgets to apply the GUA prefix, when ULA is also used. As far as I'm concerned, that's a bug.

making some progress with my learning :)
I can create a ULA for the interface and use that in the listen field too.
This feels better, the GLA is used purely for external traffic, and the ULA internally for IPv6 lookups.

still open for feedback if I'm being crazy/stoopid here. It wouldn't be the first time! :-D

Found the issue and all is working well.

If anyone else runs into the same problem I did here is the fix.

If you are running HA units and your creating a wan ipv6 carp address you must leave leading 0's on. So you cant take the leading 0's off to shorten the address. :0001 cannot be :1 shortening works fine everywhere else but with the carp IP for some reason you cannot do this. I found a thread from 4 years ago on redmine that was very similar to this issue and there was some activity on it from a few weeks ago so I'm wondering if the issue has resurfaced. Either way I'm glad i got it working now :)

If anyone from netgate sees this I am running the current stable version as of today. 5/15/20

I see that there are no clear instructions for this scenario. As I like experiments I'll try to find it out just by trying to do this. Of curse it will help if somebody could tell that there is no way to configure pfSense behind NAT to properly handle IPv6 with tunnel broker (HE)

Just for quick summary this is my setup:
Internet <--> ISP modem (as bridge) <--> pfSense on real device (main router with public IP) <--> pfSense on virtual box (test router) <--> virtual box test network

And as mentioned in first post I plan to learn by configuring IPv6 on "test router".

@JKnott thanks, I might try that to experiment. However, it seems this has been verified and input as a bug on this thread.

Hoping maybe the Netgate folks get to it in a future release... properly getting track interface to work with multiple IP addresses on a LAN interface including GUA and ULA. Definitely some funky routing and "which interface gets priority or sends the traffic and can route" going on... both on the pfSense side (which they can control), as well as the various client OS's (Windows, Mac, Linux, etc). All of them do it differently. Windows machines here always seem to ping everything just fine... Mac's not so much.

If anyone finds a fix / workaround (possibly a script to pull and add the ULA VIP after 5-10 seconds whenever the WAN goes up/down)... let me know and I'd be happy to test it.

Best Regards,

dg6464

@JKnott I misunderstood that I needed an extra interface to tie the VIP to. But I see I can just create one and tie it to the WAN interface. I just confirmed this works as expected.

Hello, and sorry for the delay.

I reverted to using my ISP native IPv6 and fixed the horrible stability issues by settings LAN's MSS to 1440 after reading https://forum.netgate.com/topic/73573/massive-http-ipv6-connectivity-issues and fiddling with values.

I kept using NPt, and used /64 on all interfaces.

All seem perfect now, including multi-wan load balancing. Thank you for your help.

@JKnott

Good advice, thank you :-)

I bought a copy today.