XG-7100 IPv6 on CARP or Virtual IPs (VIP)

msc-blue

Hi *,

we do own 2 XG-7100 devices and tried to setup a /48 ipv6 prefix delegation on our device.
A small /125 subnet is part of our WAN interface.
The /48 subnet is routed to our prefix::2 and a /64 subnet is configured on our DMZ interface

If we configure the ::2 ipv6 ip to our WAN interface, ping to the device is working as expected.
If we configure the ::2 ipv6 ip as VIP or CARP to have a high available setup ping stops to work.
We do have ipv6 firewall rules like allow any ipv6 on WAN.

We did check tcpdump on console like this: tcpdump -pnni lagg0.4090 (WAN) and do not see traffic to ::2 or our /64 subnet after switching the ::2 ipv6 ip to VIP or CARP mode.

Any hints on how to configure VIP / CARP on a XG-7100 carp cluster?

Thanks in advance

junicast

You get a /48 prefix via prefix delegation and try to configure IP addresses from that range on your WAN side?
That's not how it works. There are multiple possibilities how your upstream provider assigns IP addresses. Please provide more information about that. How is your WAN side configured on both devices?

Typical scenario would be that you get a static /64 for your WAN interface. Pick one IP out of that range for every device + one for CARP. Your upstream provider may then route that /48 of yours via a specific IP within that /64 range. Make sure to use this IP as CARP IP.

JKnott

@pmisch said in XG-7100 IPv6 on CARP or Virtual IPs (VIP):

Typical scenario would be that you get a static /64 for your WAN interface.

Actually, I have a /128 on the WAN side, not a /64. On the other hand, I have a /64 link local address, which is the one that's actually used for routing. On IPv6, it's not necessary to use a routeable address on the WAN interface, as routing is normally done with the link local address. The routeable address, if assigned, is used for diagnostics, such as ping or traceroute. The router would work just as well without that address.

msc-blue

@pmisch said in XG-7100 IPv6 on CARP or Virtual IPs (VIP):

thank you for your reply @pmisch

You get a /48 prefix via prefix delegation and try to configure IP addresses from that range on your WAN side?

I will describe the situation in more details. There is no dynamical assignment provided, everything has to be configured static. no SLAAC / RA.
The pfsense devices are located in a datacenter. The hoster assigned us a /48 prefix.
The hoster told us he reserved the first /64 subnet of the /48 delegation for himself, to route the rest of our range to
prefix::2
he also assigned a /125 mask (instead of /64) to this first subnet so we do have this:
::1 uplink of the provider /125
::2 target of prefix::/48 in the provider router (this should be our CARP IP) /125
::3 pfsense #1 (WAN interface) /125
::4 pfsense #2 (WAN interface) /125

we've configured a /64 subnet on our DMZ interface of the pfsense cluster.
so prefix:1::1 is our IP of pfsense #1 (DMZ interface)

if we configure the ::2 on our pfsense #1 (WAN) interface native (without CARP) instead of the ::3 we do see traffic in tcpdump -pnni lagg0.4090 (WAN) directing to prefix:10::1 if we ping it.

if we change to the above:
::2 CARP VIP #5 (master is pfsense #1)
::3 pfsense #1 (WAN interface)
::4 pfsense #2 (WAN interface)

we do not see traffic passing for prefix:1:: any more after using ::2 as CARP VIP.

msc-blue

attached output of ifconfig on WAN and DMZ interface
0_1539983052922_ifconfig-pfsense.txt

junicast

What your provider does seems a bit uncommon but it might work with the right routing entries on their side. Actually you do see that it works as long as you don't use CARP if I get you right.
Two questions:

When you configure both devices fully besides CARP, does each node work fully. I mean are both nodes able to ping to the Internet via IPv6 and such things?
When you configure CARP, do you see any errors in the logs. What does the CARP Monitor show in the WebGUI?

It might be helpful if you attach the parts out of your config xml so we can inspect better what might be wrong. The interfaces part and CARP might be interesting. Also the routing tables would be of interest.

msc-blue

@pmisch

thanks for your reply.

yes, both nodes are working perfect with ipv6 (dns resolution ping from and to the internet), everything looking nice and smooth if carp is disabled on both nodes (by the way, enabling carp does not break this)
there are no errors in /var/log/system.log
CARP Monitor shows on the 1st device MASTER
CARP Monitor shows on the 2nd device BACKUP
the vhid 5 (in our case) shows the ::2 ipv6 address

more detailed information in att.txt 0_1540197572646_att1.txt

junicast

Can't find any errors in the config. It should work IMO. I would have to diagnose locally in order to find the reason. I would extensively tcpdump icmp6 and look at the neighbour cache (NDP). Might be there somewhere.

msc-blue

@pmisch

bad news. Hoped we did miss something in our configuration.
I don't think we can give access to the devices.

We did some more investigation on our end and we think you are right, it looks like there is something wrong with NDP

if we do ping ::2 (from the internet) we do see the following on the MASTER pfsense (via tcpump):

20:20:07.603082 54:e0:32:xx:xx:xx > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fe80::3a4f:xxxx:xxxx:xxxx > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has <prefix::2, length 32
20:20:07.603113 00:00:5e:00:01:05 > 54:e0:32:xx:xx:xx, ethertype IPv6 (0x86dd), length 86: fe80::<pfsenseMAC> > fe80::3a4f:xxxx:xxxx:xxxx ICMP6, neighbor advertisement, tgt is <prefix>:2, length 32

54:e0:32... / fe80::3a4f:xxxx:xxxx:xxxx is the mac addr / link local addr of our provider uplink router
00:00:5e:00:01:05 is the vhid (carp 5) of our <prefix>::2 virtual IP 5

if we do ping ::3 (from the internet) it looks different

20:27:01.445917 <pfsensemac> > 54:e0:32:xx:xx:xx, ethertype IPv6 (0x86dd), length 78: <prefix>::3 > <prefix>::1: ICMP6, neighbor advertisement, tgt is <prefix>::3, length 24
20:27:05.337187 <pfsensemac> > 54:e0:32:xx:xx:xx, ethertype IPv6 (0x86dd), length 86: <prefix>::3 > <prefix>::1: ICMP6, neighbor solicitation, who has <prefix>::1, length 32

it looks like ND on the interface ipv6 addr of our pfsense is responded with the <prefix>::3 addr.
ND on the carp ipv6 addr is responded with the link local addr of our pfsense

ndp -an looks like this

: ndp -an
Neighbor                             Linklayer Address  Netif Expire    S Flags
<prefix>:1::1                     00:08:a2:xx:xx:xx lagg0.4094 permanent R 
fe80::208:xxxx:xxxx:xxxx%lagg0.4094  00:08:a2:xx:xx:xx lagg0.4094 permanent R 
fe80::208:xxxx:xxxx:xxxx%lagg0.4093  00:08:a2:xx:xx:xx lagg0.4093 permanent R 
fe80::208:xxxx:xxxx:xxxx%lagg0.4092  00:08:a2:xx:xx:xx lagg0.4092 permanent R 
fe80::208:xxxx:xxxx:xxxx%lagg0.4091  00:08:a2:xx:xx:xx lagg0.4091 permanent R 
<prefix>::1                       54:e0:32:xx:xx:xx lagg0.4090 23h59m56s S R
<prefix>::2                       00:08:a2:xx:xx:xx lagg0.4090 permanent R 
fe80::208:xxxx:xxxx:xxxx%lagg0.4090  00:08:a2:xx:xx:xx lagg0.4090 permanent R 
<prefix>::3                       00:08:a2:xx:xx:xx lagg0.4090 permanent R 
fe80::3a4f:xxxx:xxxx:xxxx%lagg0.4090  54:e0:32:xx:xx:xx lagg0.4090 1d0h0m0s  S 
fe80::208:xxxx:xxxx:xxxx%lagg0       00:08:a2:xx:xx:xx  lagg0 permanent R

maybe the provider router is not accepting fe80:: responses for ::2 CARP address as routing target?

junicast

Does your uplink router keep sending neighbor solicitation asking who ::2 is?
If so your provider might have trouble learning the MAC address. It should not have that trouble though. Might be a problem of your provider.

There's your public IP in the last post. I guess that's an accident.

msc-blue

@pmisch
Thanks for your support. We sorted this issue out now.
Our uplink provider removed a fe80:: link local filter on their interface and out of a sudden, everything works smooth.

Yes you were right, before they did the change, we did see constant (about every 2s) neighbor solicitation asking who ::2 is.
This helped to narrow down the problem.

Issue solved.