Issues with IPV6

louis2

Today I again noticed that the IPV6 gateway was gone!

Turning off and on the gateway did re-activate IPV6, however something is definitively not OK

Below some alarms !

Louis

Aug 15 12:01:39 pfSense dpinger[332]: WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: Alarm latency 3229us stddev 4333us loss 21%
Aug 11 21:39:49 pfSense dpinger[332]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr fe80::9217:3fff:fe7f:e4a1%pppoe1 bind_addr fe80::6a05:caff:fe5a:813d%pppoe1 identifier "WAN_DHCP6 "
Aug 11 21:39:49 pfSense dpinger[99533]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 195.190.^C.D^ bind_addr 77.174.^A.B^ identifier "WAN_PPPOE "
Aug 11 21:39:49 pfSense dpinger[85113]: exiting on signal 15
Aug 11 21:39:49 pfSense dpinger[85685]: exiting on signal 15
Aug 11 21:39:47 pfSense dpinger[85685]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr fe80::9217:3fff:fe7f:e4a1%pppoe1 bind_addr fe80::6a05:caff:fe5a:813d%pppoe1 identifier "WAN_DHCP6 "

I also noticed the strange alarm below, but I do not expect it to be related.

mariatech

@louis2
Have you tried 'Disable Gateway Monitoring Action'?

Go to: System / Routing / Gateways
Edit the gateway with description Interface WAN_DHCP6 Gateway
check Gateway Action Disable Gateway Monitoring Action
Save and apply changes

We can investigate further if it doesn't work or if you want to keep this function.

louis2

@mariatech

No, its important to know (=monitor) if the gateways are working correctly!

Next to that, I do not know what I have to think from all those dpringer alarms

Gertjan

Here:
@louis2 said in Issues with IPV6:

Jul 29 13:09:24 pfSense dpinger[89346]: WAN_PPPOE 195.190.228.xxx: sendto error: 65
Jul 29 13:09:24 pfSense dpinger[89737]: WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: sendto error: 50
Jul 29 13:09:24 pfSense dpinger[89346]: WAN_PPPOE 195.190.228.xxx: sendto error: 65
Jul 29 13:09:23 pfSense dpinger[89346]: WAN_PPPOE 195.190.228.xxx: sendto error: 65
Jul 29 10:19:55 pfSense dpinger[89737]: WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: Alarm latency 3032us stddev 3890us loss 21%

the entire uplink goes bad; not only IPv6.

Btw : the only relevant lines are :Jul 31 10:20:15 pfSense dpinger[7853]: WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: Alarm latency 2900us stddev 3734us loss 22%

22% means : some IPv6 pings came back - some didn't = connection bad ?
Again, it not just IPv6, its also IPv4 - the entire WAN_PPPOE goes down ones.

All the others are dpinger exiting and restarting (because there was some WAN interface .event ?).

louis2

@gertjan

Hum:

In the GUI I never saw the IPV4-interface down, or noticed that it was down !!
I have a fiber connection from kpn.nl which is a first class provider;
I recent months I noticed a couple of times that the IPV6-GW was Red, when I logged into pfSense to check or modify something. So it does not look like a short glitch. Restarting the interface "solves" the problem.

The provider is providing the internet (and voice) via vlan6 and tv via vlan4 (I use internet only).

The arriving fiber is terminated on a provider owned fiber switch/termination point. From there the vlan's are passed towards pfSense via a managed Zyxel switch.

So apart form the fiber-switch and the vlan's transported via the rock stable managed switch there is "nothing", what could spoil the connectivity. And that used to work, it is only relatively recent that I now and than notice that the IPV6 is gone / GW is red.

Gertjan

@louis2

I had a lot of these

in my Status > System Logs > System > Gateways log page.
All on August 20.
I remember doing some testing with pfBlockerng-devel, so I force reloaded the settings. That will trigger what I would call an Interface event, and interface related tasks will restart, like dpinger.

Since 20/08 and now : not a single message from dpinger : no restart, no ping loss.

Btw : I use pfBlockerng-devel,, and only sync files (feeds) once a week, not every hour (!!), as most feeds are rarely updated anyway.
So my dpinger only gets restarted once a week (I guess)

I'm not sure if lowering the number of dpinger restarts will help for you, but I guess it's worth it to try.

My ISP in Franc is Orange, using an VDSL, IPv6 from Orange sucks, as they only give a /64 == totally useless for a multi LAN setup. I use the IPv6 from tunnel.he.net, the last free mondial IPv6 ISP with a pop in Paris.

@louis2 said in Issues with IPV6:

from kpn.nl which is a first class provider

Ik heb enige ervaring met die andere ISPs ..... niet echt great idd ;)

mariatech

@louis2

No, its important to know (=monitor) if the gateways are working correctly!

Of course, I was thinking it might just be a lazy ping response. It will still monitor the gateway but the sensitivity might need tuning. Is there any noticeable pattern to when the disruptions occur, e.g. time of day, hours between occurence? Are there any DHCP/DHCP6 logs in the time leading up to a disruption?

About the ARP warning, do you recognize the MAC address, is it one of your devices, or maybe a phone, laptop, or another router? 7c:b0:c2 belongs to Intel, the gateway appears to be Huawei. Any pattern to the occurence?

How many leases does your ISP allow, e.g. I may request two IPv4 leases, and up to five IPv6 /56 delegations. After a reset, do you get the same ip/ip6 addresses every time?

What's the Zyxel switch for? Untagging, monitoring, or is pfSense one-armed? Any IGMP configuration?

louis2

@mariatech

Related to your question: "Is there any noticeable pattern to when the disruptions occur, e.g. time of day, hours between occurrence? Are there any DHCP/DHCP6 logs in the time leading up to a disruption?"
I do not know, to analyze that I have to download the logs and analyze the them offline. I will consider that option.

Related to the ARP messages, perhaps it is related to Synching firewall event. See below. There seems to be an relation. I have no idea to which device the mac-address belongs. I also wonder why there is the word "bogon".

Aug 23 21:59:58 pfSense arpwatch[58415]: bogon 0.0.0.0 7c:b0:c2:f2:df:55
Aug 23 21:59:57 pfSense arpwatch[58415]: bogon 0.0.0.0 7c:b0:c2:f2:df:55
Aug 23 10:55:00 pfSense sshguard[22416]: Now monitoring attacks.
Aug 23 10:55:00 pfSense sshguard[73281]: Exiting on signal.
Aug 23 10:53:29 pfSense check_reload_status[481]: Syncing firewall

Related to leases. I have one fixed IPV4 address and an /48 IPV6-range. So I will constantly have the same addresses.

The Zyxel switch is my 1G-main/central switch. Related to this, it is used to separate and forward vlans from the Fiber access point to pfSense. As example originally I did forward the internet vlan (vlan6) to pfsense but not the TV-vlan (vlan4).

Gertjan

@louis2 said in Issues with IPV6:

Related to the ARP messages, perhaps it is related to Synching firewall event. See below. There seems to be an relation. I have no idea to which device the mac-address belongs. I also wonder why there is the word "bogon".

For info Arpwatch reports bogons frequently

From what I make of it : DHCP clients broadcast intially with a 0.0.0.0.0.0 MAC to trigger a DHCP server reply.
The Arpwatch 'scanner' will detect this, and add a bogon line, as he knows that 0.0.0.0.0.0 is actually known as for example aa.bb.cc.dd.ee.ff.
The bogon messages are nearly always inoffensive.

louis2

@gertjan

This morning I noticed that the IPV6-gateway was gone again Stopping and starting the wan does ^fix^ the problem, but of course that is not OK.

Since previous time this happened, I did setup a syslog server (graylog) in order to fetch all potential related alarms.

In the logging I found three messages which draw my attention. when filtering the log using message ^like "ppp"^

<12>1 2022-09-09T04:12:53.468066+02:00 pfSense.lan dpinger 76225 - - WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: Alarm latency 3558us stddev 4502us loss 22%

Using the time as new selection criteria I selected the alarms occurring around this time. Starting and ending with an alarm which IMHO is not related. Here they are:

<134>1 2022-09-09T04:12:35.879325+02:00 pfSense.lan filterlog 72340 - - 1316,,,1559467905,lagg0.13,match,pass,in,4,0x0,,1,64738,0,none,103,pim,46,192.168.13.1,224.0.0.13,datalength=26

<12>1 2022-09-09T04:12:53.468066+02:00 pfSense.lan dpinger 76225 - - WAN_DHCP6 fe80::9217:3fff:fe7f:e4a1%pppoe1: Alarm latency 3558us stddev 4502us loss 22%

<30>1 2022-09-09T04:12:53.474315+02:00 pfSense.lan rc.gateway_alarm 30961 - - >>> Gateway alarm: WAN_DHCP6 (Addr:fe80::9217:3fff:fe7f:e4a1%pppoe1 Alarm:1 RTT:3.558ms RTTsd:4.502ms Loss:22%)

<13>1 2022-09-09T04:12:53.474964+02:00 pfSense.lan check_reload_status 391 - - updating dyndns WAN_DHCP6

<13>1 2022-09-09T04:12:53.474990+02:00 pfSense.lan check_reload_status 391 - - Restarting OpenVPN tunnels/interfaces

<13>1 2022-09-09T04:12:53.474983+02:00 pfSense.lan check_reload_status 391 - - Restarting IPsec tunnels

<13>1 2022-09-09T04:12:53.474997+02:00 pfSense.lan check_reload_status 391 - - Reloading filter

<30>1 2022-09-09T04:12:55.499322+02:00 pfSense.lan unbound 21401 - - [21401:3] info: 2a02:a468:c48e:11::f051 printer.lan. AAAA IN

Note that I am a bit surprised to see messages related to OpenVPN as well .... since I am not using OpenVPN

Gertjan

@louis2 said in Issues with IPV6:

since I am not using OpenVPN

And neither IPSEC, right ?

When an interface event arrives, processes that use that interface are restarted.
If OpenVPN, or IPSEC are not running, then that's ok. Nothing happens.
You also saw "updating dyndns WAN_DHCP6" as it would be logical to do a DynDNS operation if the WAN_DHCP6 was used by the a Dyndns settings, the WAN IPv6 could have been changed, and the Dyndns (might) need an update.

I don't know if dpinger triggers an interface event when the loss is 22 %, as 78 % of the v6 ping packets still made it back, so the IPv6 connection still working, but 'not for "100 %".

Was it slowly degrading, dpinger wioll take action : a simple IPv6 interface restart (down and up) could correct the situation. Or not .... all depends why IPv6 went down (upstream).
Your IPv6 was probably getting ok (long/some) time afterwards, but the immediate dpinger stop start on the interface didn't work out. It stays down, until far later on you restart it.

My theory, of course.

louis2

@gertjan

I assume that Dyndns is related to a dynamic address, however .... I have a fixed IP{V4 address and a fixed IPV6-range.

I can imagine .... that there was a maintenance action at the provider leading (not sure at all that that is what happend), however, that the situation does not recover ...... is not ok ....

And yep simple interface restart ^fixes^ the issue ..... but that is not OK of course

louis2

@gertjan

Today the GUI again showed that the IPV6 was gone.

To test if the IPV6 was really gone, I did ping a couple of IPV6-addresses. And ..... I could reach them

Pinging 2a00:1450:400e:801::200e with 32 bytes of data:
Reply from 2a00:1450:400e:801::200e: time=4ms
Reply from 2a00:1450:400e:801::200e: time=4ms
Reply from 2a00:1450:400e:801::200e: time=4ms
Reply from 2a00:1450:400e:801::200e: time=4ms

Ping statistics for 2a00:1450:400e:801::200e:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 4ms, Maximum = 4ms, Average = 4ms

So the conclusion seems to be that the gateway is available .... however that the GUI is not correctly updated ....

Gertjan

@louis2 said in Issues with IPV6:

So the conclusion seems to be that the gateway is available

It should be the gateway.
You've see many people people using (IPv4) 8.8.8.8 - ir 8.8.8.8 is not a gateway ;)

Who are you pinging ?

To find out 'for sure' : use

grep ax | grep 'dpinger' and look at the command line used.

louis2

@gertjan said in Issues with IPV6:

grep ax | grep 'dpinger'

[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root: nslookup google.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: google.com
Address: 142.251.39.110
Name: google.com
Address: 2a00:1450:400e:800::200e

[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root: ping6 2a00:1450:400e:800::200e
PING6(56=40+8+8 bytes) 2a02:a468:c48e:1::1 --> 2a00:1450:400e:800::200e
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=0 hlim=61 time=6.048 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=1 hlim=61 time=5.979 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=2 hlim=61 time=5.980 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=3 hlim=61 time=6.106 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=4 hlim=61 time=5.989 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=5 hlim=61 time=6.030 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=6 hlim=61 time=6.032 ms
16 bytes from 2a00:1450:400e:800::200e, icmp_seq=7 hlim=61 time=6.020 ms

--- 2a00:1450:400e:800::200e ping6 statistics ---
12 packets transmitted, 12 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 5.979/6.024/6.106/0.036 ms
[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root: grep ax | grep 'dpinger'
^C
[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root:

Gertjan

@louis2 said in Issues with IPV6:

grep ax | grep 'dpinger'

sorry,

ps ax | grep 'dpinger'

My IPv6 interface (he.net' :

78401 - Is 2:11.06 /usr/local/bin/dpinger -S -r 0 -i HENETV6_TUNNELV6 -B 2001:470:beef:5c0::2 -p /var/run/dpinger_HENETV6_TUNNELV6~2001:470:beef:5c0::2~2001:470:beef:5c0::1.pid -u /var/run/dpinger_HENETV6_TUNNELV6~2001:470:beef:5c0::2~2001:470:beef:5c0::1.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:beef:5c0::1

where 2001:470:beef:5c0::1 is actually my IPv6 gateway

I can ping it - like dpinger :

[22.05-RELEASE][admin@pfSense.overhere.net]/root: ping6 2001:470:beef:5c0::1
PING6(56=40+8+8 bytes) 2001:470:beef:5c0::2 --> 2001:470:beef:5c0::1
16 bytes from 2001:470:beef:5c0::1, icmp_seq=0 hlim=64 time=45.301 ms
16 bytes from 2001:470:beef:5c0::1, icmp_seq=1 hlim=64 time=43.746 ms
16 bytes from 2001:470:beef:5c0::1, icmp_seq=2 hlim=64 time=44.482 ms
^C

louis2

@gertjan

[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root: ps ax | grep 'dpinger'
31465 - Is 0:09.22 /usr/local/bin/dpinger -S -r 0 -i WAN_PPPOE -B 77.174.abc.def -p /var/ru
31795 - Is 0:07.53 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP6 -B fe80::6a05:caff:fe5a:813d
88478 1 S+ 0:00.00 grep dpinger
[2.7.0-DEVELOPMENT][admin@pfSense.lan]/root:

louis2

A couple of months past after I did create this topic. I hoped it would be solved in the FreeBSD14 release.

However, I am running 2.7 FreeBSD14 now ...... and the problem is still there

blueuser

@louis2 I had the same issue and these settings fixed it. Make sure DHCP6 is selected in 'IPv6 Configuration Type' and Gateway Monitoring is turned off for IPv6. Then release and renew WAN in Status->Interfaces.

louis2

@blueuser

I do not fully understand the behavoir. But I was all ready trying take make a change, when I had some trouble and also a system crash. So I will perhaps try later.

However,

turned off gateway monitoring for IPv6, seems a bad idea to me. And apart from that no idea where to find that setting
my provider interface is ppoe based and some settings you are suggesting are strange in relation to ppoe
I have a fixed /48 range
when turning on advanced dhcp options, I noticed a field ^Prefix Interface^ really do not know what to imagine there. One of my many interfaces was listed there, no idea why that one. If there is something like a ^Prefix Interface^ what ever it may be, I would expect the WAN interface to have that functionallity