LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata
-
I've finally made the move to dual stack about a month ago and I love it (why didn't I do this sooner?). However, there's been a handful of times that I've noticed IPv6 over the public internet no longer worked. I haven't been able to narrow it down in call cases, but I've found two scenarios that reproduce this issue every time.
- Rebooting pfSense
- Reinstalling Suricata
In both scenarios, the pfSense box itself can reach the public internet over IPv6. I can ping and ping6 from the box itself, and both my WAN_DHCP and WAN_DHCP_IPV6 show as up and healthy. However, all of the devices on my VLAN's within my network no longer can reach the public internet over IPv6. IPv4 still works fine, but IPv6 does not.
When re-installing Suricata, I end up seeing the success screen saying the package was installed successfully, and within a couple of minutes IPv6 no longer works. I don't think Suricata is the issue here, but it's doing something (restarting a service maybe) that creates this issue.
On my WAN interface, I'm using DHCP6 with a prefix size of 56, send prefix hint checked, and debug checked.
On my VLAN's I'm using Track Interface, tracking the WAN, and assigning a prefix ID.
I enable the DHCPv6 server for each VLAN with everything at the default, and I enable the RA and set it to assisted with everything else at the default.
When this issue occurs, I'm able to get IPv6 working again on all devices within my network by releasing my WAN DHCP and the requesting it again. I get back the same IPv4 and IPv6 addresses when I do that. Nothing appears (from what I can see) to change from my ISP. But immediately after doing that all of my devices can talk over IPv6 again.
I'm running the latest 2.4.5_1 and I have the following packages installed:
- acme
- Avahi
- Cron
- freeradius3
- Netgate_Coreboot_Upgrade
- nut
- openvpn-client-export
- pfBlockerNG-devel
- pimd
- suricata
- Telegraf
Any advice on a setting I may have configured incorrectly or something I can try testing would be appreciated.
-
Some interesting things I see in my logs around the time this occurs:
/rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1593301731] unbound[9850:0] error: bind: address already in use [1593301731] unbound[9850:0] fatal error: could not open ports'
/rc.linkup: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0.30 lagg0.20 lagg0.70 lagg0.60 lagg0.80 lagg0.50 lagg0.10 lagg0.40 lagg0.90 lagg0.91' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.1 Copyright 2004-2018 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp
-
check your firewall logs, by any chance are you seeing drops on the default IPv6 drop all rule?
-
@RMO said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
check your firewall logs, by any chance are you seeing drops on the default IPv6 drop all rule?
Yes actually, many on UDP 53, and TCP 80 and 443.
The DNS queries to my resolver are a local IPv6 address. Why would pfSense be rejecting this though? With my rule setup, I have an Allow TCP/UDP on VL30_Mobile net (VLAN) to Any rule towards the end of my rules, with the absolute last two rules being my default deny rules for IPv4 an IPv6. Does this mean that my device is no longer considered part of the VL30_Mobile net network that I have defined as the source in my pass rule, and therefore it bypasses that rule and gets caught in the default deny?
If so, what can I do about that?
-
Yeah, check the other post I made (https://forum.netgate.com/topic/154856/multiple-ipv6-bugs-quirks-in-pfsense), try forcing a reload of your firewall rules. The prefix you get delegated from your ISP might not get loaded into the "LAN net" variable (at least, that is my theory), thus making pfSense not being able to match the traffic to any rule that uses it, such as the default pass rules. Forcing a reload of the rules, for example by disabling and enabling some rules, often fixes it for me, until the next reboot that is.
-
Thank you for the reply, and yes, issue #1 from your post is the same thing I'm running into. Although I can also reproduce it by re-installing Suricata in addition to rebooting. And instead of saving my firewall rules, I got my IPv6 working again by releasing and renewing the DHCP lease on the WAN.
I'm just guessing at this point, but it's possible this is a race condition of some sort? The firewall rules are loaded before the PD through DHCP has been recorded, and therefore our IPv6 addresses are missing from the LAN networks like you and I are both seeing. When you save your firewall rules, or I renew my DHCP lease, it reloads the firewall rules, but the PD has already been recorded, so things appear to start working again.
-
You may have to do some testing. I ran into a problem with my ISP last year, where they had a problem providing the prefix at the head end. Run Packet Capture on the WAN interface and try pinging something like ipv6.google.com. Do you see the pings going out? If not, it's something on the pfSense box or local LAN. If they go out, but don't see anything coming back, the problem is elsewhere.
-
@JKnott said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
You may have to do some testing. I ran into a problem with my ISP last year, where they had a problem providing the prefix at the head end. Run Packet Capture on the WAN interface and try pinging something like ipv6.google.com. Do you see the pings going out? If not, it's something on the pfSense box or local LAN. If they go out, but don't see anything coming back, the problem is elsewhere.
I mentioned in my OP that when this happens, I can both ping and ping6 from the pfSense box itself. It's only my local network that can't talk over IPv6 to the public internet.
Interestingly enough, I did what you suggested in RMO's thread and enabled Do not allow PD/Address release and that appears to have fixed my issue. Should it have though? I restarted pfSense and reinstalled Suricata, and my IPv6 connection from my LAN/VLAN interfaces to the public internet worked fine both times.
-
@OffstageRoller
Interesting, but doesn't that just bypass the core issues though? I imagine checking "Do not allow PD/Address release" just makes pfSense cache the PDs through a reboot. You'd still run into the same issue if you for example shut down your pfSense for enough time for the PD to expire from the delegating router (usually after 1-3 days) -
@OffstageRoller said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
I mentioned in my OP that when this happens, I can both ping and ping6 from the pfSense box itself. It's only my local network that can't talk over IPv6 to the public internet.
You have to test various things to isolate the problem. You were able to ping from pfSense, so that shows the WAN connection works. When you try from the LAN and watch the WAN, you can determine if the problem is with pfSense or elsewhere. This sort of thing is just basic troubleshooting. You try to isolate where the problem is coming from.
Interestingly enough, I did what you suggested in RMO's thread and enabled Do not allow PD/Address release and that appears to have fixed my issue. Should it have though?
I would expect that might be the case if you had addresses hard coded somewhere and the prefix changed.
-
@RMO said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
@OffstageRoller
Interesting, but doesn't that just bypass the core issues though? I imagine checking "Do not allow PD/Address release" just makes pfSense cache the PDs through a reboot. You'd still run into the same issue if you for example shut down your pfSense for enough time for the PD to expire from the delegating router (usually after 1-3 days)I don't know enough about this to say one way or the other.
I generally make it a goal not to check boxes that aren't the default unless I fully understand what they do and it's solving a problem I have. This is one of those examples where I don't feel comfortable checking this box. While this appears to bypass the issue I ran into, it doesn't seem like this is the actual solution?
-
@JKnott said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
@OffstageRoller said in LAN unable to talk over WAN IPv6 after reboot or reinstall of Suricata:
I mentioned in my OP that when this happens, I can both ping and ping6 from the pfSense box itself. It's only my local network that can't talk over IPv6 to the public internet.
You have to test various things to isolate the problem. You were able to ping from pfSense, so that shows the WAN connection works. When you try from the LAN and watch the WAN, you can determine if the problem is with pfSense or elsewhere. This sort of thing is just basic troubleshooting. You try to isolate where the problem is coming from.
That was something RMO asked initially, and yes, my default deny IPv6 rule is blocking my LAN from reaching the internet over IPv6. Almost all of my rules have the source (LAN net) that matches the interface where the rule exists. And it appears that after a reboot, the devices that have DHCP6 addresses are not considered part of that LAN net source, and therefor they get caught by the default deny rule.
Interestingly enough, I did what you suggested in RMO's thread and enabled Do not allow PD/Address release and that appears to have fixed my issue. Should it have though?
I would expect that might be the case if you had addresses hard coded somewhere and the prefix changed.
But they're not hard coded, and my IPv6 prefix does not change :).
However, after a reboot, pfSense does not appear to be storing the IPv6 prefix in my "net" source rules I mentioned above. And it's only after I renew my DHCP lease (which doesn't change the lease), that that my pass rules start allowing IPv6 through, that something gets updated within pfSense and my prefixes for each /64 LAN are now stored in the "LAN net" rule. Hopefully that makes sense?
I'm happy to test other scenarios to help narrow things down further.