(solved) Bridged Interfaces Flapping after discon/recon
-
2.2-BETA (amd64)
built on Fri Nov 21 17:06:59 CST 2014Ive bridged my WAN to opt1 in order to bypass NAT for 1 device. If I reboot the unit while everything is connected all is fine. If I unplug a cable and plug it back in that port flaps.
Watchguard XTM505 em0 and em1
-
https://forum.pfsense.org/index.php?topic=66908.15
edit- removed redundant sentence.
-
I wasn't able to replicate that in ESX with vmxnet nor e1000, thought that was replicable across all e1000-type NICs previously when it was an issue. I'll try it on physical hardware and see.
-
If it doesn't show up for you Ill blow this thing up and reload. Ive been playing with this particular build (test box) for a while and may have broke something…
Nov 25 09:58:48 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:58:48 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:58:50 kernel: em0: link state changed to UP Nov 25 09:58:50 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em0'' Nov 25 09:58:50 check_reload_status: Linkup starting em0 Nov 25 09:58:51 php-fpm[9437]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.18.20.242 ) Nov 25 09:58:51 check_reload_status: rc.newwanip starting em0 Nov 25 09:58:52 php-fpm[9437]: /rc.newwanip: rc.newwanip: Info: starting on em0. Nov 25 09:58:52 php-fpm[9437]: /rc.newwanip: rc.newwanip: on (IP address: 172.18.20.242) (interface: WAN[wan]) (real interface: em0). Nov 25 09:58:52 kernel: em0: link state changed to DOWN Nov 25 09:58:52 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup stop em0'' Nov 25 09:58:52 check_reload_status: Linkup starting em0 Nov 25 09:58:52 check_reload_status: Reloading filter Nov 25 09:58:53 php-fpm[9437]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.18.20.242 ) Nov 25 09:58:53 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:58:53 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:58:55 kernel: em0: link state changed to UP Nov 25 09:58:55 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em0'' Nov 25 09:58:55 check_reload_status: Linkup starting em0 Nov 25 09:58:56 php-fpm[9437]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.18.20.242 ) Nov 25 09:58:56 check_reload_status: rc.newwanip starting em0 Nov 25 09:58:57 php-fpm[9437]: /rc.newwanip: rc.newwanip: Info: starting on em0. Nov 25 09:58:57 php-fpm[9437]: /rc.newwanip: rc.newwanip: on (IP address: 172.18.20.242) (interface: WAN[wan]) (real interface: em0). Nov 25 09:58:57 kernel: em0: link state changed to DOWN Nov 25 09:58:57 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup stop em0'' Nov 25 09:58:57 check_reload_status: Linkup starting em0 Nov 25 09:58:57 check_reload_status: Reloading filter Nov 25 09:58:58 php-fpm[9437]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.18.20.242 ) Nov 25 09:58:58 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:58:58 php-fpm[9437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:59:00 kernel: em0: link state changed to UP Nov 25 09:59:00 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em0'' Nov 25 09:59:00 check_reload_status: Linkup starting em0 Nov 25 09:59:00 kernel: em1: promiscuous mode disabled Nov 25 09:59:00 kernel: bridge0: link state changed to DOWN Nov 25 09:59:00 kernel: em0: promiscuous mode disabled Nov 25 09:59:01 php-fpm[64658]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.18.20.242 ) Nov 25 09:59:01 check_reload_status: rc.newwanip starting em0 Nov 25 09:59:01 check_reload_status: Syncing firewall Nov 25 09:59:02 php-fpm[64658]: /rc.newwanip: rc.newwanip: Info: starting on em0. Nov 25 09:59:02 php-fpm[64658]: /rc.newwanip: rc.newwanip: on (IP address: 172.18.20.242) (interface: WAN[wan]) (real interface: em0). Nov 25 09:59:02 check_reload_status: Reloading filter Nov 25 09:59:03 php-fpm[64658]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 09:59:03 php-fpm[64658]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 10:00:14 check_reload_status: updating dyndns firebox Nov 25 10:00:14 check_reload_status: Restarting ipsec tunnels Nov 25 10:00:14 check_reload_status: Restarting OpenVPN tunnels/interfaces Nov 25 10:00:14 check_reload_status: Reloading filter Nov 25 10:00:15 php-fpm[64658]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 10:00:15 php-fpm[64658]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). -
Yesterdays build bridging was broken so I couldn't test this but using todays build I'm not seeing this. I'm using an XTM5, identical hardware to yours so I'd suggest something else is happening.
I'm bridging several em NICs as LAN though so not quite the same setup as you.Steve
-
Thanks Steve! :)
Im downloading the latest snap to it right now and will re-test.
This is a 4GB CF card install. I need to get a drive installed in this thing instead with all the writes Im doing.
-
Edited while you posted. ::)
Since you have bridged WAN the situation is different. The issue before IIRC was interfaces were being reloaded that didn't need to be. A bridged interface, that has no IP, or a static IP interface were not supposed to be reset but got included acciudentally. The WAN interface however may be different. How exactly do you have the bridge setup?Also I see there's a new snapshot. I'm running: built on Tue Nov 25 01:23:50 CST 2014
Steve
-
I need to get a drive installed in this thing instead with all the writes Im doing.
In case you weren't being sarcastic… it's nothing to worry about. Each time you flash the card it's only 1 write to each bit. Would be a lot faster to a HD though. Updating to CF takes ages. ;)
Steve
-
Yea- mostly sarcasm but looking for more speed is the reason… ;)
Latest built on Tue Nov 25 11:18:23 CST 2014
I have WAN assigned to EM0 (Static Address with a /29)
I have OPT1 and named it SERVER EM1 (set to none)
I have a bridge with both WAN and Server as the interfacesRight now I have this desktop with the third host address of the /29 static connected to the "Server" port.
WAN is connected to a LAN port on another router with no DHCP and /29.
-
Do you see any linkup events on your system logs related to this interface?
-
@ermal:
Do you see any linkup events on your system logs related to this interface?
I just removed Siproxd before I tried to connect again. These are the logs I get-
Nov 25 15:17:29 kernel: em1: link state changed to UP Nov 25 15:17:29 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em1'' Nov 25 15:17:30 php-fpm[63464]: /rc.linkup: Hotplug event detected for SERVER(opt1) but ignoring since interface is configured with static IP ( ) Nov 25 15:17:30 check_reload_status: rc.newwanip starting em1 Nov 25 15:17:31 php-fpm[63464]: /rc.newwanip: rc.newwanip: Info: starting on em1. Nov 25 15:17:31 php-fpm[63464]: /rc.newwanip: rc.newwanip: on (IP address: ) (interface: SERVER[opt1]) (real interface: em1). Nov 25 15:17:31 kernel: em1: link state changed to DOWN Nov 25 15:17:31 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup stop em1'' Nov 25 15:17:31 php-fpm[63464]: /rc.newwanip: IP has changed, killing states on former IP 0.0.0.0. Nov 25 15:17:31 php-fpm[63464]: /rc.newwanip: Resyncing OpenVPN instances for interface SERVER. Nov 25 15:17:31 php-fpm[63464]: /rc.newwanip: Creating rrd update script Nov 25 15:17:31 check_reload_status: Linkup starting em1 Nov 25 15:17:32 php-fpm[26989]: /rc.linkup: Hotplug event detected for SERVER(opt1) but ignoring since interface is configured with static IP ( ) Nov 25 15:17:33 kernel: em1: link state changed to UP Nov 25 15:17:33 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em1'' Nov 25 15:17:33 check_reload_status: Linkup starting em1 Nov 25 15:17:33 php-fpm[63464]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages. Nov 25 15:17:33 check_reload_status: Starting packages Nov 25 15:17:33 check_reload_status: Reloading filter Nov 25 15:17:34 php-fpm[26989]: /rc.linkup: Hotplug event detected for SERVER(opt1) but ignoring since interface is configured with static IP ( ) Nov 25 15:17:34 check_reload_status: rc.newwanip starting em1 Nov 25 15:17:34 kernel: em1: link state changed to DOWN Nov 25 15:17:34 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup stop em1'' Nov 25 15:17:34 check_reload_status: Linkup starting em1 Nov 25 15:17:34 php-fpm[26989]: /rc.start_packages: Restarting/Starting all packages. Nov 25 15:17:34 php-fpm[26989]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 15:17:34 php-fpm[26989]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan). Nov 25 15:17:35 php-fpm[26989]: /rc.newwanip: rc.newwanip: Info: starting on em1. Nov 25 15:17:35 php-fpm[26989]: /rc.newwanip: rc.newwanip: on (IP address: ) (interface: SERVER[opt1]) (real interface: em1). Nov 25 15:17:35 php-fpm[26989]: /rc.newwanip: IP has changed, killing states on former IP 0.0.0.0. Nov 25 15:17:35 php-fpm[26989]: /rc.newwanip: Resyncing OpenVPN instances for interface SERVER. Nov 25 15:17:35 php-fpm[26989]: /rc.newwanip: Creating rrd update script Nov 25 15:17:35 php-fpm[36180]: /rc.linkup: Hotplug event detected for SERVER(opt1) but ignoring since interface is configured with static IP ( ) Nov 25 15:17:37 php-fpm[26989]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages. Nov 25 15:17:37 check_reload_status: Starting packages Nov 25 15:17:37 check_reload_status: Reloading filter -
Just installed the latest snap. built on Tue Nov 25 16:45:15 CST 2014
Looks like your revision fixed it Ermal..
Nov 25 16:26:31 syslogd: kernel boot file is /boot/kernel/kernel
Nov 25 16:26:36 kernel: em1: link state changed to UP
Nov 25 16:26:36 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em1''
Nov 25 16:26:37 php-fpm[59471]: /rc.linkup: Linkup detected on disabled interface…Ignoring -
So your interface is assigned but disabled hence its not being readded on the bridge.
Is this analysis correct? -
Assigned and enabled but with none as the configuration type.
-
Heh fixed, sorry for the disruption.
Try next snapshot.Normally the interface will be added to the bridge during newwanip event triggering.
-
Loading the new snap right now. :)
2.2-BETA (amd64)
built on Tue Nov 25 16:45:15 CST 2014 seemed to fix it though… -
I had never really thought about bridging the WAN side until now. Just thinking out loud here…
Would there be any advantage, or disadvantage, to adding both physical interfaces to the bridge and assigning the bridge itself as the WAN? The same way it's often done LAN side.Steve
-
I had never really thought about bridging the WAN side until now. Just thinking out loud here…
Would there be any advantage, or disadvantage, to adding both physical interfaces to the bridge and assigning the bridge itself as the WAN? The same way it's often done LAN side.Steve
That is my first choice as well. With my primary box (2.1.x) Ive tried and cannot get the box to receive an address from my ISP via DHCP when I assign WAN to the bridge. Possibly something Im doing wrong or missing..??
I have a couple of DHCP available and a Static address that Ive purchased from them. I only use one DHCP but should be able to register two macs with the cable modem for DHCP before I have to re-boot yet I do reboot it while trying.. I hand the static to the server directly and control access with firewall rules.
With my test box Im trying to reproduce this in a lab setting. :)
-
Loading the new snap right now. :)
2.2-BETA (amd64)
built on Tue Nov 25 16:45:15 CST 2014 seemed to fix it though…Still good.
Below- physically unplug and reconnect event.
Nov 26 08:43:57 syslogd: kernel boot file is /boot/kernel/kernel Nov 26 08:44:04 kernel: em1: link state changed to DOWN Nov 26 08:44:04 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup stop em1'' Nov 26 08:44:08 kernel: em1: link state changed to UP Nov 26 08:44:08 devd: Executing '/usr/local/sbin/pfSctl -c 'interface linkup start em1'' Nov 26 08:44:08 check_reload_status: Linkup starting em1 -
Can you be more clear here!
-
@ermal:
Can you be more clear here!
Sorry- ADHD.
As of 2.2-BETA (amd64)
built on Tue Nov 25 16:45:15 CST 2014The interfaces no longer flap when disconnected and reconnected.
I now have the latest as of now snaphot built on Wed Nov 26 08:02:53 CST 2014 and can report that the interfaces still behave and do not flap when disconnected and reconnected.
-
One thing that might be an issue is that the bridge interface, because it's not a physical device, doesn't have a MAC address. In order to operate like any other interface a MAC is generated when the bridge is created. However the MAC may be obviously fake and can change at each boot. Either of those could be causing an issue for dhcp. You can spoof the MAC on the bridge to prevent it though.
This was true in 2.1.X anyway, not tried it in 2.2.Steve
-
You can spoof the MAC on the bridge to prevent it though.
This was true in 2.1.X anyway, not tried it in 2.2.Steve
I have tried spoofing the mac but Im not as good at taking notes as I should be. With the American holiday tomorrow I may spend some time messing again with the primary box and see if I can make it work. It seems like I am just missing something. Ill do my best to document better as I try. ;D
Thanks!
-
In that circumstance, it may or may not be better to assign the bridge itself as WAN. Probably doesn't matter either way. One thing that'll bite you in that circumstance is the bridge has a virtual MAC address that's randomly generated when the bridge is created, hence will be different at each boot unless you specify a MAC under Interfaces>(whichever is the bridge). What I typically do there in situations where you don't want the bridge interface's MAC to ever change is just statically configure the MAC it sets automatically so it'll stick to that.
Your ISP not allowing a different MAC than your physical WAN NIC is probably why it didn't work that way. Setting the bridge to something specific, and doing whatever you need to do to change MACs with your ISP (usually just power cycle your modem) should work.
-
@cmb:
Your ISP not allowing a different MAC than your physical WAN NIC is probably why it didn't work that way. Setting the bridge to something specific, and doing whatever you need to do to change MACs with your ISP (usually just power cycle your modem) should work.
Thanks Chris!
I played with it for about a half a day recently and just had no luck. Being that it is the primary I have to pick and choose the downtime so tomorrow looks good for a re-review.
With the Watchguard Im using the Macs decrease the last octlet from interfaces 0-3. I usually just go one higher or lower when I clone. Ill do a full reboot of everything tomorrow and see If I can make it work. It is working just fine the way it is now but at this point Im curious. :)
-
Yep, Chris explained it better. ;)
I'd be interested to hear either if this helps you connect or if you think it actually helps at all.Steve
-
I did a quick writeup that I hope makes sense-
https://forum.pfsense.org/index.php?topic=84680.0
