WAN dropping out on OPT1 connection
-
When I connect a cable into OPT1, the WAN connection is dropped (port lights go out) and when I remove the cable, the WAN connection restores. It is very likely that this is self-inflicted due to me missing a key piece of information.
First, this is the system that is working. My general goal is to iteratively move the IOT stuff the main network. Eventually I'll replace the Internet Router with a netgate appliance, to do that i'll need to replace DDNS, house-to-house VPNs, etc.
In this picture the PF Sense is acting a firewall and router between the 192.168.1/24 and 192.168.20/24 subnet and is not running the NAT. I entered a route for the 192.168.20/24 subnet into the internet router and have routability between the networks and firewall rules. I was pretty happy this basically worked.
The next step is to use VLANs on Cisco switches to segment the IP cameras onto a VLAN and then have this VLAN feed into the SG-1100 OPT port as a seperate subnet. With the first step configuring the VLAN /Ports 192.1681.1.91 switch to get the 1 IP camera moved over. Which leads to this picture:
I've configured the OPT port, set it's ip address, enabled DHCP server on it. Added the VLAN 100 to the Cisco switch, assigned ports 6 and 7 to it and untagged ports. Reconfigured Port #9 to be a Access Port instead of a Trunk. When I connect the cable from OPT to #6, the OPT interface comes up and then the WAN interface goes down. By down I mean the lights turn off on both the Cisco Switch port and the PFSense port
In Pfsense logs I see this:
Jan 9 20:58:02 check_reload_status 378 Starting packages
Jan 9 20:58:02 php-fpm 13633 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 192.168.10.1 - Restarting packages.
Jan 9 20:58:00 php-fpm 13633 /rc.newwanip: Creating rrd update script
Jan 9 20:58:00 php-fpm 13633 /rc.newwanip: Resyncing OpenVPN instances for interface IOT10.
Jan 9 20:57:55 php-fpm 13633 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Jan 9 20:57:55 php-fpm 13633 /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jan 9 20:57:55 php-fpm 13633 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. ''
Jan 9 20:57:51 check_reload_status 378 Reloading filter
Jan 9 20:57:51 check_reload_status 378 Restarting OpenVPN tunnels/interfaces
Jan 9 20:57:51 check_reload_status 378 Restarting ipsec tunnels
Jan 9 20:57:51 check_reload_status 378 updating dyndns WAN_DHCP
Jan 9 20:57:51 rc.gateway_alarm 5061 >>> Gateway alarm: WAN_DHCP (Addr:192.168.1.254 Alarm:1 RTT:1.083ms RTTsd:.445ms Loss:22%)
Jan 9 20:57:46 php-fpm 13633 /rc.newwanip: rc.newwanip: on (IP address: 192.168.10.1) (interface: IOT10[opt1]) (real interface: mvneta0.4092).
Jan 9 20:57:46 php-fpm 13633 /rc.newwanip: rc.newwanip: Info: starting on mvneta0.4092.
Jan 9 20:57:45 check_reload_status 378 Reloading filter
Jan 9 20:57:45 check_reload_status 378 rc.newwanip starting mvneta0.4092
Jan 9 20:57:45 php-fpm 13633 /rc.linkup: Hotplug event detected for IOT10(opt1) static IP (192.168.10.1 )
Jan 9 20:57:44 kernel e6000sw0port1: link state changed to UP
Jan 9 20:57:44 check_reload_status 378 Linkup starting $e6000sw0port1
Jan 9 20:57:40 check_reload_status 378 Reloading filter
Jan 9 20:57:40 php-fpm 80132 /rc.linkup: Shutting down Router Advertisment daemon cleanly
Jan 9 20:57:38 php-fpm 80132 /rc.linkup: DEVD Ethernet detached event for wan
Jan 9 20:57:37 kernel e6000sw0port3: link state changed to DOWNOn the Cisco Side I see this (i have a clock sync issue it seems :-()
2147481912 2022-Jan-09 21:12:18 Warning %STP-W-PORTSTATUS: gi2: STP status Forwarding
2147481913 2022-Jan-09 21:12:14 Informational %LINK-I-Up: gi2
2147481914 2022-Jan-09 21:12:10 Warning %LINK-W-Down: gi6
2147481915 2022-Jan-09 21:11:39 Warning %STP-W-PORTSTATUS: gi3: STP status Forwarding
2147481916 2022-Jan-09 21:11:35 Informational %LINK-I-Up: gi3
2147481917 2022-Jan-09 21:11:26 Warning %LINK-W-Down: gi3
2147481918 2022-Jan-09 21:11:25 Informational %LINK-I-Up: gi3
2147481919 2022-Jan-09 21:10:43 Warning %LINK-W-Down: gi7
2147481920 2022-Jan-09 21:10:16 Warning %STP-W-PORTSTATUS: gi7: STP status Forwarding
2147481921 2022-Jan-09 21:10:12 Informational %LINK-I-Up: gi7
2147481922 2022-Jan-09 21:10:09 Warning %LINK-W-Down: gi3
2147481923 2022-Jan-09 21:06:31 Warning %STP-W-PORTSTATUS: gi3: STP status Forwarding
2147481924 2022-Jan-09 21:06:29 Warning %LINK-W-Down: gi7
2147481925 2022-Jan-09 21:06:27 Informational %LINK-I-Up: gi3
2147481926 2022-Jan-09 21:05:41 Informational %AAA-I-DISCONNECT: http connection for user pete, source 192.168.1.81 destination 192.168.1.91 TERMINATED
2147481927 2022-Jan-09 21:03:28 Warning %STP-W-PORTSTATUS: gi6: STP status Forwarding
2147481928 2022-Jan-09 21:03:24 Informational %LINK-I-Up: gi6
2147481929 2022-Jan-09 21:03:15 Warning %LINK-W-Down: gi3
2147481930 2022-Jan-09 21:03:14 Warning %LINK-W-Down: gi2
2147481931 2022-Jan-09 21:03:13 Informational %LINK-I-Up: gi3
2147481932 2022-Jan-09 21:03:09 Warning %STP-W-PORTSTATUS: gi7: STP status Forwarding
2147481933 2022-Jan-09 21:03:05 Informational %LINK-I-Up: gi7
2147481934 2022-Jan-09 21:02:58 Warning %LINK-W-Down: gi3 -
FIXED: Turns out the cable between OPT1 and the Switch was a crossover cable. Replacing it with a proper cable solved the issue. Not sure, why this causes WAN port to turn off - as I’d expect the ports to be electrically independent. Maybe is causes all to flip polarity? Anyways….
-
Hmm, the ports on the 1100 are auto-MDI/X so I would not expect a cross-over cable to make any difference there.
From the way it's connected though I'd have to guess it was creating a packet loop somehow that was shutting off ports. Since the LAN was not affected that was probably happening in the Cisco switch. I'd expect to see that logged there though.Steve
-
I’ve since temporarily separated OPT1 from the Cisco switch as I realized I had more studying of VLANs to do before success was possible - too many confounding issues - dhcp, trunks, routes, etc.
Anyways, with OPT1 going into an isolated switch. I still see WAN lights go out when hot swapping the OPT1 cables, and have to reboot. When I’m not doing that everything is stable. Next step, I’m going to temporarily put a unmanaged switch/ hub between SG WAN and Cisco Switch so I can see which side is turning out the light.
-
But you can still access the LAN when that happens yes?
That's hard to explain since all three ports are connected the same way in the 1100.
-
Yes, when the WAN drops out, the LAN is still up and I can connect to the pfsense console from that subnet.
-
Hmm, well I have no explanation for that. I can only guess something low level in the on-board switch maybe. I test the 1100 with cross-over cables at every release and have never seen anything like that.
Steve
-
Is there one MAC address for the SG-1100, hence plugging it in to the Cisco Switch twice, is creating a MAC address conflict?
If that’s correct, then looks like I’m approaching this incorrectly.
This is what I’m trying:
WAN / dhcp client -> Cisco Access default VLAN1 -> Asus Router -> Internet
LAN / dhcp server <- Linksys WiFi bridges <- wireless IOT devices
OPT1 /<- Cisco Trunk port (vlan 100) <- IP Cameras
CAMERA_VLAN / dhcp server < 0T,3 (switch config)Items 1+2 are working, 3+4 are not working - unable to get DHCP to give out addresses - I suspect I’m using it in a way not intended since most folks would feed all the VLANs in - Versus trying to use VLAN1 as WAN and VLAN 100 as a subnet?
Should I try configuring the WAN port as a Cisco Trunk and then configure the CAMERA_VLAN off that and remove the OPT1 cable? Other recommendations or docs I should read?
-
That might happen but you said you were still seeing the issue with OPT connected to a completely isolated switch which seems to rule it out.
If you're using a port on the Cisco switch for tagged and untagged traffic you may have to set the port as multi use. Cisco seem to like doing things like that.
Also I assume by VLAN 1 you actually mean untagged ? Because actually using VLAN 1 tagged can cause all sorts of problems.
-
I have not be able to reproduce the behavior again, unless I use the crossover cable and have both OPT1 and WAN going into same switch - which I now believe to be an invalid configuration. MAC addresses need to be unique at the physical LAN subnet level. Hence the marvel with 1 Mac and 3 ports works as long each port is plugged into a different physical subnet.
Now that I understand the Sg-1100, pfsense, VLANs better. I have the system working with 1 cable from WAN to Cisco Switch with the untagged VLAN being the WAN interface and the tagged 100 VLAN being the camera interface.
