Steelhead CX770 Port issues
-
I have two identical Riverbed Steelhead CX770 devices configured with pfSense 2.8. Both pfSense deployments are exactly the same (same rules, interfaces, etc.). The only difference is that on one of them I'm using the AUX (igb5) and on the other the LAN0_0 (igb0) to connect the firewalls to the the internal LAN. The firewall that uses the igb0 port routes traffic to the internet and othe VLANs without any issues. The firewall that uses the igb5 port seems to have no connectivity at all. If I plug in a laptop into the port, I'm able to connect to the management UI without any issues but if I connect anything else, no traffic seems to flow even though the link and traffic LEDs are lit up and blinking.
The CX770 devices are configured to use all ports as "No-Bypass" so all ports are available as individual ports to pfSense. Has anyone encountered this issue with the CX770 devices? If so, how can this be resolved so all ports are functional? Based on testing I've done on all ports, it looks like the igb4 and igb5 ports are fully functional. The other ports have what appears to be connectivity issues. This is a real head scratcher as other users with the same hardware are able to use all ports without any restrictions.Any asssistance from the community will be greatly appreciated!
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
Mmm, with by-pass ports first make sure it's actually connected the way you think it is.
Using either the console directly or connecting via another port run
ifconfig igb5at the command line. Make sure it changes the link status when you connect and disconnect igb5.However if you're able to connect to the webgui using a client connected to igb5 then it probably is connected as you think. And the client is in the expected subnet etc. In which case it's probably a firewall rule on the AUX interface causing a problem. It could also be a NAT rule if you're not using automatic outbound NAT. But if that were the case you would still be able to connect to other local subnets just not anything external.
-
@stephenw10 thank you for thew reply. Yes, I already checked all of the items you suggest. Everything looks as it should; that's the reason I posted here for suggestions since I don't know what else to do. On my original post I made a slight typo: the only two ports fully working are the igb4 and igb0. All other ports are exhibiting the behavior I described.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
OK, so both boxes are configured the same?
How are the ports setup in each case? 5 internal subnets and 1 WAN?
If you can access the webgui but nothing else it pretty much has to be a config issue.
-
@stephenw10, on the firewall that's online, there are 7 VLANs all in igb0 (LAN0_0). That is working without any issues. On the firewall I'm trying to bring online, I configured it exactly the same but all VLANs are on igb5 (AUX). On both cases the there is only one WAN port configured with static IP addresses provided by my ISP. The inter-VLAN rules are exactly the same on both firewalls. The rules are the same throughout all VLANs with the exception of VLAN 101 and 401. Those two VLANs have additional ports open for the Omada Controller.
Both firewalls show the same issue, though. The only ports fully functional are igb4 (PRI) and igb0 (LAN0_0). Oh, the firewall currently online is running pfSense+ 25.07.1. Since I'm not able to get another pfSense+ software, I need to downgrade to version CE 2.8; that's the whole point on bringing the other firewall online. Since I'm bringing a new firewall online, I thought about using the ports the way it makes sense to me. After the softrware is downgraded to CE 2.8, I'm planning on using both firewalls in a HA configuration.I'll bet anything (well, almost everything) that if I switch the LAN port to igb0 on the new firewall, everything will work without any issue. While that is a logical solution, it still leaves me with unusable ports.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
Just to be clear though on the new device you are able to access the webgui using a client connected to a VLAN on igb5?
On any/all VLAN there?
If you're able to do that then it isn't a problem with the port or the NIC driver. It pretty much has to be a config issue at that point.
So a bad subnet mask somewhere or a missing route etc.
What error do you get when you try to connect to something external?
-
@stephenw10, no, I'm not able to access the webui from any VLAN on igb5 even though the rules and everything else are the same with the firewall already working with the VLANs on igb0. I have attached the rule set from the firewall that works with the VLANs attached to igb0.
Also, the following image shows you how the new firewall is configured:
Let me know if you need any additional information.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
@martimun said in Steelhead CX770 Port issues:
I'm not able to access the webui from any VLAN on igb5
Ah OK. Can you ping any IP on pfSense even? Do clients on those VLANs get a DHCP lease in the correct subnet? (assuming dhcp is in use).
Or do clients on LAN_AUX work? So using igb5 but without VLAN tagging?
-
@stephenw10, on the firewall where the VLANs are on port igb0 yes, I can. When connecting my laptop to LAN_AUX, I assign an IP address in the range of the IP address on LAN_AUX. The gateway is the .254 IP address that LAN_AUX has assigned. The only thing I can do is connect to the webui interface. I have tried every single possible combination imaginable. I even configured a Dell Force10 S55T spare with the same config as my production switch just to make sure there is nothing wrong with the switch config.
Just for testing purposes, I'll switch all VLANs on the new firewall to port igb0 (like in the firewall in production) just to confirm that the new firewall works as intended. If that works, that means there is something really weird going on with the igb5, igb1, igb2, and igb3 on the actual hardware. That's why I posted in this forum; since this particular Steelhead device is fairly popular as a pfSense firewall/router and knowing that people got them to work successfully, I'd like to see if someone here can sort me out.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
Hmm, sure seems like something low level not working with that port.
So you are not using pfSense as the DHCP server for those subnets?
-
@stephenw10, no. I have a pair of Windows domain controllers doing DHCP duty. On the production firewall, all nodes in every VLAN can get a DHCP lease except for VLAN 101. That's the server VLAN and every node has its own static IP address. I have to agree; there is something really wrong with the ports at the hardware level. And, it's not just one port; it's four of them and in two different units. Too much of a coincidence if you ask me. Yesterday, I haven't had time to transfer the VLANs to the igb0 port on the firewall that I'm bringing online. Later today, I'll do the transfer and report back with the results.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
Yeah I mean this really seems like it's the bypass NICs at work to me. Except that you confirmed pfSense sees igb5 get and lose link when you connect to it?
If so then I would run a packet capture on igb5 to see if anything is actually coming in on it.
The fact you can connect to the gui but only the gui on igb1 pretty much has to be a firewall rule issue.
-
@stephenw10, since I'll be switching interfaces, I'll also do a full reset of the firewall so I can reenter all rules. I'll also run Wireshark on the new firewall to see how things flow (or not!) through the ports. It really is a shame that these ports are not working as expected because the hardware is rock solid for a pfSense install. At least I can feel good that I purchased these two CX770s brand new as a bundle for only $100 on eBay. They came from a government surplus site in their original box and all the accessories.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
Mmm, all the by-pass ports I've ever encountered disconnect the internal interfaces when they are in by-pass mode. But I could imagine (maybe) a mode where it loops back the data lines while the MII connections from the PHY still report the link status. But for that to happen you would need both NICs in the by-pass pair to be linked.
-
@stephenw10, just finished switching all VLANs to the igb0 port. Guess what? ! It all works without any issues. Something is very wrong with the ports. It's not just the bypass ports; the AUX (igb5) which is controlled separately doesn't work. At this point I'm out of ideas. I also ran Wireshark and all traffic stops at the igb5 port. ICMP and HTTP traffic successfully reach the IP address on igb5 but nothing else goes through. I rechecked the BIOS settings to make sure the igb0-3 are not in bypass mode.
Anyone who has gotten a CX770 to work successfully, I would appreciate any assistance you can give. This is truly bizarre!
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
@martimun said in Steelhead CX770 Port issues:
I also ran Wireshark and all traffic stops at the igb5 port. ICMP and HTTP traffic successfully reach the IP address on igb5 but nothing else goes through
When testing from where?
-
@stephenw10, everything was working and when I returned to the laptop directly connected to the CX770, nothing worked anymore. When I looked at the NAT setting on the working firewall, I noticed a couple rules that were created automatically. On the new firewall (pfSense CE 2.8.1), even though it was set to automatically create NAT rules, it wasn't creating them. As soon as I put NAT rules in place for all the nets everything started working! I'm streaming a series and watched three episodes without any interruptions. Now I'm happy.
The firewall that was in place and running on pfSense+, I need to downgrade. The original plan was to use igb4 for the WAN (which works) and igb4 for the LAN. I'll create the NAT rules manually so I can, hopefully, rule out the NAT config.
Is it possible that there may be a glitch with pfSense CE and automatic NAT rules? pfSense+ did not have this issue.
Stephen, let me finish rebuilding the firewall so I can post the results here. Once all is clear this subject can be closed. Thank you so much for all the feedback and assistance.
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава! -
@martimun said in Steelhead CX770 Port issues:
Is it possible that there may be a glitch with pfSense CE and automatic NAT rules?
Very unlikely. We would have thousands of complaining threads. The code there is almost all the same between CE and Plus
Much more likely is that what triggers the automatic outbound NAT rules is not in place. And that is that pfSense will add NAT rules on every WAN for all internal subnets but to determine which is a 'WAN' it checks for a gateway on the interface config. If you add a WAN gateway separately, in System > Routing > Gateways, it will work as a gateway but will not be on the interface directly and hence you won't get any OBN rules for it. I've seen that mistake many, many times!
-
@stephenw10, thank you for the clarification! Now I see exactly what happened! On the new firewall I actually created the WAN gateway to reflect the circuit ID from Spectrum Business whereas on the old firewall I left it untouched. But still it doesn't explain why any inter-VLAN/internal traffic was not moving which could still point to a physical port issue.
I still don't trust those ports though. Today, I will rebuild the old firewall to see if I can use all ports as intended. I'll post the results as soon as I finish. Again, thank you for your input and guidance!
Martin M. Mune
US Army Combat Veteran
Operation Iraqi FreedomVolunteer Soldier
International Legion for the Defense of UkraineСлава Україні!
Героям Слава!
