pfSense replacing a Cisco Router - not acting as expected
-
TL;DR;
I have a Layer3 switch with several VLANs that can communicate with each other. I had a dedicated VLAN for outgoing internet traffic connected to a Cisco router, which was then connected to two WAN devices (comcast in bridge mode, and an LTE network device they threw in for free with my business subscription) with very basic routing config (I had two configs set up for it, one was to failover to my LTE connection when Comcast went down, the other had NAT rules to forward 22,80,443 into the dmz VLAN - I could only get one or the other to work, which was part of the reason I was replacing it). I replaced that router with a pfSense box, and now internet no worky.
Longer version:
Basic Diagram
Comcast connected to WAN, Verizon connected to OPT1, Aruba connected to LAN. Two Trunked 10G Lines to ESX. One 10G line on VLAN10 to TrueNAS. Everything else is on gig access ports. No VLAN configuration in pfSense.
pfSense Routes: (note, I had tried creating individual /24 routes instead of the /16 to see if that was the problem, but it won't allow me to create a 192.168.0.0/24 because it says it conflicts with the current LAN IP address, and disable/enabling the rest of them doesn't make any difference. Disabling the /16 breaks the internet for everything I moved to VLAN1000)
My Aruba config:
# # Configuration file for ArubaOS # ArubaOS Version 7.4.1.12 72393 version 7.4 enable secret "*" hostname "ArubaCore" clock timezone PST -8 controller config 10 ip access-list eth validuserethacl permit any ! netservice svc-dhcp udp 67 68 netservice svc-dns udp 53 netservice svc-ftp tcp 21 netservice svc-h323-tcp tcp 1720 netservice svc-h323-udp udp 1718 1719 netservice svc-http tcp 80 netservice svc-https tcp 443 netservice svc-icmp 1 netservice svc-kerberos udp 88 netservice svc-natt udp 4500 netservice svc-ntp udp 123 netservice svc-sip-tcp tcp 5060 netservice svc-sip-udp udp 5060 netservice svc-sips tcp 5061 netservice svc-smtp tcp 25 netservice svc-ssh tcp 22 netservice svc-telnet tcp 23 netservice svc-tftp udp 69 netservice svc-vocera udp 5002 ip access-list stateless allowall-stateless any any any permit ! ip access-list session cplogout user alias controller svc-https dst-nat 8081 ! ip access-list stateless cplogout-stateless user alias controller sys-svc-https dst-nat 8081 ! ip access-list stateless dhcp-acl-stateless any any svc-dhcp permit ! ip access-list stateless dns-acl-stateless any any svc-dns permit ! ip access-list stateless http-acl-stateless any any svc-http permit ! ip access-list stateless https-acl-stateless any any svc-https permit ! ip access-list stateless icmp-acl-stateless any any svc-icmp permit ! ip access-list stateless logon-control-stateless any any svc-icmp permit any any svc-dns permit any any svc-dhcp permit any any svc-natt permit ! ip access-list session validuser network 169.254.0.0 255.255.0.0 any any deny any any any permit ! user-role ap-role ! user-role authenticated access-list stateless allowall-stateless ! user-role denyall ! user-role denydhcp ! user-role guest access-list stateless http-acl-stateless access-list stateless https-acl-stateless access-list stateless dhcp-acl-stateless access-list stateless icmp-acl-stateless access-list stateless dns-acl-stateless ! user-role guest-logon ! user-role logon access-list stateless logon-control-stateless ! user-role preauth ! user-role stateful-dot1x ! ! crypto ipsec transform-set default-boc-bm-transform esp-aes256 esp-sha-hmac crypto ipsec transform-set default-rap-transform esp-aes256 esp-sha-hmac mgmt-user admin root maskingthelonghexstringthatwashere ntp server 192.168.10.52 ntp server 192.168.10.51 firewall disable-stateful-h323-processing ! ip domain lookup ! aaa authentication mac "default" ! aaa authentication dot1x "default" ! aaa server-group "default" auth-server Internal set role condition role value-of ! aaa profile "default" ! aaa authentication captive-portal "default" ! aaa authentication vpn "default" ! aaa authentication mgmt ! aaa authentication wired ! web-server ! papi-security ! aaa password-policy mgmt ! traceoptions ! probe-profile "default" protocol icmp ! qos-profile "default" ! policer-profile "default" ! ip-profile route 0.0.0.0 0.0.0.0 192.168.0.2 0 ! lcd-menu ! interface-profile ospf-profile "default" area 0.0.0.0 ! interface-profile pim-profile "default" ! interface-profile igmp-profile "default" ! stack-profile ! ipv6-profile ! activate-service-firmware ! aruba-central ! rogue-ap-containment ! interface-profile switching-profile "default" ! interface-profile switching-profile "TrunkPort" switchport-mode trunk ! interface-profile switching-profile "vlan10" access-vlan 10 native-vlan 10 ! interface-profile switching-profile "vlan1000" access-vlan 1000 native-vlan 1000 ! interface-profile switching-profile "vlan11" access-vlan 11 native-vlan 11 ! interface-profile switching-profile "vlan13" access-vlan 13 native-vlan 13 ! interface-profile switching-profile "vlan15" access-vlan 15 native-vlan 15 ! interface-profile switching-profile "vlan9" access-vlan 9 native-vlan 9 ! interface-profile tunneled-node-profile "default" ! interface-profile poe-profile "default" ! interface-profile poe-profile "poe-factory-initial" enable ! interface-profile enet-link-profile "default" ! interface-profile lacp-profile "FreeNAS" group-id 0 mode active ! interface-profile lldp-profile "default" ! interface-profile lldp-profile "lldp-factory-initial" lldp transmit lldp receive ! interface-profile mstp-profile "default" ! interface-profile pvst-port-profile "default" ! interface-profile dhcp-relay-profile "MainDHCP" helper-address 192.168.10.52 ! vlan-profile dhcp-snooping-profile "default" ! vlan-profile mld-snooping-profile "default" ! vlan-profile igmp-snooping-profile "default" ! vlan-profile igmp-snooping-profile "igmp-snooping-factory-initial" ! spanning-tree mode mstp ! gvrp ! mstp ! lacp ! vlan "1" igmp-snooping-profile "igmp-snooping-factory-initial" ! vlan "9" description "VLAN0009" ! vlan "10" description "VLAN0010" ! vlan "11" description "VLAN0011" ! vlan "13" description "VLAN0013" ! vlan "15" description "VLAN0015" ! vlan "1000" description "VLAN1000" ! interface gigabitethernet "0/0/0" description "LivingRoomPort1" switching-profile "vlan1000" ! interface gigabitethernet "0/0/1" description "FrontPorchCam1" switching-profile "vlan11" ! interface gigabitethernet "0/0/2" description "OfficeWindowLeft1" switching-profile "vlan1000" ! interface gigabitethernet "0/0/3" description "OfficeWindowLeft2" switching-profile "vlan1000" ! interface gigabitethernet "0/0/4" description "OfficeWindowRight1" switching-profile "vlan1000" ! interface gigabitethernet "0/0/5" description "OfficeWindowRight2" switching-profile "vlan1000" ! interface gigabitethernet "0/0/6" description "VanessaOffice1" switching-profile "vlan1000" ! interface gigabitethernet "0/0/7" description "GarageCam1" switching-profile "vlan11" ! interface gigabitethernet "0/0/8" description "FrontCornerCam1" switching-profile "vlan11" ! interface gigabitethernet "0/0/9" description "MasterBedroomPI" switching-profile "vlan1000" ! interface gigabitethernet "0/0/10" description "OldRasPi" switching-profile "vlan1000" ! interface gigabitethernet "0/0/11" description "MasterBedroom" switching-profile "vlan1000" ! interface gigabitethernet "0/0/12" description "vSphere Mgmt" switching-profile "vlan10" ! interface gigabitethernet "0/0/13" description "TempUPS" switching-profile "vlan10" ! interface gigabitethernet "0/0/14" description "Physical Domain Controller" switching-profile "vlan10" ! interface gigabitethernet "0/0/24" description "FreeNAS iLo" switching-profile "vlan15" ! interface gigabitethernet "0/0/25" description "vSphere iLo" switching-profile "vlan15" ! interface gigabitethernet "0/0/26" ! interface gigabitethernet "0/0/27" ! interface gigabitethernet "0/0/28" ! interface gigabitethernet "0/0/29" description "AppleTimeCapsule" switching-profile "vlan15" ! interface gigabitethernet "0/0/30" description "PhilipsHueBridge" switching-profile "vlan1000" ! interface gigabitethernet "0/0/31" description "HDHomeRun" switching-profile "vlan10" ! interface gigabitethernet "0/0/32" ! interface gigabitethernet "0/0/33" ! interface gigabitethernet "0/0/36" ! interface gigabitethernet "0/0/37" ! interface gigabitethernet "0/0/38" ! interface gigabitethernet "0/0/39" ! interface gigabitethernet "0/0/40" ! interface gigabitethernet "0/0/41" ! interface gigabitethernet "0/0/44" description "ArubaWiFi-2" switching-profile "vlan1000" ! interface gigabitethernet "0/0/45" description "ComcastAdminConnection" switching-profile "vlan1000" ! interface gigabitethernet "0/0/46" description "ArubaWiFi-1" switching-profile "vlan1000" ! interface gigabitethernet "0/0/47" description "InternetUplink" switching-profile "vlan1000" ! interface gigabitethernet "0/1/0" description "TrueNAS" switching-profile "vlan10" ! interface gigabitethernet "0/1/1" ! interface gigabitethernet "0/1/2" description "vSphere Trunk Port" switching-profile "TrunkPort" ! interface gigabitethernet "0/1/3" description "vSphere Trunk Port" switching-profile "TrunkPort" ! interface vlan "10" ip address 192.168.10.1 255.255.255.0 ! interface vlan "1000" ip address 192.168.0.1 255.255.255.0 ! interface vlan "11" dhcp-relay-profile "MainDHCP" ip address 192.168.11.1 255.255.255.0 ! interface vlan "13" dhcp-relay-profile "MainDHCP" ip address 192.168.13.1 255.255.255.0 ! interface vlan "15" dhcp-relay-profile "MainDHCP" ip address 192.168.15.1 255.255.255.0 ! interface vlan "9" ip address 192.168.9.1 255.255.255.0 ! interface mgmt ! interface port-channel "0" switching-profile "vlan10" enet-link-profile pc_default ! device-group ap ! interface-group gigabitethernet "default" apply-to ALL lldp-profile "lldp-factory-initial" poe-profile "poe-factory-initial" ! snmp-server view ALL oid-tree iso included snmp-server group ALLPRIV v1 read ALL notify ALL snmp-server group ALLPRIV v2c read ALL notify ALL snmp-server group ALLPRIV v3 noauth read ALL notify ALL snmp-server group AUTHPRIV v3 priv read ALL notify ALL snmp-server group AUTHNOPRIV v3 auth read ALL notify ALL snmp-server enable trap process monitor log endPS, thanks for the link to Dia! I really hate trying to draw diagrams in paint :)
-
@jonthewise you have your lan on a 192.168/16 but your downstream networks are 192.168.x/24 networks?
Here is a good diagram for using a downstream router/L3 switch routing.
-
@johnpoz What I was trying to do was create a single route to my LAN, rather than routes to each VLAN... but it seems like that doesn't work. Unfortunately, if I switch the /24 routes on and the /16 route off, I can no longer access pfSense from inside my network, and everything on the 192.168.0.0 vlan no longer connects to the internet. I don't really want anything on the 192.168.0.0 vlan anyway though, I just moved them there to temporarily get internet access working until I could figure out what was wrong with the routing.
on Serverthehome, it was suggested that I had an overlapping route and if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16 route will work, just by changing the gateway on PFSense, and making the new PFSense LAN IP the default route on the Aruba, eg make the Aruba VLAN1000 10.0.0.1 and pfSense 10.0.0.2, set the default route to 10.0.0.2 on Aruba, and add a static route to 192.168.0.0/16 via 10.0.0.1 on pfSense.
I'm in a webinar right now, so I can't test just yet, but this seems plausible. I'm not entirely sure why I didn't have this issue on the Cisco router though...
Looking at the diagram you posted kinda confuses me. I don't want pfSense to deal with multiple VLANs, I want to use it as a router/firewall between my WAN connection (and eventually add failover to the LTE one once I get the first part working) and my LAN, which should be entirely routed on the Aruba.
I have some time in about an hour to try changing the transit stuff, and I will follow up afterwards.
-
@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:
if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16
I would concur with this.. If you want to use summary route like 192.168/16, then using a transit network that is not in the 192.168 range would make sure your not overlapping.
You could also use some other summary route for your downstream that doesn't include your transit network.. For example if your transit was 192.168.0/24
Use could use some summary routes to include your other 192.168.x/24 networks vs specific /24 like 192.168.2/23
All depends what those downstream networks are and how you can summary it up.. But yeah easy would be to just use different range for your transit so you could use 192.168/16
-
@johnpoz Okay. Then something else that is confusing to me;
As seen in the screenshot, I created /24 routes for each of the VLANs that I wanted to have access to the internet, but when I turn those on, and turn the /16 off, I still can't reach the internet from inside my LAN, and it breaks access for all the things I moved to VLAN1000 to limp along. If I try creating a route on my pfSense router of 192.168.0.0/24 to point to 192.168.0.1 (the interface on Aruba) it throws the error seen here:
I think if I turn off the /16 and move all the stuff that was on VLAN15 and subsequently moved to VLAN1000 back on VLAN15, this would accomplish the same thing, but that didn't seem to work for me.
In a half hour I'm going to reload pfSense from scratch and start clean, but if I can't set a route on pfSense for the transit VLAN, I don't see how changing the subnet from 192.168.0.0 to anything else is going to make a difference?
-
Okay, well that absolutely did not work
On my Aruba, I changed the default route to 10.0.0.2/24 and made it's VLAN1000 interface ip 10.0.0.1
Reloaded a fresh pfSense install. Set the LAN to 10.0.0.2/24 with 10.0.0.1 gateway and created a route of 192.168.0.0/16 -> 10.0.0.1
Still no internet on my internal LAN networks... I'm typing this from a laptop that I manually put on the 10.0.0.0 network with 10.0.0.2 for the gateway and dns.
Ideas?
-
@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:
Still no internet on my internal LAN networks
And did you edit your rules on this 10.0.0.1 interface to allow for these downstream networks? Did you adjust unbound ACLs to include your downstream networks. The auto ACLs for unbound are normally only the directly attached networks.
Also if you had messed with auto outbound nat, switch to manual for example then the downstream networks would not be auto added to outbound nat.
-
@johnpoz Did I do what? Point me to some docs?
I reloaded pfSense, so I haven't messed with anything except what I literally spelled out above. I was under the impression for basic internet access this would just work... but obviously my setup is a little more complicated than just using pfSense as a router
-
@jonthewise holy shit! All I had to do was add a firewall rule?!
I wonder if this was my problem the whole time?
See, I assumed that the LAN net was just saying anything that came through that interface :/
Adding the 192.168.0.0/16 rule (copying the default lan rule and changing the address) seems to have resolved the problem!
So thank you so much for pushing me in the right direction. I really appreciate all the assistance!
-
@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:
All I had to do was add a firewall rule?!
Yeah - which was clearly spelled it in the picture I posted about setting up a downstream router ;)
How you had it before where your route overlapped with the connected network - while could work in a device that is designed or configured for that. Its not good idea to do it that way, because it can be confusing. If a router gets traffic destined for network xyz, and its directly attached to a xyz network, why would it send it to some downstream router to get there, when its directly attached to this xyz network..
Now in theory it could arp for this host IP, on its attached xyz network, and doesn't see it send it on via some route..
-
@johnpoz I guess I can see that it says that now that it's all highlighted and has a big red arrow pointing at it... It just wasn't clear to me (I really don't specialize in networking) that what it meant was that there needed to be a firewall rule in place to allow traffic. Here I was trying to figure out why I couldn't ping the gateway from my devices, and thinking it was a routing issue, when it was just being blocked by a firewall rule
How I originally had it set up would have worked because I had created /24 routes for each of the VLANs on my Aruba and tried turning off the /16 as a part of my troubleshooting, and had I figured out the darned firewall rule was missing, everything would have worked just fine with the transit VLAN living on 192.168.0.0/24 and all my other VLANs on their respective 192.168.X.X subnets, and there wouldn't have been any overlapping routes. That being said, I'm fine with having my transit on the 10.0.0.0/24 and using a 192.168.0.0/16 route to the LAN, and it also looks a little cleaner.
But regardless, chalk this one up to operator error. It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)
Thanks again. I'm sure I would have never figured this out on my own!
-
@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:
It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)
That should say layer2, but when I try to edit it's flagging my post as spam and won't let me save
