pfSense replacing a Cisco Router - not acting as expected

jonthewise

TL;DR;

I have a Layer3 switch with several VLANs that can communicate with each other. I had a dedicated VLAN for outgoing internet traffic connected to a Cisco router, which was then connected to two WAN devices (comcast in bridge mode, and an LTE network device they threw in for free with my business subscription) with very basic routing config (I had two configs set up for it, one was to failover to my LTE connection when Comcast went down, the other had NAT rules to forward 22,80,443 into the dmz VLAN - I could only get one or the other to work, which was part of the reason I was replacing it). I replaced that router with a pfSense box, and now internet no worky.

Longer version:

Basic Diagram
Comcast connected to WAN, Verizon connected to OPT1, Aruba connected to LAN. Two Trunked 10G Lines to ESX. One 10G line on VLAN10 to TrueNAS. Everything else is on gig access ports. No VLAN configuration in pfSense.
alt text

pfSense Routes: (note, I had tried creating individual /24 routes instead of the /16 to see if that was the problem, but it won't allow me to create a 192.168.0.0/24 because it says it conflicts with the current LAN IP address, and disable/enabling the rest of them doesn't make any difference. Disabling the /16 breaks the internet for everything I moved to VLAN1000)

alt text

My Aruba config:

#
# Configuration file for ArubaOS
# ArubaOS Version 7.4.1.12 72393
version 7.4
enable secret "*"
hostname "ArubaCore"
clock timezone PST -8
controller config 10
ip access-list eth validuserethacl
  permit any
!
netservice svc-dhcp udp 67 68
netservice svc-dns udp 53
netservice svc-ftp tcp 21
netservice svc-h323-tcp tcp 1720
netservice svc-h323-udp udp 1718 1719
netservice svc-http tcp 80
netservice svc-https tcp 443
netservice svc-icmp 1
netservice svc-kerberos udp 88
netservice svc-natt udp 4500
netservice svc-ntp udp 123
netservice svc-sip-tcp tcp 5060
netservice svc-sip-udp udp 5060
netservice svc-sips tcp 5061
netservice svc-smtp tcp 25
netservice svc-ssh tcp 22
netservice svc-telnet tcp 23
netservice svc-tftp udp 69
netservice svc-vocera udp 5002
ip access-list stateless allowall-stateless
  any any any  permit
!
ip access-list session cplogout
  user   alias controller svc-https  dst-nat 8081
!
ip access-list stateless cplogout-stateless
  user   alias controller sys-svc-https  dst-nat 8081
!
ip access-list stateless dhcp-acl-stateless
  any any svc-dhcp  permit
!
ip access-list stateless dns-acl-stateless
  any any svc-dns  permit
!
ip access-list stateless http-acl-stateless
  any any svc-http  permit
!
ip access-list stateless https-acl-stateless
  any any svc-https  permit
!
ip access-list stateless icmp-acl-stateless
  any any svc-icmp  permit
!
ip access-list stateless logon-control-stateless
  any any svc-icmp  permit
  any any svc-dns  permit
  any any svc-dhcp  permit
  any any svc-natt  permit
!
ip access-list session validuser
  network 169.254.0.0 255.255.0.0 any any  deny
  any any any  permit
!
user-role ap-role
!
user-role authenticated
 access-list stateless allowall-stateless
!
user-role denyall
!
user-role denydhcp
!
user-role guest
 access-list stateless http-acl-stateless
 access-list stateless https-acl-stateless
 access-list stateless dhcp-acl-stateless
 access-list stateless icmp-acl-stateless
 access-list stateless dns-acl-stateless
!
user-role guest-logon
!
user-role logon
 access-list stateless logon-control-stateless
!
user-role preauth
!
user-role stateful-dot1x
!
!

crypto ipsec transform-set default-boc-bm-transform esp-aes256 esp-sha-hmac
crypto ipsec transform-set default-rap-transform esp-aes256 esp-sha-hmac


mgmt-user admin root maskingthelonghexstringthatwashere


ntp server 192.168.10.52
ntp server 192.168.10.51

firewall disable-stateful-h323-processing
!
ip domain lookup
!
aaa authentication mac "default"
!
aaa authentication dot1x "default"
!
aaa server-group "default"
 auth-server Internal
 set role condition role value-of
!
aaa profile "default"
!
aaa authentication captive-portal "default"
!
aaa authentication vpn "default"
!
aaa authentication mgmt
!
aaa authentication wired
!
web-server
!
papi-security
!
aaa password-policy mgmt
!
traceoptions
!
probe-profile "default"
   protocol icmp
!
qos-profile "default"
!
policer-profile "default"
!
ip-profile
   route 0.0.0.0 0.0.0.0 192.168.0.2 0
!
lcd-menu
!
interface-profile ospf-profile "default"
   area 0.0.0.0
!
interface-profile pim-profile "default"
!
interface-profile igmp-profile "default"
!
stack-profile
!
ipv6-profile
!
activate-service-firmware
!
aruba-central
!
rogue-ap-containment
!
interface-profile switching-profile "default"
!
interface-profile switching-profile "TrunkPort"
   switchport-mode trunk
!
interface-profile switching-profile "vlan10"
   access-vlan 10
   native-vlan 10
!
interface-profile switching-profile "vlan1000"
   access-vlan 1000
   native-vlan 1000
!
interface-profile switching-profile "vlan11"
   access-vlan 11
   native-vlan 11
!
interface-profile switching-profile "vlan13"
   access-vlan 13
   native-vlan 13
!
interface-profile switching-profile "vlan15"
   access-vlan 15
   native-vlan 15
!
interface-profile switching-profile "vlan9"
   access-vlan 9
   native-vlan 9
!
interface-profile tunneled-node-profile "default"
!
interface-profile poe-profile "default"
!
interface-profile poe-profile "poe-factory-initial"
   enable
!
interface-profile enet-link-profile "default"
!
interface-profile lacp-profile "FreeNAS"
   group-id 0
   mode active
!
interface-profile lldp-profile "default"
!
interface-profile lldp-profile "lldp-factory-initial"
   lldp transmit
   lldp receive
!
interface-profile mstp-profile "default"
!
interface-profile pvst-port-profile "default"
!
interface-profile dhcp-relay-profile "MainDHCP"
   helper-address 192.168.10.52
!
vlan-profile dhcp-snooping-profile "default"
!
vlan-profile mld-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "igmp-snooping-factory-initial"
!
spanning-tree
   mode mstp
!
gvrp
!
mstp
!
lacp
!
vlan "1"
   igmp-snooping-profile "igmp-snooping-factory-initial"
!
vlan "9"
   description "VLAN0009"
!
vlan "10"
   description "VLAN0010"
!
vlan "11"
   description "VLAN0011"
!
vlan "13"
   description "VLAN0013"
!
vlan "15"
   description "VLAN0015"
!
vlan "1000"
   description "VLAN1000"
!
interface gigabitethernet "0/0/0"
   description "LivingRoomPort1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/1"
   description "FrontPorchCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/2"
   description "OfficeWindowLeft1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/3"
   description "OfficeWindowLeft2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/4"
   description "OfficeWindowRight1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/5"
   description "OfficeWindowRight2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/6"
   description "VanessaOffice1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/7"
   description "GarageCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/8"
   description "FrontCornerCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/9"
   description "MasterBedroomPI"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/10"
   description "OldRasPi"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/11"
   description "MasterBedroom"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/12"
   description "vSphere Mgmt"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/13"
   description "TempUPS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/14"
   description "Physical Domain Controller"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/24"
   description "FreeNAS iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/25"
   description "vSphere iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/26"
!
interface gigabitethernet "0/0/27"
!
interface gigabitethernet "0/0/28"
!
interface gigabitethernet "0/0/29"
   description "AppleTimeCapsule"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/30"
   description "PhilipsHueBridge"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/31"
   description "HDHomeRun"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/32"
!
interface gigabitethernet "0/0/33"
!
interface gigabitethernet "0/0/36"
!
interface gigabitethernet "0/0/37"
!
interface gigabitethernet "0/0/38"
!
interface gigabitethernet "0/0/39"
!
interface gigabitethernet "0/0/40"
!
interface gigabitethernet "0/0/41"
!
interface gigabitethernet "0/0/44"
   description "ArubaWiFi-2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/45"
   description "ComcastAdminConnection"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/46"
   description "ArubaWiFi-1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/47"
   description "InternetUplink"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/1/0"
   description "TrueNAS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/1/1"
!
interface gigabitethernet "0/1/2"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface gigabitethernet "0/1/3"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface vlan "10"
   ip address 192.168.10.1 255.255.255.0
!
interface vlan "1000"
   ip address 192.168.0.1 255.255.255.0
!
interface vlan "11"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.11.1 255.255.255.0
!
interface vlan "13"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.13.1 255.255.255.0
!
interface vlan "15"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.15.1 255.255.255.0
!
interface vlan "9"
   ip address 192.168.9.1 255.255.255.0
!
interface mgmt
!
interface port-channel "0"
   switching-profile "vlan10"
   enet-link-profile pc_default
!
device-group ap
!
interface-group gigabitethernet "default"
   apply-to ALL
   lldp-profile "lldp-factory-initial"
   poe-profile "poe-factory-initial"
!

snmp-server view ALL oid-tree iso included
snmp-server group ALLPRIV v1 read ALL notify ALL
snmp-server group ALLPRIV v2c read ALL notify ALL
snmp-server group ALLPRIV v3 noauth read ALL notify ALL
snmp-server group AUTHPRIV v3 priv read ALL notify ALL
snmp-server group AUTHNOPRIV v3 auth read ALL notify ALL

snmp-server enable trap

process monitor log
end

PS, thanks for the link to Dia! I really hate trying to draw diagrams in paint :)

johnpoz

@jonthewise you have your lan on a 192.168/16 but your downstream networks are 192.168.x/24 networks?

Here is a good diagram for using a downstream router/L3 switch routing.

jonthewise

@johnpoz What I was trying to do was create a single route to my LAN, rather than routes to each VLAN... but it seems like that doesn't work. Unfortunately, if I switch the /24 routes on and the /16 route off, I can no longer access pfSense from inside my network, and everything on the 192.168.0.0 vlan no longer connects to the internet. I don't really want anything on the 192.168.0.0 vlan anyway though, I just moved them there to temporarily get internet access working until I could figure out what was wrong with the routing.

on Serverthehome, it was suggested that I had an overlapping route and if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16 route will work, just by changing the gateway on PFSense, and making the new PFSense LAN IP the default route on the Aruba, eg make the Aruba VLAN1000 10.0.0.1 and pfSense 10.0.0.2, set the default route to 10.0.0.2 on Aruba, and add a static route to 192.168.0.0/16 via 10.0.0.1 on pfSense.

I'm in a webinar right now, so I can't test just yet, but this seems plausible. I'm not entirely sure why I didn't have this issue on the Cisco router though...

Looking at the diagram you posted kinda confuses me. I don't want pfSense to deal with multiple VLANs, I want to use it as a router/firewall between my WAN connection (and eventually add failover to the LTE one once I get the first part working) and my LAN, which should be entirely routed on the Aruba.

I have some time in about an hour to try changing the transit stuff, and I will follow up afterwards.

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16

I would concur with this.. If you want to use summary route like 192.168/16, then using a transit network that is not in the 192.168 range would make sure your not overlapping.

You could also use some other summary route for your downstream that doesn't include your transit network.. For example if your transit was 192.168.0/24

Use could use some summary routes to include your other 192.168.x/24 networks vs specific /24 like 192.168.2/23

All depends what those downstream networks are and how you can summary it up.. But yeah easy would be to just use different range for your transit so you could use 192.168/16

jonthewise

@johnpoz Okay. Then something else that is confusing to me;

As seen in the screenshot, I created /24 routes for each of the VLANs that I wanted to have access to the internet, but when I turn those on, and turn the /16 off, I still can't reach the internet from inside my LAN, and it breaks access for all the things I moved to VLAN1000 to limp along. If I try creating a route on my pfSense router of 192.168.0.0/24 to point to 192.168.0.1 (the interface on Aruba) it throws the error seen here:

alt text

I think if I turn off the /16 and move all the stuff that was on VLAN15 and subsequently moved to VLAN1000 back on VLAN15, this would accomplish the same thing, but that didn't seem to work for me.

In a half hour I'm going to reload pfSense from scratch and start clean, but if I can't set a route on pfSense for the transit VLAN, I don't see how changing the subnet from 192.168.0.0 to anything else is going to make a difference?

jonthewise

Okay, well that absolutely did not work

On my Aruba, I changed the default route to 10.0.0.2/24 and made it's VLAN1000 interface ip 10.0.0.1

Reloaded a fresh pfSense install. Set the LAN to 10.0.0.2/24 with 10.0.0.1 gateway and created a route of 192.168.0.0/16 -> 10.0.0.1

Still no internet on my internal LAN networks... I'm typing this from a laptop that I manually put on the 10.0.0.0 network with 10.0.0.2 for the gateway and dns.

Ideas?

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

Still no internet on my internal LAN networks

And did you edit your rules on this 10.0.0.1 interface to allow for these downstream networks? Did you adjust unbound ACLs to include your downstream networks. The auto ACLs for unbound are normally only the directly attached networks.

Also if you had messed with auto outbound nat, switch to manual for example then the downstream networks would not be auto added to outbound nat.

jonthewise

@johnpoz Did I do what? Point me to some docs?

I reloaded pfSense, so I haven't messed with anything except what I literally spelled out above. I was under the impression for basic internet access this would just work... but obviously my setup is a little more complicated than just using pfSense as a router

jonthewise

@jonthewise holy shit! All I had to do was add a firewall rule?!

I wonder if this was my problem the whole time?

alt text

See, I assumed that the LAN net was just saying anything that came through that interface :/

Adding the 192.168.0.0/16 rule (copying the default lan rule and changing the address) seems to have resolved the problem!

So thank you so much for pushing me in the right direction. I really appreciate all the assistance!

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

All I had to do was add a firewall rule?!

Yeah - which was clearly spelled it in the picture I posted about setting up a downstream router ;)

How you had it before where your route overlapped with the connected network - while could work in a device that is designed or configured for that. Its not good idea to do it that way, because it can be confusing. If a router gets traffic destined for network xyz, and its directly attached to a xyz network, why would it send it to some downstream router to get there, when its directly attached to this xyz network..

Now in theory it could arp for this host IP, on its attached xyz network, and doesn't see it send it on via some route..

jonthewise

@johnpoz I guess I can see that it says that now that it's all highlighted and has a big red arrow pointing at it... It just wasn't clear to me (I really don't specialize in networking) that what it meant was that there needed to be a firewall rule in place to allow traffic. Here I was trying to figure out why I couldn't ping the gateway from my devices, and thinking it was a routing issue, when it was just being blocked by a firewall rule

How I originally had it set up would have worked because I had created /24 routes for each of the VLANs on my Aruba and tried turning off the /16 as a part of my troubleshooting, and had I figured out the darned firewall rule was missing, everything would have worked just fine with the transit VLAN living on 192.168.0.0/24 and all my other VLANs on their respective 192.168.X.X subnets, and there wouldn't have been any overlapping routes. That being said, I'm fine with having my transit on the 10.0.0.0/24 and using a 192.168.0.0/16 route to the LAN, and it also looks a little cleaner.

But regardless, chalk this one up to operator error. It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)

Thanks again. I'm sure I would have never figured this out on my own!

jonthewise

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)

That should say layer2, but when I try to edit it's flagging my post as spam and won't let me save