pfSense replacing a Cisco Router - not acting as expected

jonthewise

Hey all. So I have just replaced my old Cisco 2921 with a little pfSense build on a desktop box with an added 4 port NIC card. All just GB connections, so nothing fancy there. I thought it would just be a case of remove and replace. That didn't seem to work though.

Firstly, my entire network lives on an Aruba S2500 with VLANs segregated off for various things, my home lab runs on VLAN10, IP Cams on VLAN11, my DMZ on VLAN13, and then the WiFi (Aruba access points, no fancy config - though I would like to be able to put different WiFi networks on different VLANs eventually, right now they're just sitting on access ports with no trunking) and desktops on VLAN15, and finally VLAN1000 (VLAN1 was already taken ;)) for the subnet that my router lives on.

I have a pretty straightforward subnet config, 192.168.(VLAN#).X/24 so lab on 192.168.10.X, etc. The router was at 192.168.0.2 (all of the .1 IPs are assigned to the Aruba VLAN interfaces) and the Aruba was set up with a default route of 192.168.0.2 to pass traffic to the router. DHCP was handled with an IP Helper pointing to my domain controller, and each subnet had the appropriate .1 gateway for that subnet configured. This all worked out swimmingly.

When I hooked up the pfSense box though, everything did not "just work." As a workaround, and to get the internet working again before the wife got home from work, I've basically turned on DHCP in pfSense, moved the WiFi access points, her PC's switch port, and my desktop to VLAN1000. First I had to add a LAN gateway for 192.168.0.0/24 pointing to 192.168.0.1 in pfSense to be able to even access the web console (before I moved everything to VLAN1000), and then I added a domain override to the DNS resolver for my local domain and an access list to allow 192.168.0.0/16 to do DNS lookups (though since the only thing connecting to pfSense is VLAN1000, don't know if the access list part was required) so that everything can still work (mainly the shared storage on my TrueNAS server for the windows file shares that my wife would be looking for).

Now I really want to get everything back onto the correct VLANs, but I'm not entirely sure where to start.

Right now, none of my lab systems can access the internet, even though the default route on the switch still points to the same IP (not sure if I mentioned that I just gave pfSense the same IP that the cisco was using) but nothing is routing. So I'd like to start by resolving that...

And once that is all done, I want to enable failover - maybe I'll start another thread for that after I get this working though - I have a Comcast internet gateway in bridge mode, and they gave me free Verizon LTE as a backup - I had the failover working on the Cisco using route maps, but I could never get failover and NAT working at the same time (so I had two configs that I had to switch between, basically making it a manual failover whenever I lose Comcast service) - I don't really expect incoming traffic from the LTE backup (though I do have Dynamic DNS, so it could maybe), but at least when it's on Comcast I like to have NAT into my DMZ to connect to my Horizon gateway and access my desktop remotely without having to change the config to make failover work.

heper

Requirements for solution:
Tldr version
Drawing of network
Screenshots of pfsense settings

bingo600

@jonthewise
I concur with the below.

@heper said in pfSense replacing a Cisco Router - not acting as expected:

Requirements for solution:
Tldr version
Drawing of network
Screenshots of pfsense settings

Maybe use dia for the drawing.
https://forum.netgate.com/topic/166945/free-network-diagram-drawing-tool-for-win-mac-or-linux/1

Re. Missing internet.
Is your Vlan1000 (I suppose that's the Modem ISP Lan/Vlan) ,
Connected to the pfSense WAN interface (untagged) ?

Is your pfSense connected PC's on Vlan15 also unable to connecto to the internet ?

Normally pfSense would only make "automatic" NAT rules for ip-nets configured as interfaces.
Meaning any nets not known via pfSense interfaces has to be added manually to NAT.
Use the HYBRID NAT mode

But please make a drawing now.

/Bingo

jonthewise

TL;DR;

I have a Layer3 switch with several VLANs that can communicate with each other. I had a dedicated VLAN for outgoing internet traffic connected to a Cisco router, which was then connected to two WAN devices (comcast in bridge mode, and an LTE network device they threw in for free with my business subscription) with very basic routing config (I had two configs set up for it, one was to failover to my LTE connection when Comcast went down, the other had NAT rules to forward 22,80,443 into the dmz VLAN - I could only get one or the other to work, which was part of the reason I was replacing it). I replaced that router with a pfSense box, and now internet no worky.

Longer version:

Basic Diagram
Comcast connected to WAN, Verizon connected to OPT1, Aruba connected to LAN. Two Trunked 10G Lines to ESX. One 10G line on VLAN10 to TrueNAS. Everything else is on gig access ports. No VLAN configuration in pfSense.
alt text

pfSense Routes: (note, I had tried creating individual /24 routes instead of the /16 to see if that was the problem, but it won't allow me to create a 192.168.0.0/24 because it says it conflicts with the current LAN IP address, and disable/enabling the rest of them doesn't make any difference. Disabling the /16 breaks the internet for everything I moved to VLAN1000)

alt text

My Aruba config:

#
# Configuration file for ArubaOS
# ArubaOS Version 7.4.1.12 72393
version 7.4
enable secret "*"
hostname "ArubaCore"
clock timezone PST -8
controller config 10
ip access-list eth validuserethacl
  permit any
!
netservice svc-dhcp udp 67 68
netservice svc-dns udp 53
netservice svc-ftp tcp 21
netservice svc-h323-tcp tcp 1720
netservice svc-h323-udp udp 1718 1719
netservice svc-http tcp 80
netservice svc-https tcp 443
netservice svc-icmp 1
netservice svc-kerberos udp 88
netservice svc-natt udp 4500
netservice svc-ntp udp 123
netservice svc-sip-tcp tcp 5060
netservice svc-sip-udp udp 5060
netservice svc-sips tcp 5061
netservice svc-smtp tcp 25
netservice svc-ssh tcp 22
netservice svc-telnet tcp 23
netservice svc-tftp udp 69
netservice svc-vocera udp 5002
ip access-list stateless allowall-stateless
  any any any  permit
!
ip access-list session cplogout
  user   alias controller svc-https  dst-nat 8081
!
ip access-list stateless cplogout-stateless
  user   alias controller sys-svc-https  dst-nat 8081
!
ip access-list stateless dhcp-acl-stateless
  any any svc-dhcp  permit
!
ip access-list stateless dns-acl-stateless
  any any svc-dns  permit
!
ip access-list stateless http-acl-stateless
  any any svc-http  permit
!
ip access-list stateless https-acl-stateless
  any any svc-https  permit
!
ip access-list stateless icmp-acl-stateless
  any any svc-icmp  permit
!
ip access-list stateless logon-control-stateless
  any any svc-icmp  permit
  any any svc-dns  permit
  any any svc-dhcp  permit
  any any svc-natt  permit
!
ip access-list session validuser
  network 169.254.0.0 255.255.0.0 any any  deny
  any any any  permit
!
user-role ap-role
!
user-role authenticated
 access-list stateless allowall-stateless
!
user-role denyall
!
user-role denydhcp
!
user-role guest
 access-list stateless http-acl-stateless
 access-list stateless https-acl-stateless
 access-list stateless dhcp-acl-stateless
 access-list stateless icmp-acl-stateless
 access-list stateless dns-acl-stateless
!
user-role guest-logon
!
user-role logon
 access-list stateless logon-control-stateless
!
user-role preauth
!
user-role stateful-dot1x
!
!

crypto ipsec transform-set default-boc-bm-transform esp-aes256 esp-sha-hmac
crypto ipsec transform-set default-rap-transform esp-aes256 esp-sha-hmac


mgmt-user admin root maskingthelonghexstringthatwashere


ntp server 192.168.10.52
ntp server 192.168.10.51

firewall disable-stateful-h323-processing
!
ip domain lookup
!
aaa authentication mac "default"
!
aaa authentication dot1x "default"
!
aaa server-group "default"
 auth-server Internal
 set role condition role value-of
!
aaa profile "default"
!
aaa authentication captive-portal "default"
!
aaa authentication vpn "default"
!
aaa authentication mgmt
!
aaa authentication wired
!
web-server
!
papi-security
!
aaa password-policy mgmt
!
traceoptions
!
probe-profile "default"
   protocol icmp
!
qos-profile "default"
!
policer-profile "default"
!
ip-profile
   route 0.0.0.0 0.0.0.0 192.168.0.2 0
!
lcd-menu
!
interface-profile ospf-profile "default"
   area 0.0.0.0
!
interface-profile pim-profile "default"
!
interface-profile igmp-profile "default"
!
stack-profile
!
ipv6-profile
!
activate-service-firmware
!
aruba-central
!
rogue-ap-containment
!
interface-profile switching-profile "default"
!
interface-profile switching-profile "TrunkPort"
   switchport-mode trunk
!
interface-profile switching-profile "vlan10"
   access-vlan 10
   native-vlan 10
!
interface-profile switching-profile "vlan1000"
   access-vlan 1000
   native-vlan 1000
!
interface-profile switching-profile "vlan11"
   access-vlan 11
   native-vlan 11
!
interface-profile switching-profile "vlan13"
   access-vlan 13
   native-vlan 13
!
interface-profile switching-profile "vlan15"
   access-vlan 15
   native-vlan 15
!
interface-profile switching-profile "vlan9"
   access-vlan 9
   native-vlan 9
!
interface-profile tunneled-node-profile "default"
!
interface-profile poe-profile "default"
!
interface-profile poe-profile "poe-factory-initial"
   enable
!
interface-profile enet-link-profile "default"
!
interface-profile lacp-profile "FreeNAS"
   group-id 0
   mode active
!
interface-profile lldp-profile "default"
!
interface-profile lldp-profile "lldp-factory-initial"
   lldp transmit
   lldp receive
!
interface-profile mstp-profile "default"
!
interface-profile pvst-port-profile "default"
!
interface-profile dhcp-relay-profile "MainDHCP"
   helper-address 192.168.10.52
!
vlan-profile dhcp-snooping-profile "default"
!
vlan-profile mld-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "default"
!
vlan-profile igmp-snooping-profile "igmp-snooping-factory-initial"
!
spanning-tree
   mode mstp
!
gvrp
!
mstp
!
lacp
!
vlan "1"
   igmp-snooping-profile "igmp-snooping-factory-initial"
!
vlan "9"
   description "VLAN0009"
!
vlan "10"
   description "VLAN0010"
!
vlan "11"
   description "VLAN0011"
!
vlan "13"
   description "VLAN0013"
!
vlan "15"
   description "VLAN0015"
!
vlan "1000"
   description "VLAN1000"
!
interface gigabitethernet "0/0/0"
   description "LivingRoomPort1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/1"
   description "FrontPorchCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/2"
   description "OfficeWindowLeft1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/3"
   description "OfficeWindowLeft2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/4"
   description "OfficeWindowRight1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/5"
   description "OfficeWindowRight2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/6"
   description "VanessaOffice1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/7"
   description "GarageCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/8"
   description "FrontCornerCam1"
   switching-profile "vlan11"
!
interface gigabitethernet "0/0/9"
   description "MasterBedroomPI"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/10"
   description "OldRasPi"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/11"
   description "MasterBedroom"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/12"
   description "vSphere Mgmt"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/13"
   description "TempUPS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/14"
   description "Physical Domain Controller"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/24"
   description "FreeNAS iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/25"
   description "vSphere iLo"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/26"
!
interface gigabitethernet "0/0/27"
!
interface gigabitethernet "0/0/28"
!
interface gigabitethernet "0/0/29"
   description "AppleTimeCapsule"
   switching-profile "vlan15"
!
interface gigabitethernet "0/0/30"
   description "PhilipsHueBridge"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/31"
   description "HDHomeRun"
   switching-profile "vlan10"
!
interface gigabitethernet "0/0/32"
!
interface gigabitethernet "0/0/33"
!
interface gigabitethernet "0/0/36"
!
interface gigabitethernet "0/0/37"
!
interface gigabitethernet "0/0/38"
!
interface gigabitethernet "0/0/39"
!
interface gigabitethernet "0/0/40"
!
interface gigabitethernet "0/0/41"
!
interface gigabitethernet "0/0/44"
   description "ArubaWiFi-2"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/45"
   description "ComcastAdminConnection"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/46"
   description "ArubaWiFi-1"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/0/47"
   description "InternetUplink"
   switching-profile "vlan1000"
!
interface gigabitethernet "0/1/0"
   description "TrueNAS"
   switching-profile "vlan10"
!
interface gigabitethernet "0/1/1"
!
interface gigabitethernet "0/1/2"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface gigabitethernet "0/1/3"
   description "vSphere Trunk Port"
   switching-profile "TrunkPort"
!
interface vlan "10"
   ip address 192.168.10.1 255.255.255.0
!
interface vlan "1000"
   ip address 192.168.0.1 255.255.255.0
!
interface vlan "11"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.11.1 255.255.255.0
!
interface vlan "13"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.13.1 255.255.255.0
!
interface vlan "15"
   dhcp-relay-profile "MainDHCP"
   ip address 192.168.15.1 255.255.255.0
!
interface vlan "9"
   ip address 192.168.9.1 255.255.255.0
!
interface mgmt
!
interface port-channel "0"
   switching-profile "vlan10"
   enet-link-profile pc_default
!
device-group ap
!
interface-group gigabitethernet "default"
   apply-to ALL
   lldp-profile "lldp-factory-initial"
   poe-profile "poe-factory-initial"
!

snmp-server view ALL oid-tree iso included
snmp-server group ALLPRIV v1 read ALL notify ALL
snmp-server group ALLPRIV v2c read ALL notify ALL
snmp-server group ALLPRIV v3 noauth read ALL notify ALL
snmp-server group AUTHPRIV v3 priv read ALL notify ALL
snmp-server group AUTHNOPRIV v3 auth read ALL notify ALL

snmp-server enable trap

process monitor log
end

PS, thanks for the link to Dia! I really hate trying to draw diagrams in paint :)

johnpoz

@jonthewise you have your lan on a 192.168/16 but your downstream networks are 192.168.x/24 networks?

Here is a good diagram for using a downstream router/L3 switch routing.

jonthewise

@johnpoz What I was trying to do was create a single route to my LAN, rather than routes to each VLAN... but it seems like that doesn't work. Unfortunately, if I switch the /24 routes on and the /16 route off, I can no longer access pfSense from inside my network, and everything on the 192.168.0.0 vlan no longer connects to the internet. I don't really want anything on the 192.168.0.0 vlan anyway though, I just moved them there to temporarily get internet access working until I could figure out what was wrong with the routing.

on Serverthehome, it was suggested that I had an overlapping route and if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16 route will work, just by changing the gateway on PFSense, and making the new PFSense LAN IP the default route on the Aruba, eg make the Aruba VLAN1000 10.0.0.1 and pfSense 10.0.0.2, set the default route to 10.0.0.2 on Aruba, and add a static route to 192.168.0.0/16 via 10.0.0.1 on pfSense.

I'm in a webinar right now, so I can't test just yet, but this seems plausible. I'm not entirely sure why I didn't have this issue on the Cisco router though...

Looking at the diagram you posted kinda confuses me. I don't want pfSense to deal with multiple VLANs, I want to use it as a router/firewall between my WAN connection (and eventually add failover to the LTE one once I get the first part working) and my LAN, which should be entirely routed on the Aruba.

I have some time in about an hour to try changing the transit stuff, and I will follow up afterwards.

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

if I move my transit vlan to a different subnet range such as 172.16.X.X or 10.X.X.X, then my /16

I would concur with this.. If you want to use summary route like 192.168/16, then using a transit network that is not in the 192.168 range would make sure your not overlapping.

You could also use some other summary route for your downstream that doesn't include your transit network.. For example if your transit was 192.168.0/24

Use could use some summary routes to include your other 192.168.x/24 networks vs specific /24 like 192.168.2/23

All depends what those downstream networks are and how you can summary it up.. But yeah easy would be to just use different range for your transit so you could use 192.168/16

jonthewise

@johnpoz Okay. Then something else that is confusing to me;

As seen in the screenshot, I created /24 routes for each of the VLANs that I wanted to have access to the internet, but when I turn those on, and turn the /16 off, I still can't reach the internet from inside my LAN, and it breaks access for all the things I moved to VLAN1000 to limp along. If I try creating a route on my pfSense router of 192.168.0.0/24 to point to 192.168.0.1 (the interface on Aruba) it throws the error seen here:

alt text

I think if I turn off the /16 and move all the stuff that was on VLAN15 and subsequently moved to VLAN1000 back on VLAN15, this would accomplish the same thing, but that didn't seem to work for me.

In a half hour I'm going to reload pfSense from scratch and start clean, but if I can't set a route on pfSense for the transit VLAN, I don't see how changing the subnet from 192.168.0.0 to anything else is going to make a difference?

jonthewise

Okay, well that absolutely did not work

On my Aruba, I changed the default route to 10.0.0.2/24 and made it's VLAN1000 interface ip 10.0.0.1

Reloaded a fresh pfSense install. Set the LAN to 10.0.0.2/24 with 10.0.0.1 gateway and created a route of 192.168.0.0/16 -> 10.0.0.1

Still no internet on my internal LAN networks... I'm typing this from a laptop that I manually put on the 10.0.0.0 network with 10.0.0.2 for the gateway and dns.

Ideas?

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

Still no internet on my internal LAN networks

And did you edit your rules on this 10.0.0.1 interface to allow for these downstream networks? Did you adjust unbound ACLs to include your downstream networks. The auto ACLs for unbound are normally only the directly attached networks.

Also if you had messed with auto outbound nat, switch to manual for example then the downstream networks would not be auto added to outbound nat.

jonthewise

@johnpoz Did I do what? Point me to some docs?

I reloaded pfSense, so I haven't messed with anything except what I literally spelled out above. I was under the impression for basic internet access this would just work... but obviously my setup is a little more complicated than just using pfSense as a router

jonthewise

@jonthewise holy shit! All I had to do was add a firewall rule?!

I wonder if this was my problem the whole time?

alt text

See, I assumed that the LAN net was just saying anything that came through that interface :/

Adding the 192.168.0.0/16 rule (copying the default lan rule and changing the address) seems to have resolved the problem!

So thank you so much for pushing me in the right direction. I really appreciate all the assistance!

johnpoz

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

All I had to do was add a firewall rule?!

Yeah - which was clearly spelled it in the picture I posted about setting up a downstream router ;)

How you had it before where your route overlapped with the connected network - while could work in a device that is designed or configured for that. Its not good idea to do it that way, because it can be confusing. If a router gets traffic destined for network xyz, and its directly attached to a xyz network, why would it send it to some downstream router to get there, when its directly attached to this xyz network..

Now in theory it could arp for this host IP, on its attached xyz network, and doesn't see it send it on via some route..

jonthewise

@johnpoz I guess I can see that it says that now that it's all highlighted and has a big red arrow pointing at it... It just wasn't clear to me (I really don't specialize in networking) that what it meant was that there needed to be a firewall rule in place to allow traffic. Here I was trying to figure out why I couldn't ping the gateway from my devices, and thinking it was a routing issue, when it was just being blocked by a firewall rule

How I originally had it set up would have worked because I had created /24 routes for each of the VLANs on my Aruba and tried turning off the /16 as a part of my troubleshooting, and had I figured out the darned firewall rule was missing, everything would have worked just fine with the transit VLAN living on 192.168.0.0/24 and all my other VLANs on their respective 192.168.X.X subnets, and there wouldn't have been any overlapping routes. That being said, I'm fine with having my transit on the 10.0.0.0/24 and using a 192.168.0.0/16 route to the LAN, and it also looks a little cleaner.

But regardless, chalk this one up to operator error. It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)

Thanks again. I'm sure I would have never figured this out on my own!

jonthewise

@jonthewise said in pfSense replacing a Cisco Router - not acting as expected:

It would seem most people that use pfSense either connect to a layer 1 network, or actually know something about networking (okay, I know a little bit, but mostly just enough to get myself into trouble LOL)

That should say layer2, but when I try to edit it's flagging my post as spam and won't let me save