Use pfSense as drop-in replacement for Cisco/ZyXel (LAN no internet)



  • Our school is very impressed with pfSense so far and now we're trying to replace our old Cisco with pfSense.

    Our current old setup:

    WAN1, WAN2 (public IP, two lines) ->
      Cisco 2921 (192.168.0.10, NAT) ->
      Nortel 4584GT switch (192.168.0.40) -> (one fiber)
      Nortel 1624 core switch (192.168.0.1) -> (many fibers)
      different subnets… (192.168.2.x etc)

    Our goal is to use pfSense 2.1 as a drop-in replacement for Cisco:

    WAN1, WAN2 (public IP, two lines) ->
      pfSense (192.168.0.10, NAT, squid, filter, limiter, load-balance, failover) ->
      Nortel 4584GT switch (192.168.0.40) -> (one fiber)
      Nortel 1624 core switch (192.168.0.1) -> (many fibers)
      different subnets... (192.168.2.x etc)

    But after I set this up, our LAN does not have internet access. Here are the symptoms:

    1. pfSense has internet, can ping 0.x subnet but tracert 192.168.2.1 goes directly out into the internet
    2. 0.x subnet has no internet, but if I change a client's GW from 0.1 (core switch) to 0.10 (pfSense), then it has internet
    3. 2.x subnet has no internet, can ping 0.x subnet, but it can't ping pfSense (0.10)
    4. Cloning the Cisco's MAC address into pfSense does not help
    5. Adding/removing a GW 192.168.0.1 to the LAN side in pfSense does not help
    6. Allow all using rules or "pfctl -d" does not help
    7. Toggle on/off the block private/blogon network on my two GW's in pfSense does not help
      8 ) Changing the interface assignments (msk0, re0, re1) does not help (the NIC cards already work in transparent bridge mode)
    8. I think my default route is correct, it goes directly upstream to ISP GW

    I'm a programmer, so I hope this is not a stupid question and I also don't know why our current network has this setup. I also want to do as little change as possible because this is an established network and every router/swtich/etc has a mesh of wires, the only thing I can cleanly replace is the Cisco (it only has two WAN input and one output). Also, not shown is a ZyXel firewall that will be eliminated.

    I've goggled and gone through many posts. What should I try now?

    Thank you in advance for helping!!

    (PS: For the past year, we are successful in using pfSense as a transparent bridge limiter, it is working like a charm, so now we're getting more agressive and hope to replace the old Cisco with pfSense, we really need the firewall, filtering, caching, failover capabilities)



  • Sounds like 192.168.0.0/24 subnet is the one directly connected to pfSense LAN. You should let clients on that LAN get DHCP from pfSense and learn that pfSense is their gateway and DNS server. I guess there is a layer 3 switch in your network (192.168.0.1?) - stop that from giving the DHCP.
    Add a gateway for the layer3 switch. Do NOT actually set that as a "gateway" on the LAN interface page.
    Add static routes (System->Routing, Routes) to point to the layer3 switch for the other subnets that are behind the layer3 switch.
    Make sure firewall rules on LAN allow traffic from those subnets behind the layer3 switch.

    Post your full network topology, routes, rules if you need more help.



  • Yes, pfSense sits in the 0.0/24 subnet, we (the vendor long ago) put all core network equipment on the 0.0 subnet, the servers on another subnet, teachers/students on another etc. The important infrastructure subnets are all static (no DHCP), only the client/teacher/student subnets run DHCP.

    I'm hoping to just use pfSense as a drop-in replacement without modifying any other things, I'm hesitant to change other things because I don't have a complete understanding of the network. The tall rack is like a giant blinking spaghetti monster ready to swallow me any minute (very intimidating to a non-network admin).

    What makes it so dangerous is that I know enough to modify the system, but I don't know enough to completely know what is going, I'd easily shoot myself in the foot, yikes! >_<


    I'm going to try your suggestions one of these nights (when everyone is gone) and report back.

    Thank you so much!!



  • different subnets probably means different VLANs.
    are they routed on the cisco or are they routed on the switches ?



  • Yes, the different subnets are on different VLANs, but I think they're done by the switches.

    The Cisco does not do VLAN, but I found that the Cisco config file has a bunch of routing lines that phil.davis alluded to. So I think those are what I'm missing, I'm going to try to find a quiet night to add static routes to pfSense and report back.

    Thank you heper!



  • The VLANs might be trunked from the switch to the Cisco, and so the Cisco (to become pfSense) LAN physical port might actually be carrying multiple VLANs - although if the Cisco config has route commands in it then probably the VLANs are not trunked to it.
    In any case, make sure you understand what layer2 (VLAN) and/or layer3 (routing) things the Cisco is doing before you disconnect it. And I would find the way to login to the smart switch/es and understand their config also. Then you should be able to draw a logical network map, and follow a few cables to make a physical network map. Then you might feel less intimidated and you can be the one who eats the spaghetti monster.



  • Bingo, our LAN now has internet! Thanks to phi.davis' instructions:

    1. In System>Routing>Gateways, add new a GW on the LAN for the core switch (192.168.0.1) but do Not set it as the GW on the LAN interface
    2. In System>Routing>Routes, add static routes for all the subnets, e.g. 192.168.3.0/24 and use the LAN GW above

    I think I'm just one step away from replacing the Cisco, but I can't seem to get NAT to work. The Cisco is also doing NAT for several public IP's.

    I've tried several permutations of setting 1:1 NAT, virtual IP and port forwarding. My current understanding is that I should do both 1:1 NAT and "IP Alias" virtual IP's. But when I try to view our school's website, I get the pfSense's login page. I went in the server room around 10 PM last night so I didn't get a lot of time to try many things before midnight struck.  I'll continue to google, read through the forums and try again one night.

    I can't wait to fire up pfSense and pump those good filtering, limiting, load-balancing, failover juices through our school's inter tubes. ^_^



  • I think you are talking about inbound NAT to a public server that you offer… and you can keep working on that.
    But that made me think about outbound NAT also. pfSense is not going to know to automatically NAT those other internal subnets when they go out to the internet on WAN - I expect traffic from those will currently be pumpe out to the internet with the private IP in the source IP, which is not very useful.
    I expect you will need to switch NAT Outbound to Manual and then add rules to NAT source 192.168.3.0/24 to WAN address...



  • Hi all, I finally got most of it to work after a lot of trial and error! A lot had to be done late at night while people are gone. >_<

    1. Setup virtual IP's:

    Firewall > Virtual IP > Add > Select "IP Alias" and enter the public static IP in "Address(es)" > Save

    1. Setup 1:1 NAT:

    Firewall > NAT > 1:1 > Add > Enter "External" and "Internal" IP's and enable "NAT Reflection" > Save

    1. Config system:

    System > Advanced > Firewall/NAT > Select "Enable (Pure NAT)" and check "Enable NAT Reflection for 1:1…" and "Enable automatic outbound NAT..." > Save

    1. Change "Default Allow LAN" rule:

    Firewall > Rules > LAN > Default Allow LAN > Add load balanced GW and change source from "LAN subnet" to "Any" (because I have many LAN subnets)

    1. Open ports on one WAN:

    Firewall > Rules > WAN1 > Add allow rules for HTTP, SSH etc. (I only need to do this on WAN1, because the virtual IP's can only be defined on one WAN)

    Lemme know if I'm doing something wrong.

    Everything seems to be working now, so I've unplugged the Cisco and Zyxel. Yay!

    (Strangely, my download speed is WAN1 + WAN2 occasionally on speedtest.net, both WAN's fire up in one test)


Log in to reply