Will pfSense Allow Me To Build A Dream Out Of This Network Nightmare??



  • Let me say up front that I apologize for the length of this post, but I wanted to give as much detail as possible..

    First a little background..

    (refer to this diagram http://sjfm.us/temp/network_topology2.jpg for specifics)

    I have a little computer shop at a local Flea Market. In addition to my shop's NET needs, I also provide WiFi Hotspots for the entire Flea Market and I admin 2 different IP based Security Surveillance systems, one for the Flea Market and one for my shop..

    There are three main subnets in play here..

    10.1.10.xxx
    192.168.20.xxx
    192.168.1.xxx

    As you can see by the above diagram, there are also 3 distinct LANs in play.

    The Shop's LAN (upper left, 192.168.20.xxx)
    The FM Office LAN (upper right, 10.1.10.xxx)
    The Grid (lower, a mix of 10.1.10.xxx, 192.168.1.xxx and 192.168.20.xxx subnets)

    Now, here is what I need to happen..

    The EXETER workstation (upper center) has 3 NICs in it and must have complete unfettered access to all 3 subnets AND to the Internet. That is my main workstation.

    The FM Wifi Hotspot grid (192.168.1.xxx) must be completely isolated and ONLY have Internet Access. It cannot be allowed access to the 192.168.20.xxx and 10.1.10.xxx subnets

    The YORKTOWN workstation is the Shop's Security Surveillance server. It's on the 192.168.20.xxx subnet, but it needs to have access to a couple of The GRID's .20.xxx IP Cameras.. The workstations on the .20.xxx subnet must have NET access and that's all that is required. I also have (not pictured in the diagram) a Linksys WRT54G running DD-WRT that provides Wifi access to the net from the shop. This has a DHCP running but causes some problems for other subnets. More detail on that later..

    Which brings us to…..

    The LEXINGTON workstation (sensing a pattern??  [​IMG]) is the FM Security Surveillance server and has access to the 10.1.10.xxx IP cameras from The GRID and has it's Internet access thru the FM Office Comcast Account.

    I have this setup and it does appear to work OK. The 192.168.1.xxx WiFi routers do give NET access to the masses, but sometimes (for no apparent reason) the DHCP server from the 192.168.20.XXX DD-WRT Linksys sometimes "gets in the way and gives out .20.xxx IPs to computers connecting that SHOULD have .1.xxx IPs. That DD-WRT router ALSO seems to give out it's IP (192.168.20.5) as the gateway for ALL connections. The gateways SHOULD be .20.1 and .1.1 for the associated subnets…

    So, basically I am left with a big mess that sometimes ALMOST works as required, but there are times (usually at the most inopportune moment) when the whole thing collapses..

    Now, I have been told that VLANs are where I need to be. All of this actually DID start out as a VLAN project..

    I put in a LINKSYS RV082 to act as the "train yard for all the various connections...

    I got some assistance from a tech friend in Chicago and he sent me an updated diagram on how HE would do things:

    http://sjfm.us/temp/Topo2.jpg

    My current setup is kind of an amalgamated version of the first diagram and the second diagram.

    The weird thing is, when I set things up that way, the ONLY way it would work is if I had everything on VLAN1...

    I am wondering if my best course of action is simply to pull everything down/off/out and start from scratch.. One of the biggest problems I noticed when working with VLANs is that, with the RV082, DHCP was not available except for the default LAN, in this case 192.168.1.1

    I am thinking I might replace the RV082 with a simple Linksys WRT54G running DD-WRT because I know that DD-WRT will allow VLANs with corresponding DHCP service..

    My biggest problem in all of this is that the networking I want to do is way above my pay grade. I know enough just to be dangerous as the current mess surely indicates.  :D

    If anyone has any words of wisdom (beyond sitting down and crying.. tried that. didna help..  [​IMG]) I would be immensely grateful..

    Michale



  • I'll take the first stab at this.  First of all, the dumb switches you have are not going to help you with VLANing.  I'd do one of two things with them.

    1:  Toss them and buy manageable switches.  For small offices, I prefer used HP ProCurve equipment.  You haven't really told us how many ports you're needing, so I guess what you purchase is up to you.  You can get some nice 1810-24G switches used for $150 + S&H

    2.  Keep them and still obtain more used manageable switches from the link above (or where you'd prefer).  The "dumb" switches should be used "downstream" of the managed switches.  With the managed switches, you'll be able to create VLANs.  Then if you need more ports on a specific VLAN, take the dumb switch and throw it on the port that has the VLAN in question.  Set the managed switch port to "UNTAGGED" (So it doesn't add a VLAN tag to the packet) and the dumb switch will see it as just another switch port.  This would probably be good for your security cameras.

    Also, which devices have static IPs?  Do any of them?  Can we change IPs to something a bit more sane?  For my solution to work, we will have to change the IPs.  Do both SMC Cable Modems have multiple ports?

    To me it looks like you're thinking in terms of "types of devices" (cameras, grids??).  You need to start thinking in terms of VLANs/Subnets and function.  Then, this should become much easier.  I'm just going to put in a list of what I think the current setup is:

    FleaMarket_Pub_WiFi:  192.168.1.x  DIRECT internet access (How Many APs????)
    ShopSecServer(YorkTown):  192.168.20.whatever
    ShopWorkStations:  Also in 192.168.20.x
    ShopSecCams:192.168.20.x
    FleaMarket_Cameras:  10.1.10.x
    FleaMarket_SecServer: 10.1.10.15
    Exeter: Must have access to everything except FleaMarket_Pub_WiFi?

    If I have this right, it sounds like you want each subnet to only have access to itself and the internet and not other subnets.  Is that correct?  The only machine that should be able to talk to other subnets is EXETER?  You can get all fancy dancy with the subnets and VLANs and VLAN tags etc.  Or just make it relatively simple.

    My Scenario (requires HP ProCurve Manageable Switch):

    I'd get a pfSense box with at least 3 (INTEL ONLY!!!) NIC cards:

    pfSense NIC0 (DHCP) ->  FleaMarket_Comcast_Port0
    pfSnese NIC1 (DHCP) ->  Shop_Comcast_Port0
    pfSense NIC2 (VLAN1 192.168.20.1/24) ->  HP Switch_Port0 Tagged VLAN1
                          (VLAN2 192.168.21.1/24) -> HP Switch Port0 Tagged VLAN2
    FleaMarket_Pub_WiFi -> FleaMarket_Comcast_Port1 (192.168.1.x/24)

    VLAN SETUP:

    VLAN1:  Shop_LAN (192.168.20.x/24)
    VLAN2:  FleaMarket_LAN (192.168.21.x/24)

    On the switch, you configure 2 VLANs and assign ports as needed and untagged.  You can also set which port can be in which VLAN and a single port can be in multiple VLANs.  This is the key.  pfSense NIC2 will actually be split into 2 VLANs.  This means it has 2 VLAN interfaces that tie to NIC2.

    Set up each VLAN to be DHCP starting at a higher IP Address.  Maybe 192.168.20.100 and 192.168.21.100.  Then you can set your security servers to static IPs in their respective subnet.  And workstations can all grab an IP from the DHCP pool in their subnet.

    The EXETER workstation can have a NIC in each VLAN.

    As far as limiting access from FM VLAN to Shop VLAN, that would be taken care of on the pfSense box.  I'll have to look up how to do that.  My first thought would be setting the default gateway for the VLANs to their respective Comcast modem interface.  There are probably some firewall rules that would help as well.

    This explanation was as long (if not longer) than your question!  I hope it helps a bit.

    joltman



  • Thanx for the reply..  I am still trying to wrap my head around it..

    I'll try and provide a bit more detail..

    The FM WiFi is a grid of 12 Linksys Routers spread out over the entire Flea Market

    http://sjfm.us/temp/WiFiGrid.jpg

    The goal here is to insure that anyone who connects to the DD-WRT hotspots can ONLY get out into the Internet.  That they cannot access ANY other user on the Grid or any workstation from my shop or the FM Office…  It would be a bonus if I could throttle those hotspot connections to specific settings.

    As you indicate, Exeter must have full access to all the subnets.  Cross connections between 192.168.20.xxx and 10.1.10.xxx is not a big issue..

    The important part is to insure that the WiFi users are completely isolated from 192.168.20.xxx and 10.1.10.xxx

    A couple other minor goals is to allow cameras from the 10.1.10.xxx grid to be able to be displayed  on the Yorktown (192.168.20.xxx) Surveillance server and to open a port thru the Shop SMC Gateway (192.168.1.1) to YORKTOWN.  As it is config'ed right now, the SMC won't let me open a port to a different subnet.

    Thanx again for the reply..  Hopefully I haven't confused things even more.  :D



  • I agree with Joltman here, the only thing I would add to the situation is do you have static IPs from Comcast? If you do you should apply those to the WAN port of your PfSense Router otherwise you are double natting which in most cases is not good. You could put your PfSense router in the DMZ of your SMCs if you don't have static IPs. If you are not sure if you have static IP I would check with Comcast you may have them and not even realize that you do.

    My second point would be why do you have two connections to Comcast? Because nothing in your setup nor anything that you articulated indicates that you want to do some type of policy based routing, is this just for speed? If that is the case you could just call Comcast and have them up your speed tier and that would probably be less money then renting two modems and paying  for two different services. (Make sure your modems are D.O.C.S.I.S 3.0 compliant)

    Thirdly having certain vlan communicate with others or having them isolated so that they can only communicate with the internet is relatively easy. The first step though is to make sure that everything is working and that everyone can get online and that everything can communicate with everything else. Once you do that, then you want to start applying your firewall rules to block traffic to the appropriate area. Aliases are a good way of doing this when making firewall rules which we will sure discuss in later communications.

    Lastly I would connect all my access point to the same switch (Managed if you can) that way you know where all your APs are connected. Later on when you want to grow your network you can make your SSIDs per VLAN with dd-wrt. When every you do a major overhaul like this you want to engineer for what you need now and what you might need in the future.

    I have a couple of those HP-Procurve Switches that joltman is talking about and I love them. I do prefer Cisco but to get the same port speed in Cisco would cost X times more money. If all you need is 100 Mbps then I would go with Cisco Catalyst 2950 (http://www.ebay.com/itm/CISCO-WS-C2950T-24-PORT-FAST-ETHERNET-CATALYST-SWITCH-GIGABIT-UPLINKS-MANAGED-/290984521010?pt=US_Network_Switches&hash=item43c0076d32)but remember you want to build for now and the future. If money is not a problem then I like Cisco Catalyst 3750Gs (http://www.ebay.com/itm/GOOD-CISCO-WS-C3750G-24TS-S1U-V03-Ethernet-Network-Switch-/390759948188?pt=US_Network_Switches&hash=item5afb1b9f9c)but you are looking at around $800-$1000 to do the same thing that the HP can do for a tenth of the price. Good luck, and depending on how you answer some of the questions above I will propose a network diagram that could fit your needs.

    P.S. How are all your access points connected? Are they all homeran back to some centralized point or are you using WDS? (Homerun is the best way to do that, in my opinion)



  • Hi Mike,

    Thanx for the reply…

    The reason there are two NET access is because the Flea Market has one for their office and I have a totally distinct and separate one for my shop...

    Other than LEXINGTON having NET access, the 10.1.10.xxx NET access is incidental to the whole project.  Sometimes when my line is having issues, I use their NET Access as "back up" to trouble shoot the problems.  But, other than that and Lexington, the office NET doesn't enter the picture..

    I do have a static IP from Comcast.  Right now, it's assigned to the RV082 and the SMC Gateway is set up in Bridge Mode..

    As to Firewall rules, that's where my lack of knowledge shines.  Right now everything (somewhat) works without the use of Firewall Rules.  I realize that not having rules is likely a big part of why it's not perfect...

    The Wifi HotSot grid is connected hardwire in serial with a single line coming into my shop and connected to the RV082..

    I can go into more detail later.  Being at a Flea Market, Sat/Sun are my busiest times.

    Thanx again for the input..

    Michale



  • I understand why you have 2 Internet Connections.  It makes sense as you are 2 distinct businesses.  It sounds to me like you have a business agreement in place where you admin the network for the FM.  I'm also assuming the FM knows you sometimes use their internet connection to do the odd troubleshooting.  However, we do need some questions answered in order to better assist.

    1.  Why do you need static IPs?  It sounds unnecessary given the info you have provided thus far.  Static IPs are really only useful if you are serving out websites/services to the internet.  If you aren't then you're spending money right now that you don't need to be spending.  Comcast charges more for static IPs on business accounts.  I'd call your Comcast rep (1-800-Comcast) to find out for sure.  Looking at your very first diagram, you say you have a static IP assigned to the Linksys R082 router.  The IP you show in the diagram is 192.168.1.2.  While you have probably statically assigned the "privately routable" address to the router (192.168.1.2) it doesn't appear you have any "publicly routable" static IP addresses on the Comcast SMC devices.

    Also, if you have the occasional need to get into the network from a remote location, then you still don't need static IPs.  I have a free account through afraid.org that gives me a Dynamic DNS name on the internet.  pfSense has a built-in Dynanic DNS software.  Whenever Comcast changes my IP address, pfSense sees the address has changed, and updates the new IP with Afraid.org and I can still access my home network (by name) from anywhere in the world that has internet.  Example:

    joltman.homenet.org (This is not my afraid.org address, but it's an example) resolves to 1.1.1.1

    If Comcast changes my address to 2.2.2.2, then pfSense will talk to afraid.org and update joltman.homenet.org to 2.2.2.2.

    2.  We do need to know how the DD-WRT routers are connected back to the "network closet".  Are they a direct CAT5e/6 line back to the closet for each AP, or are they using WDS like mikeisfly said?  I agree with mikeisfly here in that they should be "homerunned" back to the closet.  Why?  More bandwidth for each AP and to each user.  If they're in a wireless WDS config now, you're using part of the wireless bandwidth to talk to each AP and back to the "network closet".  It's an inefficient use of bandwidth.  If they're not homerunned, I would suggest running CAT6 to each location.  Why CAT6?  You would be able to run Power over Ethernet to each AP.  They make little RJ45 connector boxes that have power inputs/outputs on them.  They're called PoE Injectors.  You could power each AP from the closet.  This would be beneficial if you had a UPS in that closet.  Not necessary, but a super cool thing to have.  ;D

    3.  How many cameras are we talking?  They're all IP based cameras I take it?  Are they also PoE?  Are they WiFi cameras or wired?  Are they all homerunned back to the closet?  If WiFi are they using the same APs that run the FM_Public_WiFi?  If that's the case, same IP range?

    If you are homerunning all the APs and cameras back to the closet, then you'll need a switch that can handle the amount of APs.  I count 11 APs.  That's a lot.  It actually sounds like you'll need more than one 24 port switch.  Also, mikeisfly is correct.  Cisco products are very highly rated, but they're expensive.  Especially considering 2 small businesses.  I would stick with cheap, used, HP ProCurves.  Probably the 1810-24G series.  A few of them and you'll be happy.

    Since you want to throttle the connection to the FM_Public_WiFi, and you're not using any publicly routable static IPs on the Comcast SMC devices, I would recommend also purchasing 2 Motorola Surfboard SB6121 modems and using those as your modems in the FM and shop.  Why?  Because you're paying a monthly fee to Comcast for the SMC modems/routers and you're not doing anything requiring a static IP.  You can call Comcast to confirm, and tell them you'll be purchasing your own modems and that you'd like to return the SMC modems to them in exchange for lowering your monthly bill since you won't be renting the modems from them anymore.  I did some searching yesterday, and it does appear like you can do that.  But again, call them to be sure that they'll do it for you as well as lowering the monthly bill.

    I'm changing my design for your network just a bit.  Before, I said you could have the FM APs talk directly to the Comcast SMC modems.  Since I'm recommending to get rid of those, we'll have to add another VLAN/VLAN Interface to the pfSense box and consequently, another VLAN to the switches.  Not a big deal since the "inside" facing NIC2 on pfSense will already be VLAN'd.

    Since the FM_Public_WiFi will be on the pfSense box now, and you want to throttle those users, we're now looking at QoS (Quality of Service).  I have no experience with this on pfSense.  However, it most certainly can be done, and there is a whole subforum dedicated to QoS on pfSense.

    Before we get too far into design (drawing pictures), I'm hoping you can answer those questions for us.  I've got a mental picture right now, but we'll see how that changes with those answers.  Thanks!



  • With Comcast the only way to get a public IP on your PfSense is to have a static IP. If you call them up and tell them that you want a public IP on your PfSense firewall but you don't want to pay for a static IP they may come out and swap your SMC (IP gateway) with a standard cable modem. Traditionally they don't like to do that though.

    Sorry didn't read joltman's last comment fully before creating my post. I am in total agreement. Like I said above if you don't want to buy the modems and you call Comcast they can swap the IP gateways for a regular cable modem. Just let them know that the IP gateway will not work for your situation because you need public IPs but you don't need a static IP.



  • 1.  I don't really need a static IP at this point.  I do have future plans to assign a domain name to my IP. That's why I have it in place now.

    2.  I have updated the http://sjfm.us/temp/WiFiGrid.jpg graphic to reflect how things are wired.  A Cat5e wire brings the grid into my shop and is hooked to the Linksys RV082 router.

    3.  We're talking at least 40 cameras..  All static IP, all hardwired into the grid.  All but 3 or so are of the 10.1.10.xxx subnet.  The 3 (or so) are of the 192.168.20.xxx subnet.

    Hope this clarifies things a bit…

    Thanx again for the replies..

    Michale


Log in to reply