General network design and pfSense setup



  • Hi,

    I'd like some advice on my network and some help with pfSense setup.

    I have a Dell PowerEdge R510 which is running XenServer - For now this will be the only physical server

    I also have a Dell PowerConnect 2848 which will be my main switch.

    On the server I plan to run:
    -pfSense as my main and only router
    -A pair of production web / database servers (Both Windows and Linux)
    -A pair of development web / database servers (Both Windows and Linux)
    -Active directory / DNS
    -A NAS server
    -Some game servers

    I want to divide my network up into a few areas controlling access. Rather than using subnets to control access I plan on using physical ports and VLANs.

    Here's my main groups - Keep in mind that some groups in terms of access are all grouped together, these groups show the IP address ranges for each type:

    Here are some of the IPs which have been assigned to various devices:

    The idea is that there are five main network areas:
    -The internet
    -DMZ
    -Inter-Server
    -LAN
    -Internet-only LAN (If possible I'd like the devices to not even be able to see each other)

    Here's what I intend on doing inside the server:

    (The internet-only network will also go out of the LAN interface via VLANs)

    I'd like to setup the pfSense router to allow access to the DMZ from the LAN, so that I may use FTP on and look at the web servers but that would be subject to all of the normal LAN firewall rules - I'd probably add another interface to the AD server which connects to my LAN for local access to that.

    Is this design any good and how could it be improved ?

    I've also tried doing part of this, right now just the LAN and DMZ with a Ubuntu machine on the DMZ. I'm currently using the router to access the internet from the LAN and that's working great - But the Ubuntu machine has no connectivity to the router - I can't even ping 192.168.3.1. If I bridge the LAN and DMZ interfaces in pfSense, it connects fine so the problem is somewhere within pfSense.

    I've tried adding both firewall rules and NAT outbound rules (But as it's all on the same subnet, the default NAT rules seem to allow everything for it anyway) - Here's my Firewall rules for the DMZ interface and the interface setup - As you can see I tried adding a rule to allow everything but it didn't work:

    Thanks,
    -Andrew.



  • I noticed your DMZ interface setting had a 192.168.3.0/16 instead of a /24 like you had outlined above in your ip table. Which are you using?



  • Oh sorry that detail was in the Excel document but I accidentially cut it off when I took the screenshots.

    I'm using 192.168.0.0/16 - Everything's on the same subnet.

    Am I looking at subnets the wrong way - Should I be using /24 ?



  • It seems like you're using an over-complicated setup for a small environment.  I wouldn't allocate an entire /24 address space if you don't intend to use it, especially when you're only using 4 out of the entire 254 addresses.  Is your main intention to segregate everything from one another?



  • The list of devices is not exhaustive, but yes you're right I wouldn't be using all 254 addresses.

    The segregation in terms of what other devices each device can access would be done through the port / interface / VLAN it's connected to - The IP addresses are simply there to provide logical grouping for me, or at least that was the idea.

    I was going to space each address out by 10, hence why they are all .10, .20, .30 etc - The reason for this is in case a device has more than one interface connected to the same network. An example of this is the Server's Dell management interfaces. This means up to 25 devices per group.



  • Just a few more questions:

    1. Do all the clients/servers/management need access to the whole network?
    2. Do all the clients/servers/management need access to the internet?

    Personal suggestion would be to segregate through vlaning, the management portion of the dell server in a different vlan than anything else and use /24 addresses instead of a /16

    example of a customer setup was setup with vlan'ing as followed:

    Everything was trunked to pfsense through one ethernet adapter. Each subnet was vlan'd via switch.

    Lan                - all clients and access to the server they need           [192.168.100/24]
    admin network - anything that needed administrative network             [192.168.101/24]
    wireless          - guest wireless segregated from local lan                   [192.168.102/24]
    cam system     - security system/ w remote viewing                         [192.168.103/29]
    ext servers      - web server, email, etc                                          [192.168.104/28]

    all access through network was controlled through the pfsense firewall rules, time rules, etc.  Internal access was granted to certain clients to certain resources on the internal network.



    1. No, any internet-only devices (Phones etc) don't need access to anything else. - I'l like every machine on the main LAN to have full access though.
    2. Yes, even the management network as some of the interfaces can download firmware from the internet.

    Ah I see, so using the subnet to segregate the network and having each subnet as a different level / group of access rather than just a logical grouping ?

    The only issue I can see is that I wanted DHCP ranges - I could use /23 and the last bit of the third block could distinguish between static and DHCP, what do you think of that idea ?

    As for the groups I could use these which are a lot simpler than my initial design:
    -Infrastructure (Switches and Server Management)
    -DMZ (Internet side of servers)
    -Inter-Server (Private server inter-connect)
    -LAN (Main network)
    -Internet-Only (For simple internet devices)

    Could the fact that I wasn't using different subnets be the reason my router setup wasn't working ?

    Also how would the more private virtual servers be handled, such as the Active Directory server, this is only needed by the LAN devices - Should it be connected to the DMZ and access to it be controlled through the router or should it just be connected to the LAN instead ? This would mean that any servers which use it for access control would need to connect to it from the DMZ to the LAN which kind of defeats the point of the DMZ. I could always connect it to the LAN and the Inter-Server network, but would this also defeat the point of the DMZ ?

    (I've never setup an Active Directory server before, so please forgive me if I'm looking at it wrong.)



  • To answer your questions in order:
    @yngndrw:

    The only issue I can see is that I wanted DHCP ranges - I could use /23 and the last bit of the third block could distinguish between static and DHCP, what do you think of that idea ?

    I used the first 1-199 for static addresses, 200-245 as dynamic, then 246-253 as server allocation, 254 as the firewall/dhcp

    @yngndrw:

    Could the fact that I wasn't using different subnets be the reason my router setup wasn't working ?

    Probably, looks that the allocations weren't correct

    @yngndrw:

    Also how would the more private virtual servers be handled, such as the Active Directory server, this is only needed by the LAN devices - Should it be connected to the DMZ and access to it be controlled through the router or should it just be connected to the LAN instead ? This would mean that any servers which use it for access control would need to connect to it from the DMZ to the LAN which kind of defeats the point of the DMZ. I could always connect it to the LAN and the Inter-Server network, but would this also defeat the point of the DMZ ?

    (I've never setup an Active Directory server before, so please forgive me if I'm looking at it wrong.)

    Any device (access points, servers, printers, etc.) that are being used mainly on one network should be allocated to that network.  For example, your LAN AD server should be on the LAN. A web server though or email server though should be kept in a DMZ for security reasons so if those systems become compromised they're not going to normally access the LAN network.  If normal access (constant access) isn't required, then segregation is best through firewall rule sets.



  • Thank you very much for your help, I will try this kind of setup over the weekend when I have a bit more free time.



  • Hi,

    I've made some changes but am a little confused at the pfSense setup side of things.

    I'll list all of the changes I've made for completeness:

    1. Change to VMware - I wanted the tools and XenServer's tools don't support FreeBSD
    2. Changed my network to 192.168.*/23, where the star is a subnet which defines the access (I have Modem, Infrastructure, DMZ, Inter-Server, LAN, Internet-Only) - The 24th bit is 1 for DHCP or 0 for Static ranges, which allows both DHCP and Static devices to be on the same subnet as they should be
    3. I have no VLANs set up currently

    I have my VMware Management interface on the physical LAN network at the address 192.168.4.10 - My computer is on the same physical network but has the address 192.168.10.10 and the pfSense LAN DHCP server is at 192.168.10.1.

    As the VMware Management interface is on a different subnet, I cannot access it and need to use a router to connect the two subnets. This is where I'm stuck - Do both subnets need to be on different interfaces as currently they are both on the LAN interface.

    I will also have my switch on the Infrastructure subnet at 192.168.4.1, so I need access to the entire 192.168.4.* Infrastructure subnet from the entire 192.168.10.* LAN subnet. Later on I will sort out access control.

    Any ideas as to what I can do ? I've tried adding both NAT Outbound rules as well as Firewall Rules on the LAN interface with no luck, but I'm a bit clueless here.

    Thanks.



  • @yngndrw:

    I have my VMware Management interface on the physical LAN network at the address 192.168.4.10 - My computer is on the same physical network but has the address 192.168.10.10 and the pfSense LAN DHCP server is at 192.168.10.1.

    As the VMware Management interface is on a different subnet, I cannot access it and need to use a router to connect the two subnets. This is where I'm stuck - Do both subnets need to be on different interfaces as currently they are both on the LAN interface.

    I will also have my switch on the Infrastructure subnet at 192.168.4.1, so I need access to the entire 192.168.4.* Infrastructure subnet from the entire 192.168.10.* LAN subnet. Later on I will sort out access control.

    Any ideas as to what I can do ? I've tried adding both NAT Outbound rules as well as Firewall Rules on the LAN interface with no luck, but I'm a bit clueless here.

    It looks as if you're using the /16 subnet for everything again.  In order to have everything work together, since you're going to be segregating everything later, you're going to use the /16 subnet, then everything will be able to talk to each other. You need a router (aka pfsense) to work between subnets if not connecting everything together since you don't have a router elsewhere in the design or network.  It looks as if you just want everything to access everything else.  This can be achieved using your original plan of a /16.  There is no segregation in the network so if any client becomes infected, it now has complete control of whatever it can get into on the /16 network (which is why I suggested vlaning as above).



  • No I'm using /23 - The LAN stuff is on 192.168.10.* and the Infrastructure stuff is on 192.168.4.* which are both separate and segregated subnets. As they are both on different subnets, the router needs to route traffic between the subnets but this part I'm struggling with.

    I will use VLANs to lock down the access to physical switch ports, but thought it was best to take one step at a time and hence for now I'm not using VLANs.

    Am I misunderstanding things here ?



  • @yngndrw:

    No I'm using /23 - The LAN stuff is on 192.168.10.* and the Infrastructure stuff is on 192.168.4.* which are both separate and segregated subnets. As they are both on different subnets, the router needs to route traffic between the subnets but this part I'm struggling with.

    I will use VLANs to lock down the access to physical switch ports, but thought it was best to take one step at a time and hence for now I'm not using VLANs.

    The subnets are going to have to be vlan tagged and then added as interfaces on the pfsense itself.  You can add them under [Interfaces>assign>VLAN] then you have to add the interface on [interface assignments].  Since you're going to use the /24 addressing, all routing has to be done under the firewall rules tabs for each vlan name.



  • Ah okay thanks I'll try doing all of that tonight.

    I misunderstood routers - I thought they could route between two subnets on the same interface and didn't realise they had to be on separate interfaces.



  • On pfSense they need to be different interfaces, on a normal router though they don't need to be.



  • Ah I see, I wonder if it's a software limitation or a hardware limitation ? (I.e. If a normal router has slightly different network hardware which allows for this.)

    Anyway, I only have limited time each night on a week day so it might take me a few nights to learn about setting up VLANs and getting it to work - While I'm at it I will probably increase the VM's disk space as with 1GB it has 85% usage which leaves far too little head room for if there are any changes down the line.

    Thanks again for your help.



  • Sorry it took so long to get things moving, but I've just managed to split the subnets onto their own VLANs / interfaces and the firewall rules are now working. (Just a basic set of any to any rules for now.)

    I had an issue with my PowerConnect 2848 not sending VLAN traffic down the trunk with STP enabled so I had to disable that annoyingly. Another annoying thing about this switch is that you can't select the management VLAN, it's always on the default VLAN 1 - You also can't assign anything to this VLAN, it automatically assigns ports to it if the port isn't on another VLAN. Therefore to get access to it from my Infrastructure VLAN, I will have to have a short patch cable between two of the ports, one on each VLAN.

    Anyway I now have the following networks: (All are /23)

    • Modem - 192.168.2.* Static, 192.168.3.* DHCP - Separate physical network from everything else so no VLAN
    • Infrastructure - 192.168.4.* Static - VLAN 40
    • DMZ - 192.168.6.* Static - VLAN 60
    • Inter-Server - 192.168.8.* Static - VLAN 80 (Not used in pfSense)
    • LAN - 192.168.10.* Static, 192.168.11.* DHCP - VLAN 100
    • Internet-Only - 192.168.250.* Static, 192.168.251.* DHCP - VLAN 2500

    I was wondering how I'd sort out access control for the 'Infrastructure' network but then decided that every LAN device should have access. I will keep them using separate subnets / VLANs to keep the IP addresses split, but will allow full access via the firewall rules. It would also allow me to block the Infrastructure network from having internet access. (Or more specifically, allow me to block the internet from accessing it.)

    The main issue I have now with the design is my wireless access point. (Well actually there will be two, one for LAN and one for Internet-Only which will be WEP due to the Nintendo DS limitations.) I want all of my management pages to only be accessible from the Infrastructure network, but the access points will be connected to the LAN and Internet-Only networks respectively - How do I go about splitting the management part of these ?

    The next question is pfSense related - If I want one-way access between networks what do I do ? My main concern is that if I wanted access from LAN to DMZ but not from DMZ to LAN, I wouldn't be able to get responses to my from-LAN requests. Is this where states and the interface that you define firewall rules comes from ?

    My plan for the firewall rules are:
    Infrastructure has the default anti-lockout rule
    Infrastructure -> The Modem's subnet (?) - I.e. Internet access
    DMZ -> The Modem's subnet (?) - I.e. Internet access
    LAN -> All
    Internet-Only -> The Modem's subnet (?) - I.e. Internet access

    I would then have NAT port forwarding / 1:1 rules to allow access to the DMZ services. (Would I need firewall rules for these as well, or is that part automatic ?)

    Finally, for the Internet-Only network - I'd like it if the devices on that network could not talk to each other (I.e. They could only access the internet), is there a way to do this ? I was thinking that they would possibly need to be on a /32 subnet, but I'm not sure how they would then use the router as a gateway.



  • @yngndrw:

    If I want one-way access between networks what do I do ? My main concern is that if I wanted access from LAN to DMZ but not from DMZ to LAN, I wouldn't be able to get responses to my from-LAN requests.

    Firewall rules apply on input to pfSense. If a access attempt is allowed by a firewall rule then the rule effectively allows the "back traffic" as long the original access is active. Hence for the case you described, you can have a rule on the LAN interface allowing access to DMZ but not have a firewall rule on DMZ allowing access to LAN.

    @yngndrw:

    Finally, for the Internet-Only network - I'd like it if the devices on that network could not talk to each other (I.e. They could only access the internet), is there a way to do this ? I was thinking that they would possibly need to be on a /32 subnet, but I'm not sure how they would then use the router as a gateway.

    The only way you can really do this properly is to place each device on its own interface to pfSense (or its own VLAN which effectively becomes its own pfSense interface). The small number of VLAN capable switches I have encountered all support many more VLANs than they have ports.



  • Ah okay then thank you - The firewall rules system makes more sense to me now.

    I think I will leave the Internet-Only network as it is - I don't want to have to setup a pfSense interface for each restricted port and there would still be a problem of what happens over the wireless network. It's not really needed to have that level of isolation anyway.


Locked