2x pfSense Routers, 1x ISP



  • Good morning,

    I have been trying to set up a second pfSense for testing purposes that would share my single ISP line. The lab environment would also be used for experimenting with pfSense itself, hence the need for a second pfSense rather than segregating networks behind just one. Any help is appreciated and thank you in advance, I am hoping to improve my understanding of pfSense so please do be blunt if what I am doing is impossible or inadvisable/poorly considered.

    ISP details are Zen (UK) fibre service, multiple public IPs, PPPoE, behind the ISP supplied BT Openreach modem.

    Attempt 1, individual PPPoE - Connect the ISP modem to a plain switch, connect pfSense WANs to switch.

    Both work individually but when the other is powered up the "2nd" WAN will not connect - either of them will connect but if one pfSense (doesn't matter which) has already established a link the other one will not. I googled whether it was possible to have multiple PPPoE connections on one line in general, and what I found seemed to say that it isn't impossible so I gave it a go, but without luck.

    Thinking it may be my ISP trying to assign the same WAN IP twice I tried following this post and forcing an IP to a PPPoE WAN interface -  https://forum.pfsense.org/index.php?topic=47296.msg248927#msg248927. Unfortunately I haven't been able to get this to work, the IP on WAN is always the one assigned by the ISP.

    Attempt 2, shared PPPoE - Share 1st pfSense WAN interface with 2nd pfSense WAN NIC.

    I think I had this working many many years ago, but with a different ISP that did not use PPPoE. Back then I would set up the WAN interfaces as static IPs and set the gateway on each as advised by the ISP. I don't recall exactly how but I may have bridged an OPT interface with WAN on the first pfSense and plugged this OPT directly to the WAN of the second pfSense, putting them both on the same network directly behind the bridged modem. However it was done, each WAN interface had its own public IP and behaved as if they had individual internet connections.

    With PPPoE the concept isn't quite the same, I think if I try and do the same thing now the 2nd WAN will be "in front of" the established PPPoE connection and I need to put my 2nd WAN "behind" it, somewhat like a virtual IP with a second/third/etc public IP on a single pfSense.

    Any suggestions very much appreciated, thank you.



  • @jonesr:

    Good morning,

    I have been trying to set up a second pfSense for testing purposes that would share my single ISP line. The lab environment would also be used for experimenting with pfSense itself, hence the need for a second pfSense rather than segregating networks behind just one. Any help is appreciated and thank you in advance, I am hoping to improve my understanding of pfSense so please do be blunt if what I am doing is impossible or inadvisable/poorly considered.

    You cant have two pppoe connections sharing 1 physical line AFAIK.

    Your options would be.
    1. Setup a basic firewall which handles the ppoe to your fibre then give your two pfsense firewalls a different RFC1918 ip address each, port forward your multiple IP addresses between the two pfsense boxes, they go in Firewall, Virtual IP's on pfsense (if you dont know and Zen should tell you which is the gateway ip address).
    2. Assuming your switch can handle vlans, set aside  two physical ports on your switch to plug your modem into alternatively, each port is a unique vlan and goes to one of your pfsense boxes wan interface. This way for testing you will need to swap the cable from your modem into the other port, the other pfsense you use for testing takes over and goes. This will probably be best if you intend to use your same multiple ip addresses on both pfsense boxes, and you want to test and build up firewall rules et al over time.
    3. If you switch doesnt support vlans and port tagging, you could do option two virtually by running pfsense on ESXi if your hardware is approved/supported, if not VMware workstation, WMware player or Oracles VirtualBox as some examples, and then just swap the virtual interfaces around if your switch doesnt support vlans. You can do some nice little tricks with virtualisation software but all your eggs are in one virtual box as the attack vector shifts to the virtualisation platform.

    When setting up pfsense, you can have a device with one nic and setup multiple vlans which are then assigned at the console/serial port to correspond with the vlans on your switch so you dont ever have to have the basic nic used for anything, just assign it off to a dead end vlan.

    Dont use Vlan 1 for anything, its just for setting up switches, put everything on its own separate vlan to isolate each device lanside as much as possible, use as little encryption lan side as possible and packet capture everything otherwise you can not account for any encrypted transmissions going over your lan unless you can brute force crack encryption.

    Put another way you build a big firewall fortress around your house, the postman delivers a parcel. If you break security at the firewall/fortress ie unwrap the parcel you can see if its a gift or bomb. If you let that parcel wrapped all the way up to your house like a secure connection on your webbrowser, only when that encrypted transmission is decrypted, ie parcel unwrapped at your house do you find out if its a gift or bomb ready to exploit a zero day in your web browser. Its a big risk allowing encrypted data all the way in past your firewall now a days when considering DuQu2 had three previously unknown zero days on windows alone and noone other than Kasperksy have found traces of it so far. Personally I think the USB bus is the main risk on both windows and linux/bsd even when drives are not mounted and would not be surprised to learn DuQu is exploiting this.

    Squid (as I found out last night) does SSLBump which will break encryption at the firewall so you might also be interested in that.

    Whilst its tempting for the convenience of it, dont use Radius servers et al as this just spreads the risk of your switch getting hacked from the switch onto the switch and firewall. Not saying pfsense can be hacked, but a zero day in freebsd basically increases the eggs in one basket thinking.

    The more you isolation and where possible use one pc/laptop (live CD's are useful with no hard drives) to configure each component in your network the better imo.

    hth.



  • Hi firewalluser, thank you for the suggestions.

    • I was hoping to avoid this firewall behind a firewall situation.

    • This is essentially how it is now, I have to choose which pfSense works by disconnecting the other. I don't think it makes any difference to your suggestion but to clarify, I won't be using the same public IPs on both, each pfSense will have IPs dedicated to them. As I won't be using the same IP twice, all the more reason to have both working simultaneously.

    • I do have the simple switch I mentioned, and another VLAN capable switch for my internal network. But either way it sounds like there is no simultaneous solution.

    When setting up pfsense, you can have a device with one nic and setup multiple vlans which are then assigned at the console/serial port to correspond with the vlans on your switch so you dont ever have to have the basic nic used for anything, just assign it off to a dead end vlan.

    Do you mean router-on-a-stick? I am familiar with this yes, but I wasn't clear on "just assign it (the basic NIC) off to a dead end vlan". Do you mean when flipping between the pfSenses reassign a WAN to a non-existent VLAN to "park" it rather than disable/delete it?

    Unfortunately my ISP tech support is 9-5 Mon-Fri, I am waiting to check regarding the possibility of multiple PPPoE sessions. This time tomorrow I hope to know more. Thanks again for the advice so far.



  • @jonesr:

    When setting up pfsense, you can have a device with one nic and setup multiple vlans which are then assigned at the console/serial port to correspond with the vlans on your switch so you dont ever have to have the basic nic used for anything, just assign it off to a dead end vlan.

    Do you mean router-on-a-stick?

    Sorry if I wasnt clear, what I mean is, in pfsense the nic will be known as say em0, em1, or rl0, rl1 etc etc, those are not using vlans in pfsense, but em0_vlan1000 is a vlan tagged as 1000 from the em0 nic in pfsense, so if you were not aware, you dont have to use the em0 nic for anything in pfsense, just in case you think you might have to use/assign em0 to an interface and thus to a vlan that you dont want to use.

    I am familiar with this yes, but I wasn't clear on "just assign it (the basic NIC) off to a dead end vlan". Do you mean when flipping between the pfSenses reassign a WAN to a non-existent VLAN to "park" it rather than disable/delete it?

    That is an alternative in case you wanted to see if anything comes out of it, and you have port mirror facilities in your switch to log it.

    Unfortunately my ISP tech support is 9-5 Mon-Fri, I am waiting to check regarding the possibility of multiple PPPoE sessions. This time tomorrow I hope to know more. Thanks again for the advice so far.

    I think Zen offer or used to some years ago bonded broadband connections which is different to this, but how would you put two pppoe connections in pfsense down one wan interface? The facility doesnt exist AFAIK so unless theres something which could be done via say the Virtual IP's this will be a new one on me but I've not had the pleasure of fibre yet.



  • I use Zen FTTC myself.

    You cannot have two simultaneous PPPoE sessions.

    You cannot get Internet access by putting another device on the FTTC bridge's Ethernet interface. If you want to put a router behind a router, IPv4 Internet access is only possible using NAT or, if you have a routed IP block, one or more of the public IP addresses from your router IP block.

    I do not believe Zen sell bonded FTTC. In any event, bonding (or multilink PPP) is about combining two WAN connections on one endpoint. You are looking at options for one WAN connection on two endpoints.

    If your connection is on BT Wholesale WBMC backhaul it might be possible to set up an IGMP proxy and get access to the YouView test channel. This would only be an educational exercise, as Zen do not sell YouView.

    There are a couple of interesting things you can try.

    Native IPv6 is possible if you sign up to the Zen IPv6 trial. It's worth signing up anyway to get your account IPv6 enabled - it will continue to work normally even if you don't configure pfSense for IPv6 operation. I'll endeavour to write a guide on how to set up IPv6 on Zen later, as it is a little tricky to configure correctly in pfSense 2.2.4.

    MTU 1500 operation is possible if the Ethernet interface you use as the WAN is jumbo capable. This requires the RFC 4638 patch, which I have made significantly easier to install over the weekend. I'll be posting new instructions in the next couple of days. I hope RFC 4638 support will be built in to the forthcoming pfSense 2.3.



  • Thanks both for the further advice, David_W I called Zen earlier today, you are quite right I cannot have multiple PPPoE sessions.

    So I have gone to Attempt 3: Router Behind Router

    I do have VLANs on some interfaces, but for this pfSense to pfSense link I have a dedicated NIC for the outside-pfSense-WAN, outside-pfSense-OPT, and inside-pfSense-WAN. I have set up a VIP and 1:1 NAT for all inside-pfSense outgoing traffic to appear to the internet as coming from one of my public IPs (confirmed with whatismyip.com), and a firewall rule allowing "IPv4 , ,inside-pfSense-WAN,,,none, " in other words allow all traffic destined for the VIP to the inside-pfSense-WAN (confirmed by watching blocked traffic to port 80 in the inside-pfSense, my attempt from my mobile phone on its 4G network to reach a non existent webserver). I also have a rule on outside-pfSense-OPT to block all traffic to an alias that defines all of the outside-pfSense internal networks.

    So this appears so far in my very basic testing to work, but again I would very much appreciate any guidance/sanity check. Normally I could never consider the "allow all in" rule on the outside-pfSense but I hope if it is as described I am not doing anything foolish.

    David_W,

    If you want to put a router behind a router, IPv4 Internet access is only possible using NAT or, if you have a routed IP block, one or more of the public IP addresses from your router IP block.

    Would you mind explaining this in more detail, I wasn't quite sure what you meant - that sounds like what I was hoping to achieve with Attempt 2?

    Also thank you for suggesting the IPv6 trial, I think I registered my interest some time ago but forgot all about it so I have chased Zen on this. I look forward to any documentation you would be kind enough to provide.



  • Routed Block would be your IP address block assigned by Zen, one of those will probably be a gateway IP address (at least thats what BT do) and you've set them up in your Virtual IP's, but your gateway IP would need to be setup as a gateway. Zen should have told you which of the IP addresses is your gateway ip as you normally get 1 extra IP which is the gateway ip address for routing purposes.

    On the "allow all in" point, what device do you have plugged into the Fibre connection? Are you using a router or modem? I'm assuming modem (or router in modem only mode) as you are handling the pppoe and thus pfsense is logging into Zen for you, but correct me if I'm wrong.

    If using a modem (or router in modem only mode), then you can do what I do which is have a default block all rule, and then add specific rules which allow traffic out. I do this for all interfaces including a block all rule to each interface so no vlan can talk to each other. Its time consuming setting them up, so if you are using Resolver aka Outbound (which is default in 2.2.4) you'll need to create a rule [edit on your first pfsense to let the 2nd pfsense out] to let it talk to either the root DNS servers or using the DNS provided by Zen. If you only plan on using Zen's DNS you can lock the rule down to the IP addresses Zen uses for their DNS servers. You could create an Firewall, Alias, add one called Zen DNS, and then add their DNS IP addresses in there. Then in your rule to let DNS out, you would choose the Destination as the Zen DNS alias.

    It really depends on how much you want to lock down really, but I also log every rule this way off to a seperate syslog server (syslog-ng is probably best but you'll need to alter its defaults to accept long or wide syslog messages) but then you can spot any anomalies, like something unexpected trying to get net access of sorts, although not always possible if it decides to use the Resolver to look up a domain name, then looks up subdomains which only that DNS server would know about,. Either way only by logging everything including the time between certain syslog messages if you dont run freeBSD in debug mode, could on a single core cpu be an indicator of some extra code running on the firewall which you dont know about. Its harder to spot code injected into a device at the OS level with multi cores  as the OS decides what core the threads run on when started unless over ridden by the software itself, but certainly running an OS like FreeBSD/pfsense on a single core cpu will make everything right down to the memory registers run as clock work so you can spot zero days much much easier, especially if its on its own standalone device and not virtualised which is akin to having all your eggs in one basket then.

    Probably easiest thing to do, is create a default block rule, then keep checking the firewall logs for what device wants to get out and allow the rules that way. Its like realtime learning then as you become familiar with what your devices want.

    Also back up often, in full and by section, its easy to tweak configs in the XML backup file where need be, and to be 100% sure something has taken, clear the Diagnostics, States and if its still not working as expected, do a reboot. You'll get to know when a reboot is necessary and when its not, but its a good way to check if things would work normally after a power outage, like if you plan to use snort or suricata.

    Also worth having either the console display on if not using serial, as not all things pop up in the system logs, if you plan to use them for keeping an eye on things. In fact from a hacking a firewall perspective, considering how many rack servers exist with many not having a console plugged in, you can increase your chances of getting away with some hacks if they only pop up on the console and not in system logs or other logs.

    fwiw.



  • @firewalluser:

    Routed Block would be your IP address block assigned by Zen, one of those will probably be a gateway IP address (at least thats what BT do) and you've set them up in your Virtual IP's, but your gateway IP would need to be setup as a gateway. Zen should have told you which of the IP addresses is your gateway ip as you normally get 1 extra IP which is the gateway ip address for routing purposes.

    Zen use the address immediately below the broadcast address as the gateway address.

    I have a routed /28 from Zen: x.y.z.112/28, which is 16 addresses x.y.z.112 to x.y.z.127. Most Zen routed IPv4 blocks are /29, which are 8 addresses.

    The lowest address, x.y.z.112, is the network address. Theoretically this is unusable, but in practice you can normally get away with using the network address as a normal address.

    The highest address, x.y.z.127, is the broadcast address.

    x.y.z.126 is the Zen gateway address. You need to leave PPPoE on dynamic addressing as the IP address of the remote gateway changes according to which device at Zen terminates your connection. Zen will always allocate your gateway address to your end via PPPoE.

    The gateway address is the address of the WAN interface of my pfSense box, which I use for VPNs. I also use this address for NAT for any device that doesn't have a dedicated address.

    I use the /28 as two /29s. The lower /29 is used for a routed IP DMZ. The remaining addresses in the upper /29 are used for 1:1 NAT to give certain devices a dedicated public IP address.



  • Thats pretty much how BT are, funnily I just gave them a call as I can now get fibre but their sales lady couldnt answer the question, put me through to technical and I go cut off. Not a good start considering I had some affiliate questions for them.



  • This is just a quick reply to again thank all for replying, and to say I will be spending some time on this at the weekend and hope to have news - Dave_W, I have also joined the Zen IPv6 trial and have seen your notes in a thread on the Zen forums, thank you for this too hopefully I will be able to get pfSense working as you have.


Log in to reply