Any pfSense guru to help me choosing a setup for a small network ?



  • Hello all,

    I need the help from some networking gurus wandering in this forum :).

    Here is the story :

    My father runs a small company (4 people, 8 computers under Ubuntu 14.04 LTS), but they have no tech guy so I'm trying to give them a hand on setting up their network, even if I'm just an amateur when it comes to network issues (My big advantage is that my time doesn't cost them anything. And my father still believes that when it comes to computer, I know everything. I don't want to deceive him  ;) ).

    Their activities leads them to exchange large files with their clients (1 - 3 GB) on a daily basis.

    The main trouble is that they're located in the countryside, with some terribly weak internet links. At the moment, they have three DSL-lines, with about 10 Mbits/s downlink and 1 Mbits/s uplink.

    Their office main router is a Cisco RV325, which should be dual-WAN, but we have never been able to get it working . So on top of it, they have added a Peplink Balance 20, which works better regarding dual-WAN as long as you work with multi-threaded applications. The Peplink Balance is connected to the WAN1 port of the RV325, and the third DSL is on the WAN2 of the RV325. WAN1 is used to send and receive files, and WAN2 for their daily use of internet.

    But even with this complicated setup (sometimes it work, sometimes not), it takes at least 4 or 5 hours to send their files to their clients. And it's really a pain in the *ss when an upload fails, for any reason. They may lose an entire day of work, just because of file transfers taking to long.

    So I have been starting to think about a way to deport their work on a cloud server. The idea is that only this remote server will have to exchange files with their clients, taking advantage of its strong internet link. And that I am able to setup an efficient and secure link between their office and this server, so that they can manipulate their files remotely, without having to download/upload them, except in very specific situations.

    After a lot of reading on the web, I found that maybe pfSense could be the good solution for that. And this is where I need your input, at least to confirm that what I intend to suggest them is ok, before they spend any money on it.

    The setup I imagine for the moment is like that :

    • A local pfSense router which will be in charge of creating a secured multi-wan tunnel to their remote servers. It will also be the DHCP server for the local network.
    • A remote server running pfSense which will be the router & firewall for their remote network, and will connect them to the internet through a VPN. I would also like it to be their DNS server, and to do some internet caching that may speed up their traffic.
    • A remote server in charge of running their applications and storing their data in an encrypted volume.

    With this setup, what I imagine is that from the local network, they can connect to the remote server (in a secure and multi-wan way) with VNC, access their data with the decryption key stored on their local PC, manipulate them with remote applications. Whenever they need to upload or download new data, they just have to tell the server to do it.

    Well, I'm quite sure that I can setup something like that. But I am stuck on a few questions about security and performance :

    As far as they are concerned with security, having their files stored on the cloud, I would like that their outbound traffic goes through a VPN provider like AirVPN (so that the IP of their servers does not spread anywhere). If I install AirVPN on their local computers, that will drive to a VPN tunnel inside another VPN tunnel, and I'm pretty sure that performance will take a big hit. What would be the best is that my remote pfSense manages 3 tunnels : One with the local pfSense router, one with the remote server for apps and data, and one with the VPN provider.

    Is it possible to do that ?

    Another big question for me is that for some of their clients, they also must connect to their clients through SSH tunnels. At the moment, they just use putty on their local computer to open the tunnel. With my new setup, they would need to open the SSH tunnels on the remote server. That is starting to be too complicated for me about how to do that :).

    And the last question that I can't answer now is that sometime, they need to cut the VPN service. For example, when they connect to their banking service, they need to shutdown OpenVPN otherwise, the website doesn't let them in. What will happen if the VPN is on the router ? Is it possible to declare some routes for specific websites ?

    More generally speaking, do you think I am in the right direction, using pfSense as the heart of this setup ? Do you have any hardware / software / providers suggestions that I could look into ?

    I have to choose :

    • The local router. For the moment, I think about the SG-4860 (https://netgate.com/products/sg-4860.html), keeping the Cisco as local switch, and getting rid of the Peplink.
    • A hosting service in Europe. I saw that AWS has a pfSense AMI. I could go with that but for their applications, I need a dedicated server and AWS doesn't offer that. I think that would be better to have the pfSense server on the same network that the application server.
    • A VPN provider. I am currently on AirVPN, because they have a pfSense setup on their website.
    • The remote access software : VNC ? Is it secured enough (as long as the traffic will go through the VPN tunnel) ?
    • The encryption software.
    • Maybe some affordable support from a networking guru to set up all of that in a way that I understand so that I can help my father if he gets stuck

    Any thoughts about this setup, or suggestions about it (or another) will be greatly appreciated :)

    Thanks.



  • I would get on each side the following device for the WAN and LAN part, to get rid of any kind of fiddling
    and work around;

    • DrayTek Vigor 130 or any other pure modem that is supported by your ISP
      Alternatively it could also a used one from eBay matching your DSL standard
    • pfSense SG-4860 & mSATA & WIFi if needed
      Alternatively a self made appliance by Supermicro C2x58
    • Cisco SG200-10 as the LAN Switch
      Could also be refurbished ones, but managed!

    On the side with one or more Internet connections you could set up a proper multi-WAN
    solution with adequate load balancing using policy based routing. And the VPN tunnels
    could be created with IPsec to get the most out of that appliances and get benefit from
    the AES-NI support.

    Get rid of the Peplink and the other stuff in front of the WAN.



  • Hi BlueKobold

    Thanks a lot for sharing your thoughts.

    As I am not native english speaking, I'm not always sure I understand well. So I made a little diagram of what the network should be like (I've put only 2 WANs instead of 3):

    Did I understood well ?

    • At the moment, their modems are Zyxel ZY-VMG1312. I'm not sure they have the same abilities of the Draytek, but I know they can be bridged, and offers PPPoE connections (They have 2 different ISP, and one of them is in PPPoE. But that can be changed if needed).
    • The SG-4860 would come in place of the Peplink, and would be responsible for the wan bonding and the IPSec tunnel connection. It will also serve as DHCP server and DNS server (And internet cache if we add the mSATA SSD).
    • The Cisco SG200-10 would be the main switch for the network, replacing the Cisco RV325.

    Just to be sure, behing the RV325, they have 2 unmanaged switchs, mainly for distance reasons (D-Link PGS-1008P). Do they need to be replaced by managed switches too ? And regarding Wifi, they have AP connected to theses two D-Link switches.

    In the cloud, the pfSense server would be in charge of managing the IPSec tunnel, and connecting to the VPN service (Is it interesting to have some internet cache there too ?).

    Besides those questions, and as I have no experience with pfSense, do you think it is realistic to suggest them such an organisation ? That, once set up, it has any chance to work fluently without an internal tech-guy ?

    Thanks again for your help.


  • LAYER 8 Global Moderator

    Why do you want/need this pfsense in the cloud??

    That sure is a lot of moving parts.. All of which are failure points.. Why do you need to try and hide your IP like that?

    I think someones tinfoil hat is a size or 2 too small..  It seems to be way too freaking tight ;)



  • I won't comment on the "tinfoil hat" bit, but I'll agree with johnpoz that on the cloud end, you probably don't need pfSense there. You should be able to set up a basic firewall on the server itself, and set up an IPSEC tunnel between the server in the cloud and pfSense in the office.

    The only way I can see you needing pfSense on the cloud end is if there are multiple servers that you're trying to protect with a single firewall, rather than each server running its own firewall (and if you're going to do this, there would need to be some private networking between the servers, so they're all on the same private network address range). But if you're only using one server for the purpose of sharing files, then keep it simple and use the operating system firewall and IPSEC/VPN capabilities.



  • To be clear, I am not sure of anything at the moment. That's why I hope to be enlighted with this forum knowledge :)

    What leads me to think that I need a pfSense in the cloud :

    1. WAN bonding : I've read a lot about WAN bonding (but I did not experiment a lot myself), and WAN bonding is the very first reason why we entered this whole thinking, because they are really in need of better performance. At the moment, the best results we got in WAN bonding is with the Peplink balance. This is not a real bonding, the 3 WANs have different IP, and most of the applications they use will go through only 1 WAN. But they can send or receive 3 files at the same time. And when speed is really needed, some applications which can multithread file transfers (like BitKinex) can use the 3 wans at the same time.

    I have talked with some Peplink guys and they told me about Speedfusion bandwidth bonding, which does some packet-level bonding. But to benefit from that, you have to have the same routers at each ends of the link. We cannot do that, but it leads me to think that having the same technology on each side would give better performance. I thought that 2 multiWAN pfsense would be able to create a unique tunnel between them. Maybe I am completely wrong on that point.

    2. The need of security : I suggest to move their entire work in the cloud, and they are not really at ease with that (and me neither). Their local computers would mostly act as "remote controllers" to launch commands on the remote servers. If I only put a local firewall, it does protect nothing, as far as their applications and data would be on the remote servers. So it's in the cloud that the firewalling is needed (in my understanding).

    IF you have any better suggestion, without the cloud pfsense (or the local pfsense), I would be happy to hear about it. Because at the moment, I also feels that this too complicated, especially with my very basic knowledge about networking. But I don't see any other option.

    Anyway, thanks for sharing your thoughts.



  • @virgiliomi : Thanks for your message. We haven't decided how many servers will be in the cloud. We are trying to lower costs, because this is a company launched by 4 retired people just to get a better pension, they are not into investing in tech :). But I'm pretty sure that yes, we will need at least 2 servers in the cloud to cover their CPU needs.

    As far as pfsense doesn't require a high-end CPU, I thought that installing it on a small dedicated server in front of the big servers would be a better protection (and see my point about WAN bonding too) and less maintenance (as they don't need to log on the firewall, I wouldn't fear they break everything :) ).


  • LAYER 8 Global Moderator

    "But they can send or receive 3 files at the same time"

    Huh??  So what are these links you currently have. What are speeds?

    The best and easiest solution when your pipe is not fat enough is to get a fatter pipe ;)  Adding overhead of vpn tunnels be it ipsec or openvpn does not make for better performance ;)

    Here is a cool tool for figuring out the overhad of a ipsec based upon the different factors that would make your tunnel, if NAT-T, your encryption used the integrity used both on esp and ah, etc.

    https://cway.cisco.com/tools/ipsec-overhead-calc/

    If what your looking to do is maximize the use of the pipe, putting it inside a vpn would not be first choice that is for sure.

    As to sending or getting 3 files at a time..  Not sure where you go that idea that you can only do 1 thing in a connection at a time?



  • @Johnpoz

    As I told in my first message, they have 3 DSL lines with almost the same characteristics : 10 Mbits/s down, and 1 Mbits/s up. They are installed in an old farm in the countryside, 4 kms away from the first village, and have no other options for their internet access (We thought about satellite, but as far as they have huge volumes of data to exchange, the prices are way too high).

    I have heard about the overhead, and I know that 10+10+10 down will probably end up with 23 or 24 instead of 30. But this is anyway better than what they have today.

    I'm not sure to explain myself clearly, so I'll try to make it more clear :

    A client send them a file of X Gb. They download it, work on it, and then send it back to the client 2 or 3 days later. They have no control about how the client want the file back. Sometimes the client has a SFTP service. Sometimes it is just HTTP. Some clients just use WeTransfer services for example.

    If they have 3 clients for the same week, when sending back the files, they are able to use the 3 WANS at the same time for a total uploading bandwidth of about 3 Mb/s (Well, sometimes it works, sometimes not, but that's not the point)

    But if they have only 1 client on a given week with a big file (this is the most frequent case), and that this client offers only HTTP service, they can send the file back at a maximum speed of 1 Mb/s, because they can only use 1 WAN. Sometimes it takes 20+ hrs to send back the files. For their clients, this is one more day of delay.

    So if I can bond the 3 DSL, even if I don't get 3 Mb/s, but only 2,3 Mb/s, I can cut this delay by more than a half. If they finish their work at the evening, launch the upload during the night, and the client gets his file the next morning, they have gained one day.

    This was my first thought of how to accelerate their work.

    The second thought was that they may not be forced to D/L and U/L all of their files and that if I can put their applications in the cloud, on a server connected to a huge backbone, only the server would have to D/L and U/L these files. And that with a remote desktop viewer, they can manipulate these files the way they do it locally, and then just ask the server to send the file back.

    It won't work in all cases, but again, that is maybe another 50% saving on their bandwidth if half of the files can be worked this way.


  • LAYER 8 Global Moderator

    Seems like they are on the wrong place for such a business model if you ask me ;)

    Not trying to be a smartass or anything…  But come on.. Who runs a business in a location that requires movement of large files both up and down in a location where they can only get a 1mbps uplink connection??  Someone wasn't thinking ;)

    Here is what I would do.. If your not having any issues with the download using your 10+10+10.. Then stick the file on a usbstick and mail it too them :)  Or have someone drive to the closest place that has a real internet connection and upload the file..

    Sorry but if your talking 3mbps up max without any overhead even using the Full pipe GB / 3mbps = LONG TIME!! Faster cheaper easier to just mail them them file overnight..  A usbstick weights nothing, cost of express overnight them not very much..

    Or just move the business to where you can get a connection with speeds that actually viable for a business model where your uploading large files..

    Or sure your model of working on the files remotely with the server on cloud is good idea for moving large amounts of data..



  • At first I want to clear up some things here;

    • Peplink is offering devices and services that are acting likes MLPPP (MPLS) at the WAN Port.
      If you are using pfSense you can´t bond in real life that 3 WAN connections together without
      having them from only one ISP and that must be also offering you that MLPPP service too!
      Please note, that is not a can be or should be, that is a must be situation!

    • pfSense is offering such a MLPPP (MPLS) function too, likes also MirkoTik with their RouterOS
      but this must be also supported from the ISP, and Peplink is using for that their own devices
      to realize something likes such a service, its not the same we are talking here about!

    And so it might be sounding nice if peoples are speaking about bonding WAN interfaces together
    but in real 10 MBit/s + 10 MBit/s + 10 MBit/s = 10 MBit/s + 10 MBit/s + 10 MBit/s and not 30 MBit/s
    please don´t forget this! Only if your ISP is offering you such a Service and you get also matching
    devices for that services, you may get 10 MBit/s + 10 MBit/s + 10 MBit/s = 30 MBit/s!

    In this case here I would think multi WAN (3x) and policy based routing will be the solution.
    And for that you may not need any Cloud services and/or other things like shown in the picture.

    A single line with poor or low throughput is only able to speed up by the ISP and never on your
    (customer) side, please don´t think I am kidding you, but this question will be surely 100%
    returning and discussed very hard every month in other forums such as the administrator.de
    and/orthe  MikroTik.com forums, often more then two times a month!

    Get higher suited Internet connections or use load balancing over the three lines, perhaps
    you may ask in your region for a LTE 100 uplink and put it in that multi WAN situation?

    I have heard about the overhead, and I know that 10+10+10 down will probably end up with 23 or 24 instead of 30. But this is anyway better than what they have today.

    Sure it is likes it is, but then Peplink with their model is the best bet you could get or you ask for
    a really nice LTE 100 or 200 MBit/s link and put it in the game.




  • Hmm sorry, but just my two cents….  I may have totally misunderstood things, but if not...

    Besides network architecture.... why don't you just use something like rsync to sincronize the files?, it will just send the binary differences between the source and target files and you will surely gain time. Of course it depends on the changes made to the files, but I think they will probably be much less than the original file size.


  • LAYER 8 Global Moderator

    ^ hmm, that might actually be pretty good idea to be honest..

    You could pull down the file to a server in the cloud with real internet connection.  Then possible to bring it down to your location with rsync or something else that does diff.. Maybe something as simple to use as dropbox.  Modify the file, then with say dropbox it would only send up the differences.  Then the customer could either pull the file from your dropbox, or you could remote to that server and send the file to them, etc.

    Not sure how much your actually changing the file, or if your creating a completely new file based upon the contents of that file?  But if your just making changes to it the diff sync might be very good idea..  And save you loads of bandwidth and time.



  • maybe you can try this,  https://www.youtube.com/watch?v=tqbnjgbtDl0 is a different aproach for what you want, and probably can use a pfsense behind that router and change to a new topology.


  • LAYER 8 Global Moderator

    ^ sure there you go – that will for sure help ;) ROFL heheeheh


  • LAYER 8 Netgate

    With such crappy internet they should probably put everything into AWS and just RDP in from the farmhouse.

    It's kind of a push on cost whether you use AWS VPC IPsec VPN or a pfSense instance there and VPN to that.

    pfSense on AWS starts to shine when you need multiple VPNs into a VPC. Those costs start to multiply while IPsec to pfSense on AWS is a static cost depending on the instance size regardless of the number of tunnels. With pfSense there you can also do things like remote access VPN straight to your VPC.


  • LAYER 8 Netgate

    That iTel looks like a pretty good service if the use case is right.


Log in to reply