Emulation of VSAT speed with pfSense
-
Hello dear community members. A few weeks ago I was charged with a task to find a way to limit our office Internet to a specific rack we are building with the bandwidth and latency of a slow satellite connection (VSAT). I was playing a lot and read tons of howto's and followed tutorials one after the other, however I had some issues that I could not resolve myself. I've been working as an IT for all my life, however I did never played with pfSense or used any other software as bandwidth limiting/shaping.
Let me give some more detailed information and what steps I took to make it work. Most likely I have made a simple logic mistake, however I am unable to find it.
The WAN interface of the box is behind our private network and have the following info:
IP: 192.168.9.112
Mask: 24
Gateway: 192.168.9.254The LAN interface of the box is set to communicate with the newly built rack and has the following details:
IP: 192.168.170.1
Mask: 24
Gateway: N/AThe VSAT speed I am trying to emulate is: Up: 3Mbps/Down 1Mbps with latency ~700ms.
The goal is to allow traffic from any LAN interface to Internet using the bandwidth and latency limiter as well as to allow connections to the new rack LAN from the WAN interface (which is our private network)
Following a few simple tutorials, I created two limiters in Firewall->Traffic Shaper->Limitters. One was called VSATLimitUP having bandwidth of 3Mbps and 350ms latency. The other was called VSATLimitDown having 1Mbps limit with latency 350 (the reason I put 350 instead of 700 was the fact that the ping reply became doubled prior to in/out communication so I split the desired latency in two and set the result to both limiters).
From there everything seemed to be simple, creating a rule in the firewall with the following settings:
Action: Pass
Interface: LAN
Protocol: any
Source: LAN Subnet
Destination: WAN Subnet
Description: LAN to WAN over VSAT
In = VSATLimitUP & Out = VSATLimitDownMoved the rule above all others and tested to ping. From here I had different results - once I had full LAN speed of <1ms to lan/wan or had a proper latency of 700ms to any IP (lan or wan). It is good if I have a ping to WAN with the latency, however to the LAN I should not be limited and should have <1ms to any device behind 192.168.170.0/24 network.
A similar WAN rule was created:
Action: Pass
Interface: WAN
Protocol: any
Source: WAN Subnet
Destination: LAN Subnet
Description: WAN to LAN over VSAT
In = VSATLimitDown & Out = VSATLimitUP (reversed order of the LAN rule as per most manuals and tutorials)As a result, I was able to get some traffic being limited, however it did not apply to the limits I set (3Mbps = ~250-300KBps and 1Mbps ~110-120KBps) I get around 50-60K of download speed and is not affected by any change in the limitters.
On the other hand, any attempt to access the LAN network from the WAN (192.168.9.X to 192.168.170.X) is blocked even though on the client remote windows machine I added a route: route add 192.168.170.0 mask 255.255.255.0 192.168.9.112
So far I am able to access the pfSense over the WAN (I added another rule from any to This Firewall) and ping it, but am unable to ping the second interface of the LAN (192.168.170.1) I feel like I am missing a rule to pass the traffic from 192.168.9.112 to 192.168.170.1 but tried to do that with no limitters and was not able to.
On top of that, looking at our network syslog I noticed that the box is trying to have connections to external network around midnight almost every minute. I went back to the firewall on port 53. I went back to the firewall and stopped any DNS services as well as NTP protocol thinking that the box would stop doing that. The next morning I discovered that there was still the same attempt to go out of the box to the same IP's. I thought that this might be the update attempt, but the box was already updated manually by me, so there is some connection that is still blocked and I need to stop this service as it is flooding the network and slowing down the ASA we have as a firewall.
So any shared thoughts about how to setup the limiters in my case and make the traffic goes both ways from LAN (192.168.170.0/24) to Internet and reverse, but at the same time not limiting the ping to any local LAN IP's is highly appreciated. I know there are tons of howto's out there, however most of them are reverse to my need - reducing the bandwidth and reducing the latency instead of increasing it.
Here are two of the best howto's I found so far and used as a guide but I was not able to fully accomplish the task:
http://www.squidworks.net/2012/08/pfsense-2-0-limiting-users-upload-and-download-speeds-by-limiting-bandwidth/
https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/Any info, any spotted mistakes or needed corrections would make my day ;)
-
WAN subnet is not the internet. It is the WAN interface subnet. Internet is usually dest any.
You need to make rules that set limiters on any interface connections are initiated from.
So if you are only testing from connections initiated by hosts on LAN, you probably want something like:
Action: Pass
Interface: LAN
Protocol: any
Source: LAN Subnet
Destination: WAN Subnet
Description: LAN to WAN Subnet
No limiters set. This is LAN to WAN subnet traffic and is local trafficFollowed by:
Action: Pass
Interface: LAN
Protocol: any
Source: LAN Subnet
Destination: any
Description: LAN to Internet over VSAT
In = VSATLimitUP & Out = VSATLimitDownIf connections are coming into WAN, like with port forwards or something, the easiest place to set the limiters are on the rules that are already there passing traffic. You might want to make another firewall rule above them that passes traffic from WAN subnet dest port forward target that does not set limiters if you are testing incoming connections from local hosts on WAN subnet.
-
Thank you very much Derelict for the detailed instructions. Most likely I am getting confused by the fact that the WAN interface for me is taken as the Internet source, however looking at your info seems like I am wrong as the WAN and LAN are two different private local networks where pfSense would act just as a bridge from the first network to the second.
Do you think that I should apply similar rules to allow access from the WAN interface to the LAN network (after the pfSense box)? To make my task even harder, I was instructed today to provide certain IP's with no limitations accessing the 192.168.170.0/24 network but the rest should be passed through the same VSAT limiters. The whole idea is to simulate and incoming and outgoing traffic over VSAT speed where the defined IP's would be used to manage some of the devices on the LAN net and need to have the full speed for that.
Knowing your info, I will give the box a try and let's see what is going to happen.
Btw, do you have any information about why a freshly installed pfSense box would have a minute interval attempts to connect to external networks?
Here is the log I got from our firewall:
–-------------------------------------------------------
192.168.9.254 Jan 10 11:08:25 Jan local6 warning 10 2017 11:08:19 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:202.12.27.33/53 to INSIDE_V109:192.168.9.112/13849; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254 Jan 10 11:08:25 Jan local6 warning 10 2017 11:08:19 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:217.149.76.228/53 to INSIDE_V111:192.168.11.10/57447; packet length 519 bytes exceeds configured limit of 512 bytes
192.168.9.254 Jan 10 11:08:25 Jan local6 warning 10 2017 11:08:19 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:202.12.27.33/53 to INSIDE_V109:192.168.9.112/37048; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254 Jan 10 11:08:26 Jan local6 warning 10 2017 11:08:20 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:192.58.128.30/53 to INSIDE_V109:192.168.9.112/25684; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254 Jan 10 11:08:26 Jan local6 warning 10 2017 11:08:20 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:192.58.128.30/53 to INSIDE_V109:192.168.9.112/19010; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254 Jan 10 11:08:26 Jan local6 warning 10 2017 11:08:20 %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:198.97.190.53/53 to INSIDE_V109:192.168.9.112/18967;As far as this box is intended to be used ONLY as a limiter to emulate the VSAT speed and no other services are required, I turned off the DNS services however it seems that I am missing still something and need to make sure that this connections stop as we have some important update tasks happening during the night hours and an eventual drops might overwhelm the network...
-
Those logs to me look like the ASA has a 512-byte limit on DNS responses which seems nonsensical to me. not sure what to say about those ASA logs.
Seems to me that if you had a better understanding about how pfSense rules applied on interfaces this would be pretty simple. it is not hard to make traffic sourced from one set of addresses be excluded from limiters, for instance.
Nothing between LAN and WAN is bridged. It is all routed.
It looked like you had a misunderstanding about what WAN network meant.
Look at the diagram in my signature:
LAN Network: 172.25.232.0/24
Host A: 172.25.232.100
WAN network: 172.25.228.0/24
WAN address: 172.25.228.5If you make a rule that passes traffic source 172.25.232.100 dest WAN network you are not passing traffic from Host A to the internet. You are passing traffic from source 172.25.232.100 to dest 172.25.228.0/24
In order to pass traffic to the internet you need to pass traffic from source 172.25.232.100 to destination any. The firewall will be looking at the source and destination addresses of the packets with no regard to the intermediate interface addresses.
Interface rules are evaluated from the top down and when a match is found processing stops and the traffic is blocked or passed based on that match. If there is no match, the traffic is blocked by the default deny rules.
So in your latest scenario you said this:
I was instructed today to provide certain IP's with no limitations accessing the 192.168.170.0/24 network but the rest should be passed through the same VSAT limiters.
That does not sound difficult to do but I need more information, like the source address/interface of the traffic in question. You might need to produce a diagram. Id doesn't need to be a Rembrandt but should include the pfSense interfaces and subnets and their subnets and where this VSAT limiter is placed.
As far as this box is intended to be used ONLY as a limiter to emulate the VSAT speed and no other services are required, I turned off the DNS services however it seems that I am missing still something and need to make sure that this connections stop as we have some important update tasks happening during the night hours and an eventual drops might overwhelm the network…
Nothing pfSense does by default should interrupt any normal network operations.
-
Thank you very much for your willing to help and assist Derelict. You can't be more certain that I am having some confusion about the way firewalls in general worked which would cause some confusion on the terminology used in pfSense. No offence to the devs and all the people using it, there is nothing wrong with the names or else, it is just me who needs to adapt and better understand the approach. I checked out your diagram and quickly draw one for our needs. I am sure, that a million words could not explain as well as a single diagram, so did my best to make it clear what is the structure, what is the goal etc.
So as per the diagram, VESSEL LAN and Office LAN need to communicate in both directions with the VSAT delay of 128Kbps/64 and latency of 700ms. Same thing to apply when the VESSEL LAN goes to Internet.
Hope this would make more sense. Meanwhile I had to reinstall the box as the hard drive I used failed so once I am back online I will try to implement the rules you suggested and see if my better understanding now would make everything work as intended..
Once again, thanks for finding some time to write the info and most of all - for the willing to share it as this is what makes a product good - its nice and sharing community :)
-
OK. I applied the rules as per your instructions adding the speeds that the management wants to have (128Kbps Down and 64Kbps up) and it seems to be working.
The issue left to deal with is the ping that is still showing 7-8ms to google (the download is limited though) and to allow access from the Office LAN (setup as WAN) to the VESSEL LAN.
I added a route on my computer for the test:
route add 192.168.170.0 mask 255.255.255.0 192.168.9.112
however that does not let the traffic through.
I also added a rule Any to LAN Net with no specific ports and no limiters and moved it to the top, but am still not able to communicate from Office LAN to Vessel LAN over pfSense. Something is still missing and looking at the logic - it is another rule on the box to allow the traffic.
From my computer (192.168.9.148) I am able to ping 192.168.9.112 but am not able to ping 192.168.170.1 (the second pfSense Interface). Shouldn't the rule Any to Vessel LAN allow it? (the rule has been saved and applied. I even tried to reboot the box but that did not help either).
-
Could the access be blocked due to the RFC1918 as both of my networks are private 192.168.x.x or I would have seen the rules in LAN/WAN?
-
I checked out this morning and the blockage of RFC1918 was not active. What questioned me was the fact that I can ping the WAN interface but not the LAN of the pfsense even though I added a rule from WAN Net to LAN Net as well as from IP to IP. On the same time I am able to ping from pfsense a host on the LAN and WAN network.
-
In that case wou want to be sure block RFC1918 is disabled on both LAN and WAN. I want to be 100% sure the following is clear:
LAN network = 192.168.170.0/24
WAN Network = 192.168.9.0/24
any = any including the internetAre you just trying to use pfSense to simulate the VSAT here or are you developing something that will be deployed using pfSense?
If the former, you should probably:
Disable NAT and place pass any any any rules on WAN to match LAN.
If the latter you want to keep WAN rules there.
Your diagram says: Limit both ways communication 128Kbps/64Kbps latency 700ms.
I presume that really means 128Kbps into WAN (download) and 64Kbps out WAN (upload)
You have lots of potential issues with that design. First is that I presume all of the hosts on 192.168.9.0/24 have 192.168.9.254 as their default gateway. That means 192.168.9.254 will receive all traffic destined for 192.168.170.0/24 and will have to hairpin it mack out the interface it arrived on and send it to 192.168.9.112. Not ideal.
Are you concerned only with connections originating from Vessel LAN out (web browsing, pinging google, etc) or are you also trying to limit connections going to Vessel LAN originating from the outside (The internet or Office LAN). That dictates where the rules go.
Why do they care about full-speed access from certain hosts on 192.168.170.0/24 - that is only complicating your testing with a requirement that cannot exist in the real world. They should figure out what they really want you to do. I would personally belay that request until everyone is limited, then bypass the limiters for certain traffic after it's all working how you want. Smack someone on the noggin there.
So, ignoring that requirement, I would make two limiters:
From_VSAT (which is LAN downloads)
Bandwidth: 128 Kbit/s
Mask: None
Delay: 350msTo_VSAT (which is LAN uploads)
Bandwidth: 64 Kbit/s
Mask: None
Delay: 350msConnections from LAN out:
Floating rules:
Action: Match
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: any
Source: any
Dest: any
In/Out pipe: To_VSAT/From_VSATConnections from WAN in:
Bandwidth: 64 Kbit/s
Mask: None
Delay: 350msConnections from LAN out:
Floating rules:
Action: Match
Interface: WAN
Direction: in
Address Family: IPv4
Protocol: any
Source: any
Dest: any
In/Out pipe: From_VSAT/To_VSATAt least I think that's what you want. I might have time to try that here today.
-
Looks like you need to set the gateway on the floating WAN out rule.
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=45 time=763.081 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=45 time=762.011 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=45 time=763.005 ms–- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 762.011/762.699/763.081/0.487 ms:)
-
Note also this is going to perform horribly compared to what you will actually experience on a VSAT link. They will put your traffic through performance enhancers that keep the TCP sessions on both sides happy so they don't think they are really being subjected to 800ms of latency.
Seems to be working here:
 -
Derelict, thank you hugely for the info. I was allowed today to order the definitive guide of pfSense which should be in my possession in the next few days. I hope the value of the book would support the development and the author.
A brief story of the reasons to need the emulations. We have on the company roof an actual VSAT antenna that was planned to be used for our needs to emulate a vessel that has the same identical hardware. Due to the winter season, the antenna was suspended. As far as we need to imitate the speed of VSAT to the rack, I decided to use pfSense of which I have only heard that can do the job easily and supports the functionality out of the box.
Now back to the networks. 192.168.170.1 presently the LAN interface of pfSense on the actual vessel is the VSAT connection with the defined speeds that is used for incoming and outgoing communication. While in the office and working as an IT, we need to connect to the vessels and fix ongoing issues or test/improve existing functionality. Because of the fact that we connect over the VSAT to the vessel, this is why I am trying to duplicate all the conditions just like we are connecting to a real ship. Achieving this would allow us to run some tests over a rack that is physically in the office instead of testing something in a real working environment where if the things go wrong, would affect the ship operations.
In other words, 192.168.170.X devices need to go to the Internet through 192.168.170.1 -> 192.168.9.112 and any 192.168.9.X interface to go to 192.168.170.X device through 192.168.9.112 -> 192.168.170.X
The reason to ask for a few IP's to have unlimited access is that sometimes to test new functionality, we need to be able to transfer large files which in a real environment can take days. In a scenario where the rack is near by us, we can copy the files and bring them on USB/HDD, however, if we allow 1-2 IP's access to 192.168.170.X to be passing through the real LAN (with no limiters) - the job is done perfectly. In a case, where you think this would confuse the situation, we can say that a communication 192.168.170.0 <–>192.168.9.112 <--> the Internet with the VSAT limit would work fine as we all will have an access to the firewall administrative interface so can temporary manipulate the box to have full LAN speed and once finished to return it to proper values.
I will try what you suggested first place tomorrow morning and inform how that goes. Once we figure it out, I am planning to write down a descriptive howto for our needs but will share it here, so that the next time somebody is looking for similar functionality/structure, knows where to find it.
Thanks again for your superb and prompt assistance. As far as I had some dealing with BSD/Linux, the firewalls/networking were never my strongest part to be honest. Slowly I am getting there but as of now, I feel like pfSense can replace quite a few of our office ASA boxes that we use sort of for the same functionality (if less than pfSense can offer with the proper setup and knowledge).
-
Figured it was something like that. Bypassing limiters on those WAN in rules will be pretty trivial once you know the limiters are working.
Still going to totally suck compared to the optimized connections the VSAT provider will give you.
-
Derelict, for sure there is nothing better than the original equipment as the initial idea was to have a 1x1 copy of a vessel in the office in order to have prompt tests metrics. However we've been told that the satellite dish might be nonoperational for a while and I did not want to loose more time but start the planned work. Having an emulation of VSAT simply put would eliminate the possibility to troubleshoot the modem, ACU, and PCU but having a limited speed would at least be something compared to nothing so far. I took the challenge to start with pfSense (even after knowing about the lack of advanced networking skills) in order to start improving that and most of all - learn. As you said, once the basic principles are acknowledged, we might start thinking of other ways to use pfSense other than just have it as a simple traffic shaper, however, it is just the beginning :)
With a risk to repeat myself, I really really really appreciate your shared info and will come back tomorrow as soon as I try it. Either with the how-to manual or with a question ;)
-
Btw, I see several videos in Internet testing pfSense in a virtual environment. Could you share what you find as the best approach to start with as I thought to play some tests at home and not wait for the office working hours but to start installing now 2 VM's and setting up a pfSense seemed a bit of overwhelming? So do you consider HyperV, ESX or VirtualBox as a good alternative to start with or there are other tools that would make it easier to just fire up and test? I also heard of some testings with GNS3 but did not play with it yet…
-
I am testing this in XenServer. I believe it works fine in all of those listed.
https://doc.pfsense.org/index.php/Category:Virtualization
https://forum.pfsense.org/index.php?board=37.0
-
Hello again.
I reset the box to factory settings just to make sure no previous rules or settings are activated and messing up with the result. Here is what steps I took:
1. Went to Limiters and as per your instructions created the FROM_VSAT and TO_VSAT (proper values added).
2. Went to add a floating rule: MATCH-WAN-OUT-IPv4-ANY-ANY-ANY-TO_VSAT/FROM_VSAT (here I had to select the GW as you mentioned)
3. Added another floating rule: MATCH-WAN-IN-IPv4-ANY-ANY-ANY-FROM_VSAT/TO_VSAT
4. Disabled NAT
5. Made sure private and bogus networks are not blocked on both WAN & LANTested to ping from 192.168.9.148 and was able to reach successfully the 192.168.170.10 (a client PC)!!!
I was able to ping from 192.168.170.10 the 192.168.9.148 successfully as well!!
However, I was not able to browse the Internet from 192.168.170.10 or any other device on the network (and yes, I made sure there is no firewall blocking the connection, nor antivirus etc.) Before today, I had a connection from the 192.168.170.0 network to Internet, however was not able to get from the 192.168.9.0 network to 192.168.170.0 but now is something different. Did I miss something?
-
Just set the WAN interface with a gateway to, IIRC, the .9 address.
None of those floating match rules have anything to do with actually passing traffic.
Disabling NAT is probably not what you want to do. If you do you will have to add a route for the IP subnet on LAN side of pfSense to the edge router.
I see where I said disable NAT below but that was for a specific scenario. Reenable auto nat and you will probably be fine.
-
Hello Derelict.
Sorry for the delay of my response as I promised to comeback and advise if it works or not.
I applied your rules on a fresh pfSense install and started testing. The PC had Internet until I applied the FROM_VSAT and TO_VSAT to WLAN out traffic. After setting up the in/out settings, I was able to resolve, ping and traceroute to the host, but could not browse the page itself.
What I did was to add a rule in Firewall>Rules>LAN with the following:
Action: Pass
Interface: LAN
Address Family: IPv4 (we do not use IPv6 in the company)
Protocol: any
Source: any
Destination: any
Description: LAN ANY TO ANY
In/Out: To_VSAT / From_VSATSaved the rule, moved it to the top, applied the settings and after that all works like a charm. Incoming connections are limited to the speed and latency setup in limiters for both directions but at the same time any LAN traffic is unlimited so a ping from the LAN Net to LAN Net is not affected. I find it useful to have an option to apply the limiters to the LAN network as well for some tests, where in production environment a need to troubleshoot a slow network/latency is needed.
I am extremely grateful to your help and I hope that this short tutorial plus your extensive instructions would be useful for others who need to emulate VSAT or just any other bandwidth, latency, package drop etc. limitations. The options are limitless and it is up to us to see how can we use it.
