A double-NAT topology employing a second pfSense instance allows you to use ALTQ schedulers on an "edge" pfSense machine to shape traffic from any number of LANs or DMZs, against any number of WANs, by first routing any desired Internet-bound traffic that reaches the edge pfSense machine on its LAN / DMZ interfaces to one of a set of gateways on internal "NAT" interfaces, where those interfaces are wired to corresponding interfaces a second "NAT" pfSense machine (e.g. a virtual appliance), whose only job is to NAT that traffic to one of a set of private addresses according to some Manual Outbound NAT rules and then pass it back via yet another interface on the edge pfSense machine, from where the edge machine can shape the traffic, and ultimately route it to a WAN interface - NATting again along the way.
The concept is to aggregate all Internet-bound LAN/DMZ traffic reaching the edge machine onto a single interface per WAN, rather than directly NATting it and routing it to the Internet as would normally happen. Unfortunately it can't all be done in one pfSense instance, and the second pfSense instance, i.e. the NAT machine, is required for two main reasons:
For the double-NAT solution, here is an example topology using 2 WANs and 3 LANs. (It assumes as a precondition that the edge machine is already set up, i.e. just NATs as normal to WAN1 or WAN2 from each of the LANs):
On the edge machine let's say we already have:
WAN1 at 211.0.0.1/30 (a public IP on the Internet) WAN2 at 222.0.0.1/30 (a public IP on the Internet) LAN1 at 10.1.0.1/24 LAN2 at 10.2.0.1/24 LAN3 at 10.3.0.1/24Now let's add some internal routing interfaces to help us with our goal of shaping traffic in the desired aggregated manner. These interfaces are also on edge pfSense instance:
NAT1 at 192.168.1.1/24 NAT2 at 192.168.2.1/24 NAT1_RET at 192.168.10.1/24 NAT2_RET at 192.168.11.1/24Now let's create a second instance of pfSense and call it the "NAT machine" with interfaces as follows:
WAN1 at 192.168.10.2/24 gateway 192.168.10.1 (default) WAN2 at 192.168.11.2/24 gateway 192.168.11.1 NAT1 at 192.168.1.2/24 NAT2 at 192.168.2.2/24WAN1 and WAN2 have private (RFC1918) IP addresses but they are still WANs from the perspective of the NAT machine, because it will be using them to reach the Internet via the gateway IPs shown. I mention this only because some readers may have the expectations that WANs must have public IPs, however that isn't a requirement.
Notice WAN1 on the NAT machine and NAT1_RET on the edge machine are the same network segment (the same CIDR network). The intention is that their interfaces be wired together, so let's wire them together.
Similarly, wire up WAN2 <=> NAT2_RET, NAT1 <=> NAT1, NAT2 <=> NAT2.
In our own implementation of all this, inside a mini datacenter with 32 network segments, all our pfSense instances were virtual machines so we did this wiring virtually using VLANs on a vSphere distributed switch, and each pfSense machine just had one 'internal' interface operating in trunk mode, with VLAN interfaces defined as needed on it for the individual networks. The overhead was negligible and we proved our pfSense appliances could switch 9.8Gb/s on a 10GbE link at 1500 MTU, so performance with VLANs was simply not an issue. Equally, we could saturate multiple 1GbE Internet links in the double-NAT configuration without CPU usage on any machine going over about 5%.
Notice at the same time that all the NAT machine addresses end in ".2" whereas the NAT1, NAT2, NAT1_RET and NAT2_RET addresses on the edge machine all end in ".1" This is only for memorability as you can then think of the edge machine as the ".1" machine, and the NAT machine as the ".2" machine, on each NAT or RET CIDR network; in reality the IP addresses you use can be anything obviously.
The next step happens on the NAT machine, and is crucial to the traffic shaping design:
Add three Virtual IPs on WAN1: 192.168.10.11, 192.168.10.12, 192.168.10.13. These will be used as the NAT addresses for traffic received by the NAT machine on NAT1 interface, from the LAN1, LAN2 and LAN3 CIDR networks respectively. Similarly, add three Virtual IPs on WAN2: 192.168.11.11, 192.168.11.12, 192.168.11.13. These will be used as the NAT addresses for traffic received by the NAT machine on NAT2 interface, from the LAN1, LAN2 and LAN3 CIDR networks. Use Manual Outbound NAT on the NAT machine. Delete all pre-existing rules; we will be creating our own rule set from scratch. Create simple 2 helper rules first, these are just to allow the NAT machine itself to reach the Internet:** WAN1 interface: NAT1 source network --> NAT via WAN1 address (i.e. 192.168.10.1)
** WAN2 interface: NAT2 source network --> NAT via WAN2 address (i.e. 192.168.11.1)
** PS make sure you choose either WAN1 or WAN2 as default gateway as best suits your needs. It's mostly a choice about which Internet connection you want the NAT machine itself to use as everything else is going to use policy routes. Now create rules to re-present traffic received on the NAT1 and NAT2 interfaces as separate, discrete IP addresses on either the NAT1_RET and NAT2_RET interfaces, according to the source CIRD network address:
** WAN1 interface rule: NAT 10.1.0.1/24 network via 102.168.10.11
** WAN1 interface rule: NAT 10.2.0.1/24 network via 102.168.10.12
** WAN1 interface rule: NAT 10.3.0.1/24 network via 102.168.10.13
** WAN2 interface rule: NAT 10.1.0.1/24 network via 102.168.11.11
** WAN2 interface rule: NAT 10.2.0.1/24 network via 102.168.11.12
** WAN2 interface rule: NAT 10.3.0.1/24 network via 102.168.11.13
** What's important here is how the NAT address on the NAT1_RET or NAT2_RET interfaces effectively represent the source CIDR network and the intended Internet route.
As an aside, you can be as complex as you want with the Manual Outbound NAT on the NAT machine. You could NAT based on any combination of source, dest IP, or port or protocol. This means you could ultimately prioritise UDP traffic differently to TCP traffic, on the same WAN. Or you can split traffic up by IP or port number. For each separate way you want to categorise traffic heading to a WAN, you can simply choose to add a virtual IP to the relevant NAT1_RET or NAT2_RET interface, and set up the corresponding Manual Outbound NAT rule to so that the desired traffic is NAT'ted via that new virtual IP.
Moving on, now we can complete the configuration of the edge machine:
Define 2 new gateways:** NAT1_GW on NAT1 interface at 192.168.1.2
** NAT2_GW on NAT2 interface at 192.168.2.2 Now create policy routes (i.e. advanced firewall rules that specify a gateway) on each of the LAN1, LAN2, LAN3 interfaces to set the gateway for whichever traffic you want to send out WAN1 or WAN2 on the edge machine. Specify NAT1_GW in the rule if you want the traffic to end up going out WAN1, and NAT2_GW if you want it to end up going out WAN2 A policy route can match on anything that the firewall rules support e.g. source and destination IP, protocol, port, etc. It's up to you what you want to send where, just design the rules accordingly, and specify NAT1_GW or NAT2_GW in the advanced section depending on which Internet connection you want that traffic to eventually use. The policy routes ultimately work by directing traffic appearing on each LAN to the NAT machine, which then reflects it back to the edge machine either via NAT1_RET (which we will send out WAN1 in the next steps) or NAT2_RET (which we will send out WAN2 in the next steps) A simple example of a policy route is all Internet traffic destined for port 80 or 443 uses NAT2_GW. All other traffic uses NAT1_GW. The result will be "browsing from whichever LAN has those rules is via WAN2, everything else goes via WAN1". You can apply this to any or all LAN/DMZ interfaces just by adding the applicable rules to those interfaces. Now specify policy routes on the "RET" interfaces:
** All traffic on NAT1_RET routes out WAN1_GW.
** All traffic on NAT2_RET routes out WAN2_GW.
** This, by the way, is how the traffic ultimately gets divided between WAN1 and WAN2. Put another way, by design, it's not possible for traffic arriving on NAT1_RET to go out WAN2...
That completes the networking topology.
Now you're ready to design the ALTQ shapers, and you do this in the normal way on WAN1, WAN2 and NAT1_RET and NAT2_RET. The important observation is that at no time will traffic from NAT1_RET be routed through WAN2, nor NAT2_RET traffic via WAN1 because of the policy routes on the RET interfaces. This means when designing the shapers, you can reliably think of NAT1_RET as the one and only LAN corresponding to WAN1, and similarly NAT2_RET is the one and only LAN corresponding to WAN2. It's up to you to ensure the rules on LAN1, LAN2, and LAN3 (plus any DMZ interfaces you have) properly direct traffic to NAT1_GW or NAT2_GW and not allow the default gateway to be used at any time, in order for this principle to be valid. Of course you can still break the rules if you want to and route some traffic using default gateway, it just means that traffic shaping won't be able to account for traffic that you don't explicitly send via the NAT machine.
To be clear, to set up the traffic shapers, you are best to use floating rules as follows:
Control WAN1 download bandwidth via queues on WAN1 interface Control WAN1 upload bandwidth via queues on NAT1_RET interface Control WAN2 download bandwidth via queues on WAN2 interface Control WAN2 upload bandwidth via queues on NAT2_RET interface You should obviously have one queue per category of traffic you want to shape PS the reason download is controlled on WAN and upload is controlled on LAN is that ALTQ schedulers work on 'out' traffic only, not on 'in' traffic.In terms of testing, here are some tips:
A PC running Wireshark with a NIC in promiscuous mode is a great way to see what's happening on NAT1, NAT2, NAT1_RET and NAT2_RET interfaces Or, use Packet Capture built within pfSense Or, inspect states on each interface to get an idea of which flows are active Watch queue activity via Status > Queues Check if floating rules are working by inspecting the state count and traffic volume processed, associated with each rule. If needed, log firewall rules to tatus > System Logs > Firewall to view specific policy routes in actionAll the usual firewall rules needed for security still get implemented in the usual way on WAN1, WAN2 and LAN1, 2, 3. Nothing changes there. All you're really doing is setting gateways.
This configuration works fine with pfBlockerNG as well, just make sure pfB's block rules are set to apply on each LAN/DMZ interface before the policy routes you create. I.e. make sure they appear above your rules in the list.
As a tip, we often define an IP alias on the edge machine called ipALL_LOCAL_NETWORKS that includes the CIDR networks for interfaces we consider LANs. Therefore the definition of "Internet-bound" traffic can simply be "!ipALL_LOCAL_NETWORKS" i.e. "not ipALL_LOCAL_NETWORKS", in any rule where we want to define that. Inter-LAN routing is still possible (just define the rules as you need to), and then make sure to define the appropriate NAT1_GW or NAT2_GW gateway for rules that specify !ipALL_LOCAL_NETWORKS.
Hopefully all of this helps someone else who wants to achieve multi-LAN / multi-WAN traffic shaping in a way that supports good test tooling, is readily inspectable and is easily extensible.
