Dual-path routing to the internal network?
-
Hello,
I have a multi-WAN single-LAN pfSense machine (v2.3.4) which is working basically as a [policy-based] router in connection with a single-WAN multi-LAN Fortigate (FortiOS v4.0), which is working basically as a [fast] firewall between the internal networks.
The dominant part of traffic is outbound. Inbound is negligible.
Due to increased Internet traffic I recently added a second link between the pfSense and the Fortigate (see Figure), and enabled Equal Cost Multi-path (ECMP) routing on the Fortigate (namely, source IP-based method).
I can see the Fortigate sending outbound sessions fairly evenly on both its WAN interfaces. I expect that the reply-to-session packets from pfSense arriving on exactly the same [WAN] interface for the session or, at the minimum, arriving randomly on both interfaces, so that both links are utilized roughly evenly, in both directions.
Unfortunately it is not the case. pfSense still sends all reply packets to Fortigate via only one interface (its LAN or LAN2, depends on the static route to the internal network .0.0/20 that I added manually to the pfSense).
I removed the static route and enabled Quagga-OSPF in pfSense instead. Quagga does update the system route table, but not as I hoped – it adds only one interface, either LAN or LAN2, to the route table, and all reply packets are still sent via that interface.
The Fortigate WAN1 port is 100 Mbps Ethernet, which is smaller than the total throughput of Internet links. Users in the local networks are able to surf the Web, but not as fast as they should.
Is there a solution that works in pfSense for my problem?
Thank you.
ISP 1 ISP 2 ISP 3 | | | pubIP1 | | | +-------+ | | | NAT | | | +-------+ | | .0.65 | | | | | | .0.64/30 | | | | | | .0.66 | pubIP2 | pubIP3 | +-------+-----------+-----------+-------+ | WAN OPT2 OPT3 | | NAT NAT | | | | pfSense 2.3.4 | | | | LAN LAN2 | +-------------+-----------+-------------+ .0.74 | .0.84 | | | .0.72/29 | .0.80/29 | | | .0.75 | .0.85 | +-------------+-----------+-------------+ | WAN1 WAN2 | | | | FortiOS 4.0 | | | | VLAN1 VLAN2 VLAN3 ... VLAN15| +---+-------+-------+-------+-------+---+ .1.1| .2.1| .3.1| .15.1| | | | | .1.0/24 .2.0/24 .3.0/24 .15.0/24 NOTES 1\. Outbound load balancing is enabled in the Fortigate. 2\. Quagga-OSPF is installed and configured in pfSense. 3\. The Fortigate is OSPF-capable. 4\. The Fortigate is not LACP-capable. -
Ecmp isn't available AFAIK.
Why not use gateway groups?
-
Ecmp isn't available AFAIK.
Thank you. After a day googling and reading I tend to agree with you. Probably the kernel module supporting ECMP isn't compiled into pfSense.
Why not use gateway groups?
Would you please elaborate?
I use gateway groups to load balance WAN, OPT2 and OPT3 interfaces for outbound traffic. So, I think that gateway groups will similarly load balance LAN and LAN2 interfaces for inbound traffic only.
EDIT: Thanks to your suggestion, I think I figured it out.
1. Define a gateway group (LAN, LAN2) for inbound traffic. (Alternatively, define static routes to some specific internal servers only.)
2. (For outbound traffic,) Turn off Quagga, remove the .0.0/20 static route, and set upstream gateway for LAN, LAN2 [to Fortigate's WAN1, WAN2 respectively].
LAN and LAN2 upstream gateways have been left unset (in accordance to the WebUI hint On local area network interfaces the upstream gateway should be "none"), which seems to be a mistake for my topology. Setting the upstream gateway for all interfaces turns pfSense into a symmetrical router, i.e. all interfaces are functionally equivalent (as there're no "local area network" interfaces at all).
I'll try this and report back.
-
It didn't work.
After three days of trials and errors I managed to make outbound traffics, but either via default gateway (i.e. WAN) only, or by an additional static route to the internal [network containing the] client.
The former, of course, does not solve my problem. The later would require splitting the internal network into two equal halves, with so many potential issues that can hardly be called a 'solution'.
On the other hand, a gateway group (LAN, LAN2) works flawlessly for inbound traffic via IPsec or OpenVPN with tunnel endpoints put on pfSense's WAN, OPT2 or OPT3, without affecting outbound traffics to the remote sites.
Putting it simply, pfSense can be either multi-WAN or multi-LAN, but not both.
This suggested me a solution: decompose the pfSense into a cascade of two pfSense: an 'outer' 3WAN-1LAN, and an 'inner' 1WAN-2LAN with LAN1 and LAN2 both having upstream gateway defined and no static routes at all, as described in post #2. The 'outer' pfSense would have the same gateway groups [for outbound traffics] as before. For inbound traffics, the 'inner' pfSense would have gateway groups like (LAN1, LAN2). I haven't try it. But I believe it's workable.
-
So, here is the new topology, described in previous post.
ISP_1 ISP_2 ISP_3 | | | 1 | | | +---+---+ | | | NAT | | | +---+---+ | | .0.65/30 | | | | | | .0.66/30 | 2 | 3 | +---+------+------+---+ | WAN OPT2 OPT3 | | | | pfsense 2.3.4 x86 | | | | LAN | +----------+----------+ .0.69/30 | | .0.70/30 | +----------+----------+ | WAN | | | | pfsense 2.4.3 x64 | | | | LAN1 LAN2 | +----+-----------+----+ .0.74/29 | | .0.84/29 | | .0.75/29 | | .0.85/29 +-------------+-----------+-------------+ | WAN1 WAN2 | | | | FortiOS 4.0 | | | | VLAN1 VLAN2 VLAN3 ... VLAN15| +---+-------+-------+-------+-------+---+ .1.1/24 .2.1/24 .3.1/24 ... .15.1/24 | | | |It seems to work, but I can't tell how accurately because interface statistics are obviously wrong. (see attached figure).
Any idea on how to get true statistics?
-
not sure what you think is wrong with it
you could check 'traffic totals' package
-
not sure what you think is wrong with it
Interface Stats shows ridiculous LAN traffic. It looks smaller than it should be by orders of magnitude.
As the pfSense is configured to pass all through, I expect WAN in = LAN out (i.e. LAN1 out + LAN2 out), and vice versa, with respect to both #packets and #bytes, plus/minus negligible amount of house-keeping traffic (which is initiated by or destined to the pfSense itself) and ill-formed traffic (i.e. blocked by default).
you could check 'traffic totals' package
Great.
Traffic Totals doesn't count #packets but for #bytes the invariant holds with > 99% accuracy. So, I believe Traffic Totals are valid.
According to Traffic Totals, LAN1 and LAN2 both take good shares in transmitting and receiving data (although sometimes they are pretty off from 50%). So, basically, the solution works.
Thank you very much.
-
Attached image shows in (RX) and out (TX) #bytes for LAN interfaces for the recent 4 weeks.
image url)Data collected and represented by Traffic Totals.
-
I don't see an issue. Gateway groups load balance connections not packets. So its expected to have some in balance as seen in graphs. This also means that if your isp is giving you more than 100mbits on one interface, a user behind the fortigate would still only use 100mbit per connection, limited by the speed of fortigate. This can be demonstrated when downloading big files via http, ftp etc.... Nothing would exceed 100 mbit.
I do wonder why are you not using lagg and link aggregation from the fortigate.
It seems to support up to four interfaces, giving you a nice 400mbit packet level load balanced solution.As for the two pfsense approach, I believe they can be merged if you create 2 "lan" interfaces on the same pf connected by a cross lan cable* so being able to apply different outbound rules for gateway groups.
*Nowdays cross cable are not needed due to mdi-mdix of physical interfaces, In virtual environments you only need two interfaces on the same vswitch.
-
@netblues said in Dual-path routing to the internal network?:
I don't see an issue. Gateway groups load balance connections not packets. So its expected to have some in balance as seen in graphs. This also means that if your isp is giving you more than 100mbits on one interface, a user behind the fortigate would still only use 100mbit per connection, limited by the speed of fortigate. This can be demonstrated when downloading big files via http, ftp etc.... Nothing would exceed 100 mbit.
My newest post was to visualize the result of a possible solution, not the issue.
I do wonder why are you not using lagg and link aggregation from the fortigate.
It seems to support up to four interfaces, giving you a nice 400mbit packet level load balanced solution.My Fortigate does not support any link aggregation protocol.
As for the two pfsense approach, I believe they can be merged if you create 2 "lan" interfaces on the same pf connected by a cross lan cable* so being able to apply different outbound rules for gateway groups.
Would you please elaborate? Your topology is not very easily understandable.
-
Well, the topology remains the same as you have it.
Instead of having two separate pf installations, merge them into one, and keep the
.0.69/30 <-> .0.70/30 link with separate interfaces on the same box. -
@netblues Ahah, now I understand.
Looks like a briliant optimization. If works, it should save one piece of hardware.
I'll try it once I go to physical machine. (I had to put pfSense on virtual machines since I'm short of switch port. And in merging the VMs I see no benefits.)
-
Having pf on vm's gives another layer of redunduncy, but thats another story.
Maintaining one system does have its benefits (upgrades, troubleshooting etc)
And routing the packets in and out of virtual interfaces does consume unnecessary cycles. I can't tell if this has any measureable degradation whatsoever in any case.
I do have second thoughts if that would work in the end, becauseit all boils down to a common routing table so traffic would never pass through the lans :(