Can lagg be done between geographically separated pfsense machines?
-
Are the ISPs just giving you L2 connections between your locations..
-
Don't know. How do I find out? ADSL modems are used.
-
If your typical adsl modems are being used then no its not just an L2 connection ;)
What exactly are you wanting accomplish?? You do understand that lagg is not 1+1=2 for speed right.. And normally the load balancing that happens across a lagg would be done via the mac of the talking devices. So on an end to end sort of connection there would only really ever be 2 mac's so only 1 connection would be used for any sort of load sharing. So while such a setup might be a way to get redundancy it would not be the normal practice for such a setup.
-
Trying to outperform multi-wan per-connection round robin which only uses one network interface if you only use one connection, such as with a vpn or anything tunneled. So how do you pass one connection through two network interfaces and get double the bandwidth with geographically separated machines is my question.
Can't ethernet traffic somehow be repeated from one site to another?
-
"So how do you pass one connection through two network interfaces and get double the bandwidth with geographically separated machines is my question."
You don't!! you just stated its 1 connection - you can not magically bond these 2 different connections into 1 fat pipe.. If you want a fatter pipe - get a fatter pipe!
You can for sure load share across your 2 connections.. to ramp up to using both pipes fully… If you have enough clients and sessions being created.. But there is is no way to take 2 connections and turn it into 1 fat pipe.. Lagg doesn't even do this on a lan.. Be you call it etherchannel, portchannel, etc. It doesn't turn multiple pipes into a fatter 1.. As I stated its not 1+1=2, its 1 and 1..
-
there is is no way to take 2 connections and turn it into 1 fat pipe.. Lagg doesn't even do this on a lan.. Be you call it etherchannel, portchannel, etc. It doesn't turn multiple pipes into a fatter 1.. As I stated its not 1+1=2, its 1 and 1..
Need a clarification of the above: you have two pfsense systems with two ethernet cables between them, and lagg set up. If one pfsense instance begins to send UDP packages to the other pfsense instance, half the packages go via frames in one cable and half via frames in the other, therefore twice the speed, right? But if a single TCP connection is established, all packages go via frames in one cable you are saying. Ok but what's to stop people from writing software that separates the TCP packages sending half via frames in one cable and half via frames in the other, and where the second pfsense instance puts the packages back together?
EDIT: just found someone who says this: "OpenVPN: tunnels either IP or ethernet frames on top of UDP". Is this and multi-wan the way to go?
-
" If one pfsense instance begins to send UDP packages to the other pfsense instance, half the packages go via frames in one cable and half via frames in the other, therefore twice the speed, right?"
No - not sure where you got this idea from…
It does not work this way!!
-
Ok that won't happen with lagg, but will it happen with multi-wan? Ie: in multi-wan, will half of the UDP packets go via one available route and half via the other? UDP is connection-less. Multi-wan does per-connection round-robin, right? But presented with connection-less UDP packets, it won't round-robin them?
If multi-wan won't round-robin UDP packets, any inspiration from the following for alterative more elaborate setups with several virtual machines if need be?
OpenVPN: tunnels either IP or ethernet frames on top of UDP".
-
Not sure where you got the idea that UDP is going to round robin out your different interfaces?
Not sure where your quote is from but sure you can use either tcp or udp for a tunnel.. But that is not going to split traffic across multiple interfaces frame by frame… It just doesn't work that way.. Just because you add another interface does not make it 1+1=2
Sorry it just does not work like that!!
-
It's common sense to me that pfsense's multi-wan should round-robin UDP packets (not that UDP packets would round-robin themselves across network interfaces without multi-wan if that's what crossed your mind). Otherwise how would you load-balance UDP with pfsense?
Regarding OpenVPN, I was thinking something far more elaborate. Two separate UDP VPN's, two respective VM's, and "ethernet over UDP" means that the VM's present ethernet interfaces to a third VM that runs pfsense and does lagg without knowing anything about OpenVPN, it just thinks its two ethernet cables extend to the other side of the world where another pfsense with lagg runs.
-
Your sense is wrong ;) Be it common or not.. You really need to look into how frames leave a nic.. Not sure where you got the idea that freebsd or pfsense load shares UDP Or any other protocol out different wan interfaces or lan for that matter based on frame or packet, etc. Where if your sending 2, 1 goes out one nic and the 2nd goes out the other… It just doesn't work that way..
There is an algorithm that uses hashing to determine which nic it sends out traffic.. This could be mac based, ip based, url based, random, etc. etc.. But not sure where you got the idea that if device A is talking to B over pfsense - and pfsense has 2 nics in any sort of configuration that it would send even if UDP a frame out 1 nic, and then next frame out the other nic, etc.
Even if you could do such a thing - you run into an issue with out of order packets. So this can be huge hit on overall bandwidth, etc. etc. This is why when a session of any sort of traffic leaves a specific nic, all traffic associated should and would leave the same nic or you highly increase the chances of out of order packets..
If you need a FATTER pipe then you need to get a FATTER pipe - that is all there is too it.. Now you can leverage your 2 connections so that different streams of data go across the different connections (paths) to get to the other side.. But this will not give any specific client 2x the bandwidth just because there 2x your connections. But yes there are multiple methods to load share across these connections so that some clients go one way, other clients go the other to load share across the connections.
-
If you really wonder where I get my ideas, here's a hint: I am a software developer. Software does what you design it to do. There may be a layer to interface with existing software and standards, but the rest is open-ended. If pfsense does not have a package for load balancing where consecutive UDP packets go via different routes yet, someone can develop it. You have not given a good reason why this would not work, UDP packets are already allowed to arrive out of order in a single route and it is the duty of another protocol on top to reassemble them, as TCP would do on top of UDP in OpenVPN's context. I believe such load balancing as I imagine has already been done for frames in the protocol you mentioned, in its random mode. What exactly do you think random means in the context of that protocol?
EDIT: just found the pf firewall does it for UDP, it allows random redirects:
https://forum.pfsense.org/index.php?topic=46202.msg242313#msg242313
-
"You have not given a good reason why this would not work"
"you run into an issue with out of order packets." Especially!!!!! when your talking about wan connections - run by 2 different isp even.. Dude it not work that way!!!
And look at it again - that is NOT what your talking about.. Where is the 2 nics in that scenario? That is a NAT thing..
UDP DNAT based load balancing which solves the following problem:
A single external IP address must forward UDP clients to one of many internal UDP servers. -
I was not referring to his wish for inbound load balancing, but the capability for outbound-only load balancing that someone replied further down already exists in pf (and pfsense probably), can we not harness this to send packets randomly to different networks and therefore different nic's?
Maybe I ought to transfer the question to the firewall subforum, it is too much for one person and there's ego issues too getting in the way of unbiased suggestions/workarounds.
-
is your ego a problem that you think you could do something that is not possible? ;)
Even if you could send out the traffic via different nics.. You still have the issue of now increased problem of packets out of order on the receiving end!! And this is a HUGE problem for overall performance.. Even more so with UDP traffic… And you also want to put this traffic now inside a tunnel, which will have tcp traffic inside of it.. Packets out of order in this fashion will cause even more problems with most likely increased retrans from the applications using the tcp inside the udp tunnel that you are sending across 2 different higher latency paths.
I wish you all the luck in your quest - but sorry dude it just doesn't work the way you think it does.. Do your own research if you want.. If it was so easy to bond smaller wan links into a fatter pipe it would be done all the time.. The question does come up all the time in fact, and its from the same people that don't understand the underlaying factors of what is going on.
How exactly are these udp packets going to even be reordered since there is no seq number on them, etc. While this happens even when going over the same path, and can cause problems - normally its not crazy amount of out of order and few packets here or there depending on the application is not a big deal.. But your plan would exaggerate that to a high amount most likely.. Your different nics in your bond will have different queue lengths and different number packets to wait their turn to be sent out, and then different congestion on their wan paths to their destination, etc.. all of which will cause out of order packets..
To your point in the thread you linked too has to do with ADDRESS and sticky, etc. this has nothing to do with frames being sent out different nics via random, etc.
-
Distorting my proposals to mean UDP on top of TCP when in OpenVPN it is TCP on top of UDP… is yet another hit under the belt. Once I find a solution to utilise both of my ISP's for a single TCP connection which is clearly light years beyond your obviously non-programmer outlook so it cannot be done here, I intend to show it to the admin here and show your messages too.
-
Still waiting ;)
Have you looked at this? This seems a bit like what your after, but you have to use a server in the middle with FAST connection.. This would attempt to remove the out of order packets.. But you have to worry about highest latency connection, etc..
https://zehome.github.io/MLVPN/
And again they bring up my point of the packets being out of order do to different paths..
"Re-ordering is important because packets are not sent at the same speed on every path. Packets would come of order which confuses a LOT TCP. Without re-ordering enabled, expect to have the bandwidth of the slowest link."
-
If someone get MLVPN running, tell me how please :)
-
Like the Chinese say: those who say it cannot be done should not interrupt the one already doing it.
