Multi-WAN gateway doesn't work

[mastermindpro]

I have a simple 1 LAN 2 WAN network (static IP's all around) that I'm trying to get running off the Feb. 17th snapshot.  My gateway groups look to be configured correctly and all show online and functioning.  When I set the default LAN outbound rule to use the "balanced" gateway, I'd guess about 1 in 10 new states actually works.  ICMP, TCP, and UDP traffic were all tried during my testing and all exhibited the same result.  The majority of traffic doesn't make a complete loop.

When I chose one of the two failover gateways or leave the rule as default, traffic flows as I'd expect.

Anyone else seeing this behavior?

[rsingh]

yeah, i've been seeing this all through 2.0. try the latest snap. here's my work around:

lan
|
pfsense 1.2.3-rc1 - ISP1
|
pfsense 2.0 - ISP2

load balancing is done by pfsense 1.2.3 because it works very well there. sadly it is a double-nat between the two pfsense boxes.

[clarknova]

Any reason you don't use bridging in 2.0 to avoid the double-NAT?

[mastermindpro]

So, this is a known issue then?  I don't see any corresponding bug report in any status.  Seems like this would be worthy of a bug report.  Multi-WAN completely fails to handle packets appropriately…kind of a core feature.  ::)

What would the devs need from me to create a bug report?  I've never done one before, so I'd need to know the details of what is required.

[rsingh]

yeah the problem is i have no idea what causes this or what information can be used to correct this. maybe a place to start would be a routing table, your LAN rules, a traffic capture showing a packet doing something it should not - both on the input interface and output interface.

the other issue i have with this problem is that enabling promiscuous mode (tcpdump) causing things to work properly.

[rsingh]

some more investigating: some dynamic gateways have more than one interface name in the gateway selection of a rule such as OPT1 and GW_OPT1. choosing the dynamically created name (GW_OPT1) seems to work better than OPT1.

i'm doing some lab tests trying to replicate this problem with the latest snapshot (Wed Feb 24 12:14:53 EST). I setup this PBR rule under LAN:

Proto  	Source  	Port  	Destination  	Port  	Gateway  	Queue  	Schedule  	Description  	
TCP 	       * 	       * 	       * 	       80 (HTTP) 	    WAN_GATEWAY 	none 	HTTP via WAN  	
TCP 	       * 	       * 	       *         443 (HTTPS) 	GW_OPT1 	none 	  	HTTPS via OPT1  

Interfaces:
WAN (wan)                -> em1        -> 172.26.104.7
LAN (lan)                -> em0        -> 192.168.12.7
OPT1 (opt1)              -> em2        -> 192.168.250.128(DHCP)

I have a single client at 192.168.12.10. HTTPS and HTTP go through the right interfaces.

Some examples of this working in my state table:

Proto  	Source -> Router -> Destination  	State  	
tcp  	68.180.206.184:80 <- 192.168.12.10:53770  	FIN_WAIT_2:FIN_WAIT_2  	
tcp 	192.168.12.10:53770 -> 172.26.104.7:36458 -> 68.180.206.184:80 	FIN_WAIT_2:FIN_WAIT_2

Proto  	Source -> Router -> Destination  	State  	
tcp 	83.149.75.183:443 <- 192.168.12.10:44574 	FIN_WAIT_2:FIN_WAIT_2 	
tcp 	192.168.12.10:44574 -> 192.168.250.128:60854 -> 83.149.75.183:443 	FIN_WAIT_2:FIN_WAIT_2

[rsingh]

this is still working really well, http and https are being split through the two interfaces. i'll probably put this into production later this week. setting the gateways by their dynamic name helps. in 1.2.3 there was only the interface name, not both.

[mastermindpro]

This isn't working for me.  I've had my box configured from the get-go using those names, and something about it just doesn't work.  My gateways are all static, but I've been using the GW_WAN names.