2.0-RC1 (i386) 28th Feb : incoming multi-wan + load balancing issues ?

bEsTiAn

Hi,

I have the folllowing setup :

2 different ISPs
vIP for incoming traffic (as I have redundant pfsense boxes)
load balancing (as I have redundant web servers behind pfsense)
public DNS entries having one IP per ISP

When I browse to one of my websites from the Internet, I have timeouts or very slow traffic hitting web servers, intermittently and randomly.
I have a static page with 5 pictures, sometimes one random picture will get a red cross instead. Or sometimes it'll hang 10 seconds before appearing.
If I play with my hosts entry on my client, I can reach the website perfectly using every one of all the public IPs I have in my DNS.
If I look at the access log of my web servers, they only receive the traffic after the long delay, not before. So it is blocked somewhere in pfsense.

Any idea ?

bEsTiAn

anyone, any idea ?

bEsTiAn

issue persists after upgrading to
2.0-RC1 (i386)
built on Wed Mar 16 06:36:08 EDT 2011

bEsTiAn

I can pinpoint (from what I understand) this issue to the load balancer part.
I have the same setup for my DNS but no load balancing (not needed due to DNS retries) and this never fails.
I try repeatedly intodns.com/multipurpose.be and I never get a timeout.
But try to browse to www.multipurpose.be and I have randomly timeouts on some parts of the front page.

Am I really the only one facing this issue ?

eri--

without describing your setup nobody will answer.

bEsTiAn

well i thought my first post was explanatory enough ? i'll retry to explain it

2 WANs + 2 pfsense + 2 web servers

public DNS have IP of both WANs
so traffic hits though both WANs the active pfsense (carp vIP)
load balancer listener is on the carp vIPs of each WAN
it then load balances to the web servers

is it clear enough or i should maybe try to draw a visio ?

eri--

No i meant more information from system logs, firewall rules, shaper….
Helpful would be even the output of pfctl -a relayd -vsr and pfctl -a relayd -vsn

bEsTiAn

ok ! i tried both (pfctl -a relayd -vsr and pfctl -a relayd -vsn)
and both return empty ??

[2.0-RC1][root@dupond.multipurpose.be]/root(7): pfctl -a relayd -vsr
[2.0-RC1][root@dupond.multipurpose.be]/root(8): pfctl -a relayd -vsn
[2.0-RC1][root@dupond.multipurpose.be]/root(9):

eri--

Can you show the load balancing config and the output of the command ps -ax | grep relay

bEsTiAn

sure. I guess this is not disclosing too much sensitive info to the world :)

[2.0-RC1][root@dupond.multipurpose.be]/root(30): ps -ax | grep relay
28195 ?? Is 0:00.01 relayd: parent (relayd)
28676 ?? S 0:12.22 relayd: pf update engine (relayd)
28983 ?? S 3:14.42 relayd: host check engine (relayd)
10826 0 S+ 0:00.00 grep relay

[2.0-RC1][root@dupond.multipurpose.be]/root(31): cat /var/etc/relayd.conf
log updates
table <smtp_relays>{ 192.168.101.107, 192.168.101.108 }
table <web_servers>{ 192.168.101.101, 192.168.101.102 }
table <web_proxies>{ 192.168.101.101, 192.168.101.102 }
redirect "pool_squid_kpn_1" {
listen on 62.166.228.197 port 80
forward to <web_proxies>port 8080 check http '/' code 302 timeout 1000
}
redirect "pool_squid_kpn_2" {
listen on 62.166.228.198 port 80
forward to <web_proxies>port 8080 check http '/' code 302 timeout 1000
}
redirect "pool_web_voo_1" {
listen on 212.68.200.227 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_voo_2" {
listen on 212.68.200.228 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_kpn_1" {
listen on 62.166.228.203 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_kpn_2" {
listen on 62.166.228.204 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_kpn_3" {
listen on 62.166.228.195 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_kpn_4" {
listen on 62.166.228.196 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_smtp_kpn_1" {
listen on 62.166.228.203 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn_2" {
listen on 62.166.228.204 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn_3" {
listen on 62.166.228.203 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn_4" {
listen on 62.166.228.204 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_voo_1" {
listen on 212.68.200.227 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_voo_2" {
listen on 212.68.200.228 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_voo_3" {
listen on 212.68.200.227 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_voo_4" {
listen on 212.68.200.228 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_mgt" {
listen on 192.168.254.3 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_web" {
listen on 192.168.101.3 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_app" {
listen on 192.168.102.3 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_lab" {
listen on 192.168.103.3 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_braun" {
listen on 192.168.0.3 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_squid_kpn2_1" {
listen on 94.107.234.55 port 80
forward to <web_proxies>port 8080 check http '/' code 302 timeout 1000
}
redirect "pool_squid_kpn2_2" {
listen on 94.107.234.56 port 80
forward to <web_proxies>port 8080 check http '/' code 302 timeout 1000
}
redirect "pool_web_kpn2_1" {
listen on 94.107.234.53 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_web_kpn2_2" {
listen on 94.107.234.54 port 80
forward to <web_servers>port 80 check http '/' code 200 timeout 1000
}
redirect "pool_smtp_kpn2_1" {
listen on 94.107.234.53 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn2_2" {
listen on 94.107.234.54 port 25
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn2_3" {
listen on 94.107.234.53 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}
redirect "pool_smtp_kpn2_4" {
listen on 94.107.234.54 port 2525
forward to <smtp_relays>port 25 check tcp timeout 1000
}</smtp_relays></smtp_relays></smtp_relays></smtp_relays></web_servers></web_servers></web_proxies></web_proxies></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></smtp_relays></web_servers></web_servers></web_servers></web_servers></web_servers></web_servers></web_proxies></web_proxies></web_proxies></web_servers></smtp_relays>

bEsTiAn

so is there anything looking wrong ?
I need to mention that the similar setup worked perfectly using 1.2.3