CARP/VIPS issue in downloading large file

mypal · Sep 20, 2006, 1:18 AM

I have two pfsense firewalls set up with 4 interfaces each (Public, Private, Sync and DMZ). The interfaces are all running with CARP/VIPs. The issue that I have downloading or uploading a large size file (1MB +) to a webserver/qmail server is located at the DMZ from the public internet always hit a timeout when both firewalls are active. If I down one of the firewall, this issue does not occur.

Both firewall are Dell Dual Xeon/2GB RAM. The pfsense version is 1.0-Snapshot-09-12-06. Any ideas on what should I check?

Thanks
Mypal

sullrich · Sep 20, 2006, 1:28 AM

Take a look at this document: http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense and see if it brings up anything that may be setup wrong.

mypal · Sep 20, 2006, 1:48 AM

One additional thing that I forgot to mention, when I transfer file from a PC at the Private network (using VIPs gateway at Private) to DMZ network, I have no such issue.

I suspect there must be some timeout when doing it from the Internet. How can I check this?

Thanks
Mypal

billm · Sep 20, 2006, 2:03 AM

@mypal:

One additional thing that I forgot to mention, when I transfer file from a PC at the Private network (using VIPs gateway at Private) to DMZ network, I have no such issue.

I suspect there must be some timeout when doing it from the Internet. How can I check this?

Thanks
Mypal

Sounds like your CARP address(es) on the internet side are fighting for master status to me. Make sure that all the CARP IPs on the secondary are in BACKUP, not MASTER mode.

–Bill

sullrich · Sep 20, 2006, 2:04 AM

No, there is no timeout. I really have no idea what you mean.

Take a look at the wiki, I just updated the graphic so that it may be a bit better now.

hsiang · Sep 26, 2006, 8:20 AM

I also facing the same issue from the WAN link. I had set the CARP as describe. When the issue occur the secondary firewall status is showing backup.

However the external user is having time out problem when sending attachment email (150kb only), connecting to the mail server in dmz. Thing works fine when i shut down the secondary firewall totally.

sullrich · Sep 26, 2006, 4:28 PM

Sorry but this is not enough information to go on.

Please show us this command on both machines:

ifconfig

mypal · Sep 27, 2006, 2:29 AM

#ifconfig on machine1

em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.0.229 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::214:22ff:fe18:8d47%em0 prefixlen 64 scopeid 0x1
ether 00:14:22:18:8d:47
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 202.184.208.229 netmask 0xffffff00 broadcast 202.184.208.255
inet6 fe80::214:22ff:fe18:8d48%em1 prefixlen 64 scopeid 0x2
ether 00:14:22:18:8d:48
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em2: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 10.0.0.229 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::204:23ff:fed4:c338%em2 prefixlen 64 scopeid 0x3
ether 00:04:23:d4:c3:38
media: Ethernet autoselect
status: no carrier
em3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 10.10.10.229 netmask 0xffffff00 broadcast 10.10.10.255
inet6 fe80::204:23ff:fed4:c339%em3 prefixlen 64 scopeid 0x4
ether 00:04:23:d4:c3:39
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em4: flags=8802 <broadcast,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:bc
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em5: flags=8802 <broadcast,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:bd
media: Ethernet autoselect
status: no carrier
pflog0: flags=100 <promisc>mtu 33208
enc0: flags=0<> mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: em2 maxupd: 128
carp0: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.231 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 200
carp1: flags=49 <up,loopback,running>mtu 1500
inet 192.168.0.232 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 200
carp2: flags=49 <up,loopback,running>mtu 1500
inet 10.10.10.231 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 200

ifconfig on machine2

em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 10.10.10.230 netmask 0xffffff00 broadcast 10.10.10.255
inet6 fe80::204:23ff:fed4:bcfa%em0 prefixlen 64 scopeid 0x1
ether 00:04:23:d4:bc:fa
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 10.0.0.230 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::204:23ff:fed4:bcfb%em1 prefixlen 64 scopeid 0x2
ether 00:04:23:d4:bc:fb
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.0.230 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::206:5bff:feef:61f0%bge0 prefixlen 64 scopeid 0x3
ether 00:06:5b:ef:61:f0
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 202.184.208.230 netmask 0xffffff00 broadcast 202.184.208.255
inet6 fe80::206:5bff:feef:61f1%bge1 prefixlen 64 scopeid 0x4
ether 00:06:5b:ef:61:f1
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em2: flags=8802 <broadcast,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:36
media: Ethernet autoselect
status: no carrier
em3: flags=8802 <broadcast,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:37
media: Ethernet autoselect
status: no carrier
pflog0: flags=100 <promisc>mtu 33208
enc0: flags=0<> mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: em1 maxupd: 128
carp0: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.231 netmask 0xffffff00
carp: BACKUP vhid 1 advbase 1 advskew 200
carp1: flags=49 <up,loopback,running>mtu 1500
inet 192.168.0.232 netmask 0xffffff00
carp: BACKUP vhid 2 advbase 1 advskew 200
carp2: flags=49 <up,loopback,running>mtu 1500
inet 10.10.10.231 netmask 0xffffff00
carp: BACKUP vhid 3 advbase 1 advskew 200
carp3: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.201 netmask 0xffffff00
carp: BACKUP vhid 6 advbase 1 advskew 200</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>

sullrich · Sep 27, 2006, 2:36 AM

See http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense

Machine #1's advskew needs to be lower.

hsiang · Sep 27, 2006, 4:00 AM

From the GUI i had set the master CARP advertising frequency as 0 and is shown as 0 in config.xml. However from ifconfig the advskew still show as 200.

To change the value, i went to /etc/inc/interface.inc to manually change the advskew to 0. Now from ifconfig the master advskew is showing 0

carp0: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.231 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 0
carp1: flags=49 <up,loopback,running>mtu 1500
inet 192.168.0.232 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 0
carp2: flags=49 <up,loopback,running>mtu 1500
inet 10.10.10.231 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 0
carp3: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.201 netmask 0xffffff00
carp: MASTER vhid 6 advbase 1 advskew 0

Result:
after i reboot the master - the CARP Status will remain as backup, until i shut down the slave.

Question:
1. shouldn't the master take charge back as master when it alive?
2. is it a bug in the program where the advskew is binded to 200 as default??

Thanks</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running>

sullrich · Sep 27, 2006, 4:22 AM

After a reboot operation the CARP holdup timer will keep the item at 200 for atleast 2-3 minutes.

Afterwards the advskew is set to whatever is in config.xml.

hsiang · Sep 27, 2006, 8:39 AM

It seems like is not that case where the advskew will follow whatever in config.xml after 3 minutes

Here is the setting of in my config.xml for my Slave/Secondary Pfsense

<vip><mode>carp</mode>
<interface>wan</interface>
<vhid>1</vhid>
<advskew>100</advskew>
<password>carp</password>
<descr>WAN CARP</descr>
<type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>x.x.x.x</subnet></vip>
<vip><mode>carp</mode>
<interface>lan</interface>
<vhid>2</vhid>
<advskew>100</advskew>
<password>carp</password>
<descr>LAN CARP</descr>
<type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>192.168.0.232</subnet></vip>
<vip><mode>carp</mode>
<interface>opt2</interface>
<vhid>3</vhid>
<advskew>100</advskew>
<password>carp</password>
<descr>DMZ CARP</descr>
<type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>10.10.10.231</subnet></vip>

which show all is having advskew 100.

however when i issue ipconfig after the said time (actually more than 1/2 hour) it still show as below:

carp0: flags=49 <up,loopback,running>mtu 1500
inet 202.184.208.231 netmask 0xffffff00
carp: BACKUP vhid 1 advbase 1 advskew 200
carp1: flags=49 <up,loopback,running>mtu 1500
inet 192.168.0.232 netmask 0xffffff00
carp: BACKUP vhid 2 advbase 1 advskew 200
carp2: flags=49 <up,loopback,running>mtu 1500
inet 10.10.10.231 netmask 0xffffff00
carp: BACKUP vhid 3 advbase 1 advskew 200</up,loopback,running></up,loopback,running></up,loopback,running>

moffl · Dec 12, 2006, 5:54 AM

Did you find out what was causing your email/file timeout errors as i to am having this problem. The master has a advskew "0" and backup has "100" everything seems to sync up but emails over 150 - 200k hang the email client and cuases a timeout error and files over 500k just time out

hsiang · Dec 12, 2006, 7:40 AM

i had created CARP for each IP i am using for my servers. then it works fine after that. have a try.

moffl · Dec 12, 2006, 2:01 PM

@hsiang:

i had created CARP for each IP i am using for my servers. then it works fine after that. have a try.

Does this mean that you created virtual ips for each server and if so i have already done that

moffl · Dec 12, 2006, 2:08 PM

I also keep getting this in the system logs
kernel: arp_rtrequest: bad gateway 10.190.10.7 (!AF_LINK)
Dec 12 00:16:22 kernel: arp_rtrequest: bad gateway 10.190.10.6 (!AF_LINK)
Dec 12 00:16:21 kernel: arp_rtrequest: bad gateway 10.190.10.4 (!AF_LINK)
Dec 12 00:16:18 kernel: arp_rtrequest: bad gateway 10.190.10.2 (!AF_LINK)
Dec 12 00:16:17 kernel: arp_rtrequest: bad gateway 198.169.176.2 (!AF_LINK)
Dec 12 00:16:16 kernel: arp_rtrequest: bad gateway 10.190.10.25 (!AF_LINK)

sullrich · Dec 12, 2006, 3:27 PM

http://faq.pfsense.com/index.php?action=artikel&cat=1&id=167&artlang=en&highlight=bad%20gateway