CARP/VIPS issue in downloading large file



  • I have two pfsense firewalls set up with 4 interfaces each (Public, Private, Sync and DMZ). The interfaces are all running with CARP/VIPs. The issue that I have downloading or uploading a large size file (1MB +) to a webserver/qmail server is located at the DMZ from the public internet always hit a timeout when both firewalls are active. If I down one of the firewall, this issue does not occur.

    Both firewall are Dell Dual Xeon/2GB RAM. The pfsense version is 1.0-Snapshot-09-12-06. Any ideas on what should I check?

    Thanks
    Mypal



  • Take a look at this document: http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense and see if it brings up anything that may be setup wrong.



  • One additional thing that I forgot to mention, when I transfer file from a PC at the Private network (using VIPs gateway at Private) to DMZ network, I have no such issue.

    I suspect there must be some timeout when doing it from the Internet. How can I check this?

    Thanks
    Mypal



  • @mypal:

    One additional thing that I forgot to mention, when I transfer file from a PC at the Private network (using VIPs gateway at Private) to DMZ network, I have no such issue.

    I suspect there must be some timeout when doing it from the Internet. How can I check this?

    Thanks
    Mypal

    Sounds like your CARP address(es) on the internet side are fighting for master status to me.  Make sure that all the CARP IPs on the secondary are in BACKUP, not MASTER mode.

    –Bill



  • No, there is no timeout.  I really have no idea what you mean.

    Take a look at the wiki, I just updated the graphic so that it may be a bit better now.



  • I also facing the same issue from the WAN link. I had set the CARP as describe. When the issue occur the secondary firewall status is showing backup.

    However the external user is having time out problem when sending attachment email (150kb only), connecting to the mail server in dmz. Thing works fine when i shut down the secondary firewall totally.



  • Sorry but this is not enough information to go on.

    Please show us this command on both machines:

    ifconfig



  • #ifconfig on machine1

    em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.0.229 netmask 0xffffff00 broadcast 192.168.0.255
            inet6 fe80::214:22ff:fe18:8d47%em0 prefixlen 64 scopeid 0x1
            ether 00:14:22:18:8d:47
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 202.184.208.229 netmask 0xffffff00 broadcast 202.184.208.255
            inet6 fe80::214:22ff:fe18:8d48%em1 prefixlen 64 scopeid 0x2
            ether 00:14:22:18:8d:48
            media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    em2: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 10.0.0.229 netmask 0xffffff00 broadcast 10.0.0.255
            inet6 fe80::204:23ff:fed4:c338%em2 prefixlen 64 scopeid 0x3
            ether 00:04:23:d4:c3:38
            media: Ethernet autoselect
            status: no carrier
    em3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 10.10.10.229 netmask 0xffffff00 broadcast 10.10.10.255
            inet6 fe80::204:23ff:fed4:c339%em3 prefixlen 64 scopeid 0x4
            ether 00:04:23:d4:c3:39
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    em4: flags=8802 <broadcast,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:bc
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    em5: flags=8802 <broadcast,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:bd
            media: Ethernet autoselect
            status: no carrier
    pflog0: flags=100 <promisc>mtu 33208
    enc0: flags=0<> mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: em2 maxupd: 128
    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.231 netmask 0xffffff00
            carp: MASTER vhid 1 advbase 1 advskew 200
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.0.232 netmask 0xffffff00
            carp: MASTER vhid 2 advbase 1 advskew 200
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 10.10.10.231 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 200

    ifconfig on machine2

    em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 10.10.10.230 netmask 0xffffff00 broadcast 10.10.10.255
            inet6 fe80::204:23ff:fed4:bcfa%em0 prefixlen 64 scopeid 0x1
            ether 00:04:23:d4:bc:fa
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    em1: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 10.0.0.230 netmask 0xffffff00 broadcast 10.0.0.255
            inet6 fe80::204:23ff:fed4:bcfb%em1 prefixlen 64 scopeid 0x2
            ether 00:04:23:d4:bc:fb
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.0.230 netmask 0xffffff00 broadcast 192.168.0.255
            inet6 fe80::206:5bff:feef:61f0%bge0 prefixlen 64 scopeid 0x3
            ether 00:06:5b:ef:61:f0
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 202.184.208.230 netmask 0xffffff00 broadcast 202.184.208.255
            inet6 fe80::206:5bff:feef:61f1%bge1 prefixlen 64 scopeid 0x4
            ether 00:06:5b:ef:61:f1
            media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    em2: flags=8802 <broadcast,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:36
            media: Ethernet autoselect
            status: no carrier
    em3: flags=8802 <broadcast,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>ether 00:04:23:d4:c3:37
            media: Ethernet autoselect
            status: no carrier
    pflog0: flags=100 <promisc>mtu 33208
    enc0: flags=0<> mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: em1 maxupd: 128
    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.231 netmask 0xffffff00
            carp: BACKUP vhid 1 advbase 1 advskew 200
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.0.232 netmask 0xffffff00
            carp: BACKUP vhid 2 advbase 1 advskew 200
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 10.10.10.231 netmask 0xffffff00
            carp: BACKUP vhid 3 advbase 1 advskew 200
    carp3: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.201 netmask 0xffffff00
            carp: BACKUP vhid 6 advbase 1 advskew 200</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>



  • See http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense

    Machine #1's advskew needs to be lower.



  • From the GUI i had set the master CARP advertising frequency as 0 and is shown as 0 in config.xml. However from ifconfig the advskew still show as 200.

    To change the value, i went to /etc/inc/interface.inc to manually change the advskew to 0. Now from ifconfig the master advskew is showing 0

    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.231 netmask 0xffffff00
            carp: MASTER vhid 1 advbase 1 advskew 0
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.0.232 netmask 0xffffff00
            carp: MASTER vhid 2 advbase 1 advskew 0
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 10.10.10.231 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 0
    carp3: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.201 netmask 0xffffff00
            carp: MASTER vhid 6 advbase 1 advskew 0

    Result:
    after i reboot the master - the CARP Status will remain as backup, until i shut down the slave.

    Question:
    1. shouldn't the master take charge back as master when it alive?
    2. is it a bug in the program where the advskew is binded to 200 as default??

    Thanks</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running>



  • After a reboot operation the CARP holdup timer will keep the item at 200 for atleast 2-3 minutes.

    Afterwards the advskew is set to whatever is in config.xml.



  • It seems like is not that case where the advskew will follow whatever in config.xml after 3 minutes

    Here is the setting of in my config.xml for my Slave/Secondary Pfsense

    <vip><mode>carp</mode>
                            <interface>wan</interface>
                            <vhid>1</vhid>
                            <advskew>100</advskew>
                            <password>carp</password>
                            <descr>WAN CARP</descr>
                            <type>single</type>
                            <subnet_bits>24</subnet_bits>
                            <subnet>x.x.x.x</subnet></vip>
                    <vip><mode>carp</mode>
                            <interface>lan</interface>
                            <vhid>2</vhid>
                            <advskew>100</advskew>
                            <password>carp</password>
                            <descr>LAN CARP</descr>
                            <type>single</type>
                            <subnet_bits>24</subnet_bits>
                            <subnet>192.168.0.232</subnet></vip>
                    <vip><mode>carp</mode>
                            <interface>opt2</interface>
                            <vhid>3</vhid>
                            <advskew>100</advskew>
                            <password>carp</password>
                            <descr>DMZ CARP</descr>
                            <type>single</type>
                            <subnet_bits>24</subnet_bits>
                            <subnet>10.10.10.231</subnet></vip>

    which show all is having advskew 100.

    however when i issue ipconfig after the said time (actually more than 1/2 hour) it still show as below:

    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 202.184.208.231 netmask 0xffffff00
            carp: BACKUP vhid 1 advbase 1 advskew 200
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.0.232 netmask 0xffffff00
            carp: BACKUP vhid 2 advbase 1 advskew 200
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 10.10.10.231 netmask 0xffffff00
            carp: BACKUP vhid 3 advbase 1 advskew 200</up,loopback,running></up,loopback,running></up,loopback,running>



  • Did you find out what was causing your email/file timeout errors as i to am having this problem. The master has a advskew "0" and backup has "100" everything seems to sync up but emails over 150 - 200k hang the email client and cuases a timeout error and files over 500k just time out



  • i had created CARP for each IP i am using for my servers. then it works fine after that. have a try.



  • @hsiang:

    i had created CARP for each IP i am using for my servers. then it works fine after that. have a try.

    Does this mean that you created virtual ips for each server and if so i have already done that



  • I also keep getting this in the system logs
    kernel: arp_rtrequest: bad gateway 10.190.10.7 (!AF_LINK)
    Dec 12 00:16:22 kernel: arp_rtrequest: bad gateway 10.190.10.6 (!AF_LINK)
    Dec 12 00:16:21 kernel: arp_rtrequest: bad gateway 10.190.10.4 (!AF_LINK)
    Dec 12 00:16:18 kernel: arp_rtrequest: bad gateway 10.190.10.2 (!AF_LINK)
    Dec 12 00:16:17 kernel: arp_rtrequest: bad gateway 198.169.176.2 (!AF_LINK)
    Dec 12 00:16:16 kernel: arp_rtrequest: bad gateway 10.190.10.25 (!AF_LINK)




Locked