Sync only works on LAN, but not the dedicated interface or WAN



  • We have 2 identical setups running with 3 network interfaces each. We setup CARP according to the howto's that are our there, but are only able to get it to sync if it is set to sync on the LAN interface. If we switch it back to sync on the CARP interface, it stops working and gives "A communications error occurred while attempting XMLRPC sync with username admin"

    On both machines
    2.1-RELEASE (i386), built on Wed Sep 11 18:16:50 EDT 2013, FreeBSD 8.3-RELEASE-p11

    WAN has 64. IPs
    LAN is 192.168.2.0/24
    CARP is 192.168.5.0/24

    each firewall has it's own external IP and there is a 3rd, virtual IP for them
    firewalls are 192.168.2.11 and 192.168.2.12 with virtual 192.168.2.10
    the CARP IPs are 192.168.5.11 and 192.168.5.12

    WAN has gateway, LAN and CARP do not

    Some more notes
    We've already gone through the CARP troubleshooting and the Sync troubleshooting links
    We've already removing and re-setting up the CARP interfaces
    We've checked and rechecked the users and passwords, plus it works when on the LAN
    We do have an allow IPv4* * * * * * none rule for CARP on both machines
    The issue was not resolved by assigning new IPs to the CARP interfaces
    We've tried switching to HTTP, unchecking Disable webConfigurator anti-lockout, disabling DNS rebinding and HTTP_REFERER, as well as every combination there of.
    We only entered sync config IP, remote username, and pass on the master. (again, works fine if set to sync on LAN)
    We use the default port 444 for the webConfigurator

    The interesting part
    If I ssh to each firewall I can ping the other's LAN ip AND CARP ip, but if I telnet to port 444 I only get a response from the LAN ip. The CARP ip times out. (I've attached image of what shows up in the firewall log) We do get entries on the slave for each attempt to sync, so I think it's not outgoing blocked by the master. We've even tried adding allow rules with the easyrule links from the firewall entries to no avail.

    [2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(6): ping -c 1 192.168.2.12
    PING 192.168.2.12 (192.168.2.12): 56 data bytes
    64 bytes from 192.168.2.12: icmp_seq=0 ttl=64 time=0.447 ms
    –- 192.168.2.12 ping statistics ---
    1 packets transmitted, 1 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 0.447/0.447/0.447/0.000 ms

    [2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(7): ping -c 1 192.168.5.12
    PING 192.168.5.12 (192.168.5.12): 56 data bytes
    64 bytes from 192.168.5.12: icmp_seq=0 ttl=64 time=3.412 ms
    –- 192.168.5.12 ping statistics ---
    1 packets transmitted, 1 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 3.412/3.412/3.412/0.000 ms

    [2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(8): telnet 192.168.2.12 444
    Trying 192.168.2.12…
    Connected to firewall2.drive-on-in.com.
    Escape character is '^]'.
    ^]
    telnet> quit
    Connection closed.

    [2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(9): telnet 192.168.5.12 444
    Trying 192.168.5.12…
    telnet: connect to address 192.168.5.12: Operation timed out
    telnet: Unable to connect to remote host

    It's as if it is ignoring the rule to allow all traffic on the CARP interface, or if it's not being routed correctly, or if the webConfigurator server isn't listening to the CARP interface. Any help would be appreciated.

    Thank you.




  • LAN and WAN are your CARP interfaces.
    What you call CARP interfaces are actually your sync interface and you don't have CARP there.

    Step 1. Remove all virtual CARP IPs on both firewalls.
    Step 2. Add firewall rule on the SYNC interface - allow protocol ANY from SYNC Subnet to SYNC Address on both firewalls
    Step 3. Check if Webconfigurator is configured to listen on the same ports on both firewalls.
    Step 4. Configure XMLRPC sync (high availability) on Master node typing the SYNC IP of the backup firewall.
    Step 5. Add your Virtual IPs (CARP) on the Master firewall. They should be automatically created on the backup firewall.



  • Thank you for your reply.

    I apologize for the confusion. We named the dedicated sync interfaces CARP because that setup is what we dedicated for, even though technically, yes, they are used for the syncing and not the actual redundant addresses. Maybe we should have called it OPT1 or SYNC, but we didn't.

    We've already, and repeatedly, removed and readded both the virtual IPs and the allow ANY firewall rules on both firewalls. We have verified that we use the default port 444 for the webConfigurator on both firewalls. The virtual IPs that we create do work appropriately.

    I've attached an image of the firewall rule which we have on BOTH firewalls.




  • Could you please paste the ipv4 part of "netstat -nr" and first paragraph of "netstat -ban"?



  • on firewall1
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            64.250.236.49      UGS        0  1369548  bge1
    10.112.1.0/24      10.112.1.2        UGS        0    18535 ovpns1
    10.112.1.1        link#10            UHS        0        0    lo0
    10.112.1.2        link#10            UH          0        0 ovpns1
    64.250.224.2      64.250.236.49      UGHS        0        0  bge1
    64.250.224.6      64.250.236.49      UGHS        0        0  bge1
    64.250.236.48/28  link#2            U          0    79398  bge1
    64.250.236.51      link#2            UHS        0        0    lo0
    127.0.0.1          link#5            UH          0      29    lo0
    192.168.2.0/24    link#1            U          0  1572118  bge0
    192.168.2.10      link#9            UH          0        0 lan_vi
    192.168.2.11      link#1            UHS        0        0    lo0
    192.168.5.0/24    link#7            U          0        3    ue0
    192.168.5.11      link#7            UHS        0        0    lo0

    Active Internet connections (including servers)
    Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
    tcp4      0      0 192.168.2.11.7829      192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.55582    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.53972    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.42430    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.61382    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.22076    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.37698    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.56853    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.46964    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.12438    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.44246    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.11.2302      192.168.2.101.80      TIME_WAIT
    tcp4      0    52 192.168.2.11.22        10.112.1.6.59526      ESTABLISHED
    tcp4      0      0 64.250.236.51.1194    24.120.44.147.28901    ESTABLISHED
    tcp4      0    151 64.250.236.51.1194    24.120.44.147.14495    ESTABLISHED
    tcp6      0      0 *.53                  .                    LISTEN
    tcp4      0      0 *.53                  .                    LISTEN
    tcp4      0      0 *.80                  .                    LISTEN
    tcp6      0      0 *.444                  .                    LISTEN
    tcp4      0      0 *.444                  .                    LISTEN
    tcp4      0      0 64.250.236.51.1194    .                    LISTEN
    tcp4      0      0 *.22                  .                    LISTEN
    tcp6      0      0 *.22                  .                    LISTEN
    udp4      0      0 10.112.1.1.123        .
    udp6      0      0 fe80🅰:2e0:b6ff.123  .
    udp4      0      0 192.168.2.10.123      .
    udp6      0      0 fe80:7::200:ff:f.123  .
    udp4      0      0 192.168.5.11.123      .
    udp6      0      0 fe80:5::1.123          .
    udp6      0      0 ::1.123                .
    udp4      0      0 127.0.0.1.123          .
    udp6      0      0 fe80:2::2e0:b6ff.123  .
    udp4      0      0 64.250.236.51.123      .
    udp6      0      0 fe80:1::2e0:b6ff.123  .
    udp4      0      0 192.168.2.11.123      .
    udp6      0      0 *.123                  .
    udp4      0      0 *.123                  .
    udp6      0      0 *.18596                .
    udp4      0      0 *.11601                .
    udp4      0      0 *.514                  .
    udp6      0      0 *.514                  .
    udp4      0      0 *.67                  .
    udp6      0      0 *.12088                .
    udp4      0      0 *.13900                .
    udp6      0      0 *.45734                .
    udp4      0      0 *.57260                .
    udp4      0      0 .                    .
    udp6      0      0 *.11874                .
    udp4      0      0 *.54555                .
    udp4      0      0 .                    .
    udp6      0      0 *.53                  .
    udp4      0      0 .53                  .
    udp4      0      0 127.0.0.1.6969        .
    ip 4      0      0 .                    .
    ip 4      0      0 .                    .
    icm4      0      0 .                    .
    icm4    2320      0 .                    .
    icm4    2320      0 .                    .
    icm4      0      0 64.250.236.51.
            .
    ip64      0      0 .                    .
    ip64      0      0 .                    .
    icm6      0      0 .                    .
    icm6      0      0 .                    .

    on firewall2
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            64.250.236.49      UGS        0      774  bge1
    10.112.1.0/24      10.112.1.2        UGS        0      32 ovpns1
    10.112.1.1        link#10            UHS        0        0    lo0
    10.112.1.2        link#10            UH          0        0 ovpns1
    64.250.224.2      64.250.236.49      UGHS        0        0  bge1
    64.250.224.6      64.250.236.49      UGHS        0        0  bge1
    64.250.236.48/28  link#2            U          0    73069  bge1
    64.250.236.52      link#2            UHS        0        0    lo0
    127.0.0.1          link#5            UH          0      37    lo0
    192.168.2.0/24    link#1            U          0    73168  bge0
    192.168.2.12      link#1            UHS        0        0    lo0
    192.168.5.0/24    link#7            U          0        0    ue0
    192.168.5.12      link#7            UHS        0        0    lo0

    Active Internet connections (including servers)
    Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
    tcp4      0      0 192.168.2.12.46778    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.13781    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.11330    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.46078    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 64.250.236.52.444      24.120.44.147.28941    ESTABLISHED
    tcp4      0      0 192.168.2.12.38530    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.13412    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.44096    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.3913      192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.39112    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.39141    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.49058    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.26410    192.168.2.101.80      TIME_WAIT
    tcp4      0      0 192.168.2.12.22        192.168.2.231.35492    ESTABLISHED
    tcp4      0      0 64.250.236.52.444      24.120.44.147.36322    TIME_WAIT
    tcp6      0      0 *.53                  .                    LISTEN
    tcp4      0      0 *.53                  .                    LISTEN
    tcp4      0      0 64.250.236.52.1194    .                    LISTEN
    tcp6      0      0 *.444                  .                    LISTEN
    tcp4      0      0 *.444                  .                    LISTEN
    tcp4      0      0 *.22                  .                    LISTEN
    tcp6      0      0 *.22                  .                    LISTEN
    udp4      0      0 10.112.1.1.123        .
    udp6      0      0 fe80🅰:2e0:b6ff.123  .
    udp4      0      0 192.168.2.10.123      .
    udp4      0      0 64.250.236.50.123      .
    udp6      0      0 fe80:7::200:ff:f.123  .
    udp4      0      0 192.168.5.12.123      .
    udp6      0      0 fe80:5::1.123          .
    udp6      0      0 ::1.123                .
    udp4      0      0 127.0.0.1.123          .
    udp6      0      0 fe80:2::2e0:b6ff.123  .
    udp4      0      0 64.250.236.52.123      .
    udp6      0      0 fe80:1::2e0:b6ff.123  .
    udp4      0      0 192.168.2.12.123      .
    udp6      0      0 *.123                  .
    udp4      0      0 *.123                  .
    udp4      0      0 *.67                  .
    udp6      0      0 *.60472                .
    udp4      0      0 *.57100                .
    udp6      0      0 *.51300                .
    udp4      0      0 *.30519                .
    udp6      0      0 *.53                  .
    udp4      0      0 *.53                  .
    udp4      0      0 *.514                  .
    udp6      0      0 *.514                  .
    udp6      0      0 *.25859                .
    udp4      0      0 *.44756                .
    udp4      0      0 .                    .
    udp6      0      0 *.20948                .
    udp4      0      0 .30785                .
    udp4      0      0 .                    .
    udp4      0      0 127.0.0.1.6969        .
    ip 4      0      0 .                    .
    ip 4      0      0 .                    .
    icm4      0      0 .                    .
    icm4    2320      0 .                    .
    icm4    2320      0 .                    .
    icm4      0      0 64.250.236.52.
            .
    ip64      0      0 .                    .
    ip64      0      0 .                    .
    icm6      0      0 .                    .
    icm6      0      0 .                    .



  • Everything seems fine… At least I see no reason for such behavior.

    Check /var/etc/lighty-webConfigurator.conf for something like:

    ## bind to port (default: 80)
    server.bind  = "0.0.0.0"
    server.port  = 80
    $SERVER["socket"]  == "0.0.0.0:80" { }
    $SERVER["socket"]  == "[::]:80" {
     }
    

    But with 444 instead of 80 for you.
    If that's OK then I guess someone of the devs should suggest something…

    Umm, maybe "tcpdump -i um0 host 192.168.5.11" on the slave could show something useful.



  • Good suggestion.

    : cat /var/etc/lighty-webConfigurator.conf

    bind to port (default: 80)

    server.bind  = "0.0.0.0"
    server.port  = 444
    $SERVER["socket"]  == "0.0.0.0:444" { }
    $SERVER["socket"]  == "[::]:444" {

    ssl configuration

    ssl.engine = "enable"
    ssl.pemfile = "/var/etc/cert.pem"

    }

    I'll work on the tcpdump and post it as soon as I can.



  • Got busy and haven't had time to tcpdump yet. Any other ideas out there about anything else that can be looked at in the interim?


Log in to reply