Sync only works on LAN, but not the dedicated interface or WAN
-
We have 2 identical setups running with 3 network interfaces each. We setup CARP according to the howto's that are our there, but are only able to get it to sync if it is set to sync on the LAN interface. If we switch it back to sync on the CARP interface, it stops working and gives "A communications error occurred while attempting XMLRPC sync with username admin"
On both machines
2.1-RELEASE (i386), built on Wed Sep 11 18:16:50 EDT 2013, FreeBSD 8.3-RELEASE-p11WAN has 64. IPs
LAN is 192.168.2.0/24
CARP is 192.168.5.0/24each firewall has it's own external IP and there is a 3rd, virtual IP for them
firewalls are 192.168.2.11 and 192.168.2.12 with virtual 192.168.2.10
the CARP IPs are 192.168.5.11 and 192.168.5.12WAN has gateway, LAN and CARP do not
Some more notes
We've already gone through the CARP troubleshooting and the Sync troubleshooting links
We've already removing and re-setting up the CARP interfaces
We've checked and rechecked the users and passwords, plus it works when on the LAN
We do have an allow IPv4* * * * * * none rule for CARP on both machines
The issue was not resolved by assigning new IPs to the CARP interfaces
We've tried switching to HTTP, unchecking Disable webConfigurator anti-lockout, disabling DNS rebinding and HTTP_REFERER, as well as every combination there of.
We only entered sync config IP, remote username, and pass on the master. (again, works fine if set to sync on LAN)
We use the default port 444 for the webConfiguratorThe interesting part
If I ssh to each firewall I can ping the other's LAN ip AND CARP ip, but if I telnet to port 444 I only get a response from the LAN ip. The CARP ip times out. (I've attached image of what shows up in the firewall log) We do get entries on the slave for each attempt to sync, so I think it's not outgoing blocked by the master. We've even tried adding allow rules with the easyrule links from the firewall entries to no avail.[2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(6): ping -c 1 192.168.2.12
PING 192.168.2.12 (192.168.2.12): 56 data bytes
64 bytes from 192.168.2.12: icmp_seq=0 ttl=64 time=0.447 ms
–- 192.168.2.12 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.447/0.447/0.447/0.000 ms[2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(7): ping -c 1 192.168.5.12
PING 192.168.5.12 (192.168.5.12): 56 data bytes
64 bytes from 192.168.5.12: icmp_seq=0 ttl=64 time=3.412 ms
–- 192.168.5.12 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.412/3.412/3.412/0.000 ms[2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(8): telnet 192.168.2.12 444
Trying 192.168.2.12…
Connected to firewall2.drive-on-in.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed.[2.1-RELEASE][username@firewall1.drive-on-in.com]/home/username(9): telnet 192.168.5.12 444
Trying 192.168.5.12…
telnet: connect to address 192.168.5.12: Operation timed out
telnet: Unable to connect to remote hostIt's as if it is ignoring the rule to allow all traffic on the CARP interface, or if it's not being routed correctly, or if the webConfigurator server isn't listening to the CARP interface. Any help would be appreciated.
Thank you.
-
LAN and WAN are your CARP interfaces.
What you call CARP interfaces are actually your sync interface and you don't have CARP there.Step 1. Remove all virtual CARP IPs on both firewalls.
Step 2. Add firewall rule on the SYNC interface - allow protocol ANY from SYNC Subnet to SYNC Address on both firewalls
Step 3. Check if Webconfigurator is configured to listen on the same ports on both firewalls.
Step 4. Configure XMLRPC sync (high availability) on Master node typing the SYNC IP of the backup firewall.
Step 5. Add your Virtual IPs (CARP) on the Master firewall. They should be automatically created on the backup firewall. -
Thank you for your reply.
I apologize for the confusion. We named the dedicated sync interfaces CARP because that setup is what we dedicated for, even though technically, yes, they are used for the syncing and not the actual redundant addresses. Maybe we should have called it OPT1 or SYNC, but we didn't.
We've already, and repeatedly, removed and readded both the virtual IPs and the allow ANY firewall rules on both firewalls. We have verified that we use the default port 444 for the webConfigurator on both firewalls. The virtual IPs that we create do work appropriately.
I've attached an image of the firewall rule which we have on BOTH firewalls.
-
Could you please paste the ipv4 part of "netstat -nr" and first paragraph of "netstat -ban"?
-
on firewall1
Destination Gateway Flags Refs Use Netif Expire
default 64.250.236.49 UGS 0 1369548 bge1
10.112.1.0/24 10.112.1.2 UGS 0 18535 ovpns1
10.112.1.1 link#10 UHS 0 0 lo0
10.112.1.2 link#10 UH 0 0 ovpns1
64.250.224.2 64.250.236.49 UGHS 0 0 bge1
64.250.224.6 64.250.236.49 UGHS 0 0 bge1
64.250.236.48/28 link#2 U 0 79398 bge1
64.250.236.51 link#2 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 29 lo0
192.168.2.0/24 link#1 U 0 1572118 bge0
192.168.2.10 link#9 UH 0 0 lan_vi
192.168.2.11 link#1 UHS 0 0 lo0
192.168.5.0/24 link#7 U 0 3 ue0
192.168.5.11 link#7 UHS 0 0 lo0Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.2.11.7829 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.55582 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.53972 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.42430 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.61382 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.22076 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.37698 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.56853 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.46964 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.12438 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.44246 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.11.2302 192.168.2.101.80 TIME_WAIT
tcp4 0 52 192.168.2.11.22 10.112.1.6.59526 ESTABLISHED
tcp4 0 0 64.250.236.51.1194 24.120.44.147.28901 ESTABLISHED
tcp4 0 151 64.250.236.51.1194 24.120.44.147.14495 ESTABLISHED
tcp6 0 0 *.53 . LISTEN
tcp4 0 0 *.53 . LISTEN
tcp4 0 0 *.80 . LISTEN
tcp6 0 0 *.444 . LISTEN
tcp4 0 0 *.444 . LISTEN
tcp4 0 0 64.250.236.51.1194 . LISTEN
tcp4 0 0 *.22 . LISTEN
tcp6 0 0 *.22 . LISTEN
udp4 0 0 10.112.1.1.123 .
udp6 0 0 fe80
:2e0:b6ff.123 .
udp4 0 0 192.168.2.10.123 .
udp6 0 0 fe80:7::200:ff:f.123 .
udp4 0 0 192.168.5.11.123 .
udp6 0 0 fe80:5::1.123 .
udp6 0 0 ::1.123 .
udp4 0 0 127.0.0.1.123 .
udp6 0 0 fe80:2::2e0:b6ff.123 .
udp4 0 0 64.250.236.51.123 .
udp6 0 0 fe80:1::2e0:b6ff.123 .
udp4 0 0 192.168.2.11.123 .
udp6 0 0 *.123 .
udp4 0 0 *.123 .
udp6 0 0 *.18596 .
udp4 0 0 *.11601 .
udp4 0 0 *.514 .
udp6 0 0 *.514 .
udp4 0 0 *.67 .
udp6 0 0 *.12088 .
udp4 0 0 *.13900 .
udp6 0 0 *.45734 .
udp4 0 0 *.57260 .
udp4 0 0 . .
udp6 0 0 *.11874 .
udp4 0 0 *.54555 .
udp4 0 0 . .
udp6 0 0 *.53 .
udp4 0 0 .53 .
udp4 0 0 127.0.0.1.6969 .
ip 4 0 0 . .
ip 4 0 0 . .
icm4 0 0 . .
icm4 2320 0 . .
icm4 2320 0 . .
icm4 0 0 64.250.236.51. .
ip64 0 0 . .
ip64 0 0 . .
icm6 0 0 . .
icm6 0 0 . .on firewall2
Destination Gateway Flags Refs Use Netif Expire
default 64.250.236.49 UGS 0 774 bge1
10.112.1.0/24 10.112.1.2 UGS 0 32 ovpns1
10.112.1.1 link#10 UHS 0 0 lo0
10.112.1.2 link#10 UH 0 0 ovpns1
64.250.224.2 64.250.236.49 UGHS 0 0 bge1
64.250.224.6 64.250.236.49 UGHS 0 0 bge1
64.250.236.48/28 link#2 U 0 73069 bge1
64.250.236.52 link#2 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 37 lo0
192.168.2.0/24 link#1 U 0 73168 bge0
192.168.2.12 link#1 UHS 0 0 lo0
192.168.5.0/24 link#7 U 0 0 ue0
192.168.5.12 link#7 UHS 0 0 lo0Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.2.12.46778 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.13781 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.11330 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.46078 192.168.2.101.80 TIME_WAIT
tcp4 0 0 64.250.236.52.444 24.120.44.147.28941 ESTABLISHED
tcp4 0 0 192.168.2.12.38530 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.13412 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.44096 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.3913 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.39112 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.39141 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.49058 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.26410 192.168.2.101.80 TIME_WAIT
tcp4 0 0 192.168.2.12.22 192.168.2.231.35492 ESTABLISHED
tcp4 0 0 64.250.236.52.444 24.120.44.147.36322 TIME_WAIT
tcp6 0 0 *.53 . LISTEN
tcp4 0 0 *.53 . LISTEN
tcp4 0 0 64.250.236.52.1194 . LISTEN
tcp6 0 0 *.444 . LISTEN
tcp4 0 0 *.444 . LISTEN
tcp4 0 0 *.22 . LISTEN
tcp6 0 0 *.22 . LISTEN
udp4 0 0 10.112.1.1.123 .
udp6 0 0 fe80:2e0:b6ff.123 .
udp4 0 0 192.168.2.10.123 .
udp4 0 0 64.250.236.50.123 .
udp6 0 0 fe80:7::200:ff:f.123 .
udp4 0 0 192.168.5.12.123 .
udp6 0 0 fe80:5::1.123 .
udp6 0 0 ::1.123 .
udp4 0 0 127.0.0.1.123 .
udp6 0 0 fe80:2::2e0:b6ff.123 .
udp4 0 0 64.250.236.52.123 .
udp6 0 0 fe80:1::2e0:b6ff.123 .
udp4 0 0 192.168.2.12.123 .
udp6 0 0 *.123 .
udp4 0 0 *.123 .
udp4 0 0 *.67 .
udp6 0 0 *.60472 .
udp4 0 0 *.57100 .
udp6 0 0 *.51300 .
udp4 0 0 *.30519 .
udp6 0 0 *.53 .
udp4 0 0 *.53 .
udp4 0 0 *.514 .
udp6 0 0 *.514 .
udp6 0 0 *.25859 .
udp4 0 0 *.44756 .
udp4 0 0 . .
udp6 0 0 *.20948 .
udp4 0 0 .30785 .
udp4 0 0 . .
udp4 0 0 127.0.0.1.6969 .
ip 4 0 0 . .
ip 4 0 0 . .
icm4 0 0 . .
icm4 2320 0 . .
icm4 2320 0 . .
icm4 0 0 64.250.236.52. .
ip64 0 0 . .
ip64 0 0 . .
icm6 0 0 . .
icm6 0 0 . . -
Everything seems fine… At least I see no reason for such behavior.
Check /var/etc/lighty-webConfigurator.conf for something like:
## bind to port (default: 80) server.bind = "0.0.0.0" server.port = 80 $SERVER["socket"] == "0.0.0.0:80" { } $SERVER["socket"] == "[::]:80" { }But with 444 instead of 80 for you.
If that's OK then I guess someone of the devs should suggest something…Umm, maybe "tcpdump -i um0 host 192.168.5.11" on the slave could show something useful.
-
Good suggestion.
: cat /var/etc/lighty-webConfigurator.conf
bind to port (default: 80)
server.bind = "0.0.0.0"
server.port = 444
$SERVER["socket"] == "0.0.0.0:444" { }
$SERVER["socket"] == "[::]:444" {ssl configuration
ssl.engine = "enable"
ssl.pemfile = "/var/etc/cert.pem"}
I'll work on the tcpdump and post it as soon as I can.
-
Got busy and haven't had time to tcpdump yet. Any other ideas out there about anything else that can be looked at in the interim?