ntpd uses virtual IP - even in backup mode

Hi,

I have up and running a two node CARP system. Seems to work fine so far.

To have the time synced on both nodes (for DHCP) I enabled ntpd on both nodes with the same configuration.

What I see now is the ntp-packets going out on both nodes with the same source IP which is the virtual IP of the red interface. Even then when I enable maintenance mode on the backup node.
Interface Status:

Status
    up
MAC Address
    00:1a:8c:13:7e:39
IPv4 Address
    192.168.10.202
Subnet mask IPv4
    255.255.255.0
Gateway IPv4
    192.168.10.1
IPv6 Link Local
    fe80::21a:8cff:fe13:7e39%em2
DNS servers
    127.0.0.1
MTU
    1492

And a packet caputre on the red interface while in maintenance mode shows:

21:20:31.072367 00:1a:8c:13:7e:39 > 4c:09:d4:c3:63:62, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 7132, offset 0, flags [none], proto UDP (17), length 76)
    192.168.10.99.47791 > 46.4.16.145.123: [bad udp cksum 0x09ea -> 0x118a!] NTPv4, length 48
	Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 6 (64s), precision -21
	Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
	  Reference Timestamp:  0.000000000
	  Originator Timestamp: 0.000000000
	  Receive Timestamp:    0.000000000
	  Transmit Timestamp:   3739029631.072245284 (2018/06/26 21:20:31)
	    Originator - Receive Timestamp:  0.000000000
	    Originator - Transmit Timestamp: 3739029631.072245284 (2018/06/26 21:20:31)

Obviously this causes issues. As you can see 192.168.10.99 is not the configured IP for the interface.

So what can I do to tell ntpd to use the real IP instead of the virtual? Or just use virtual interface in case it is CARP master?

Any clues?

Thanks!

/KNEBB

Additionally,

if I stop the ntpd service and update my time with

ntpdate 0.de.pool.ntp.org

and do a packet capture in parallel it will send out with the correct (not virtual) IP:

21:36:29.705460 IP (tos 0x0, ttl 64, id 27139, offset 0, flags [none], proto UDP (17), length 76)
    192.168.10.202.123 > 78.47.234.59.123: NTPv4, length 48
	Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3 (8s), precision -6
	Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
	  Reference Timestamp:  0.000000000
	  Originator Timestamp: 0.000000000
	  Receive Timestamp:    0.000000000
	  Transmit Timestamp:   3739030589.705351263 (2018/06/26 21:36:29)
	    Originator - Receive Timestamp:  0.000000000
	    Originator - Transmit Timestamp: 3739030589.705351263 (2018/06/26 21:36:29)

So this is definetly an issue of ntpd!

Another addition:

ntpd knows the configuration item

query source

Is there any way to add this to the configuration of ntpd? I tried to stop ntpd, edit /var/etc/ntpd.conf but when starting the file got overwritten.

Otherwise, is there any possibility to disable ntpd? It is causing serious issues with the virtualIP.

jimp

What exactly do you have set in your NTP configuration?

What exactly do you have set in your Outbound NAT?

There are a couple possible issues here, but the most common error would be Outbound NAT rules that have too broad a source (e.g. any) which are causing traffic from the firewall itself to also get NAT applied.

@jimp said in ntpd uses virtual IP - even in backup mode:

What exactly do you have set in your Outbound NAT?

That was indeed the point. Sometimes it's just dump not to check everything.

Switched back to automated outbound NAT and now it's working fine.

Thanks!

jimp

The better solution there is to setup Hybrid Outbound NAT so that your internal networks, or ideally an alias of them, get outbound NAT to your CARP VIP.

You won't get seamless HA failover unless the NAT states use the CARP VIP. But you can make it easy and get that done with ~2 rules per WAN using an alias. I like making an RFC1918 alias containing 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8. Then I switch to Hybrid Outbound NAT and then setup two rules:

NAT on WAN from RFC1918, any port, to any host, udp/500, translate to the WAN CARP VIP, static port
NAT on WAN from RFC1918, any port, to any host/any port, translate to the WAN CARP VIP

@jimp :

Could you explain what the difference would be to the fully automated NATing?
Doesn't fully automated nat all local nets to the outbound IP?

And when I create the above rules, I have the same as before or am I wrong? When NATing to the CARP IP I have again my ntpd being mapped to CARP IP even in backup mode?

jimp

Automatic outbound NAT creates mappings to the interface address. In HA, each node has a different interface address. The CARP VIP is shared.

So if you have state synchronization setup, a NAT mapping from LAN to the WAN IP address will not be a valid state on the secondary node, so the connections would die during failover.

If you NAT to the CARP VIP, that is shared, so the states are valid for both HA nodes.

The difference in my example rules is the strict source. Traffic like NTP from the firewall itself will be sourced from the interface address on WAN, which is separate. Those kind of things do not need to fail over as they are handled independently. So with the example rules I gave, only the local/private/LAN traffic gets NAT applied.

@jimp Thanks, got it so far.

What I am wondering- when I set it now to hybrid and add some manual rules as you suggested (has been more or less equal to mine before) I am facing the same issue again.
The ntp packets are maped to CARP IP instead of interface IP.

ROT is my WAN interface and the 192.168.10.99 is the CARP IP of the WAN interface.
So it is the same I would say. Still, the ntp packets get nated to 192.168.10.99 - CARP.

Why?

jimp

What do you have set in the NTP settings? Are there any interfaces selected explicitly?

Based on what you show, it should not be translating the WAN traffic. You may need to reset the states table for that to take effect, however. Go to Diagnostics > States, Reset States tab and reset them.

Hi,

yes I had the interfaces restricted - I did not want the ntpd to LISTEN on the WAN interface.

Reseting state did not help- same issue.

But attaching ntpd to the WAN interface did the trick.

Now having hybrid NAT and proper ntpd source IP.

Thanks& Greetings