Update to 2.5.0 broke DHCP relay

fwcheck · Feb 19, 2021, 3:17 PM

I can confirm your findings. The DHCP-Relay setup was/is broken on update.

I used

/usr/local/sbin/dhcrelay -id vmx1 -iu vmx0 -a -m replace IP_dhcp-server1 IP_dhcpsever2

as a quick fix. (one WAN (vmx0), one LAN-Interface (vmx1))

I was very delighted to see that within 2.4.5_p1 the DHCP-Handling was much better. I did not have time to look deeply into this problem, so this is a quick fix.

viktor_g · Feb 19, 2021, 3:57 PM

@fwcheck said in Update to 2.5.0 broke DHCP relay:

I can confirm your findings. The DHCP-Relay setup was/is broken on update.

I used

/usr/local/sbin/dhcrelay -id vmx1 -iu vmx0 -a -m replace IP_dhcp-server1 IP_dhcpsever2

as a quick fix. (one WAN (vmx0), one LAN-Interface (vmx1))

I was very delighted to see that within 2.4.5_p1 the DHCP-Handling was much better. I did not have time to look deeply into this problem, so this is a quick fix.

Could you provide vmx0 and vmx1 IP addresses and IP_dhcp-server1, IP_dhcpsever2 to check ?

fwcheck · Feb 19, 2021, 4:11 PM

@viktor_g
Short:
vmx0 is a private IP
vmx1 is a public IP
Both dhcp Servers are upstream (behind vmx0)

Later today i will do a clean update install (2.4.4_p3 -> 2.5.0) and can check that again. I will come back to this.

viktor_g · Feb 19, 2021, 4:18 PM

@fwcheck we need to know IP addresses to test
you can change the network part of address if you don't want to show it

jimp · Feb 19, 2021, 4:43 PM

To diagnose this we are going to need better info. You can redact some things but we need to know:

The interfaces involved, and their subnets
All of the DHCP relay settings used

If you redact anything replace them with appropriate dummy addresses but keep at least enough to uniquely identify them (e.g. 10.0.0.1/24 -> x.x.x.1/24, 10.4.1.2/24->y.y.y.2/24).

Bonus points for getting the process output from 2.4.x and 2.5.x to compare what it's trying to run in each case.

fwcheck · Feb 19, 2021, 7:44 PM

I am not quite sure but i think within 2.4.5_p1 the dhcp was supplying adresses to openvpn-clients via a bridge. I will look into the old configuration ans supply info on monday afternoon.
Redundant carp setup
looks basically like this:
vmx0 192.168.0.3/29
vmx1 1.1.1.1/24 (a public subnet)
dhcp-server 10.2.1.14
dhcp-server 10.2.1.13

thesurf · Feb 19, 2021, 8:48 PM

@jimp

lets try here to give you a quick headsup since it is alread 9pm in germany.

Building setup:

Building a  ---- LWL dark fiber ---- Head office

Some quick drawing from draw.io

Total

The Carp and pfSense cluster didn't play a role here. I disabled carp in testing.

What I have seen on fwint3 that the dhcp request arrive as usal on the clients interface. But nothing is send out to the dhcp server.

There are more vlans that have ther own interface assigned in pfSense and other components, but they can be left out.

The cluster fwint3 and fwint4 have been updated without any problems from 2.4.5p1 to 2.5.0. Routing traffic everything works but not the dhcp forwarding.

I think I do not have logs of 2.4.5p1. And on 2.5.0 the dhcp forward did not start. There is nothing in the dhcp log nor is there something in the syslog. If I disable dhcp forwarding and start dhcp server on the fwint3 I see dhcp logs and leases go out and work like they did when they where forwarded.

Hope that helps. I you need something let me know.

thesurf · Feb 19, 2021, 8:49 PM

@jimp
Hi forgot. All Interface are intel x520 nic. So Interfaces will be ix0 and ix1 on each firewall. On top that is the vlan so ix0.8 and so on. NO Bridge.

johnsdixon · Feb 20, 2021, 5:58 PM

Also occurring on VMware ESXi 6.7 hosted environment with VMXNET3 cards. Single DHCP server target, with relays from six different networks (some VLAN, some 'direct' connect). Appears that all do not pass the DHCP packets to the server (none received in the external server logs), but packets are seen by packet capture on the interfaces concerned on pfSense.
Happy to build a test 2.5.0 environment to provide further info and logs if necessary.

viktor_g · Feb 21, 2021, 5:25 PM

@johnsdixon said in Update to 2.5.0 broke DHCP relay:

Happy to build a test 2.5.0 environment to provide further info and logs if necessary.

very good, we need to know:

DHCP Relay configuration (screenshot or <dhcrelay> part from your config.xml);
Routing table (netstat -rn output);
Interfaces IP addresses (ifconfig output);

thesurf · Feb 22, 2021, 10:46 AM

@viktor_g said in Update to 2.5.0 broke DHCP relay:

@johnsdixon said in Update to 2.5.0 broke DHCP relay:

Happy to build a test 2.5.0 environment to provide further info and logs if necessary.

very good, we need to know:

DHCP Relay configuration (screenshot or <dhcrelay> part from your config.xml);

Routing table (netstat -rn output);

Interfaces IP addresses (ifconfig output);

Hi, network plan I have already posted abouve.

Here are the requeired data:

[2.5.0-RELEASE][root@fwint3.XXXXXXXX.local]/conf: netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.10.65.3         UGS    lagg0.65
10.10.55.0/28      link#19            U      lagg0.11
10.10.55.2         link#19            UHS         lo0
10.10.56.0/30      link#8             U          bge1
10.10.56.1         link#8             UHS         lo0
10.10.64.0/28      link#18            U      lagg0.64
10.10.64.4         link#18            UHS         lo0
10.10.64.6         link#18            UHS         lo0
10.10.65.0/28      link#17            U      lagg0.65
10.10.65.4         link#17            UHS         lo0
10.10.65.6         link#17            UHS         lo0
127.0.0.1          link#10            UH          lo0
192.168.8.0/24     link#14            U       lagg0.8
192.168.8.252      link#14            UHS         lo0
192.168.8.254      link#14            UHS         lo0
192.168.11.0/24    link#19            U      lagg0.11
192.168.11.252     link#19            UHS         lo0
192.168.11.254     link#19            UHS         lo0
192.168.24.0/24    link#15            U      lagg0.24
192.168.24.252     link#15            UHS         lo0
192.168.24.254     link#15            UHS         lo0
192.168.71.0/24    link#20            U      lagg0.71
192.168.71.252     link#20            UHS         lo0
192.168.71.254     link#20            UHS         lo0
192.168.109.0/24   link#16            U      lagg0.10
192.168.109.252    link#16            UHS         lo0
192.168.109.254    link#16            UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#10                       UH          lo0
fe80::%bge1/64                    link#8                        U          bge1
fe80::529a:4cff:fe94:ad73%bge1    link#8                        UHS         lo0
fe80::%lo0/64                     link#10                       U           lo0
fe80::1%lo0                       link#10                       UHS         lo0
fe80::%lagg0/64                   link#13                       U         lagg0
fe80::faf2:1eff:fe34:9540%lagg0   link#13                       UHS         lo0
fe80::%lagg0.8/64                 link#14                       U       lagg0.8
fe80::faf2:1eff:fe34:9540%lagg0.8 link#14                       UHS         lo0
fe80::%lagg0.24/64                link#15                       U      lagg0.24
fe80::faf2:1eff:fe34:9540%lagg0.24 link#15                      UHS         lo0
fe80::%lagg0.109/64               link#16                       U      lagg0.10
fe80::faf2:1eff:fe34:9540%lagg0.109 link#16                     UHS         lo0
fe80::%lagg0.65/64                link#17                       U      lagg0.65
fe80::faf2:1eff:fe34:9540%lagg0.65 link#17                      UHS         lo0
fe80::%lagg0.64/64                link#18                       U      lagg0.64
fe80::faf2:1eff:fe34:9540%lagg0.64 link#18                      UHS         lo0
fe80::%lagg0.11/64                link#19                       U      lagg0.11
fe80::faf2:1eff:fe34:9540%lagg0.11 link#19                      UHS         lo0
fe80::%lagg0.71/64                link#20                       U      lagg0.71
fe80::faf2:1eff:fe34:9540%lagg0.71 link#20                      UHS         lo0

[2.5.0-RELEASE][root@fwint3.XXXXXXXXXXX.local]/conf: ifconfig
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether b4:96:91:33:de:40
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether b4:96:91:33:de:41
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether b4:96:91:33:de:42
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether b4:96:91:33:de:43
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        hwaddr f8:f2:1e:34:95:41
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 50:9a:4c:94:ad:72
        media: Ethernet autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: sync
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 50:9a:4c:94:ad:73
        inet6 fe80::529a:4cff:fe94:ad73%bge1 prefixlen 64 scopeid 0x8
        inet 10.10.56.1 netmask 0xfffffffc broadcast 10.10.56.3
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
        pfsync: syncdev: bge1 syncpeer: 10.10.56.2 maxupd: 128 defer: off
        syncok: 1
        groups: pfsync
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0 prefixlen 64 scopeid 0xd
        laggproto failover lagghash l2,l3,l4
        laggport: ixl0 flags=5<MASTER,ACTIVE>
        laggport: ixl1 flags=0<>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.8: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LANMaMue
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.8 prefixlen 64 scopeid 0xe
        inet 192.168.8.252 netmask 0xffffff00 broadcast 192.168.8.255
        inet 192.168.8.254 netmask 0xffffff00 broadcast 192.168.8.255 vhid 8
        groups: vlan
        carp: MASTER vhid 8 advbase 5 advskew 1
        vlan: 8 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.24: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: PrintMaMue
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.24 prefixlen 64 scopeid 0xf
        inet 192.168.24.252 netmask 0xffffff00 broadcast 192.168.24.255
        inet 192.168.24.254 netmask 0xffffff00 broadcast 192.168.24.255 vhid 24
        groups: vlan
        carp: MASTER vhid 24 advbase 5 advskew 1
        vlan: 24 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.109: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: VoiceMaMue
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.109 prefixlen 64 scopeid 0x10
        inet 192.168.109.252 netmask 0xffffff00 broadcast 192.168.109.255
        inet 192.168.109.254 netmask 0xffffff00 broadcast 192.168.109.255 vhid 109
        groups: vlan
        carp: MASTER vhid 109 advbase 5 advskew 1
        vlan: 109 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.65: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: CarrierVlanLWL
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.65 prefixlen 64 scopeid 0x11
        inet 10.10.65.4 netmask 0xfffffff0 broadcast 10.10.65.15
        inet 10.10.65.6 netmask 0xfffffff0 broadcast 10.10.65.15 vhid 65
        groups: vlan
        carp: MASTER vhid 65 advbase 5 advskew 1
        vlan: 65 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.64: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: CarrierVLanWlan
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.64 prefixlen 64 scopeid 0x12
        inet 10.10.64.4 netmask 0xfffffff0 broadcast 10.10.64.15
        inet 10.10.64.6 netmask 0xfffffff0 broadcast 10.10.64.15 vhid 64
        groups: vlan
        carp: MASTER vhid 64 advbase 5 advskew 1
        vlan: 64 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: MgmtMaMue
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.11 prefixlen 64 scopeid 0x13
        inet 192.168.11.252 netmask 0xffffff00 broadcast 192.168.11.255
        inet 192.168.11.254 netmask 0xffffff00 broadcast 192.168.11.255 vhid 11
        inet 10.10.55.2 netmask 0xfffffff0 broadcast 10.10.55.15
        groups: vlan
        carp: MASTER vhid 11 advbase 1 advskew 0
        vlan: 11 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0.71: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ErfasserLAN
        options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether f8:f2:1e:34:95:40
        inet6 fe80::faf2:1eff:fe34:9540%lagg0.71 prefixlen 64 scopeid 0x14
        inet 192.168.71.252 netmask 0xffffff00 broadcast 192.168.71.255
        inet 192.168.71.254 netmask 0xffffff00 broadcast 192.168.71.255 vhid 71
        groups: vlan
        carp: MASTER vhid 71 advbase 1 advskew 0
        vlan: 71 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I have deactivated dhcp server and enabled dhcp forwarding again with the values as there where and saved it:

but unter /conf/config.xml I cat NOT find anything about dhcp forwarding. Even the gui said it was saved. (Did I look on the wrong place?)

This is out of the 2.4.4p1 config

<dhcrelay>
	<enable></enable>
	<interface>opt1,opt3,opt7,opt8</interface>
	<agentoption></agentoption>
	<server>192.168.1.28,192.168.1.27</server>
</dhcrelay>

johnsdixon · Feb 22, 2021, 11:42 AM

@viktor_g
Also from my environment.. (first set under 2.4.5_1, second under 2.5.0).

<dhcrelay>
        <enable></enable>
        <interface>lan</interface>
        <agentoption></agentoption>
        <server>192.168.99.1</server>
</dhcrelay>

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            100.64.44.1        UGS        vmx0
100.64.44.0/24     link#1             U          vmx0
100.64.44.3        link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.0.0/16     100.64.44.1        UGS        vmx0
192.168.192.0/24   link#2             U          vmx1
192.168.192.1      link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#3                        UH          lo0
fe80::%vmx0/64                    link#1                        U          vmx0
fe80::20c:29ff:fe24:ebd7%vmx0     link#1                        UHS         lo0
fe80::%vmx1/64                    link#2                        U          vmx1
fe80::20c:29ff:fe24:ebe1%vmx1     link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0c:29:24:eb:d7
	hwaddr 00:0c:29:24:eb:d7
	inet6 fe80::20c:29ff:fe24:ebd7%vmx0 prefixlen 64 scopeid 0x1
	inet 100.64.44.3 netmask 0xffffff00 broadcast 100.64.44.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0c:29:24:eb:e1
	hwaddr 00:0c:29:24:eb:e1
	inet6 fe80::20c:29ff:fe24:ebe1%vmx1 prefixlen 64 scopeid 0x2
	inet 192.168.192.1 netmask 0xffffff00 broadcast 192.168.192.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: enc
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog

And from 2.5.0, immediately after upgrading.

-<dhcrelay>
<enable/>
<interface>lan</interface>
<agentoption/>
<server>192.168.99.1</server>
</dhcrelay>

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            100.64.44.1        UGS        vmx0
100.64.44.0/24     link#1             U          vmx0
100.64.44.3        link#1             UHS         lo0
127.0.0.1          link#4             UH          lo0
192.168.0.0/16     100.64.44.1        UGS        vmx0
192.168.192.0/24   link#2             U          vmx1
192.168.192.1      link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#4                        UH          lo0
fe80::%vmx0/64                    link#1                        U          vmx0
fe80::20c:29ff:fe24:ebd7%vmx0     link#1                        UHS         lo0
fe80::%vmx1/64                    link#2                        U          vmx1
fe80::20c:29ff:fe24:ebe1%vmx1     link#2                        UHS         lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=e000bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0c:29:24:eb:d7
	inet6 fe80::20c:29ff:fe24:ebd7%vmx0 prefixlen 64 scopeid 0x1
	inet 100.64.44.3 netmask 0xffffff00 broadcast 100.64.44.255
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=e000bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0c:29:24:eb:e1
	inet6 fe80::20c:29ff:fe24:ebe1%vmx1 prefixlen 64 scopeid 0x2
	inet 192.168.192.1 netmask 0xffffff00 broadcast 192.168.192.255
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync

johnsdixon · Feb 22, 2021, 3:51 PM

@victor_g
Correction: Having moved the backup file from 2.4.5 across the divide, it is as below..
Errors introduced by rekeying and not reading what's on the screen.

-<dhcrelay>
    <enable/>
    <interface>lan</interface>
    <agentoption/>
    <server>192.168.123.1</server>
</dhcrelay>

k60010 · Feb 22, 2021, 1:21 PM

Our dhcp relay service failure on 2.5.0 update seems to be hardware specific.

Netgate XG-1537 = success 2 out 2 (version 21.02)
VMware 6.5 = success 4 out of 4
Supermicro 1U server (not sure of flavor, rear facing ports) 1 out of 1
Supermicro CSE-505-203B = fail 2 out of 2
Supermicro SYS-5018D-FN8T = fail 4 out of 4

Have not tried bare metal reload on failures yet.

fwcheck · Feb 22, 2021, 3:33 PM

I think i found the root cause.

DHCP-Server is Upstream (behind) WAN.
DHCP-Relay for example only on LAN.

At least i found a hint in Syslog:

Feb 22 16:01:17 check_reload_status 363 Syncing firewall
Feb 22 16:01:18 php-fpm 326 /services_dhcp_relay.php: No suitable upstream interfaces found for running dhcrelay!

I guess i know where the problem resides.

Our default configuration sets the dhcp-relay only for the interfaces, not for wan. Our DHCP-Servers resides are mostly upstream on the WAN side. We have some firewalls where that is different.

/etc/inc/services.inc

    $srvifaces = array();
    foreach ($srvips as $srcidx => $srvip) {
            $destif = guess_interface_from_ip($srvip);
            if (!empty($destif) && !is_pseudo_interface($destif)) {
                    $srvifaces[] = $destif;
            }
    }

    /* Check for relays in the same subnet as clients so they can bind for
     * either direction (up or down) */
    $srvrelayifs = array_intersect($dhcrelayifs, $srvifaces);

    /* The server interface(s) should not be in this list */
    $dhcrelayifs = array_diff($dhcrelayifs, $srvifaces);

    /* Remove the dual-role interfaces from up and down lists */
    $srvifaces = array_diff($srvifaces, $srvrelayifs);
    $dhcrelayifs = array_diff($dhcrelayifs, $srvrelayifs);

    /* fire up dhcrelay */
    if (empty($dhcrelayifs) && empty($srvrelayifs)) {
            log_error(gettext("No suitable downstream interfaces found for running dhcrelay!"));
            return; /* XXX */
    }
    if (empty($srvifaces) && empty($srvrelayifs)) {
    # Error is here 
            log_error(gettext("No suitable upstream interfaces found for running dhcrelay!"));
            return; /* XXX */
    }

My dhcp-Server resides outside of any net within the firewall, therefore $servifaces
is empty, resulting in the error in syslog.

My fix is to explicit add the upstream if there is none. I am not quite sure if this is the best variant. I would think that fixing guess_interface_from_ip() might be a better way.

    if (empty($srvifaces)){
            $srvifaces[] = "vmx0";
    }
    if (empty($srvifaces) && empty($srvrelayifs)) {
            log_error(gettext("No suitable upstream interfaces found for running dhcrelay!"));
            return; /* XXX */
    }

If there is anything else you need to know please let me know.

johnsdixon · Feb 22, 2021, 3:51 PM

@victor_g
Further investigation on the upgraded 2.5.0 production environment shows (in /var/log/dhcpd.log)

Feb 22 15:41:48 wight dhcrelay[82265]: Internet Systems Consortium DHCP Relay Agent 4.4.2
Feb 22 15:41:48 wight dhcrelay[82265]: Copyright 2004-2020 Internet Systems Consortium.
Feb 22 15:41:48 wight dhcrelay[82265]: All rights reserved.
Feb 22 15:41:48 wight dhcrelay[82265]: For info, please visit https://www.isc.org/software/dhcp/
**Feb 22 15:41:48 wight dhcrelay[82265]: Unsupported device type 24 for "lo0"**
Feb 22 15:41:48 wight dhcrelay[82265]:
Feb 22 15:41:48 wight dhcrelay[82265]: If you think you have received this message due to a bug rather
Feb 22 15:41:48 wight dhcrelay[82265]: than a configuration issue please read the section on submitting
Feb 22 15:41:48 wight dhcrelay[82265]: bugs on either our web page at www.isc.org or in the README file
Feb 22 15:41:48 wight dhcrelay[82265]: before submitting a bug.  These pages explain the proper
Feb 22 15:41:48 wight dhcrelay[82265]: process and the information we find helpful for debugging.
Feb 22 15:41:48 wight dhcrelay[82265]:
Feb 22 15:41:48 wight dhcrelay[82265]: exiting.

So perhaps the default upgrade is adding lo0 to the dhcrelay startup process?

The equivalent from the 2.4.5_1 environment is as follows:

Feb 22 15:47:00 wight dhcrelay: Internet Systems Consortium DHCP Relay Agent 4.4.1
Feb 22 15:47:00 wight dhcrelay: Copyright 2004-2018 Internet Systems Consortium.
Feb 22 15:47:00 wight dhcrelay: All rights reserved.
Feb 22 15:47:00 wight dhcrelay: For info, please visit https://www.isc.org/software/dhcp/
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx4/00:0c:29:10:bf:e3
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx4/00:0c:29:10:bf:e3
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx7.888/00:0c:29:10:bf:15
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx7.888/00:0c:29:10:bf:15
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx6/00:0c:29:10:bf:ed
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx6/00:0c:29:10:bf:ed
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx5/00:0c:29:10:bf:0b
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx5/00:0c:29:10:bf:0b
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx3/00:0c:29:10:bf:01
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx3/00:0c:29:10:bf:01
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx2/00:0c:29:10:bf:d9
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx2/00:0c:29:10:bf:d9
Feb 22 15:47:00 wight dhcrelay: Listening on BPF/vmx1/00:0c:29:10:bf:f7
Feb 22 15:47:00 wight dhcrelay: Sending on   BPF/vmx1/00:0c:29:10:bf:f7
Feb 22 15:47:00 wight dhcrelay: Sending on   Socket/fallback

fwcheck · Feb 22, 2021, 3:54 PM

Ok i have to add, that this is a solution (WAN is always vmx0) for my case. The physical interface of another system has sure another name,
<interfaces>
<wan>
<enable></enable>
<if>vmx0</if>
Therefore more general it is s.th. like
$config[interfaces][wan][if]
..

johnsdixon · Feb 22, 2021, 4:52 PM

@fwcheck There's definitely something odd going on.
In the test scenario, I have a similar environment, with DHCP server on the dirty (WAN) side, using vmx0. This seems to be working correctly, as I get vmx0 in the $srvifaces list.
But no lo0 in any list.

Now adding more elements to the test environment to find the thing that triggers lo0 to get added to the list.

Gertjan · Feb 23, 2021, 7:07 AM

@johnsdixon said in Update to 2.5.0 broke DHCP relay:

But no lo0 in any list.

Isn't that a good thing ?
lo0 is the local host or 127.0.0.1
dhcrelay can't operate on "lo0" :

@johnsdixon said in Update to 2.5.0 broke DHCP relay:

Feb 22 15:41:48 wight dhcrelay[82265]: Unsupported device type 24 for "lo0"

which is rather logic.

johnsdixon · Feb 23, 2021, 10:59 AM

@gertjan But my production environment generates a startup command for the DHCP relay with lo0 included.
This is not there in 2.4.5_1, but following an upgrade to 2.5.0 this appears, and there is no functioning DHCP relay process started by default in that situation. What I'm trying to do is work out what is triggering the inclusion of the lo0 within the startup process.
There is no lo0 anywhere in my config, nor has disabling services (eg. squid, OpenVPN) on the production configuration gained me working DHCP forwarding.