Cisco CBS220 switch randomly rebooting - Potential DHCP Issue?

[giminik]

Hi everyone,

I’m experiencing a frustrating issue with my Cisco CBS220-24P-4G switch. It works well overall, but it reboots randomly every 7 to 40 days. I have already updated the firmware to the latest version.

After some research, I found suggestions pointing to a potential power supply issue. To address this, I tested the switch with two different UPS units, but the problem persisted. Another potential cause mentioned is a DHCP issue, as discussed in this Reddit post. I haven't yet tried switching to a static IP.

I checked the logs on both the switch and pfSense+ at the time of the last reboot. The switch went down at 19:10:37 (confirmed by my surveillance cameras). In the pfSense+ DHCP logs, there’s a DHCP request at 19:10:31, which seems too coincidental to ignore. It looks like something crashed at 19:10:41. Could this crash be causing the switch's DHCP client to fail and trigger a reboot?

Here are the relevant logs:

Jul 5 19:10:31	kea-dhcp4	13245	INFO [kea-dhcp4.dhcpsrv.0xfbfa9418200] EVAL_RESULT Expression pool_opt6_0 evaluated to 1
Jul 5 19:10:31	kea-dhcp4	13245	INFO [kea-dhcp4.leases.0xfbfa9418200] DHCP4_LEASE_ALLOC [hwtype=1 74:11:b2:9f:9a:20], cid=[01:74:11:b2:9f:9a:20], tid=0x87d9c0c: lease 10.0.1.2 has been allocated for 7200 seconds
Jul 5 19:10:41	dhcpleases	38813	Sending HUP signal to dns daemon(85903)
Jul 5 19:10:41	dhcpleases	38813	Could not deliver signal HUP to process 85903: No such process.
Jul 5 19:10:42	kea-dhcp4	13245	INFO [kea-dhcp4.dhcp4.0xfbfa9412000] DHCP4_SHUTDOWN server shutdown
Jul 5 19:10:42	kea-dhcp4	13245	INFO [kea-dhcp4.lease-cmds-hooks.0xfbfa9412000] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful
Jul 5 19:10:42	kea-dhcp4	13245	INFO [kea-dhcp4.hooks.0xfbfa9412000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
Jul 5 19:10:42	kea-dhcp4	44721	INFO [kea-dhcp4.hosts.0xe500b612000] HOSTS_BACKENDS_REGISTERED the following host backend types are available:
Jul 5 19:10:42	kea-dhcp4	44721	WARN [kea-dhcp4.dhcpsrv.0xe500b612000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.

Three minutes later, switch finished to boot and asked its IP address.

Jul 5 19:13:44	kea-dhcp4	63683	INFO [kea-dhcp4.dhcpsrv.0x2c04ca16600] EVAL_RESULT Expression pool_opt6_0 evaluated to 1
Jul 5 19:13:44	kea-dhcp4	63683	INFO [kea-dhcp4.leases.0x2c04ca16600] DHCP4_LEASE_ADVERT [hwtype=1 74:11:b2:9f:9a:20], cid=[01:74:11:b2:9f:9a:20], tid=0x52c2f921: lease 10.0.1.2 will be advertised
...
Jul 5 19:13:44	kea-dhcp4	63683	INFO [kea-dhcp4.dhcpsrv.0x2c04ca16600] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
Jul 5 19:13:44	kea-dhcp4	63683	INFO [kea-dhcp4.dhcpsrv.0x2c04ca16600]
...
Jul 5 19:13:44	kea-dhcp4	63683	INFO [kea-dhcp4.leases.0x2c04ca16600] DHCP4_LEASE_ALLOC [hwtype=1 74:11:b2:9f:9a:20], cid=[01:74:11:b2:9f:9a:20], tid=0x52c2f921: lease 10.0.1.2 has been allocated for 86400 seconds

Any help or suggestions would be greatly appreciated!

[johnpoz]

@giminik well putting the devices on different upses not really a check if the devices power supply has issues.Did you like switch it to a full sinewave ups vs simulated? But again that could have zero to do with anything if the power supply is just flaky..

But per your thread would be simple enough to test if dhcp is the problem, per the thread they disabled the IPv6 stack on the switch as well.. I would do what they did disable the ipv6 on the switches management IP and set the IPv4 to static..

[giminik]

I just turn this setting after my first post. So I have to wait a long time to know.

But are these logs normal?

Jul 5 19:10:41	dhcpleases	38813	Sending HUP signal to dns daemon(85903)
Jul 5 19:10:41	dhcpleases	38813	Could not deliver signal HUP to process 85903: No such process.
Jul 5 19:10:42	kea-dhcp4	13245	INFO [kea-dhcp4.dhcp4.0xfbfa9412000] DHCP4_SHUTDOWN server shutdown

[johnpoz]

@giminik have no idea what the normal logs for kea look like to be honest, its preview - it is missing many features currently.. no registration, no options, etc.. See the release notes or the blog.

Unless all your doing is the most basic of handing out an ip, would suggest you switch back to isc..

https://www.netgate.com/blog/netgate-adds-kea-dhcp-to-pfsense-plus-software-version-23.09-1

Basic functionality is present in version 23.09, but the Kea implementation lacks the following DHCP server features:

    Local DNS Resolver/Forwarder Registration for static and dynamic DHCP clients
    Remote DNS server registration
    DHCPv6 Prefix Delegation
    High Availability Failover
    Lease statistics/graphs
    Custom DHCP options

[coxhaus]

@giminik My guess is you have some kind of power problem. Maybe an overloaded circuit or overloaded UPS.
I run a Cisco CBS350 switch and it never goes down. I am using a Smart APC UPS.

[johnpoz]

@coxhaus is your dhcp or static.. While I am a fan of dhcp reservation vs setting static for sure for most everything - I don't recall ever setting a switch to dhcp ever in real life, ie work.

My switches at home are all static. I don't have any cbs switches, couple of SG300s for me. Some cheap soho netgear and tplink, oh and a sg250 sitting on shelf if I ever need for setup or something.

[JonathanLee]

Is it over heating have you cleaned the fans? The Cisco Switches and even older Cisco PIX firewalls seem like they run forever, but dust in the fans make them overheat

[tgl]

I'm with @johnpoz here: I would never have a switch or similar basic network infrastructure device getting its address via DHCP. If you do that, your entire network can go down, or at least become un-manageable, the moment the DHCP server has a hiccup.

Having said that, I suspect the timing of this crash is only coincidental ... unless you have more log entries showing that the pattern repeats?

Flaky power supply could be it, and I also like @JonathanLee's recommendation to open the thing up and clean it.

[giminik]

I already had this problem before migrating to kea.
Switch is a brand new, no dust, and it provides approximately 15% of the poe budget.

It is running for 8 months. I only noticed the problem a few months ago, by chance. We don't always see a 2-minute break over a 40-day period. And when I change a setting, I have to wait that much to see if the problem persists. I switched the IP to static, we'll see in a few weeks.

[tgl]

@giminik If it's new, I'd RMA it while it's still under warranty. "It randomly reboots" should be plenty enough justification to get it replaced.

[giminik]

On the cisco forums I saw people having this problem, did an RMA and had the problem again. It seems more like a software problem, moreover in the logs, before the reboot we clearly see the message "REBOOTING" which excludes a power supply problem. Cisco does not communicate about this.

I'm pretty confident about the static IP config. I have to wait about one month to be fixed.

[JonathanLee]

@giminik it’s the NVRAM some of them are bad

[mvikman]

What's the firmware version in switch?

Firmware 2.0.2.14 Release Notes lists "CBS 220 keeps rebooting after 1 - 2 weeks of uptime." in resolved issues.
https://www.cisco.com/c/dam/en/us/td/docs/switches/lan/csbss/CBS220/Release-Note/b_cbs220-rn-v2_0_2_14.pdf

[giminik]

@mvikman Damn, I hadn't seen this new version, although I was going to see it regularly. I'm on the version just before this one (2.0.2.12). So I'm going to upgrade. Thank you for enlightening me :)

[mvikman]

@giminik Just out of curiosity, did the switch firmware upgrade solve this or was it something else?

[giminik]

@mvikman For now, the problem has not manifested itself, but I must wait around forty days to be sure.