6100-upgrade to 26.03 and can no longer log in via GUI or SSH

ddbnj

The router is still routing correctly.

The webgui login page loads correctly, once I enter my login credentials, the screen blinks and goes back to empty fields.

When I try to SSH, I get the following:

login as: admin
Keyboard-interactive authentication prompts from server:
| Password for admin@pfsense.devan.click:
End of keyboard-interactive prompts from server
Mon Apr 13 20:38:30 2026 (78571): Fatal Error Unable to create opcache lock file                                                                                   in /tmp: No space left on device (28)
*** *** *** *** *** ***

Password MUST be changed.
If the password has already been changed in the GUI, press ctrl-c to cancel.
*** *** *** *** *** ***
Mon Apr 13 20:38:30 2026 (79825): Fatal Error Unable to create opcache lock file                                                                                   in /tmp: No space left on device (28)
Mon Apr 13 20:38:30 2026 (80197): Fatal Error Unable to create opcache lock file                                                                                   in /tmp: No space left on device (28)

My other site has not upgraded and has no issues. Not sure if it's hardware failure or upgrade related.

Any tips appreciated. I haven't tried anything yet.

Thanks,

Devan

ddbnj

Halted system via power button.

Powered on and it works. Drive is 98% empty.

Grabbed a backup config file.

Anyone have any ideas on what happened and how to prevent it?

Thanks,

Devan

SteveITS

@ddbnj hmm actually ignore all that, I reread the error message. Sounds like something got stuck with PHP opcache. Have not seen that before.

stephenw10

Yeah seeing the login page but nothing after that usually implies a PHP issue. I'd guess it either stopped entirely or all the PHP processes were tied up.

Check the system logs when it happened.

ddbnj

@stephenw10

Thanks for replying.

It happened again 5 days later. No openvpn clients could connect.

Same issue with tmpfs full.

The /tmp directory is full of empty subdirectories:

gosnowflake-cgo123456789

I increased the max ram size and it lasts longer before crashing but tmpfs is still growing every day.

I'm sending logs to an external server but all I'm seeing is attempted logins to openvpn.

"find . -type d | wc -l"

133010 empty directories in /tmp

ddbnj

It's creating two directories every second.

What process spawns these directories?

stephenw10

You have Nexus enabled?

ddbnj

@stephenw10

nope

ddbnj

Troubleshooting steps:

killed watchdog
stopped OpenVPN
stopped frr mgmt.
stopped haproxy
stopped dpinger
stopped arpwatch
stopped radvd
stopped dhcpd
stopped ntpd
stopped syslogd
stopped udpbroadcastrelay
stopped unbound
stopped wireguard

unbound, dhcpd, radvd, dpinger all restarted by themselves.

Router is still creating gosnowflake directores 2 per second.

Every other service is off.

Rolling back to 25.11.1...

stephenw10

And definitely not happening in 25.11.1 with the same packages/services enabled?

stephenw10

If you're able to test again try to see what's creating them using something like: ps -auxwwd | grep snowflake

If it's happening almost continually you should be able to catch it.

ddbnj

@stephenw10

Before I restored to 25.11.1 I tried ps -aux | grep snow but didn't catch anything. I didn't use the -i caps switch. my bad.

I'm in 25.11.1 and don't see any tmp directory creation. Only been up for 20 minutes. All same services running.

I'll upgrade back to 26.03 tomorrow.

stephenw10

Mmm, it looks likely to be this but AFAIK we don't ship it: https://docs.snowflake.com/en/developer-guide/golang/go-driver

Do you have any 3rd party packages installed?

Other suggestions from our devs here are to try using fstat or install lsof and see what's holding the files.

ddbnj

@stephenw10

I don't have anything else installed. Everything is from the pfsense package manager.

stephenw10

Further suggestion is to run:

dtrace -n 'fbt::kern_mkdirat:entry /strstr(copyinstr(arg2), "gosnowflake") != 0/ { printf("%s (pid %d) created %s", execname, pid, copyinstr(arg2)); }'

That should show whats creating then straight away if it's doing so that quickly.

ddbnj

@stephenw10 said in 6100-upgrade to 26.03 and can no longer log in via GUI or SSH:

dtrace -n 'fbt::kern_mkdirat:entry /strstr(copyinstr(arg2), "gosnowflake") != 0/ { printf("%s (pid %d) created %s", execname, pid, copyinstr(arg2)); }'

Awesome.

0 67432 kern_mkdirat:entry telegraf (pid 34233) created /tmp/gosnowflake-cgo3048104734

Seems telegraf is the issue. This only happens on 26.03.

I am disabling the service for now.

ddbnj

@stephenw10

Thank you for your help.

ddbnj

@SteveITS

Thank you

stephenw10

Ooo interesting. Do you have anything custom configured there that might have been causing it?

ddbnj

@stephenw10

I was using telegraf to send discrete data to influxdb to be graphed into grafana. I also was using it as an alert if my wan failback to LTE activated.