6100-upgrade to 26.03 and can no longer log in via GUI or SSH

stephenw10

Further suggestion is to run:

dtrace -n 'fbt::kern_mkdirat:entry /strstr(copyinstr(arg2), "gosnowflake") != 0/ { printf("%s (pid %d) created %s", execname, pid, copyinstr(arg2)); }'

That should show whats creating then straight away if it's doing so that quickly.

ddbnj

@stephenw10 said in 6100-upgrade to 26.03 and can no longer log in via GUI or SSH:

dtrace -n 'fbt::kern_mkdirat:entry /strstr(copyinstr(arg2), "gosnowflake") != 0/ { printf("%s (pid %d) created %s", execname, pid, copyinstr(arg2)); }'

Awesome.

0 67432 kern_mkdirat:entry telegraf (pid 34233) created /tmp/gosnowflake-cgo3048104734

Seems telegraf is the issue. This only happens on 26.03.

I am disabling the service for now.

ddbnj

@stephenw10

Thank you for your help.

ddbnj

@SteveITS

Thank you

stephenw10

Ooo interesting. Do you have anything custom configured there that might have been causing it?

ddbnj

@stephenw10

I was using telegraf to send discrete data to influxdb to be graphed into grafana. I also was using it as an alert if my wan failback to LTE activated.

stephenw10

Hmm, so nothing to the snowflake cloud specifically?

ddbnj

@stephenw10

correct.

mi8088

We were experiencing the same problem on a Netgate 4100 running 26.03.1-RELEASE (amd64).

Fixed it by logging in via ssh, running the dtrace commands above to verify that it was indeed the telegraf service that was causing the issue, then using ps and kill to kill the service. Then manually running rm -rf /tmp/gosnowflake-cgo9* while stepping through all the numbers to get small enough amount of directories each time. Solved the login problem, but not the underlying issue.

stephenw10

How do you have telegraf cofigured?

marcosm

For reference:
https://github.com/influxdata/telegraf/issues/19072
https://github.com/snowflakedb/gosnowflake/issues/1800

mi8088

Incorrectly, it seems..

We have used Telegraf previously, but on two of our firewalls, this was not disabled when the server was shut off.

This means: Telegraf was installed and enabled, but there would be not be any reply from the server/IP address associated with "InfluxDB server".

On one of the firewalls, we upgraded to 26.03.1 - with ensuing problems as above (affecting the GUI, not SSH logins).

On the other firewall, version 26.03 was still installed, no problems with login there - even though there are many(*) gosnowflake directories.

(*) Count is not easy to get, since there are so many files. There were far above 500'000 files, though.

I'm going to skip my Telegraf config, since the linked issues show the root cause is upstream. There are extra items there.

mi8088

It seems that disabling Telegraf and stopping the service was not enough to get the directory creation to stop - I had to also remove the Telegraf package.

stephenw10

Hmm, that's odd. I wouldn't expect anything to be running with it disabled. Perhaps something had failed to stop? Had it been rebooted?

mi8088

Hi, sorry for the slow reply.

Yes, that is possible - I did not reboot inbetween. It was easier to get rid of Telegraf, as it was not in use anyway.