Investigating how did I locked myself out of pfsense

scotrod

Greetings,

I have pfsense 24.11 VM acting as my router and FW for ~2 years now. Few days ago I was setting OpenVPN and ntopng, and while doing so, I locked myself out of the web interface. This happened shortly after installing ntopng, setting the admin password (for ntopng, not for pfsense) and observing the collected data. Suddendly, I couldn't logon to the dashboard from my PC, so I tried logging in from my smartphone, again, without success.

I rebooted the system, and I still couldn't logon. So what I did was to reset the admin password from the console access I had (pfsense is hosted on my esxi home server).

I started checking OpenVPN logs - nothing unusual besides my own logons. I went and checked the Configuration History. This is where it gets interesting, since the last two events before the event that I generated while reseting the admin account password from console are marked as "Unknown" (refer to the attached screenshot)

I enabled SSH access and downloaded the whole /var/log directory so I can investigate this further. I opened an old config of mine to check the hash of the admin password. Unfortunately, the format was different on my old config (sha512-hash) while the new one seems to be bcrypt-hash.

I further checked the system.log downloaded from pfsense and noticed that both my PC and smartphone had been blocked by pfsense, noted in the following events:

Jan  7 12:14:22 PFSENSE_HOSTNAME php-fpm[88815]: /index.php: webConfigurator authentication error for user 'admin' from: 192.168.2.32
Jan  7 12:14:22 PFSENSE_HOSTNAME sshguard[45238]: Attack from "192.168.2.32" on service unknown service with danger 10.
Jan  7 12:14:26 PFSENSE_HOSTNAM Ephp-fpm[88815]: /index.php: webConfigurator authentication error for user 'admin' from: 192.168.2.32
Jan  7 12:14:26 PFSENSE_HOSTNAME sshguard[45238]: Attack from "192.168.2.32" on service unknown service with danger 10.
Jan  7 12:14:31 PFSENSE_HOSTNAME php-fpm[91835]: /index.php: webConfigurator authentication error for user 'admin' from: 192.168.2.32
Jan  7 12:14:31 PFSENSE_HOSTNAME sshguard[45238]: Attack from "192.168.2.32" on service unknown service with danger 10.
Jan  7 12:14:31 PFSENSE_HOSTNAME sshguard[45238]: Blocking "192.168.2.32/32" for 120 secs (3 attacks in 9 secs, after 1 abuses over 9 secs.)

I have also checked all events from 12:06 (the first change marked as "Unkown" until 12:36 (the admin account password reset that I did). Besides some noise, here is the only notable event:

Jan  7 12:07:17 PFSENSE_HOSTNAME php-fpm[72992]: /pkg_edit.php: Configuration Change: 
Jan  7 12:07:17 PFSENSE_HOSTNAME check_reload_status[542]: Syncing firewall

It looks like I haven't entered the correct password, which I am quite sure is not true, as I've been using that password for quite a lot of time on this device.

Unfortunately I no longer have access to all of the config changes I have screenshotted, because I had the default limiter of 30, and I didn't downloaded them all back then.

Is there anything further that I can look up to?

Thanks.

stephenw10

Check the config diffs for each change in the history. See if it's actually setting a different password somewhere.

Also I assume you were able to access the login screen still? Just unable to login successfully there?

scotrod

@stephenw10 Hey and thanks for the suggestion.

Unfortunately the default limiter of 30 that I had left had already rotated so I no longer can do these cross-checks directly on pfsense. Luckily, in the heat of the situation back then I downloaded 3 configs around the "Unknown" changes.

The bcrypt-hash values from the 3 versions were different. That got me worried additionally, because neither of them had the same values as my current hash value. This was unexpected, because when I did the PW reset, I pasted the same old password. So, I expected the same hash value. I have it-tools installed locally so I was able to cross check my password against the two set of hash values - success. I guess what happened is that every time PW reset/change is done, the hash is salted. Or at least that would explani the difference.

Now for the real problem. I grabbed the three hash values, and cross-checked them versus the OpenVPN user PW that I set that day - again, success. What happened was that I resetted my admin account PW, instead of my VPN user PW (that was not vanilla VPN setup. I had that an year ago, I just disabled it and now re-enabled it).

TL;DR: Wrong account PW reset.

stephenw10

Ah, OK so you think you just set the wrong account rather than some backend bug?

scotrod

That would be the simplest solution, yes. The two account fields are next to each other, so it's completely possible that I clicked on the admin account menu, instead of my VPN user. That, plus the fact that I compared the bcrypt-hash from the admin account, with the password value from my own VPN account, and they matched. I really do think what happened was a simple user error.

I think the last piece of the puzzle would be someone to confirm whether the b-crypt-hash value is supposed to be different every time one changes account password, even if the new password is the same as the old one.

stephenw10

bcrypt includes salting so I'd expect a different hash every time it is generated even from the same password.

scotrod

That settles is then. The only thing that's bugging me, is that 30 row config history doesn't include the password change for my admin account.

I just changed my admin password (again, to my old password) and the following entry was populated in the config history

This is not visible in my previous history, and the simplest explination again would be rotation. I may have done that change before the earliest visible event (10:09:36). Are you aware of changing the admin PW from the web interface gets logged anywhere? That would solve it.

stephenw10

I'd certainly expect it to be shown like that because you are editing the admin user. The page is correctly setup to pass a config change string so it is logged.

'Unknown' is shown when a page/script does not pass a string back to the config write.

scotrod

Thanks. Is that event logged somewhere? I pulled all the logs from /var/log so if its logged, I could find the exact event.

JonathanLee

I see unknown when you make changes to packages sometimes.

scotrod

Okay, as expected, this event is tracked in the system.log. After resetting the admin password today, I checked and can confirm that event is recorded there. Unfortunately I couldn't find such event in my previous logs. Log retention is not a problem since the system.log goes back to Oct'24.

That really doesn't make sense, because at some point I had different admin password. The hash proves it (I compared one of the hashes from my admin password with the password of my other VPN user. This is where I accidentally changed the admin PW instead of my VPN user PW; so, for a short period of time my admin password had the password of my VPN user). At the end there may be a backend bug here. I had a different admin password at some point and I'm not seeing any notable events at least in the system.log