Vouchers getting Expired before remaining Time

refugeesonline

Hi!

I tried to investigate this a little deeper. Here is what I suspect: If voucher rolls are created running 2.4.x the upgrade to 2.5.x deletes the db-files in /var/db. On the first login of an existing voucher, the respective db-files will be re-created, but with wrong activation time stamps. (No evidence. just a thought!)

What helps:

Delete all voucher rolls. Upgrade to 2.5.x. Create all voucher rolls again. Check if all corresponding db-files are present in /var/db.

However, there is a BIG disadvantage: All voucher history gets lost, especially vouchers which were expired will be valid again unless you manually expire them!!!

BR,

Volker

Gertjan

@refugeesonline

You might be right
I can't / didn't test an upgrade, as I'm on 2.5.1 already. I'm not using voucher neither.
DB files ate filled with the time info, when vouchers are used or activated.

But :

With the option set, DB files are backed up ....

wazim4u

I have Fresh Install and everything created from Scratch. I have two site still running development version of 2.5 and there is no issue

I planned to upgrade them to 2.5 stable but facing problem with vouchers early expiring in two new deployments ( Built on FEB-2021 ). so Postponed it.

There is some kind of bug unable to trace yet.

viktor_g

@wazim4u said in Vouchers getting Expired before remaining Time:

I have Fresh Install and everything created from Scratch. I have two site still running development version of 2.5 and there is no issue

What is the actual snapshot date?
Could you provide more information and create a bugreport?
https://docs.netgate.com/pfsense/en/latest/development/bug-reports.html

May be related to https://redmine.pfsense.org/issues/97

wazim4u

@viktor_g

Voucher system working perfectly on this snapshot

2.5.0.a.20210107.2142 ( Development )

up to 2500 captive portal users renew monthly basis and there is not a single issue reported about voucher's early expiry.

refugeesonline

Aaargghhh. It happended again. Even the way I though will work (delete vouchers, upgrade to 2.5.1, create vouchers again) is failing. Around 280 vouchers expired way before the end of its validity period. And we have a lot of happy customers on our hotline ... :-(

The system was rebooted due to a power outage and this reboot may have caused the corruption of the voucher db. Just a thought ...

I am going to roll back all the respective sites to 2.4.5 now. This bug is severe - at least if vouchers are being used.

Regards,

Volker

viktor_g

Please create a bugreport:
https://docs.netgate.com/pfsense/en/latest/development/bug-reports.html

refugeesonline

Done!

https://redmine.pfsense.org/issues/11894

serbus

Hello!

Are you using RAM disks?

John

viktor_g

@refugeesonline said in Vouchers getting Expired before remaining Time:

Aaargghhh. It happended again. Even the way I though will work (delete vouchers, upgrade to 2.5.1, create vouchers again) is failing. Around 280 vouchers expired way before the end of its validity period. And we have a lot of happy customers on our hotline ... :-(

The system was rebooted due to a power outage and this reboot may have caused the corruption of the voucher db. Just a thought ...

I am going to roll back all the respective sites to 2.4.5 now. This bug is severe - at least if vouchers are being used.

Regards,

Volker

What is your complete configuration?
Are you using RAM disks or Captive Portal HA sync?
Please provide more info

wazim4u

I can prove now that there is serious bug in system . given below are details for two vouchers showing expired today.

Voucher : 976522494 ( Expired after 2 days )

May 6 01:14:49	logportalauth	14356	Zone: smg_camp - FAILURE: 976522494, 6c:c7:ec:4b:11:c3, 172.17.228.225, voucher expired
May 6 01:14:49	logportalauth	14356	Zone: smg_camp - 976522494 (6/50) already used and expired

May  4 04:28:01 smg logportalauth[24061]: Zone: smg_camp - Voucher login good for 516420 min.: 976522494, 6c:c7:ec:4b:11:c3, 172.17.228.225

Voucher : 7929393872 ( Expired after 6 days )

May 6 08:04:05	logportalauth	61082	Zone: smg_camp - FAILURE: 7929393872, 2e:64:bd:c8:d3:4f, 172.17.120.210, voucher expired
May 6 08:04:05	logportalauth	61082	Zone: smg_camp - 7929393872 (6/18) already used and expired

Apr 29 12:03:47 smg logportalauth[46095]: Zone: smg_camp - Voucher login good for 517452 min.: 7929393872, 52:62:b0:80:78:00, 172.17.230.152

refugeesonline

@serbus
Yes, /tmp and /var are RAM disks

refugeesonline

@viktor_g
Yes, /tmp and /var are RAM disks. No Captive Portal HA sync, only local dbs.

What I noticed: System has been upgraded, all voucher rolls were deleted by the admin. No db-files in /var/db. Then the admin created 3 voucher rolls and the corresponding db-files are present in /var/db.
Then reboot. All of the sudden 286 vouchers expired (which is weird as only around 70 were active at that time). In /var/db only the db-file for one roll (the one containing active vouchers) is present - the others disappeared. Even more strange: All remaining active vouchers were set back to original validity (43.200 mins). It looks like the reboot has had a severe impact on the /var/db contents ...
Two things are a pain in the neck: Active vouchers are expired way earlier before the nominal validity (consequently, our hotline has a lot of calls). Secondly, the remaining vouchers start again with full validity. (consequently, we are giving access for free as active vouchers are kind of reset to square one).

refugeesonline

Tried to switch off RAM disks.
Result: Captive Portal does not work anymore ....

May 6 15:11:51 logportalauth 360 Zone: zone_01 - Trying to modify DB returned error: table captiveportal has no column named traffic_quota
May 6 15:11:51 logportalauth 360 Zone: zone_01 - Trying to modify DB returned error: table captiveportal has no column named traffic_quota

I am closed to give up on this ....

Gertjan

@refugeesonline said in Vouchers getting Expired before remaining Time:

Yes, /tmp and /var are RAM disks

and then :

@refugeesonline said in Vouchers getting Expired before remaining Time:

It looks like the reboot has had a severe impact on the /var/db contents ...

True : RAM disks are lost during reboot.
edit : I presume that's a common knowledge .... Am I wrong ?

When I read all this :

I do not have the confirmation that the captive data base files are retained.

Data (files) that are kept between reboots are mentioned !
The captive portal database files are NOT mentioned ....
So ..... I vote for : they are lost.
edit : the manual confirmed this.

There is probably an issue, but using RAM disks and the using the captive portal - and using vouchers == asking for another issue.

@refugeesonline said in Vouchers getting Expired before remaining Time:

All remaining active vouchers were set back to original validity (43.200 mins).

Yep - this seems normal to me. These data base files - per portal and per role - are created when at least one of the vouchers of a roll is activated. The voucher code and the time stamp of their activation is recorded.
Loosing these file makes the voucher 'available' again with the original duration.

refugeesonline

@gertjan
I do not understand your comment. We are operating 170 sites and the problems I described ONLY appear on systems running 2.5.1 ....

Gertjan

@refugeesonline said in Vouchers getting Expired before remaining Time:

and the problems I described ONLY appear on systems running 2.5.1 ...

== the systems you upgraded. This will zap de ram disk.
Or : the other system never reboot ( ? ) .... so the RAM disk stays valid.

Also :
See https://github.com/pfsense/pfsense/commits/master/src/etc/inc/voucher.inc

The last time this file changed was may 2020. That's before 2.4.5-p1 ...
And that code decides if the log message "already used and expired" is thrown out :

May 6 08:04:05	logportalauth	61082	Zone: smg_camp - 7929393872 (6/18) already used and expired

I've created some vouchers and roles myself, and play with them.
I can't really create vouchers with a "517452 minutes " == one year ( !!!! ) and test them.

Btw : the voucher code could change tomorrow, and break the bit-nitting** of of the expiration test, so making vouchers "517452 " will always give troubles. The unknown factor is just "when".

** Just look - for the fun - at the function voucher_expire() and voucher_auth() in
/etc/inc/voucher.inc

viktor_g

@gertjan said in Vouchers getting Expired before remaining Time:

@refugeesonline said in Vouchers getting Expired before remaining Time:

Yes, /tmp and /var are RAM disks

and then :

@refugeesonline said in Vouchers getting Expired before remaining Time:

It looks like the reboot has had a severe impact on the /var/db contents ...

True : RAM disks are lost during reboot.
edit : I presume that's a common knowledge .... Am I wrong ?

When I read all this :

I do not have the confirmation that the captive data base files are retained.

Data (files) that are kept between reboots are mentioned !
The captive portal database files are NOT mentioned ....
So ..... I vote for : they are lost.
edit : the manual confirmed this.

There is probably an issue, but using RAM disks and the using the captive portal - and using vouchers == asking for another issue.

Please create a feature request for storing captive portal / voucher data:
https://docs.netgate.com/pfsense/en/latest/development/feature-requests.html

refugeesonline

@gertjan
None of the reporters of this "bug" (wazim4u and refugeesonline=me) is using a validity period of one year, we are using 43.200 mins = 30 days.
I am sorry I don't have the time to look in GitHub (although I would love to be able to contribute), I am just an operator of 170 appliances.
And concerning reboots: All our sites are refugee camps, power outages (and consequently reboots) are daily business. We never had this problem before. I can't judge whether this is a bug or a configuration problem. All I am saying: I did NOT change the configuration, I just did an upgrade to 2.5.1. and ran into problems. And I had to notice that the configuration had to be changed, too. 2.5.1 has a totally different handling of concurrent logins (which I figured by accident as many customers were complaining). So my conclusion is: For my case the upgrade caused the problem. I will have to physically visit al lot of sites and do the roll-back to 2.4.5. (Else, I am open for any advice how to do this remotely ....)

serbus

Hello!

A cursory look through some of the code indicates that some voucher info is stored in the config during a graceful shutdown/reboot. Even that wont help with ram disks, or if there is a power loss.

Check on the requirements that forced you to move from 2.4.5 to 2.5.x and see if any of those can be worked around.

Try building a fresh 2.5.x without ram disks and see how it works.

Moving back to "no ram disks" after initially using them might work with a little scripting.

Using the current rc.backup* and rc.restore* scripts as a template for manually saving and restoring the /var/db data...

While ram disks are still on...
rc.backup_db

#!/bin/sh
#

: ${DBPATH:=/var/db}
: ${CF_CONF_PATH:=/cf/conf}

: ${RAM_Disk_Store:=${CF_CONF_PATH}/RAM_Disk_Store}

# Save the logs database to the RAM disk store.
if [ -d "${DBPATH}" ]; then
	echo -n "Saving DB to RAM disk store...";

	[ -f "${RAM_Disk_Store}/db.tgz" ] && /bin/rm -f "${RAM_Disk_Store}/db.tgz"

	if [ ! -d "${RAM_Disk_Store}" ]; then
		mkdir -p "${RAM_Disk_Store}"
	fi

	/usr/bin/tar -czf "${RAM_Disk_Store}/db.tgz" -C / "${DBPATH#/}/"

	echo "done.";
fi

Turn off ram disks, reboot, manually run...
rc.restore_db

#!/bin/sh
#
: ${CF_CONF_PATH:=/cf/conf}

: ${RAM_Disk_Store:=${CF_CONF_PATH}/RAM_Disk_Store}

/usr/bin/tar -xzf "${RAM_Disk_Store}/db.tgz" -C / 2>&1

Restart captive portal and see if it picks up your newly restored /var/db

The standard system rc.restore_ramdisk_store might even catch & restore the db.tgz you created if it runs when rams disks are off (???).

This is a highly experimental process with lots of unknowns. Use at your own risk.
Maybe someone with more/better knowledge of the startup/shutdown and ram disk operations could chime in.

John