Shouldn't expired sessions be removed?

Gertjan

The portal interface isn't a package, its a core function of pfSense.

I'm pretty sure a dev would drop in after the following:
State this:
What hardware are you sing ? Free disk size ? Mem size ? Interfaces ?
What is your pfSense setup - basic use ? load ?
Did it happen after a clean install (after 'format') ?
When did it happen ? How many users ?

Start with:

(if doubts: in file /etc/rc.prunecaptiveportal copy line 50 to line 42 - and change the text to: "Portal minicron - running now"

it won't harm the activity of your pfSEnse setup - and show you when the minicron that logs of portal users, dies.

adegans

Alright, here you go:

What hardware are you sing ? 
Soekris 6501-70 - 1.6Ghz ARM, 2GB, Sata disk 250GB

Version	2.1-RELEASE (i386) 
Platform	 pfSense
CPU Type	 Genuine Intel(R) CPU @ 1.60GHz
2 CPUs: 1 package(s) x 1 core(s) x 2 HTT threads

Free disk size ? At least 200GB
Mem size ? 2GB (13% at time of writing)
Interfaces ? 4x 1Gbit (2x wan, 2x lan diff subnets)
What is your pfSense setup - basic use ? DHCP/DNS/FreeRadius (for CP)/CP (1 Zone)/NAT/Firewall/NTP
load ? Load average 0.10, 0.03, 0.01 (At time of this writing, never seen it go notably high)
Did it happen after a clean install (after 'format') ? This is a relative clean and basic setup (redid the whole thing for 2.1 because of a failed upgrade)
When did it happen ? Since install
How many users ? So far everyone who used CP

mikenl

What hardware are you using ?

ESXI 5.1 vm
Dual-Core AMD Opteron Processor 8218
2 CPUs: 2 package(s) x 1 core(s)
Load average 0.07, 0.20, 0.31
Memory usage 45% of 3051 MB
Disk usage 50% of 8.7G

Version 2.1-RELEASE (i386)
Platform pfSense
CPU Type Genuine Intel(R) CPU @ 1.60GHz
2 CPUs: 1 package(s) x 1 core(s) x 2 HTT threads

Interfaces ? 3 in use, 8 vlans not in use now
What is your pfSense setup - basic use ? DHCP DNS Radius NAT Firewall NTP Snort Squid
Did it happen after a clean install (after 'format') ? Not a clean install, upgrade from 2.0.1
When did it happen ? Since upgrade

This morning i couldn't "restart" the CP, users remain active even after a "save" on the config page. Disable/enable the CP helped, and the portal minicron is back.

Gertjan

All minicrons are still up ?

mikenl

Was away a few days, the cp minicrons are gone again :(

mikenl

Looking at the log :

Nov 10 12:02:47 lighttpd[10730]: (request.c.1133) GET/HEAD with content-length -> 400
Nov 10 11:52:51 php: /snort/snort_alerts.php: [Snort] Snort RELOAD CONFIG for lan(le1)…
Nov 10 11:52:50 check_reload_status: Syncing firewall
Nov 10 11:52:47 php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 10 11:51:46 php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 10 11:50:46 php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running

Until Nov 10 11:52:47 everything was working.
Snort reloads config for lan (captive portal is on another interface OPT1)

Gertjan

This:
@mikenl:

Nov 10 11:52:47 php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running

means that minicron invoked /etc/inc/rc.prunecaptiveportal 60 secondes later - but the instance that ran the minute before did not terminate - or didn't terminate properly.
First, it checks if another instance is already running. If so, it logs the message "Skipping CP pruning…" and stops.
The next minute (120 sec later), it will silently skip this message, and start the (another ?!) pruning process.

This boils down to the fact that the function captiveportal_prune_old(); in /etc/inc/captiveportal.inc doesn't stop in 60 seconds - or, worse, just, some how, "blows up".
This would explain that even the minicron itself is blown up.

@mikenl : I understood you use a radius authentication.
Have a look at the file /etc/inc/captiveportal.inc, find the function captiveportal_prune_old();
If you feel up to it, add this line
log_error("CP prunning process : step XX");
on several places, increment XX for every next log line.
This will indicate where in the code things go wrong.

Concentrate your 'log_error' line there where radius is involved.

I suspect that radius communication some how drops … taking down the entire pruning process.

adegans

Im using radius too, with account verification.
ALso I use timeouts and the expirydate.

Gertjan

For both of you, where is the radius server situated ?
On the LAN segment ?
Other packages like Snort, Sguid, etc do not interfere with the communication pfSense <=> Radius server ?

I don't know if it is possible for you (both): stop making use of radius solves the problem ?

edit: This http://forum.pfsense.org/index.php/topic,63791.0.html suggests putting the radius server on its own lan segment.

adegans

I'm using the FreeRadius2 on the computer itself. So it's localhost.

I can't disable it as I rely on it's timeout features for networked access (I sell access per hour using those accounts).
For a few accounts (and possible others where I just didn't notice it) I saw that the expiration date (In free radius) of the account may interfere with the expiring of the CP session.

Right now I set an account to expire the next day and define specific access such as "Ma0900-1200". Often the Ma… bit doesn't allow proper access and the session seems to remain active till the actual expiration date.

mikenl

@Gertjan:

This:

@mikenl : I understood you use a radius authentication.
Have a look at the file /etc/inc/captiveportal.inc, find the function captiveportal_prune_old();
If you feel up to it, add this line
log_error("CP prunning process : step XX");
on several places, increment XX for every next log line.
This will indicate where in the code things go wrong.

Concentrate your 'log_error' line there where radius is involved.

I suspect that radius communication some how drops … taking down the entire pruning process.

Will do.
Not using the Radiusserver is not an option, using Freeradius on a windows host in the same LAN.

mikenl

Ok, don't know if i'm getting anywhere.
To me it seems that it just stops :

Nov 12 20:38:29 	lighttpd[75519]: (request.c.1133) GET/HEAD with content-length -> 400
Nov 12 19:46:16 	php: rc.filter_configure_sync: There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.
Nov 12 19:46:16 	php: rc.filter_configure_sync: The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was '/tmp/rules.test.packages:45: syntax error /tmp/rules.test.packages:46: syntax error'
Nov 12 19:46:15 	php: rc.filter_configure_sync: There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.
Nov 12 19:46:15 	php: rc.filter_configure_sync: The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was '/tmp/rules.test.packages:46: syntax error'
Nov 12 19:46:11 	check_reload_status: Reloading filter
Nov 12 19:46:11 	php: /pkg_edit.php: Reloading Squid for configuration sync
Nov 12 19:46:09 	check_reload_status: Syncing firewall
Nov 12 19:45:22 	php: rc.filter_configure_sync: There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.
Nov 12 19:45:22 	php: rc.filter_configure_sync: The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was '/tmp/rules.test.packages:45: syntax error /tmp/rules.test.packages:46: syntax error'
Nov 12 19:45:21 	php: rc.filter_configure_sync: There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.
Nov 12 19:45:21 	php: rc.filter_configure_sync: The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was '/tmp/rules.test.packages:46: syntax error'
Nov 12 19:45:18 	check_reload_status: Reloading filter
Nov 12 19:45:18 	php: /pkg_edit.php: Reloading Squid for configuration sync
Nov 12 19:45:16 	check_reload_status: Syncing firewall
Nov 12 19:45:06 	php: /diag_logs.php: Successful login for user 'admin' from: 192.168.0.9
Nov 12 19:45:06 	php: /diag_logs.php: Successful login for user 'admin' from: 192.168.0.9
Nov 12 19:45:04 	php: rc.prunecaptiveportal: CP prunning process : step 01b
Nov 12 19:45:04 	php: rc.prunecaptiveportal: CP prunning process : step 01
Nov 12 19:45:04 	php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 12 19:44:03 	php: rc.prunecaptiveportal: CP prunning process : step 01b
Nov 12 19:44:03 	php: rc.prunecaptiveportal: CP prunning process : step 01
Nov 12 19:44:03 	php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 12 19:43:03 	php: rc.prunecaptiveportal: CP prunning process : step 01b
Nov 12 19:43:03 	php: rc.prunecaptiveportal: CP prunning process : step 01
Nov 12 19:43:03 	php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 12 19:42:02 	php: rc.prunecaptiveportal: CP prunning process : step 01b
Nov 12 19:42:02 	php: rc.prunecaptiveportal: CP prunning process : step 01
Nov 12 19:42:02 	php: rc.prunecaptiveportal: Skipping CP prunning process because previous/another instance is already running
Nov 12 19:41:02 	php: rc.prunecaptiveportal: CP prunning process : step 01b
Nov 12 19:41:02 	php: rc.prunecaptiveportal: CP prunning process : step 01

/etc/inc/captiveportal.inc :

	$radiussrvs = captiveportal_get_radius_servers();
log_error("CP prunning process : step 01");

	/* Read database */
	/* NOTE: while this can be simplified in non radius case keep as is for now */
	$cpdb = captiveportal_read_db();
log_error("CP prunning process : step 01b");

adegans

I gave in (up?) and removed Radius, time outs using the local usersystem work fine.

Just FYI… This to me confirms there is a problem with Freeradius.

Gertjan

@mikenl:

log_error("CP prunning process : step 01");

This is the way to go.
You should place des "step xx" everywhere.
Just before a function, and just after a function.

Like:

log_error("CP prunning process : step 01b - before");
	$cpdb = captiveportal_read_db();
log_error("CP prunning process : step 01b - after");

```After a while, when it brakes, you'll know where it entered, and never came back.

If it is a radius function call, have a look at /etc/inc/radius.inc

Btw: I'm not using Radius but I'm pretty sure many people do.
I don't think they all accept "unlimited portal access" while they are using Radius as an accounting system.
So: the questions is: they aren't aware of the situation ? Radius is working well for them ?
You both aren't using the same radius server …

mikenl: STOP using squid - your log shows far to many errors - even worse: it tries to redo the firewall itself with faulty rules. This means that one of the basic function of your pfSEnse box becomes messy. I would consider it as non-stable.

mikenl

Well, i've stopped using squid and now all seems to be well.
The CP is running for three days now whitout any issue.

Edit :

Ahum, well i made some changes to snort and :

Nov 21 08:47:04	kernel: le2: promiscuous mode enabled
Nov 21 08:46:56	php: /snort/snort_interfaces.php: [Snort] Snort START for opt1(le2)...
Nov 21 08:46:55	php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for OPT1...
Nov 21 08:46:55	php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: OPT1...
Nov 21 08:46:50	php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: OPT1 ...
Nov 21 08:46:49	php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for LAN...
Nov 21 08:46:49	php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: LAN...
Nov 21 08:46:45	php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: LAN ...
Nov 21 08:46:44	php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for WAN...
Nov 21 08:46:44	php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN...
Nov 21 08:46:40	php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ...
Nov 21 08:46:40	php: /snort/snort_interfaces.php: Toggle (snort starting) for OPT1(opt1)...
Nov 21 08:46:12	php: /snort/snort_rulesets.php: [Snort] Building new sig-msg.map file for WAN...
Nov 21 08:46:11	php: /snort/snort_rulesets.php: [Snort] Enabling any flowbit-required rules for: WAN...
Nov 21 08:46:07	php: /snort/snort_rulesets.php: [Snort] Updating rules configuration for: WAN ...
Nov 21 08:46:07	check_reload_status: Syncing firewall
Nov 21 08:45:45	kernel: le2: promiscuous mode disabled
Nov 21 08:45:45	kernel: pid 50689 (snort), uid 0: exited on signal 11
Nov 21 08:45:44	lighttpd[17615]: (connections.c.137) (warning) close: 27 Connection reset by peer
Nov 21 08:45:44	check_reload_status: Syncing firewall
Nov 21 08:45:39	php: rc.prunecaptiveportal: CP prunning process : step 02b begin

The rule : php: rc.prunecaptiveportal: CP prunning process : step 02b begin should be in the log every minute.

Gertjan

@mikenl:

Nov 21 08:45:45	kernel: pid 50689 (snort), uid 0: exited on signal 11

Please Google this "exited on signal 11".
signal 11 normally stands for "Houston, we have a problem …"
This problem could be 'software' or worse: hardware.

mikenl

Ok,
I have had zero problems the last days since i disabled squid and didn't touch the system.
Too bad, i was using squid in combination with squidguard.

Gertjan

What you could try:
If you have a spare NIC, put your radius server on that NIC.
Now, you could "run" squid. DO NOT let it handle or touch the NIC where the radius server is running.
When it goes down, making a mess on all its interfaces, it will not interact with the radius connection.

Please note: I do not advise you to run a package that goes "signal 11" or core dump, but this test would nail down the issue.

The minicron that prunes the portal users is running a high level language like PHP, it takes a boatload of code, and some time - questions radius about every users status. When squid goes down, this is interrupted, taking the minicron with it.
That is what was happening on your system (I think).

mikenl

The CP pruning process is still running since my last post.
I will now try and run squid on an interface which the radius server isn't connected to.

mikenl

The CP is still running, squid is active on another IF which the radius server isn't connected to.
And only active on 1 IF, i don't see any weird squid errors anymore.