22.05 - CP clients have connectivity issues after x amount of time

heper

@gertjan
@stephenw10
arrived at work "early" to verify the things you've discovered this weekend.
current CP settings:
idle timeout: 600minutes (to prevent anyone from idle-timeout during the entire day)
per-user-bw: 240d | 20u mbit/s

i can confirm all users are using the same pipeno:

username: Ines.
        pipeno: 2002
        MAC: 5e:50:66:f3:e8:7e
        IP: 172.16.20.57
************************************************

username: Bchir
        pipeno: 2002
        MAC: 9e:a8:ce:44:7a:e4
        IP: 172.16.23.155
************************************************

username: Jeroe
        pipeno: 2002
        MAC: 9a:7b:b9:e4:c4:1c
        IP: 172.16.20.50
************************************************

snippet of pfanchordrill:

cpzoneid_2_auth/172.16.23.164_32 rules/nat contents:
ether pass in quick proto 0x0800 from 04:b4:29:2d:30:83 l3 from 172.16.23.164 to any tag cpzoneid_2_auth dnpipe 2002
ether pass out quick proto 0x0800 to 04:b4:29:2d:30:83 l3 from any to 172.16.23.164 tag cpzoneid_2_auth dnpipe 2003

cpzoneid_2_auth/172.16.23.23_32 rules/nat contents:
ether pass in quick proto 0x0800 from 9e:36:73:93:d4:20 l3 from 172.16.23.23 to any tag cpzoneid_2_auth dnpipe 2002
ether pass out quick proto 0x0800 to 9e:36:73:93:d4:20 l3 from any to 172.16.23.23 tag cpzoneid_2_auth dnpipe 2003

cpzoneid_2_auth/172.16.23.42_32 rules/nat contents:
ether pass in quick proto 0x0800 from a4:45:19:64:0c:b8 l3 from 172.16.23.42 to any tag cpzoneid_2_auth dnpipe 2002
ether pass out quick proto 0x0800 to a4:45:19:64:0c:b8 l3 from any to 172.16.23.42 tag cpzoneid_2_auth dnpipe 2003

below limiter information in chronological order:
1 working
2 client disconnect = not working
3 client (re)connect = working

1 output /sbin/dnctl pipe show:

02002:  20.000 Mbit/s    0 ms burst 0
q133074 100 sl. 0 flows (1 buckets) sched 67538 weight 0 lmax 0 pri 0 droptail
 sched 67538 type FIFO flags 0x0 16 buckets 1 active
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        7      606  0    0   0
02003: 240.000 Mbit/s    0 ms burst 0
q133075 100 sl. 0 flows (1 buckets) sched 67539 weight 0 lmax 0 pri 0 droptail
 sched 67539 type FIFO flags 0x0 16 buckets 1 active
  0 ip           0.0.0.0/0             0.0.0.0/0       37    45954  0    0   0
02000: unlimited         0 ms burst 0
q133072 100 sl. 0 flows (1 buckets) sched 67536 weight 0 lmax 0 pri 0 droptail
 sched 67536 type FIFO flags 0x0 16 buckets 1 active
  0 ip           0.0.0.0/0             0.0.0.0/0        2      156  0    0   0
02001: unlimited         0 ms burst 0
q133073 100 sl. 0 flows (1 buckets) sched 67537 weight 0 lmax 0 pri 0 droptail
 sched 67537 type FIFO flags 0x0 16 buckets 0 active

2 output /sbin/dnctl pipe show AFTER MANUAL DISCONNECT:

02000: unlimited         0 ms burst 0
q133072 100 sl. 0 flows (1 buckets) sched 67536 weight 0 lmax 0 pri 0 droptail
 sched 67536 type FIFO flags 0x0 16 buckets 0 active
02001: unlimited         0 ms burst 0
q133073 100 sl. 0 flows (1 buckets) sched 67537 weight 0 lmax 0 pri 0 droptail
 sched 67537 type FIFO flags 0x0 16 buckets 0 active

3 output /sbin/dnctl pipe show WHEN A new(old) CLIENT (re)CONNECTS:

02002:  20.000 Mbit/s    0 ms burst 0
q133074 100 sl. 0 flows (1 buckets) sched 67538 weight 0 lmax 0 pri 0 droptail
 sched 67538 type FIFO flags 0x0 16 buckets 1 active
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       41     3143  0    0   0
02003: 240.000 Mbit/s    0 ms burst 0
q133075 100 sl. 0 flows (1 buckets) sched 67539 weight 0 lmax 0 pri 0 droptail
 sched 67539 type FIFO flags 0x0 16 buckets 1 active
  0 ip           0.0.0.0/0             0.0.0.0/0      260   374935  0    0   0
02000: unlimited         0 ms burst 0
q133072 100 sl. 0 flows (1 buckets) sched 67536 weight 0 lmax 0 pri 0 droptail
 sched 67536 type FIFO flags 0x0 16 buckets 0 active
02001: unlimited         0 ms burst 0
q133073 100 sl. 0 flows (1 buckets) sched 67537 weight 0 lmax 0 pri 0 droptail
 sched 67537 type FIFO flags 0x0 16 buckets 0 active

Gertjan

@heper said in 22.05 - CP clients have connectivity issues after x amount of time:

i doubt it's a gui issue.
because currently i'm under the impression that i might be hitting the same issue:

with or without per-user-bandwidth enabled (i guess when disabled, everyone gets added to the same unlimited pipe-pair?)

with manual disconnect on gui
when CP-client triggers the idle-timeout

In any case, captiveportal_disconnect() gets called, and this will delete conditionally delete the pipe ($pipeno) of that user.
If all users use the same pipe, then that will will impact all the users that use that pipe.
edit : removed my script, you - @heper know how to dump the connected portal user database and see pipes used per user..

edit : lol, 12 minutes to late : all your users use the same pipe number.

I did this to restore unique pipeno usage :

/usr/local/captiveportal/index.html

	$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, true);
.......
	}
	
	$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);

I changed the true for false on the first line
Added an 'else' condition.

	$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, false);
.......

	} else
		$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);

Now, if a pipe gets attributed, but the login fails, the pipe will get freed up again.

My users have now unique pipes.

heper

@gertjan said in 22.05 - CP clients have connectivity issues after x amount of time:

edit : removed my script, you - @heper know how to dump the connected portal user database and see pipes used per user..

i copy/pasted your script yesterday & adjusted it to not print_r the entire array, but only the information i wanted. =)

heper

@gertjan
i won't change the index.php or any other file on my production system unless it's by an official patch or at the request of netgate.

current testing indicates i can work around the bug.

by preventing anyone from disconnecting manually (blanked out the logout page)
by setting the idle timeout to 10 hours to cover the entire school-day
by setting the per-user-bw value's high enough because everyone uses the same pipeno

i'll keep close watch on it today, but i think/believe you found the root cause.

Gertjan

@heper
If this is the issue, there will be an official patch, and with a little luck an update for the pfSense patch package with a build in patch, so it will be a one click solution.

@heper said in 22.05 - CP clients have connectivity issues after x amount of time:

i won't change the index.php or any other file on my production system unless it's by an official patch or at the request of netgate.

I fully understand ;)

stephenw10

Yes, me too.

Just replicating that here now. Hopefully we should have something more official available soon.

Steve

heper

@gertjan

i've monitored cp-situation fairly closely the entire day.
i didn't notice any dropouts throughout the day with the workaround i have in place.

thanks for looking into this & figuring out what the issue is

stephenw10

I have replicated this and we are reviewing it. I'll open a bug report that better describes the issue momentarily:

https://redmine.pfsense.org/issues/13488

Please add anything there that may be pertinent.

Steve

heper

@stephenw10

the same thing happens when per-user-bw is disabled:

all users use the same pipe
- on disconnect that pipe gets removed

should i add this to the redmine myself or will you change the summary ?

Gertjan

@heper

Actually, bandwidth set, for all users, or not, or set individually when using FreeRadius, it doesn't matter.
The issue has nothing to do with bandwidth, but bandwidth usage made it more 'visible'.
Your video was spot on : all connected users together use the set bandwidth of one user (10 Mbits down). Looking again at the limiters list did the rest.

Solving this one is easy.

I'm often think about what jimp said ones in the past : we will have to re write all this captive portal (Italian food) 'code', he mentioned Python if I recall well. As "/etc/inc/captiveportal.inc" is a bit .... messy.

stephenw10

Ah, OK, that is likely a hangover from ipfw where all traffic had to use a pipe in order to be filtered at L2. Though the pipe was set at unlimited bandwidth.
Yes, add that or I can when I've tested it later today.

Steve

stephenw10

Ok, yup, it's exactly that. Adding it.

level4

@gertjan said in 22.05 - CP clients have connectivity issues after x amount of time:

I did this to restore unique pipeno usage :

/usr/local/captiveportal/index.html
	$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, true);
.......
	}
	
	$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);
I changed the true for false on the first line
Added an 'else' condition.
	$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, false);
.......

	} else
		$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);
Now, if a pipe gets attributed, but the login fails, the pipe will get freed up again.

My users have now unique pipes.

@Gertjan
Would you be so kind to share the modified 22.05- index.php ?
(Or a real diff)

I'm having the same issue, and would like to try this fix, but dont really understand what should be changed.

Gertjan

@level4

Sorry, never managed to make a "diff" - tried many times, but the obtained patch file isn't usefull (for me) otherwise I would use that myself, and it would be easier to communicate here, I agree.

The original code (from github):

Step 1: before the first line "$pipeno = captiveportal_get_next_dn_ruleno('auth',......", add

$auth_result = array();

Step 2 : change the (now second) line

$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, true);

for

$pipeno = captiveportal_get_next_dn_ruleno('auth', 2000, 64500, false);

Step 3 - goto the closing line 262 "}"
and add an "else" behind and remove the following empty line.

The result looks like :

level4

@gertjan Thanks!

I'm a total php noob, but;

That "} else" ..

shouldn't be a "} else {"

with a "}" below the "$auth_result =" line ?

Gertjan

@level4 said in 22.05 - CP clients have connectivity issues after x amount of time:

That "} else" ..
shouldn't be a "} else {"
with a "}" below the "$auth_result =" line ?

You can place

$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);

between { .... }, true. Like :

	.......
	} else
	{
		$auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context);
	}

Or

	.......
	} else { $auth_result = captiveportal_authenticate_user($user, $passwd, $clientmac, $clientip, $pipeno, $context); }

But as this is just one line, so no need.