Captive portal ignoring MACs in latest version and allowing all machines access
-
Something new after the above steps were taken:
If an entry in the MAC list has bandwidth limitations, but the Action is set to PASS, then only the download bandwidth is affected, and upload speed is still unrestricted.
If I change the MAC entry to "DENY" then both upload and download bandwidth is unrestricted.
So right now if I want to stop a machine from accessing the internet, I have to make sure it has a MAC address entry in the list, then I have to set it to PASS and then set the download speed to "5".... but they can still upload. This is different from yesterday when I could set any MAC entry's speed regardless of PASS or DENY and it would control both download and upload speed.
I'm honestly considering blowing the whole configuration away and starting the router configuration over from scratch. But as I mentioned before, if I restore this configuration to the last version of pfSense, it works just fine, so there's no guarantee that starting over will solve this problem.... just more time wasted.
-
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Deleted the Captive Portal profile. Result: all machines can access the Internet.
With default firewall rules on LAN, yes, that's normal.
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Created a new Captive Portal profile. Result: All machines blocked by the "Captive Portal" login prompt
All devices non authenticated should be blocked, that's normal.
Although, a second captive portal on the same LAN interface ? Even with the first interface disabled, I wouldn't test drive such a thing.@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Added one machine's MAC address: that machine no longer gets login prompt, but cannot access the Internet. Strange.
As you showed above, MACs that are added as a pass, are added to the first table "default_pipe_mac"
--- table(default_pipe_mac), set(0) --- e0:ff:49:4a:91:f5 any 2689 0 0 0 any e0:ff:49:4a:91:f5 2688 0 0 0 ....
The numbers 2689 and 2688 are the references to the pipes created for that MAC - on for each direction - your "ipfw_pipe_list.txt" above listed these two pipes as unlimited :
02689: unlimited 0 ms burst 0 q133761 100 sl. 0 flows (1 buckets) sched 68225 weight 0 lmax 0 pri 0 droptail sched 68225 type FIFO flags 0x0 16 buckets 0 active
and
02688: unlimited 0 ms burst 0 q133760 100 sl. 0 flows (1 buckets) sched 68224 weight 0 lmax 0 pri 0 droptail sched 68224 type FIFO flags 0x0 16 buckets 0 active
But : the machine can not access the anything (Internet).
Be more precise please : can it access - or visit, at least the pfSense GUI (not the portal login page, but the GUI - check you GUI firewall if that's permitted) ? Or some other close by web server / other service I mean, a gateway problem make us looking at portal problems, but (example) Internet is simply down.@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Did a "Backup" of the Captive Portal, edited the file to insert all of the MAC address items, then "Restored" the Captive Portal backup file... now all machines can access the Internet... the MAC address list is now completely ignored again.
This is what's bugging me.
No MAC's on the list : everybody is blocked => That normal.
Add one MAC : this one should pass : you can see it in the "default_pipe_mac" table twice. These two rules should match, traffic should flow, according to attached pipe constraints.
Two MAC's : identical, but for those 2.may MAC : everybody passes now .... ?? It's not even pfSense that misbehaves, it looks like you broke 'ipfw', for what I know, an industrial strength firewall .
I can't find an entry or rule or whatever that explains your issue.When no MACs or IPs are listed in any tables, a device will hit rule 02117 (and 02118 if https portal page) who interacts with the portal web server - rule 02119 permit the portal server to 'talk' to the portal interface - rule 02120 jumps to the final rule 65534 that represent the wall == block everything.
I understand that you can't tear the place down in pieces, the setup is in a live environment.
The captive portal is used by thousands - and it should have been know that adding some MAC's to the MAC "Pass or Block" page adds a free ride for everybody - I have some blocked MAC's and Passes on that MAC page : it just work for years now.
Also : I'm not a developer, but I can read Github : captive portal code didn't change between 2.4.4 p1 p2 and p3.
There is this "You are connected" 2.4.4-p3 bug which resets / flushes all tables in ipfw when an pfSense admin saves the portal settings. Authenticated users are still shown as "logged in" in the GUI, but related IP/MAC are flushed from the tables : these users can't login again (no login page is shown, just the text "You are already logged in") , because pfSense considers them as "logged in" and thus ipfw rules (tables) are NOT added for them to pass : users are locked out.
A pull request has been issued that resolves this issue.
IMHO : Your issue is not related to this one.@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
If an entry in the MAC list has bandwidth limitations, but the Action is set to PASS, then only the download bandwidth is affected, and upload speed is still unrestricted.
If I change the MAC entry to "DENY" then both upload and download bandwidth is unrestricted.It would be nice if @jimp could see this - if he could understand what situation would make this happen.
I imported into my config 200 records like (with incrementing MAC IDs):
<passthrumac> <action>pass</action> <mac>10:20:30:40:50:60</mac> <descr><![CDATA[test test]]></descr> </passthrumac>
and 50 "block" records .
I also added the MAC of my Phone, as a "pass", and my pad as a blocked device and I tested : all well - phone could communicate, pad was blocked.What I can't test is the leaking through as you described : I don't have that many real devices.
I tested different Bandwidth up and Bandwidth down values like 4321 and 1234 (Kbits) : the speed was respected.
-
The captive portal is used by thousands - and it should have been know that adding some MAC's to the MAC "Pass or Block" page adds a free ride for everybody - I have some blocked MAC's and Passes on that MAC page : it just work for years now.
Like I said before, I've used it for years, and this client has used it for years.
We have had problems with Captive Portal. Two bugs were acknowledged and fixed recently.
I'm going to reset the settings on that router and enter everything again from scratch over the weekend.
There is this "You are connected" 2.4.4-p3 bug which resets / flushes all tables in ipfw when an pfSense admin saves the portal settings. Authenticated users are still shown as "logged in" in the GUI, but related IP/MAC are flushed from the tables :
This also happens here when people try to login, but we don't use the login feature anymore so it doesn't matter.
-
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Like I said before, I've used it for years, and this client has used it for years.
I understand - I'm just repeating myself a lot (close to rambling perhaps).
You and I use the same code. I really want see this bug. Seeing is is resolving it. -
I understand - I'm just repeating myself a lot (close to rambling perhaps).
You and I use the same code. I really want see this bug. Seeing is is resolving it.If I install the latest pfSense on a separate machine and import this configuration, the problem exists also on the new machine. Perhaps I could sanitize the configuration file and post it here. Perhaps you can see something I can't. But I know it can be reproduced. It's not a complicated router-- one WAN, one LAN. No hot backup.
-
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Perhaps I could sanitize the configuration file and post it here
Start with the entire <captiveportal> </captiveportal> xml dump.
Don't post it here : drop it on a pastebin.org and send me the link.What packages are installed on this setup ?
-
@Gertjan said in Captive portal ignoring MACs in latest version and allowing all machines access:
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Perhaps I could sanitize the configuration file and post it here
Start with the entire <captiveportal> </captiveportal> xml dump.
I'll do this a bit later this morning. I think I already sent you everything except for the MAC entries.
Can you send me your email address? I can send you these files directly. Send me an email h2professor@gmail.comWhat packages are installed on this setup ?
arping*
bandwidthd
cron*
darkstat*
FTP_Client_Proxy
notes*- can be uninstalled because we never use it... installed by the client's request years ago.
-
Notes and cron cron are totally inoffensive,
Same thing for arpping, I just tested that one.
FTP_Client_Proxy : I thought that FTP was abandoned .... but I'll install it anyway.
bandwidthd and darkstat : never used them, but I'm pretty sure they do "mess" with the IP stack ....
-
FTP_Client_Proxy : I thought that FTP was abandoned .... but I'll install it anyway.
We have no choice on this one. There are legacy devices in the organization that have to communicate with legacy FTP servers externally. I'm sure there's a workaround, but I can't believe this would suddenly be a problem after using it for years.
bandwidthd and darkstat : never used them, but I'm pretty sure they do "mess" with the IP stack ....
These could be uninstalled temporarily (darkstat permanently) but again, been using them for years with no problem.
edit: I just removed Darkstat and had no effect on the problem. -
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
been using them for years with no problem.
A recent update ?
Packages are not static.
For example, it happened that FreeRadius was upgraded and breaking my portal access because "there was a small glitch" in the newer version.I don't use darkstat or bandwidthd so I don't know if they were upgraded, which could be a potential reason.
I'll install them anyway.Btw : if this setup isn't patched or changed manually, you could take a backup (config file) of the system, and remove these packages for testing purposes. re-installing them afterwards, importing the backup file and a reboot would take care of things.
I'll report back Monday, have to leave now. Weekend ;)
-
Did you resolve this? We are running into a problem that seems similar:
- We have incrementally upgraded from pfsense 1.x to 2.4.4.p3
- We have MAC addresses (but also allowed IP addresses)
- Setting bandwidth limits on the MAC addresses works, but the captive portal does not block blocked MAC addresses
- All users are able to use the network without the captive portal
- DNS works and is being done by pfSense
- The firewall allows http/https access
However, one thing is different: removing all MAC address listings does NOT fix the problem. The captive portal continues not to work.
I can generate the results of the ipfw commands above but do not what I am looking for.
-
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
Did you resolve this? We are running into a problem that seems similar
Yes, that does seem similar. We did not get this solved with the latest version. We ended up rolling back to a mirror-drive backup dated October 7, 2017 and it's working. (The configuration is 100% identical other than the software version.) We're not going to apply any updates until we figure out how to get it working on the latest version. Nobody in this forum believes there's a real problem, so it's somewhat comforting that someone else is working the same problem, although I do apologize that it had to be you.
Coincidentally my client is shutting down for a company-wide getaway this coming week so I'll be able to focus my attention on getting the most recent version working, if it's possible. I might experiment with 2.5.x experimental releases if all else fails.
I will post my findings in this forum.
Thank you
-
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
removing all MAC address listings does NOT fix the problem
Question: are you using a custom Captive Portal login page? Or the stock firewall login page?
Thank you -
@h2professor It is a custom login page.
Unlike your situation we do ask for a username and password, using the local database.
-
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
It is a custom login page
If it's no trouble, may I suggest that you delete your custom page and see if the problem resolves without it?
In a non-production setup last July (with the same configuration) we ended up in a situation where everything was working until we added the custom page. If this happens with yours as well, then we might narrow down the problem a bit.
Thanks -
No, when I delete the custom login page and use the default I get the same behavior.
-
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
I get the same behavior
okay thank you for trying. We'll proceed with our testing here and will share anything that might be useful.
Cheers -
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
Nobody in this forum believes there's a real problem,
You had side-long talks in this topic about your problem and say nobody believes you? I find that hard to believe.
As the last of this was talked about in June/July, did you @h2professor ever try to re-create the steps manually to "create" the problem you have?
After reading through, I was always stuck with "and then I re-imported my whole/part of my config and the problem was back". Is it possible to re-trace the steps for the problem to appear? As @Gertjan said: seeing is (no, not believing) half way to fix a potential bug. But as this always re-appeared after updating/re-importing the config I'm wondering if it's a problem with the configuration as it is.
Also did anyone try and open a bug report at the pfsense redmine and link this thread? Might be helpful for hunting down a potential bug!
https://redmine.pfsense.org/projects/pfsense/issues -
@JeGr said in Captive portal ignoring MACs in latest version and allowing all machines access:
I find that hard to believe
How ironic.
We tried everything that was suggested, but the client ran out of patience for the problem, so we were forced to use the last working cold mirror to get it back online.
Here's a summary:
It worked prior to 2.4.4-p3. It has worked for years.
We upgraded to p3: it no longer works.
We did a clean installation of p3: confirmed captive portal works as advertised in a basic installation on the hardware.
We imported the original configuration: no longer works.
We did a clean installation of the older 2.4.4-p1 and imported configuration: worked just fine.
We tried dozens of suggestions, including removing packages, reinstalling, cutting sections out of the configuration, etc.We had to conclude that the problem is in version 2.4.4-p3 despite people insisting that "no changes were made to the captive portal section" bla bla bla. Hence, it's not a believable problem.
I have a week with the client out of their building so I have time to figure this out, if it is possible.
-
Well, I solved the problem in my installation, but I expect it won't help @h2professor . We have some allowed IPs in "Allowed IP addresses." One of our local IPs was listed there. However, that dialog allows you to set a subnet mask, and the IP in that subnet had been set to /24, not /32. So it was allowing the entire subnet through!
In the process of finding this, I rebuilt my captive portal:
- I created a dummy interface and assigned the old captive portal there.
- I made the configurations the same for the new portal.
- I added all of the IP addresses and hostnames (but idiot that I was, I inadvertently used a /32 for the offending IP in my new build, and not a /24 like in the old one)
- I added the allowed MAC addresses one by one
- I re-added my custom login page
and sure enough the new portal worked, but the old one didn't. (I tested the new captive portal after adding each component, hoping to find the one that broke the build.)
I also exported the captive portal config for these two captive portals and compared them, but I was not able to see any significant differences.
The only thing that occurred to me was whether there might be a duplicate MAC address in the list? Due to a copy and paste error when transcribing entries I thought I had one. The current pfSense interface will not let me add a duplicate MAC address, but maybe some old one did?
-
It is not quite true to say that the XML for the two captive portal backups were identical. There were some extra lines in the old one that were not present in the new one:
<radius_protocol></radius_protocol> <redirurl></redirurl> <radiusip></radiusip> <radiusip2></radiusip2> <radiusip3></radiusip3> <radiusip4></radiusip4> <radiusport></radiusport> <radiusport2></radiusport2> <radiusport3></radiusport3> <radiusport4></radiusport4> <radiusacctport></radiusacctport> <radiuskey></radiuskey> <radiuskey2></radiuskey2> <radiuskey3></radiuskey3> <radiuskey4></radiuskey4> <radiusvendor>default</radiusvendor> <radiussrcip_attribute>wan</radiussrcip_attribute>
(
redirurl
is actually in both).I notice that
radiussrcip_attribute
is set towan
in my listing and set tolan
in yours. I tried changing mine toopt1
(my captive portal interface) in my backup file and restoring, but it made no difference. -
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
I solved the problem in my installation
That's excellent work, thank you very much. We're going to try that. It's something we haven't considered. I'll report back in a couple of days.
Cheers -
@theworkingcentre said in Captive portal ignoring MACs in latest version and allowing all machines access:
I notice that radiussrcip_attribute is set to wan in my listing and set to lan in yours. I tried changing mine to opt1 (my captive portal interface) in my backup file and restoring, but it made no difference.
When the config file version moved from 18.6 to 18.7 (august 2018), that's several updates in the past, we're 19.1 now - "radiussrcip_attribute" was created and set to 'wan' as de default value.
It's a radius related setting, and if radius isn't used it isn't important.
<captiveportal> <cpzone1> <zone>cpzone1</zone> <descr><![CDATA[cpzone1]]></descr> <zoneid>2</zoneid> <interface>opt1</interface> ...... <radiussrcip_attribute>opt1</radiussrcip_attribute> ...... </cpzone1> </captiveportal>
I'm using FreeRadius - my portal is running on OPT1.
-
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
We had to conclude that the problem is in version 2.4.4-p3 despite people insisting that "no changes were made to the captive portal section" bla bla bla. Hence, it's not a believable problem.
People don't "insist" anything. My post also wasn't meant as an insult or attack, I was simply wondering why you wrote that. The devs simply said in another thread, that they did check the portal code and there was no commit or anything else belonging to the portal code in between -p1 and -3. So it either has to be a dependency like e.g. radius or some other thing that links into the portal like IP lists etc. what @theworkingcentre wrote.
I also am not "ironic" but I find it quite a bit "unfair" to write sth like "nobody believes me" after page-long tries to help you debug the problem Also your list in "things we did do" lists "importing the configuration" EVERY time. So like I wrote above, how about simply debugging the thing with a fresh installation and configuring step by step the way it is now in your clients environment? Why do you insist of always importing the config.xml when it could very well be some thing that doesn't get imported the right way or perhaps was a problem with importing and updating this installation every time? I had small problems like that a few times. Rarely but they do exist. Changes in config format and old configs that have some duplicate lines or sth along those lines.
So why not trying one of this things instead of being sarcastic yourself and accuse us for doing "nothing"? At the end of the day we are simply other users and/or private persons here. This is not the official Netgate pfSense bug tracking support system.
And as much as I would like to see you/us succeed in tackling that problem: If your client is that sensitive and you have that narrow a margin in errors, wouldn't it be a better job to buy support for that device and ask the devs to have a look / fix it in your client's setup instead of getting angry with us for trying to help you? Or at least before going the support route submit that case with all details as a an official bug in the bugtracker? Sorry, I'm just trying to help here. And I'm not related or employed by Netgate after all. :)
Best wishes!
-
I"m still reading here .... trying to figure out.
Last couple of weeks I loaded a Hyper-VM on my 2012 Win server, it has 3 NIC's, so I can simulate and test without disturbing my companie network. Also : I'm using a second PC @home loaded with pfSense (using VM also).
Detail these 2 phrases :
@h2professor said in Captive portal ignoring MACs in latest version and allowing all machines access:
After, with 28:c6:8e:0f:95:9b set to Block
after.txt
I note that the MAC is not found in the second output.As soon as the captive portal is activated on an interface, everybody (MAC, IP, whatever) is blocked.
Even when you have this :
on the LAN interface.ipfw takes precedence of the ip firewall. ip being the firewall you set up with the GUI.
When you add a MAC on the MAC tab as a "pass" , this MAC will be part of your table "default_pipe_mac":
02100 145763334 141217081935 pipe tablearg ip from any to any MAC table(default_pipe_mac)
This is a snaphot of your "default_pipe_mac" :
dc:ef:09:9b:a8:c0 any 2671 1155484 1699152753 1560263421 any dc:ef:09:9b:a8:c0 2670 11 0 1560262129
You can see the MAC, the pipe rule numbers 2670 (down) and 2671 (up) and the number of bytes received and send.
These are the related pipe rules 2670 and 2671 :
..... 02670: unlimited 0 ms burst 0 q133742 100 sl. 0 flows (1 buckets) sched 68206 weight 0 lmax 0 pri 0 droptail sched 68206 type FIFO flags 0x0 16 buckets 0 active ..... 02671: unlimited 0 ms burst 0 q133743 100 sl. 0 flows (1 buckets) sched 68207 weight 0 lmax 0 pri 0 droptail sched 68207 type FIFO flags 0x0 16 buckets 0 active ...
Both are unlimited pipes.
Btw :
I found one (just 1) speed limiting (half a mega / s )pipe :02223: 500.000 Kbit/s 0 ms burst 0 q133295 100 sl. 0 flows (1 buckets) sched 67759 weight 0 lmax 0 pri 0 droptail sched 67759 type FIFO flags 0x0 16 buckets 0 active
Pipe 2223 : so this is device
28:c6:8e:0f:95:9b any 2223 11722 14806366 1560263421 any 28:c6:8e:0f:95:9b 2222 2 0 1560262183
is speed limited - this is the only device I found that was limited speed.
The blocked MAC list : as you might have understand, MAC's that are blocked are not present in the ipfw tables and rules.
When you add a MAC as blocked, it's been put in a list handled by the GUI.
The Captive portal web server, when intercepting a (in your case : http) visitor web browser http requests, are redirected to this page page :(some conditions have to be met, like this page must is on the same LAN segment as the captive portal - there must be a http web server that can serve the page, etc - it might, it might not. For me, using an iphone, it didn't redirect well )
If no URL, the device is blocked, for any IP, for any port, for any protocol.
But : when a MAC isn't present on the MAC tab, or it's set as a red block, it won't pass.
I advice you to use and old PC to test - make sure there is a second NIC, and setup pfSense for yourself. Although I strongly advise you to use a captive portal on a dedicated - OPT1 - NIC, and leave the LAN for administrative purposes.
When applied the minimal setup as per Netgate's video (there are 3 videos on Youtube, the Netgate channel, take a recent one that handles basic operations) no device can connect, and they will show the default Login when you use a web browser on a visiting device. https restrictions might apply.
Now, when you add ONE MAC as a pass, this device can pass to the net. Right ?
Still, no other device can pass. Right ?
Add another MAC as a pass. It passes right ?
An still, no other devices can pass.
For the fun, add a MAC of a device that you own, as a BLOCK. It can not pass, right ?
And again, other, non listed MAC's still can't pass.
Etc etc.You could even import your entire "300 MAC" list.
I would do this by exporting the config.xml - then use notepad++ to insert the block of.... <passthrumac> <action>pass</action> <mac>xx:8d:79:91:ec:52</mac> <bw_up></bw_up> <bw_down></bw_down> <descr><![CDATA[Sophie]]></descr> </passthrumac> <passthrumac> <action>pass</action> <mac>7c:bb:35:f2:a9:0e</mac> <descr><![CDATA[Serge Nouveau portable]]></descr> </passthrumac> .....
in the correct section, and import that file back in again.
Still, unlisted device you own can't pass as they are not part of the list.
I ended up listing all my devices (9) as blocked : they didn't pass.
I removed them from the MAC tab, so not listed as a pass or block : they still didn't pass.Btw : do not hesitate to reset firewall states. I don't know if it is really needed, but it would harm to reset they all the time, after changes.
My main question is : can you replicate your issue on a barebone system, after a manual minimal setup.
And if so, after which change your issue happens ?Also : export your
.... <passthrumac> <action>pass</action> <mac>xx:8d:79:91:ec:52</mac> <bw_up></bw_up> <bw_down></bw_down> <descr><![CDATA[Sophie]]></descr> </passthrumac> <passthrumac> <action>pass</action> <mac>bb:bb:35:f2:a9:0e</mac> <descr><![CDATA[Serge Nouveau portable]]></descr> </passthrumac> .....
section, and drop it in here.
Mistify all MAC's be replacing the first byte by placing 'bb', as I did above.I'll import your list.
I wonder if I see the issue then ...