[solved ] 2.3-release CP - crash report



  • Last week i've upgraded one of my production systems to 2.3
    it runs on esxi5.5.

    CP has around 200-350 logged in users on average during working hours.
    CP uses vouchers & AD2008r2 authentication.

    Every couple of days since the upgrade i get a notification of a crash report on dashboard. Luckily everything keeps working as normal.

    Can't find any clue's in system logs / portal auth log at the time of the crash report.

    
    					Crash report begins.  Anonymous machine information:
    
    amd64
    10.3-RELEASE
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    PHP Errors:
    [18-Apr-2016 08:43:25 CET] PHP Stack trace:
    [18-Apr-2016 08:43:25 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [18-Apr-2016 08:43:25 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [18-Apr-2016 08:43:25 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [18-Apr-2016 08:43:25 CET] PHP   4\. captiveportal_read_db() /etc/inc/captiveportal.inc:2093
    [18-Apr-2016 08:43:25 CET] PHP   5\. SQLite3->query() /etc/inc/captiveportal.inc:1516
    [18-Apr-2016 08:43:25 CET] PHP Stack trace:
    [18-Apr-2016 08:43:25 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [18-Apr-2016 08:43:25 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [18-Apr-2016 08:43:25 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [18-Apr-2016 08:43:25 CET] PHP   4\. portal_reply_page() /etc/inc/captiveportal.inc:2295
    [18-Apr-2016 08:43:25 CET] PHP   5\. header() /etc/inc/captiveportal.inc:1940
    [18-Apr-2016 10:47:19 CET] PHP Stack trace:
    [18-Apr-2016 10:47:19 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [18-Apr-2016 10:47:19 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [18-Apr-2016 10:47:19 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [18-Apr-2016 10:47:19 CET] PHP   4\. portal_reply_page() /etc/inc/captiveportal.inc:2295
    [18-Apr-2016 10:47:19 CET] PHP   5\. header() /etc/inc/captiveportal.inc:1940
    
    

    cp config snippet

    
        <captiveportal><cpzone><page><htmltext>........
                    <errtext>..............</errtext></htmltext></page> 
                <timeout><interface>opt6</interface>
                <maxproc><idletimeout>90</idletimeout>
                <freelogins_count><freelogins_resettimeout><enable><auth_method>radius</auth_method>
                <reauthenticateacct><httpsname>fw.mydomain.org</httpsname>
                <preauthurl><peruserbw><bwdefaultdn>9000</bwdefaultdn>
                <bwdefaultup>512</bwdefaultup>
                <redirurl><radiusip>172.16.1.20</radiusip>
                <radiusip2><radiusport><radiusport2><radiusacctport><radiuskey>*removed*</radiuskey>
                <radiuskey2><radiusvendor>default</radiusvendor>
                <radiussrcip_attribute>opt9</radiussrcip_attribute>
                <radmac_format>default</radmac_format>
                <radacct_enable><httpslogin><element><name>captiveportal-babologo300.jpg</name>
                    <size>71959</size>
                    ................
                <noconcurrentlogins><zoneid>2</zoneid>
                cpzone
                <radius_protocol>PAP</radius_protocol>
                <certref>123456789</certref>
                <blockedmacsurl><radiusip3><radiusip4><radiusport3><radiusport4><radiuskey3><radiuskey4><radiusnasid></radiusnasid></radiuskey4></radiuskey3></radiusport4></radiusport3></radiusip4></radiusip3></blockedmacsurl></noconcurrentlogins></element></httpslogin></radacct_enable></radiuskey2></radiusacctport></radiusport2></radiusport></radiusip2></redirurl></peruserbw></preauthurl></reauthenticateacct></enable></freelogins_resettimeout></freelogins_count></maxproc></timeout></cpzone></captiveportal> 
    
    

    only cp-related error i can find in system log

    
    Apr 15 10:17:39	php-fpm	69184	/index.php: Submission to captiveportal with unknown parameter zone:
    
    

    The timestamps do not match the crash report in any way

    So not a big issue, as everything works as intended … but probably something that should be investigated.



  • sent another cr today.

    
    					Crash report begins.  Anonymous machine information:
    
    amd64
    10.3-RELEASE
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    PHP Errors:
    [19-Apr-2016 08:27:51 CET] PHP Stack trace:
    [19-Apr-2016 08:27:51 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [19-Apr-2016 08:27:51 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [19-Apr-2016 08:27:51 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [19-Apr-2016 08:27:51 CET] PHP   4\. captiveportal_read_db() /etc/inc/captiveportal.inc:2093
    [19-Apr-2016 08:27:51 CET] PHP   5\. SQLite3->query() /etc/inc/captiveportal.inc:1516
    [19-Apr-2016 08:27:51 CET] PHP Stack trace:
    [19-Apr-2016 08:27:51 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [19-Apr-2016 08:27:51 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [19-Apr-2016 08:27:51 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [19-Apr-2016 08:27:51 CET] PHP   4\. portal_reply_page() /etc/inc/captiveportal.inc:2295
    [19-Apr-2016 08:27:51 CET] PHP   5\. header() /etc/inc/captiveportal.inc:1940
    [19-Apr-2016 09:04:12 CET] PHP Stack trace:
    [19-Apr-2016 09:04:12 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [19-Apr-2016 09:04:12 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [19-Apr-2016 09:04:12 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [19-Apr-2016 09:04:12 CET] PHP   4\. captiveportal_read_db() /etc/inc/captiveportal.inc:2093
    [19-Apr-2016 09:04:12 CET] PHP   5\. SQLite3->query() /etc/inc/captiveportal.inc:1516
    [19-Apr-2016 09:04:12 CET] PHP Stack trace:
    [19-Apr-2016 09:04:12 CET] PHP   1\. {main}() /usr/local/captiveportal/index.php:0
    [19-Apr-2016 09:04:12 CET] PHP   2\. radius() /usr/local/captiveportal/index.php:210
    [19-Apr-2016 09:04:12 CET] PHP   3\. portal_allow() /etc/inc/captiveportal.inc:1431
    [19-Apr-2016 09:04:12 CET] PHP   4\. portal_reply_page() /etc/inc/captiveportal.inc:2295
    [19-Apr-2016 09:04:12 CET] PHP   5\. header() /etc/inc/captiveportal.inc:1940
    
    

    starting to think it's a htmlspecialchars issue because:

    
    Apr 19 08:27:51 	logportalauth 	66147 	Zone: cpzone - USER LOGIN: Labcda.D'abcdo, e4:9a:79:e9:64:4b, 172.16.23.76 
    
    
    
    Apr 19 09:04:12 	logportalauth 	72911 	Zone: cpzone - USER LOGIN: Kabcda.Aabcd-L'Aefghi, 68:ae:20:9d:16:8a, 172.16.23.91 
    
    

    i've redacted a few "normal" characters in the usernames, because of privacy - they are students of a school.
    i'm guessing the < ' > are not properly escaped or something.  maybe adding the htmlspecialchars() at the correct location is sufficient to fix this issue.

    unfortunately i'm too stupid to know what the best approach is to fix it & as this is a production system, can't risk of breaking it.
    stacktrace indicates below as the starting point:
    https://github.com/pfsense/pfsense/blob/RELENG_2_3/src/usr/local/captiveportal/index.php#L210

    I could potentially see the query failing here: ( addslashes()  the $tmpusername perhaps ??)
    https://github.com/pfsense/pfsense/blob/RELENG_2_3/src/etc/inc/captiveportal.inc#L2089

    cant immediatly see what goes wrong here https://github.com/pfsense/pfsense/blob/RELENG_2_3/src/etc/inc/captiveportal.inc#L2295 or https://github.com/pfsense/pfsense/blob/RELENG_2_3/src/etc/inc/captiveportal.inc#L1940

    Also: should i create a redmine ticket for this or wait for confirmation by one of the coredevs ?



  • One of the problems will be at https://github.com/pfsense/pfsense/blob/master/src/etc/inc/captiveportal.inc#L2091
    where it is building an SQL query.

    $query .= " OR (username != 'unauthenticated' AND lower(username) = '{$tmpusername}')";
    

    There is not going to be any joy by directly inserting a single-quote at that point.



  • yea, just figured that out at the same time you posted (edited original post). thanks though  ;)

    oddly enough never had a crashreport for it in 2.1.x or 2.2.x



  • In SQL literal strings, a literal single-quote needs to be 2 single quotes in a row. So this should fix it:
    https://github.com/pfsense/pfsense/pull/2886

    If you can, make yourself a username with a single quote in it for testing, try using that through Captive Portal, verify the problem, then make the possible fix and test again.

    That was the only obvious place I saw where the username is used in constructing an SQL query. There might be other places where he username is used in other contexts and goes wrong when it contains a single quote.



  • yea my addslashes attempt crashed & burned, didn't know about the double-singlequote (thanks for pointing that out)

    thanks phil, that solves it. i've added your patch & hopefully it'll get merged into releng_2_3 & master



  • Even better solution in PR https://github.com/pfsense/pfsense/pull/2887 and verified to work by heper.



  • PR already merged, that was quick.
    Redmine issue https://redmine.pfsense.org/issues/6203 for tracking.


Log in to reply