Dashboard and 1.2.1



  • Some weeks ago I posted some lines about the problem of the Dashborad on a pfSense 1.2.1, that doesn't show correctly the logs.

    Someone posted the solution…

    But all the 1.2.1 section of the forum was deleted and with it the solution...

    Could someone repost it?

    Thank you!


  • Rebel Alliance Developer Netgate

    Emab,

    I noticed that the board was deleted, too! It might have been better if they'd left that forum available, but locked, so that people could search the messages.

    The short version of the fix was to change "length:" to "length\s" in /usr/local/www/widgets/include/log.inc, and the alternative was to use a copy of log.inc that I altered with more logic taken from the firewall log page. I've attached it below, as log.inc.txt but it should be log.inc. It still has some issues, but it works for me.

    If you need more details, let me know.

    P.S. Be aware that I also changed the attached log.inc to display at most 10 messages (less if there are skipped lines)

    log.inc.txt



  • Thanks, exactly what I was looking for!



  • Thank you very much!!


  • Rebel Alliance Developer Netgate

    Here is another alternate version of log.inc that could use some testing. I reworked it quite a bit, changing the regex around and going about things in a slightly different way.

    If it continues to test well, perhaps I'll submit it as a patch.

    log.inc.txt



  • @jimp:

    Here is another alternate version of log.inc that could use some testing. I reworked it quite a bit, changing the regex around and going about things in a slightly different way.

    If it continues to test well, perhaps I'll submit it as a patch.

    I'll test!

    What are the impovements?


  • Rebel Alliance Developer Netgate

    @Emab:

    I'll test!

    What are the impovements?

    Nothing major, it's mainly "behind the scenes" stuff. Most importantly, it should be able to properly interpret a wider array of log entries. The old method skipped quite a few entries, and misinterpreted others. It's also a little cleaner, obtaining more information from one regex than the old method obtained from two separate ones. It also doesn't guess at the protocol as the old method did, instead it attempts to use the protocol text found in the entry.

    If it works, I'll also need to look at the actual system status firewall logs page to see if it improves upon what is used there as well.



  • Great work!


  • Rebel Alliance Developer Netgate

    I found a few more things that need changing to fix some other Firewall Log dashboard functionality that has also become non-functional in 1.2.x.

    1. Adjust /usr/local/www/diag_logs_filter_dynamic.php in the same way as log.inc.  The same regex needs the same changes. This should fix the AJAX rule updating. (It still seems to skip some log entries but at least it does update)

    2. Adjust /usr/local/www/widgets/widgets/log.widget.php, change line 69 to this:

     [![](<?=$img;?>)](#) 
    

    That will fix being able to click the action image to determine the rule number that triggered the pass/block.

    A similar change needs to be made to /usr/local/www/diag_logs_filter_dynamic.php so that it will continue to work on the AJAX-updated rules: Replace the "if" statement at line 446 with this:

    			if($row_time > $lastsawtime) {
    			  $img = '[' . $img . "](#)";
    				$new_rules .= "{$img}||{$log_row['time']}||{$log_row['interface']}||{$log_row['src']}||{$log_row['dst']}||{$log_row['proto']}||" . time() . "||\n";
    			}
    

  • Rebel Alliance Developer Netgate

    Ok, this should make things easier.

    I rolled all of my changes into a new dashboard package now available to install!:

    • Fixes incorrect log display in System Logs if set to view forward (It did not show the right set of log entries). Reverse view worked fine.
    • Fixes dashboard log display
    • Fixes AJAX updating of dashboard log entries (works with forward and reverse)
    • Fixes clicking the red X / green arrow to obtain rule number and description
    • If you act now, as a special bonus, I'm throwing in my IPSec status fix :-)

    (patches removed since the updates are now in a new dashboard package)


  • Rebel Alliance Developer Netgate

    One bug with the AJAX support… It seems to only work properly if you view the logs in reverse order. I didn't notice it at first because all of my servers are set that way.  :)

    Still tinkering...


  • Rebel Alliance Developer Netgate

    Forgot something…

    After running the patch, you should move/delete the old files. Having the ".orig" files in the widgets/include directory will cause some problems.

    rm /usr/local/www/widgets/include/ipsec.inc.orig
    rm /usr/local/www/widgets/include/log.inc.orig
    rm /usr/local/www/widgets/widgets/log.widget.php.orig
    rm /usr/local/www/diag_logs_filter_dynamic.php.orig
    


  • Broke my dashboard. I get all this code accross the top of the page…..

    Though my Firewall Logs seem to be showing up now, further down the page after all the text, but it's working

    *************** *** 4,10 **** function get_ipsec_tunnel_sad() { /* query SAD */ - $fd = @popen("/sbin/setkey -D", "r"); $sad = array(); if ($fd) { while (!feof($fd)) { --- 4,10 ---- function get_ipsec_tunnel_sad() { /* query SAD */ + $fd = @popen("/usr/local/sbin/setkey -D", "r"); $sad = array(); if ($fd) { while (!feof($fd)) { *************** *** 14,23 **** /* format filter logs */ - function conv_clog_filter($logfile, $tail = 8) { - global $config, $nentries, $logfile; - - $logfile = "/var/log/filter.log"; /* make interface/port table */ $iftable = array(); --- 14,21 ---- /* format filter logs */ + function conv_clog_filter($logfile, $tail = 13) { + global $config, $nentries; /* make interface/port table */ $iftable = array(); *************** *** 42,104 **** $log_split = ""; - //old reg ex - //preg_match("/(.*)\s.*\spf:\s.*\srule\s(.*)\(match\)\:\s(.*)\s\w+\son\s(\w+)\:\s(.*)\s>\s(.*)\:\s.*/", $logent, $log_split); - - preg_match("/(.*)\s.*\spf:\s.*\srule\s(.*)\(match\)\:\s(.*)\s\w+\son\s(\w+)\:\s.*\slength\:.*\s(.*)\s>\s(.*)\:\s.*/", $logent, $log_split); - $logent = strtoupper($logent); $do_not_display = false; - if(stristr(strtoupper($logent), "UDP") == true) - $flent['proto'] = "UDP"; - else if(stristr(strtoupper($logent), "TCP") == true) - $flent['proto'] = "TCP"; - else if(stristr(strtoupper($logent), "ICMP") == true) - $flent['proto'] = "ICMP"; - else if(stristr(strtoupper($logent), "HSRP") == true) - $flent['proto'] = "HSRP"; - else if(stristr(strtoupper($logent), "ESP") == true) - $flent['proto'] = "ESP"; - else if(stristr(strtoupper($logent), "AH") == true) - $flent['proto'] = "AH"; - else if(stristr(strtoupper($logent), "GRE") == true) - $flent['proto'] = "GRE"; - else if(stristr(strtoupper($logent), "IGMP") == true) - $flent['proto'] = "IGMP"; - else if(stristr(strtoupper($logent), "CARP") == true) - $flent['proto'] = "CARP"; - else if(stristr(strtoupper($logent), "PFSYNC") == true) - $flent['proto'] = "PFSYNC"; - else - $flent['proto'] = "TCP"; - - $time_regex = ""; - preg_match("/.*([0-9][0-9]:[0-9][0-9]:[0-9][0-9])/", $log_split[1], $time_regex); - $row_time = strtotime($time_regex[1]); - $flent['time'] = $row_time; - $flent['act'] = $log_split[3]; - $friendly_int = convert_real_interface_to_friendly_interface_name($log_split[4]); $flent['interface'] = strtoupper($friendly_int); if($config['interfaces'][$friendly_int]['descr'] <> "") $flent['interface'] = "{$config['interfaces'][$friendly_int]['descr']}"; - $flent['src'] = convert_port_period_to_colon($log_split[5]); - $flent['dst'] = convert_port_period_to_colon($log_split[6]); - $flent['dst'] = str_replace(": NBT UDP PACKET(137)", "", $flent['dst']); - $tmp = split("/", $log_split[2]); - $flent['rulenum'] = $tmp[0]; $counter++; $filterlog[] = $flent; } --- 40,92 ---- $log_split = ""; + preg_match("/(.*)\s(.*)\spf:\s.*\srule\s(.*)\(match\)\:\s(.*)\s\w+\son\s(\w+)\:\s\((.*)\)\s(.*)\s>\s(.*)\:\s.*/", $logent, $log_split); + list($all, $ltime, $host, $rule, $action, $int, $details, $src, $dst) = $log_split; + $flent['src'] = convert_port_period_to_colon($src); + $flent['dst'] = convert_port_period_to_colon($dst); $do_not_display = false; + $proto = array(" ", "TCP"); + if (!(strpos($details, 'proto ') === FALSE)) { + preg_match("/.*\sproto\s(.*)\s\(/", $details, $proto); + } elseif (!(strpos($details, 'proto: ') === FALSE)) { + preg_match("/.*\sproto\:(.*)\s\(/", $details, $proto); + } elseif (!(strpos($details, 'next-header ') === FALSE)) { + preg_match("/.*\snext-header\s(.*)\s\(/", $details, $proto); + } + $flent['proto'] = $proto[1]; + $flent['time'] = $ltime; + $flent['act'] = $action; + $friendly_int = convert_real_interface_to_friendly_interface_name($int); $flent['interface'] = strtoupper($friendly_int); if($config['interfaces'][$friendly_int]['descr'] <> "") $flent['interface'] = "{$config['interfaces'][$friendly_int]['descr']}"; + $tmp = split("/", $rule); + $flent['rulenum'] = $tmp[0]; + $shouldadd = true; + if(trim($flent['src']) == "") + $shouldadd = false; + if(trim($flent['dst']) == "") + $shouldadd = false; + if(trim($flent['time']) == "") + $shouldadd = false; + if($shouldadd == true) { $counter++; $filterlog[] = $flent; + } + //$counter++; + //$filterlog[] = $flent; } *************** *** 159,167 **** else $img = ""; //echo "{$time_regex[1]} - $row_time > $lastsawtime
    
    "; - if($row_time > $lastsawtime) $new_rules .= "{$img}||{$log_row['time']}||{$log_row['interface']}||{$log_row['src']}||{$log_row['dst']}||{$log_row['proto']}||" . time() . "||\n"; } echo $new_rules; exit; } --- 147,157 ---- else $img = ""; //echo "{$time_regex[1]} - $row_time > $lastsawtime
    
    "; + if($row_time > $lastsawtime) { + $img = '' . $img . ""; $new_rules .= "{$img}||{$log_row['time']}||{$log_row['interface']}||{$log_row['src']}||{$log_row['dst']}||{$log_row['proto']}||" . time() . "||\n"; } + } echo $new_rules; exit; } 
    

  • Rebel Alliance Developer Netgate

    That looks almost like the contents of the patch file… not sure why it would be showing up there. Where did you put the patch file?

    Also - What version of pfSense are you using?

    And do you still happen to have the output from patch when you ran it?

    You should be able to reverse the patch with "patch --reverse < dashboardpatch.txt"



  • lol, put the file in /usr/local and ran patch <dashboardpatch.txt<br>This time when I ran the patch –reverse <dashboardpatch.txt it="" doubled="" the="" amount="" of="" code="" on="" screen="" when="" i="" go="" to="" view="" dashboard<br="">I'm running 1.2.2 and when I ran the reverse I got

    # patch --reverse <dashboardpatch.txt<br>Hmm...  Looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |diff -rub www-orig/diag_logs_filter_dynamic.php www/diag_logs_filter_dynamic.php
    |--- www-orig/diag_logs_filter_dynamic.php      2007-07-04 16:14:26.000000000 -0400
    |+++ www/diag_logs_filter_dynamic.php   2009-02-14 10:23:00.000000000 -0500
    --------------------------
    Patching file www/diag_logs_filter_dynamic.php using Plan A...
    Hunk #1 succeeded at 76.
    Hunk #2 succeeded at 444.
    Hmm...  The next patch looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |diff -rub www-orig/widgets/include/ipsec.inc www/widgets/include/ipsec.inc
    |--- www-orig/widgets/include/ipsec.inc 2007-10-05 04:40:00.000000000 -0400
    |+++ www/widgets/include/ipsec.inc      2009-02-08 21:03:25.000000000 -0500
    --------------------------
    Patching file www/widgets/include/ipsec.inc using Plan A...
    Hunk #1 failed at 4.
    1 out of 1 hunks failed--saving rejects to www/widgets/include/ipsec.inc.rej
    Hmm...  The next patch looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |diff -rub www-orig/widgets/include/log.inc www/widgets/include/log.inc
    |--- www-orig/widgets/include/log.inc   2007-07-15 13:24:00.000000000 -0400
    |+++ www/widgets/include/log.inc        2009-02-14 09:41:12.000000000 -0500
    --------------------------
    Patching file www/widgets/include/log.inc using Plan A...
    Hunk #1 failed at 14.
    Hunk #2 failed at 42.
    Hunk #3 failed at 159.
    3 out of 3 hunks failed--saving rejects to www/widgets/include/log.inc.rej
    Hmm...  The next patch looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |diff -rub www-orig/widgets/widgets/log.widget.php www/widgets/widgets/log.widget.php
    |--- www-orig/widgets/widgets/log.widget.php    2007-07-02 22:56:42.000000000 -0400
    |+++ www/widgets/widgets/log.widget.php 2009-02-13 20:37:48.000000000 -0500
    --------------------------
    Patching file www/widgets/widgets/log.widget.php using Plan A...
    Hunk #1 failed at 66.
    1 out of 1 hunks failed--saving rejects to www/widgets/widgets/log.widget.php.rej
    done</dashboardpatch.txt<br>
    ```</dashboardpatch.txt></dashboardpatch.txt<br>

  • Rebel Alliance Developer Netgate

    Ah… so it didn't apply cleanly. I wonder if they put out a new version of the dashboard package since I made the patch.

    What version of the dashboard package does it say you have installed?

    I tested and applied the patch against a 1.2.2 system before I posted the patch here...

    This might help clean stuff up:

    find /usr/local -name "*.rej" -delete
    


  • 0.6.2

    I ran that command and received

    Fatal error: Cannot redeclare get_ipsec_tunnel_sad() (previously declared in /usr/local/www/widgets/include/ipsec.inc:7) in /usr/local/www/widgets/include/ipsec.inc.orig on line 5 
    

    And now I have completely lost my dashboard. Can't reinstall the package or the gui to fix it. It'll reinstall, but I get the same error.


  • Rebel Alliance Developer Netgate

    It still has the ".orig" files sitting there, which need to be deleted (see my earlier post)

    find /usr/local -name "*.orig" -delete
    

    I'm working on a better way to test this out…



  • I found the problem…. ID10T error!!! I didn't remove the package then run find /usr/local -name "*.orig" -delete then install the package again. Thanks bud. Please let us know when you find a fix for that, maybe it's fix in 1.2.3 or 2.0

    Edit: LOL, I just noticed that my firewall logs box is working after all this :-P  Nice, Thanks!!

    Solved?



  • Strange, so I just logged into another remote PfSense box with the same issue. I applied the patch and got the same code on the page and "find /usr/local -name ".orig" -delete"
    didn't solve the problem of the code on the page but "find /usr/local -name "
    .rej" -delete" did and fixed the problem of the logs not working. It appears your dashboardpatch worked, the boxes just needed some cleanup without uninstalling and reinstalling the dashboard. I hope this helps you correct what ever it is in the script.


  • Rebel Alliance Developer Netgate

    It's probably a couple files that don't match up quite right with the most current dashboard package.

    I'll have to adjust the patch tomorrow when I'm at work with handy access to a fresh test box.


  • Rebel Alliance Developer Netgate

    Actually it looks like the patch is fine, but the formatting is off in the text file somehow when I upload it.

    I'll have to investigate that more later/tomorrow.

    Edit: updated earlier post with a better patch and removed the attachment from this one.



  • LOL, well it's all good now, no reason to patch the patched dashboard  ;)

    But if you'd like me to help you with something then let me know.


  • Rebel Alliance Developer Netgate

    I suspect it has something to do with unix/dos line termination. That, or it's losing the proper tabs/spacing when I upload it.

    I need to do some more testing, but I've also added additional functionality to the patch.

    If you have another install to test on, give each of these a try with "patch -C < patchfile.txt" that way it will not modify anything - just test the patch run.

    (attachments removed since the updates are now in a new dashboard package)


  • Rebel Alliance Developer Netgate

    Weird. Neither one of those worked on my test box.

    This works, though:
    (removed since the updates are now in a new dashboard package)



  • Huh? Did you remove your patch document? I don't see it posted anymore. I was going to run it to see what happens  ;)


  • Rebel Alliance Developer Netgate

    Yeah I removed the attachments as they were broken in some way. I don't know if it was the line termination, spacing, what, but they were mangled when I downloaded them.

    You can fetch it straight from my web server and it works that way:
    (removed since the updates are now in a new dashboard package)


  • Rebel Alliance Developer Netgate

    Just a little update: I've gotten clearance to fix the dashboard, but my current patches need a little more work/testing before I tar up a new version of the package.

    If anyone has seen anything else that doesn't work quite right, let me know.



  • Well, I dunno if it's a firefox issue or a dashboard issue but I know if i leave the dashboard open on my web browser for an extended period of time, say 4 hours plus then firefox starts using up CPU resources like crazy!!! That is until I refresh the page, move to another page or close the page.

    If you want to take that on that would be cool but unnecessary I suppose :-P


  • Rebel Alliance Developer Netgate

    I've had that happen to me as well.

    I'm not sure if that is happening due to the AJAX log view or the traffic graphs, but I think it might be the traffic graphs as it has happened to be since before I fixed the AJAX part of the log widget.

    I haven't tried this yet, but another test to see if it is the AJAX viewer would be to look at the 'dynamic' view of the filter logs. That uses much of the same code as the widget. If that also locks up after several hours, it's probably still in that code.

    I know that code had other issues before, but I think I got most of them. I know there are more, but I didn't think they caused long-term problems, just problems with large/fast updates.


  • Rebel Alliance Developer Netgate

    I left the main dashboard screen running overnight on my laptop and it was still working and responsive this morning. A little slower than it was right after a refresh, but not locked up like it had done previously.

    I'm not sure if it's changes that are up in the current package, or updates I have done since, that have improved it.

    I'm hoping to have yet another version of the dashboard out with a couple new widgets by the end of the weekend.


Log in to reply