Firewall log showing strange "pass" entry



  • Please forgive me if I make mistakes - I'm a new user on the forum.

    I have been happily using pfSense for about 1.5 years, and I think I understand how it generally works.

    Today as I am looking at the firewall logs, I noticed a strange entry that shows firewall rule as "pass"
    (which indicates firewall is "allowing" the traffic in).  Looking at more closely,  the 4294967295 seems "strange to me"
    I am wondering if there is some sort of "boundary condition" is a play.

    I tried to go to the pfSense IRC chat, but someone redirected me to ask this on the forum.

    Mar 25 14:42:05 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,10359,0,DF,6,tcp,36,105.212.87.78,[MYIPADDRESS],2123,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    This is the screenshot I took of the Firewall log page.    it's really weird because there is no such rule (@4294967295)

    Not sure if the attached picture took, but here is the screenshot I put on twitter.
    https://pbs.twimg.com/media/DZKQCQDU0AAR1g6.jpg

    Thanks!
    Vt44

    [Edit:  Please jump to Follow-up post #24 to see the Wireshark data]



  • Hmmm . . . Telnet (TCP/23) from Africa (105.212.87.78).

    How did you determine that there is no Rule 4294967295?



  • Perhaps I was incorrect in understanding how rules are numbered in pfsense, I think I do have about 230 rules total, including pfSense system default rules, and addition of user rules – and that may have led me to incorrect to assume there was no rule 4294967295 -- perhaps it could be found with the forum's assistance.

    In my past experience reading of the pfsense firewall logs, I had seen plenty of "match,block" on both (WAN/LAN) interfaces, but this was the first one I had seen that came from the WAN that said "short,pass" and I'm just trying to make sense of it and understand it better.

    Mar 25 14:42:05 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,10359,0,DF,6,tcp,36,105.212.87.78,[MYIPADDRESS],2123,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    I know pfsense firewall defaults to blocking all incoming firewall port scanniing request, and I do add one rule to block port 1-65535 on the WAN firewall IP (I know it is redundant) to just count how much background scanning traffic (packets) that are just always there.  I want to emphasize that I do not have any open ports exposed to the Internet, including the port 23  the offender was trying to access.

    I stand corrected on possibility of the existence the rule 4294967295, but I don't have user firewall rules that just says pass (external IP) to LAN ok because I know that's a no-no.

    – and I'm running 2.4.2-Release-p1 amd64 / FreeBSD 11.1-RELEASE-p6)  I think that's the latest version for everything


  • Netgate

    What does running this show:

    pfctl -vvsr | grep -A3 4294967295



  • I have rebooted the firewall after adding an alias for ASN block for AS16637 (102 IP ranges) and a corresponding floating blocking rule with that Alias.

    https://exchange.xforce.ibmcloud.com/search/AS16637

    A few hours ago, someone suggested I try :

    pfctl -vvsr | grep "short"

    and see what was matched, but nothing came of it.

    and I know you already know the response for the following

    pfctl -vvsr | grep -A3 4294967295

    was just empty response as well (since I rebooted).  But, I think I have learned something from this conversation, and try to record responses of

    pfctl -vvsr | grep -A3 RULE_NUMBER_AS_DISPLAYED_FROM_FIREWALL_LOG

    &

    pfctl -vvsr | grep RULE_NUMBER_AS_DISPLAYED_FROM_FIREWALL_LOG

    and have that handy if it ever happens again.

    Thank you for responding and for helping identifying additional steps I need to take to gather needed info to ask better questions.



  • … What do you know..

    Mar 25 21:02:12 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x28,,242,20338,0,DF,6,tcp,36,45.6.195.22,70.44.54.50,55118,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    and…

    [2.4.2-RELEASE][admin@pfSense.localdomain]/tmp: pfctl -vvsr | grep -A3 4294967295
    [2.4.2-RELEASE][admin@pfSense.localdomain]/tmp:



  • It just occurred to me that 4294967295 is one less than 232 (4294967296).

    I don't know what the significance of that might be but it is strange.  Perhaps an overflow of some sort.


  • Netgate

    Yeah I am not sure that is actually indicating passed traffic at all. I don't know that I have ever seen a log message with an error code like that.



  • Maybe the re0 interface (or whatever it's connected to) is misbehaving.

    Any I/O errors on Status > Interfaces?



  • Interesting theory..  I went and checked and it's been 23 hours since the reboot  – and the Status for WAN says"

    WAN Interface (wan, re0)
    Media    1000baseT <full-duplex,master>In/out packets    2461287/1308634 (2.76 GiB/106.79 MiB)
    In/out packets (pass)    2461287/1308634 (2.76 GiB/106.79 MiB)
    In/out packets (block)    22309/5486 (12.81 MiB/391 KiB)
    In/out errors    0/0
    Collisions    0

    LAN Interface is solid (re1) with no errors nor collisions

    Since the reboot, there has been 5 additional instances recorded in the firewall log

    Mar 26 08:25:40 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,28320,0,DF,6,tcp,36,109.73.184.113,[MYIPADDRESS],60964,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    Mar 26 11:19:30 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,241,37101,0,DF,6,tcp,36,168.228.165.29,[MYIPADDRESS],1454,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    Mar 26 12:33:37 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,243,19749,0,DF,6,tcp,36,88.220.191.225,[MYIPADDRESS],29755,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    Mar 26 15:07:07 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,2173,0,DF,6,tcp,36,109.73.184.235,[MYIPADDRESS],20600,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    Mar 26 16:23:28 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,54321,0,none,6,tcp,36,61.140.124.185,[MYIPADDRESS],45174,22,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    and, I had tried before to try to correlate but there is no "rule" found from:

    pfctl -vvsr | grep -A3 4294967295

    I do have and use Alias(es) for the purpose of ASN based blocking, for example,  Facebook ASN blocking

    whois -h whois.radb.net – '-i origin AS32934' | grep ^route | grep -v route6 | cut -d" " -f7 > ./fbblock.txt

    I primarily build aliases to block the "ASN of the "usual suspects of persistent hacking/probing" to cut down on the firewall log "noise", as I'm unlikely to have the need to visit networks of or websites located in ... say..  AS9808 or AS4837

    I don't know if there are other debugging options that can be turned on to track this, but all the monitoring indicates no "States" or connections are being successfully established by these incoming attempts, as checked by Diagnostics>States>States and filtered by IP of offender on "all" interface.

    Thanks for the responding with ideas to check, but as of now, I still have not found the culprit of the strange error message,  so I am proceeding cautiously.</full-duplex,master>



  • Good that there are no I/O errors but it's unlikely that your log entries reflect what's really happening.

    Maybe the only way to find out is to packet capture everything on the re0 interface until you see another of those log entries.

    If the packets are bad there probably isn't anything you could reliably use to filter the capture, so the output could be quite large.

    EDIT:  This is probably the source of this crap.  Three out of five of the IP addresses you mention above come back with a Mikrotik login prompt.



  • Of course, this doesn't solve the riddle of the rule number being 232-1 or the "pass" entry.

    I turned on logging for the default block rules and saw more than 100 attempts to connect to ports 21, 22, 23, 8291 in the space of two hours.  However, none of my logs had those strange entries shown in yours : oddball rule number, "short", "pass" or  "bad hdr length…"

    I'm still on 2.3.5  :-[  Perhaps there was a regression in log parsing after that.


  • Netgate

    I would pcap port 23 on WAN and see what wireshark thinks about it.



  • @Derelict:

    I would pcap port 23 on WAN and see what wireshark thinks about it.

    Yeah, that or 8291.

    I updated to 2.4.2_1 after my last post and there are still plenty of blocked 21, 22, 23, and 8291 connections.  I don't see those weird entries in my logs though, so it doesn't look like the packets are malformed.



  • Let's say I want to follow the advice and do some packet capture stuff, how would I do it?

    I will try to read up and see if I can set it up, but given the previous log entry, and the new ones below,
    I would appreciate some examples of what to use, either through the GUI or from the command line tailored to narrow down as much as possible.

    And , some quick updates.

    Mar 26 19:01:31 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,240,22619,0,DF,6,tcp,36,91.109.193.174,
    [MYIPADDRESS],54443,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    One of the previous "offender" was blocked with my new rule 58 (a new rule specifically created only to block everything coming from those previously identified "short,pass" IP addresses), I don't know if this yields any additional information or new information I can use to help packet capture beside to try to packet filter on port 23 or on port 8291.

    Mar 27 00:30:10 pfSense filterlog: 58,,,1522081663,re0,match,block,in,4,0x0,,239,43226,0,DF,6,tcp,40,109.73.179.128,[MYIPADDRESS],5233,23,0,S,781346768,,16060,,

    Mar 27 00:52:35 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,27500,0,DF,6,tcp,36,105.212.92.243,[MYIPADDRESS],24625,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    Until I receive replies and guidance, I'm off to read about how to do packet filtering and save the results to disk.



  • Diagnostics > Packet capture

    Install WireShark on your desktop to read the PCAP file that is generated/downloaded.

    Now I'm curious about why the log from your new rule is showing no strange stuff.


  • Rebel Alliance Global Moderator

    Your always going to see a shit ton of noise.. Part of the reason I turned off logging default rule and just put in my own rule to log SYN only… I don't need to see all the udp noise, etc.  Or anything that is out of state, etc.  But I do like to keep an eye on the top ports... Yeah 23, 22 all going to be heavy hitters..

    Do you have any odd ball forwards or floating rules?  There are things that are looked at before firewall rules, etc..

    https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

    If your really curious I would post up your full rule set.
    https://doc.pfsense.org/index.php/How_can_I_see_the_full_PF_ruleset

    Or if you not open to posting that public.. Sure there are a few of here that that can be trusted you could PM it too.. If you don't feel comfortable with what might be listed in it.. We could then clean it up with any sort of public IPs and repost it for the board to look over, etc.

    Part of the reason posting is want to get informed on updates to this thread.  Off the top I have nothing.. If your saying you searched your rules for that ID and not seeing it.. at a loss..

    How about atleast a screen shot of your wan, floating and forward rules.. To go along with your firewall rules if you don't feel like posting the full set..



  • There's nothing too exciting there I'm afraid.

    screenshots of floating rules and wan rules as example,  my floating rules goes a lot further below, mostly composed of ASN alias that you can compile checking the IBMcloud.com page doing query of the individual ASN and IP ranges (which I slightly massage, sort into an individual ASN Alias.    Rule description basically is "ASN#-Nation-B(lock)-Start_BLOCKDATE.

    the "shortpassblock" is currently de-activated so I can try to packet capture some data.  and before you ask about 2 copies of say FB rules,  I alternate them like two light switches (turn 2nd FB rule on, activate), then turn off 1st FB rule, then reload filter rules, to basically reset packet count.

    There is no port forwarding, only pfB DNSBL for malicious ad blocking etc.

    Last example shows a mouseover over an ASN alias, shows in the first IP description that shows the "Offender IP" that caused the ASN to be blocked.








  • Sorry, uploaded the wrong image.  Here is the Floating rule one.




  • [Edit]:  Please see follow-up post #24

    [edit] …  Learning how to packet capture deleted ...

    Below is the "new firewall log entry"

    Mar 28 01:11:32 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,25830,0,DF,6,tcp,36,168.228.167.124,[MYIPADDRESS],32068,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',



  • [Edit]  Removed trial packet capture info ..  Please see captured Wireshark info a few posts down.


  • Rebel Alliance Global Moderator

    Dude just download the packet capture from pfsense and open it with wireshark or post up the pcap file you download..  What you have posted is just gibberish not going to help anyone figure out anything.

    You can run packet capture without being logged in… just come back to it latter and stop it, etc.



  • Hi.  I am posting a small progress note.

    After some trial and errors, and some hit & misses, I think I have figured out a way to finally grabbing the packets to be examined, I will try to write them down so maybe if people have trouble like mine they will have something to refer to, so they can do the same.

    1.  I downloaded and installed latest stable version of Wireshark (for Win 7)

    2.  Learned how to copy a file remotely from pfSense box to local box, so I can move the packetcapture.cap to be examined by Wireshark.

    https://doc.pfsense.org/index.php/HOWTO:_Access_pfSense_filesystems_remotely_with_scp

    3.  Learning to use tcpdump to capture the packets from pfSense box.
            the most trouble for me initially was there was no predictability where the packet would be coming from, and I created a bunch of ssh windows each have:

    /usr/sbin/tcpdump -i re0 -p -c 100 -s 0 -w /tmp/105-212-92-243.cap ip and tcp and port 8291 and host 105.212.92.243

    using previous offender IPs, and I would see no traffic from it, but a new one would come in like:

    Mar 29 21:07:56 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,56,29244,0,none,6,tcp,36,155.133.83.54,[myIPAddress],40011,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    so, I finally settled on using:

    /usr/sbin/tcpdump -i re0 -p -s 0 -w /tmp/port8291packetcapture.cap ip and tcp and port 8291 or 23 or 22

    and was able to successful moved a copy of packet capture to examine on Wireshark.

    Now that I have cast my net correctly, now I just have to wait.

    With the example sample Wireshark screenshot,  Once I grab the appropriate packets, what do I need to share here and How would I do it?  I think I tried to upload the previous "packetcapture.cap" file but the forum would not accept it.

    if you look at the screengrab posted here, what would be the relevant information to post in the next followup, once I have the data, or how would I process the data and post follow up to it?

    [Edit]  Small correction editing out my IP address on attached picture.  The attached picture was to confirm that packets on Ports (22,23,8291) was being captured, and it seems to be working.  Now  I can wait for the pfSense filter log to show me another packet came in then I can correlate and post follow-up.



  • Netgate

    First thing to do is get a packet matched up with a log entry.



  • Got it !!!

    I have absolutely no idea what this means…  but here it is.  Please let me know if this helps.  I have the file on hand if someone needs it.

    Mar 29 23:40:23 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,231,17518,0,DF,6,tcp,36,185.51.112.191,[myIPADDRESS],13794,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',




  • WireShark also sees the packet as malformed and pfSense is reporting that: '[bad hdr length 20 - too long, > 16]'

    When logging this same sort of traffic, I haven't seen any malformed packets at all.  Given that you're not seeing any I/O errors on that interface, it's seems to me that something upstream is corrupting the packets.

    You are also getting that weird rule number and the "short,pass" reason and action.  That's probably just a result of the fact that the packet is corrupt but it would be good to know.

    The MAC address translation in the second WireShark window says that packet came from a Cisco (maybe Linksys) device.  Do you have something from one of those manufacturers in front of your pfSense box?  If so, it would be interesting to see whether the other poster with this "bad hdr" error has one of those as well.  Although, he wasn't hitting the "short,pass" or rule number problems.

    It really doesn't look like anything is getting in to your network, despite that "pass".

    It might be worth backing up your config, install a fresh copy of 2.4.3 and restoring the config to that.



  • I am feeling better today after I read your reply, because while I felt there was no unauthorized traffic coming from WAN, I could not see the underlying conditions nor the data causing the erroneous log entry, and the pfSense logger was causing me anxiety.  The mysterious "rule set" number which I knew didn't exist did not help in the matter, either.

    To address the "MAC address translation" question,  I generate a new random MAC address and use it to "spoof" in the WAN interface periodically to obtain a different IP address once in a while from my ISP.  My thinking is that if I got a new IP address,  it will make me a more elusive hacking target than if I stayed at one IP address 100% of the time for years on end.  I think that the "source" is the "real" MAC address of the WAN interface of my pfSense box.    Would this thinking be correct?

    I was thinking of updating to pfSense v2.4.3 yesterday, but,  I didn't know my issue was relating to these 2 topics listed in the new release notes:

    Added firewall rule tracking ID to rule list (in counter tooltip) and firewall rule edit page #8348
      Fixed cases where automatic or scripted rules were not getting tracking IDs #8353

    Now that I have more clarity, I went ahead and did a upgrade this morning (option #13 - upgrade from console), and I will keep an eye out on the pfSense logger.

    If the upgrade path doesn't seem to resolve the issue,  I may do a clean install in the future.  Meanwhile, I'll keep an eye out on the pfSense logger and try to continue to packet capture on those 3 ports.

    If anyone have more guidance based on reading the Wireshark data, please let me know.


  • Rebel Alliance Global Moderator

    "I generate a new random MAC address and use it to "spoof" in the WAN interface periodically to obtain a different IP address once in a while from my ISP."

    WTF???  Really???  Stop smoking whatever it is your smoking.. Nobody gives 2 shits what your IP is… Your not on an episode of Mr. Robot...

    As to pulling a sniff from pfsense - all you need to do is click the freaking download button for gosh sake...  Either open it directly or save it and open it later, etc..

    Why don't you actually post the capture?  Vs some screen shot?  If your worried about your ip you an always obfuscate that



  • Netgate

    Sounds like you're pretty clicky-clicky and have probably shot yourself in the foot somehow.



  • It seems that even though I had upgraded now to the latest version of pfSense software (2.4.3-RELEASE (amd64)),  the error (see below) persists.

    Sorry.  until 2 days ago, I had never used Wireshark, nor did I really needed to know scp,  packet capture technique, and forensic packet analysis.

    That said, in the spirit of trying to move forward to increase understanding – and hopefully increase security for all using pfSense,

    Mar 30 13:22:20 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x8,,240,49160,0,DF,6,tcp,36,105.212.95.78,24.115.69.86,31586,23,-4,S,errormsg='[bad hdr length 20 - too long,          > 16]',

    The two enclosed .pcap files (which I just learned to mark and export into .pcap) are attached.  I thought that by posting the screen grab late last night it would be enough for people to look and infer what the problem was.

    Nevertheless, here are the two files you asked for.    The "marked-033018.pcap" goes with the filterlog in this message;  and "329packetcapture.pcap" was the one mentioned in follow-up reply #24.

    I look forward to reading your analysis.

    Enjoy.
    [edit] fixed small spelling error.



    marked-033018.pcap
    329packetcapture.pcap



  • @biggsy:

    Good that there are no I/O errors but it's unlikely that your log entries reflect what's really happening.

    Maybe the only way to find out is to packet capture everything on the re0 interface until you see another of those log entries.

    If the packets are bad there probably isn't anything you could reliably use to filter the capture, so the output could be quite large.

    EDIT:  This is probably the source of this crap.  Three out of five of the IP addresses you mention above come back with a Mikrotik login prompt.

    Very interesting read with the Mikrotik you found biggsy.
    Makes a lot of sense too. Capture file has Iran, Brazil, Cambodia, Albania, China, Sweden, Korea, Japan, Turkey, South Africa, wew!
    Ports 22, 23, 8291. All trying to start handshake sending SYN packets within minutes. I did not see any SYN ACK going out so I guess PfSense is just humming along in the storm, 8)
    vt44 did you read it. You may have missed it in the rush. By any chance you running a mikrotik that let out a call for a bot party at your place. Probably not but I would get rid of the redundant blocking and simplify things. Looks confusing to me.
    Just a thought.



  • I'm running a ZOTAC ZBOX nano CI323 with Intel N3150 chip with only pfSense software, so I'm definitely not using Mikrotik.    It looks like the Mikrotik party has moved to Poland & Ukaine and added port 2000, too?  I did not have these two packets (nor port 2000) in particular captured, but will modify my filter if to grab the future data if anyone is interested in seeing the pcap file.

    Mar 30 23:09:59 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,244,33870,0,DF,6,tcp,36,176.110.150.46,[myipaddress],26210,2000,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
    Mar 31 05:05:55 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,243,59793,0,DF,6,tcp,36,195.214.197.90,[myipaddress],32909,2000,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',

    –-

    On the topic of routers,  I was actually using Google OnHub in 2016 before before I decided to "roll my own router" after reading the Ars Technica's article about it, and the overwhelming reason for the change was not because the OnHub was not a good router, but I wanted more transparency.  The Onhub was changed to bridge mode and is working as a dedicated wireless AP, but still is a candidate as my back-up router should it becomes necessary.


    Was Mr Derelict addressing this sentence to me?  If he could clarify what he was asking I would appreciate it.

    "Sounds like you're pretty clicky-clicky and have probably shot yourself in the foot somehow."

    I did want to mention that while the pfSense GUI packet capture will allow multiple IP addresses  to be specified, it does not allow for the capture of multiple ports at the same time (I tried the "22 23") -- and that was why at the end I could not use "clicky-click" interface to do packet capture and had to rely on "tcpdump" from the command line  in the end.  And, since there was no "click to download capture", I had to figure out to learn use "scp" to moves the packet capture files over.


  • Rebel Alliance Global Moderator

    Pretty much everyone on the net is seeing traffic from these bots..  But you seem to be the only one with the odd pass rule.  Most likely due to some oddball setup you have done.

    I am seeing hundreds of blocks to 8291 port.. And one to 23..

    Here is one with the odd headers

    Mar 29 17:04:02 filterlog: 131,,,1512833215,igb1,match,block,in,4,0x0,,229,24172,0,DF,6,tcp,40,105.212.82.37,64.53.xx.xx,30634,8291,-4,S,errormsg='[bad hdr length 24 - too long, > 20]',

    Mar 30 20:24:02 filterlog: 131,,,1512833215,igb1,match,block,in,4,0x0,,238,58316,0,DF,6,tcp,40,198.162.199.56,64.53.xx.xx,2514,23,-4,S,errormsg='[bad hdr length 24 - too long, > 20]',

    Why don't you actually post your full rules so we can see what your doing that is odd..

    https://doc.pfsense.org/index.php/How_can_I_see_the_full_PF_ruleset



  • Sanitized file from command : pfctl -sa

    fullruleset.txt.txt