Firewall log showing strange "pass" entry
-
I have rebooted the firewall after adding an alias for ASN block for AS16637 (102 IP ranges) and a corresponding floating blocking rule with that Alias.
https://exchange.xforce.ibmcloud.com/search/AS16637
A few hours ago, someone suggested I try :
pfctl -vvsr | grep "short"
and see what was matched, but nothing came of it.
and I know you already know the response for the following
pfctl -vvsr | grep -A3 4294967295
was just empty response as well (since I rebooted). But, I think I have learned something from this conversation, and try to record responses of
pfctl -vvsr | grep -A3 RULE_NUMBER_AS_DISPLAYED_FROM_FIREWALL_LOG
&
pfctl -vvsr | grep RULE_NUMBER_AS_DISPLAYED_FROM_FIREWALL_LOG
and have that handy if it ever happens again.
Thank you for responding and for helping identifying additional steps I need to take to gather needed info to ask better questions.
-
… What do you know..
Mar 25 21:02:12 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x28,,242,20338,0,DF,6,tcp,36,45.6.195.22,70.44.54.50,55118,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
and…
[2.4.2-RELEASE][admin@pfSense.localdomain]/tmp: pfctl -vvsr | grep -A3 4294967295
[2.4.2-RELEASE][admin@pfSense.localdomain]/tmp: -
It just occurred to me that 4294967295 is one less than 232 (4294967296).
I don't know what the significance of that might be but it is strange. Perhaps an overflow of some sort.
-
Yeah I am not sure that is actually indicating passed traffic at all. I don't know that I have ever seen a log message with an error code like that.
-
Maybe the re0 interface (or whatever it's connected to) is misbehaving.
Any I/O errors on Status > Interfaces?
-
Interesting theory.. I went and checked and it's been 23 hours since the reboot – and the Status for WAN says"
WAN Interface (wan, re0)
Media 1000baseT <full-duplex,master>In/out packets 2461287/1308634 (2.76 GiB/106.79 MiB)
In/out packets (pass) 2461287/1308634 (2.76 GiB/106.79 MiB)
In/out packets (block) 22309/5486 (12.81 MiB/391 KiB)
In/out errors 0/0
Collisions 0LAN Interface is solid (re1) with no errors nor collisions
Since the reboot, there has been 5 additional instances recorded in the firewall log
Mar 26 08:25:40 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,28320,0,DF,6,tcp,36,109.73.184.113,[MYIPADDRESS],60964,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 11:19:30 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,241,37101,0,DF,6,tcp,36,168.228.165.29,[MYIPADDRESS],1454,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 12:33:37 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,243,19749,0,DF,6,tcp,36,88.220.191.225,[MYIPADDRESS],29755,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 15:07:07 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,2173,0,DF,6,tcp,36,109.73.184.235,[MYIPADDRESS],20600,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 16:23:28 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,54321,0,none,6,tcp,36,61.140.124.185,[MYIPADDRESS],45174,22,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
and, I had tried before to try to correlate but there is no "rule" found from:
pfctl -vvsr | grep -A3 4294967295
I do have and use Alias(es) for the purpose of ASN based blocking, for example, Facebook ASN blocking
whois -h whois.radb.net – '-i origin AS32934' | grep ^route | grep -v route6 | cut -d" " -f7 > ./fbblock.txt
I primarily build aliases to block the "ASN of the "usual suspects of persistent hacking/probing" to cut down on the firewall log "noise", as I'm unlikely to have the need to visit networks of or websites located in ... say.. AS9808 or AS4837
I don't know if there are other debugging options that can be turned on to track this, but all the monitoring indicates no "States" or connections are being successfully established by these incoming attempts, as checked by Diagnostics>States>States and filtered by IP of offender on "all" interface.
Thanks for the responding with ideas to check, but as of now, I still have not found the culprit of the strange error message, so I am proceeding cautiously.</full-duplex,master>
-
Good that there are no I/O errors but it's unlikely that your log entries reflect what's really happening.
Maybe the only way to find out is to packet capture everything on the re0 interface until you see another of those log entries.
If the packets are bad there probably isn't anything you could reliably use to filter the capture, so the output could be quite large.
EDIT: This is probably the source of this crap. Three out of five of the IP addresses you mention above come back with a Mikrotik login prompt.
-
Of course, this doesn't solve the riddle of the rule number being 232-1 or the "pass" entry.
I turned on logging for the default block rules and saw more than 100 attempts to connect to ports 21, 22, 23, 8291 in the space of two hours. However, none of my logs had those strange entries shown in yours : oddball rule number, "short", "pass" or "bad hdr length…"
I'm still on 2.3.5 :-[ Perhaps there was a regression in log parsing after that.
-
I would pcap port 23 on WAN and see what wireshark thinks about it.
-
I would pcap port 23 on WAN and see what wireshark thinks about it.
Yeah, that or 8291.
I updated to 2.4.2_1 after my last post and there are still plenty of blocked 21, 22, 23, and 8291 connections. I don't see those weird entries in my logs though, so it doesn't look like the packets are malformed.
-
Let's say I want to follow the advice and do some packet capture stuff, how would I do it?
I will try to read up and see if I can set it up, but given the previous log entry, and the new ones below,
I would appreciate some examples of what to use, either through the GUI or from the command line tailored to narrow down as much as possible.And , some quick updates.
Mar 26 19:01:31 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,240,22619,0,DF,6,tcp,36,91.109.193.174,
[MYIPADDRESS],54443,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',One of the previous "offender" was blocked with my new rule 58 (a new rule specifically created only to block everything coming from those previously identified "short,pass" IP addresses), I don't know if this yields any additional information or new information I can use to help packet capture beside to try to packet filter on port 23 or on port 8291.
Mar 27 00:30:10 pfSense filterlog: 58,,,1522081663,re0,match,block,in,4,0x0,,239,43226,0,DF,6,tcp,40,109.73.179.128,[MYIPADDRESS],5233,23,0,S,781346768,,16060,,
Mar 27 00:52:35 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,27500,0,DF,6,tcp,36,105.212.92.243,[MYIPADDRESS],24625,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Until I receive replies and guidance, I'm off to read about how to do packet filtering and save the results to disk.
-
Diagnostics > Packet capture
Install WireShark on your desktop to read the PCAP file that is generated/downloaded.
Now I'm curious about why the log from your new rule is showing no strange stuff.
-
Your always going to see a shit ton of noise.. Part of the reason I turned off logging default rule and just put in my own rule to log SYN only… I don't need to see all the udp noise, etc. Or anything that is out of state, etc. But I do like to keep an eye on the top ports... Yeah 23, 22 all going to be heavy hitters..
Do you have any odd ball forwards or floating rules? There are things that are looked at before firewall rules, etc..
https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order
If your really curious I would post up your full rule set.
https://doc.pfsense.org/index.php/How_can_I_see_the_full_PF_rulesetOr if you not open to posting that public.. Sure there are a few of here that that can be trusted you could PM it too.. If you don't feel comfortable with what might be listed in it.. We could then clean it up with any sort of public IPs and repost it for the board to look over, etc.
Part of the reason posting is want to get informed on updates to this thread. Off the top I have nothing.. If your saying you searched your rules for that ID and not seeing it.. at a loss..
How about atleast a screen shot of your wan, floating and forward rules.. To go along with your firewall rules if you don't feel like posting the full set..
-
There's nothing too exciting there I'm afraid.
screenshots of floating rules and wan rules as example, my floating rules goes a lot further below, mostly composed of ASN alias that you can compile checking the IBMcloud.com page doing query of the individual ASN and IP ranges (which I slightly massage, sort into an individual ASN Alias. Rule description basically is "ASN#-Nation-B(lock)-Start_BLOCKDATE.
the "shortpassblock" is currently de-activated so I can try to packet capture some data. and before you ask about 2 copies of say FB rules, I alternate them like two light switches (turn 2nd FB rule on, activate), then turn off 1st FB rule, then reload filter rules, to basically reset packet count.
There is no port forwarding, only pfB DNSBL for malicious ad blocking etc.
Last example shows a mouseover over an ASN alias, shows in the first IP description that shows the "Offender IP" that caused the ASN to be blocked.
-
Sorry, uploaded the wrong image. Here is the Floating rule one.
-
[Edit]: Please see follow-up post #24
[edit] … Learning how to packet capture deleted ...
Below is the "new firewall log entry"
Mar 28 01:11:32 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,25830,0,DF,6,tcp,36,168.228.167.124,[MYIPADDRESS],32068,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
-
[Edit] Removed trial packet capture info .. Please see captured Wireshark info a few posts down.
-
Dude just download the packet capture from pfsense and open it with wireshark or post up the pcap file you download.. What you have posted is just gibberish not going to help anyone figure out anything.
You can run packet capture without being logged in… just come back to it latter and stop it, etc.
-
Hi. I am posting a small progress note.
After some trial and errors, and some hit & misses, I think I have figured out a way to finally grabbing the packets to be examined, I will try to write them down so maybe if people have trouble like mine they will have something to refer to, so they can do the same.
1. I downloaded and installed latest stable version of Wireshark (for Win 7)
2. Learned how to copy a file remotely from pfSense box to local box, so I can move the packetcapture.cap to be examined by Wireshark.
https://doc.pfsense.org/index.php/HOWTO:_Access_pfSense_filesystems_remotely_with_scp
3. Learning to use tcpdump to capture the packets from pfSense box.
the most trouble for me initially was there was no predictability where the packet would be coming from, and I created a bunch of ssh windows each have:/usr/sbin/tcpdump -i re0 -p -c 100 -s 0 -w /tmp/105-212-92-243.cap ip and tcp and port 8291 and host 105.212.92.243
using previous offender IPs, and I would see no traffic from it, but a new one would come in like:
Mar 29 21:07:56 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,56,29244,0,none,6,tcp,36,155.133.83.54,[myIPAddress],40011,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
so, I finally settled on using:
/usr/sbin/tcpdump -i re0 -p -s 0 -w /tmp/port8291packetcapture.cap ip and tcp and port 8291 or 23 or 22
and was able to successful moved a copy of packet capture to examine on Wireshark.
Now that I have cast my net correctly, now I just have to wait.
With the example sample Wireshark screenshot, Once I grab the appropriate packets, what do I need to share here and How would I do it? I think I tried to upload the previous "packetcapture.cap" file but the forum would not accept it.
if you look at the screengrab posted here, what would be the relevant information to post in the next followup, once I have the data, or how would I process the data and post follow up to it?
[Edit] Small correction editing out my IP address on attached picture. The attached picture was to confirm that packets on Ports (22,23,8291) was being captured, and it seems to be working. Now I can wait for the pfSense filter log to show me another packet came in then I can correlate and post follow-up.
-
First thing to do is get a packet matched up with a log entry.
