Firewall log showing strange "pass" entry
-
Yeah I am not sure that is actually indicating passed traffic at all. I don't know that I have ever seen a log message with an error code like that.
-
Maybe the re0 interface (or whatever it's connected to) is misbehaving.
Any I/O errors on Status > Interfaces?
-
Interesting theory.. I went and checked and it's been 23 hours since the reboot – and the Status for WAN says"
WAN Interface (wan, re0)
Media 1000baseT <full-duplex,master>In/out packets 2461287/1308634 (2.76 GiB/106.79 MiB)
In/out packets (pass) 2461287/1308634 (2.76 GiB/106.79 MiB)
In/out packets (block) 22309/5486 (12.81 MiB/391 KiB)
In/out errors 0/0
Collisions 0LAN Interface is solid (re1) with no errors nor collisions
Since the reboot, there has been 5 additional instances recorded in the firewall log
Mar 26 08:25:40 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,28320,0,DF,6,tcp,36,109.73.184.113,[MYIPADDRESS],60964,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 11:19:30 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,241,37101,0,DF,6,tcp,36,168.228.165.29,[MYIPADDRESS],1454,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 12:33:37 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,243,19749,0,DF,6,tcp,36,88.220.191.225,[MYIPADDRESS],29755,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 15:07:07 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,2173,0,DF,6,tcp,36,109.73.184.235,[MYIPADDRESS],20600,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Mar 26 16:23:28 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,54321,0,none,6,tcp,36,61.140.124.185,[MYIPADDRESS],45174,22,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
and, I had tried before to try to correlate but there is no "rule" found from:
pfctl -vvsr | grep -A3 4294967295
I do have and use Alias(es) for the purpose of ASN based blocking, for example, Facebook ASN blocking
whois -h whois.radb.net – '-i origin AS32934' | grep ^route | grep -v route6 | cut -d" " -f7 > ./fbblock.txt
I primarily build aliases to block the "ASN of the "usual suspects of persistent hacking/probing" to cut down on the firewall log "noise", as I'm unlikely to have the need to visit networks of or websites located in ... say.. AS9808 or AS4837
I don't know if there are other debugging options that can be turned on to track this, but all the monitoring indicates no "States" or connections are being successfully established by these incoming attempts, as checked by Diagnostics>States>States and filtered by IP of offender on "all" interface.
Thanks for the responding with ideas to check, but as of now, I still have not found the culprit of the strange error message, so I am proceeding cautiously.</full-duplex,master>
-
Good that there are no I/O errors but it's unlikely that your log entries reflect what's really happening.
Maybe the only way to find out is to packet capture everything on the re0 interface until you see another of those log entries.
If the packets are bad there probably isn't anything you could reliably use to filter the capture, so the output could be quite large.
EDIT: This is probably the source of this crap. Three out of five of the IP addresses you mention above come back with a Mikrotik login prompt.
-
Of course, this doesn't solve the riddle of the rule number being 232-1 or the "pass" entry.
I turned on logging for the default block rules and saw more than 100 attempts to connect to ports 21, 22, 23, 8291 in the space of two hours. However, none of my logs had those strange entries shown in yours : oddball rule number, "short", "pass" or "bad hdr length…"
I'm still on 2.3.5 :-[ Perhaps there was a regression in log parsing after that.
-
I would pcap port 23 on WAN and see what wireshark thinks about it.
-
I would pcap port 23 on WAN and see what wireshark thinks about it.
Yeah, that or 8291.
I updated to 2.4.2_1 after my last post and there are still plenty of blocked 21, 22, 23, and 8291 connections. I don't see those weird entries in my logs though, so it doesn't look like the packets are malformed.
-
Let's say I want to follow the advice and do some packet capture stuff, how would I do it?
I will try to read up and see if I can set it up, but given the previous log entry, and the new ones below,
I would appreciate some examples of what to use, either through the GUI or from the command line tailored to narrow down as much as possible.And , some quick updates.
Mar 26 19:01:31 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,240,22619,0,DF,6,tcp,36,91.109.193.174,
[MYIPADDRESS],54443,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',One of the previous "offender" was blocked with my new rule 58 (a new rule specifically created only to block everything coming from those previously identified "short,pass" IP addresses), I don't know if this yields any additional information or new information I can use to help packet capture beside to try to packet filter on port 23 or on port 8291.
Mar 27 00:30:10 pfSense filterlog: 58,,,1522081663,re0,match,block,in,4,0x0,,239,43226,0,DF,6,tcp,40,109.73.179.128,[MYIPADDRESS],5233,23,0,S,781346768,,16060,,
Mar 27 00:52:35 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,239,27500,0,DF,6,tcp,36,105.212.92.243,[MYIPADDRESS],24625,8291,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
Until I receive replies and guidance, I'm off to read about how to do packet filtering and save the results to disk.
-
Diagnostics > Packet capture
Install WireShark on your desktop to read the PCAP file that is generated/downloaded.
Now I'm curious about why the log from your new rule is showing no strange stuff.
-
Your always going to see a shit ton of noise.. Part of the reason I turned off logging default rule and just put in my own rule to log SYN only… I don't need to see all the udp noise, etc. Or anything that is out of state, etc. But I do like to keep an eye on the top ports... Yeah 23, 22 all going to be heavy hitters..
Do you have any odd ball forwards or floating rules? There are things that are looked at before firewall rules, etc..
https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order
If your really curious I would post up your full rule set.
https://doc.pfsense.org/index.php/How_can_I_see_the_full_PF_rulesetOr if you not open to posting that public.. Sure there are a few of here that that can be trusted you could PM it too.. If you don't feel comfortable with what might be listed in it.. We could then clean it up with any sort of public IPs and repost it for the board to look over, etc.
Part of the reason posting is want to get informed on updates to this thread. Off the top I have nothing.. If your saying you searched your rules for that ID and not seeing it.. at a loss..
How about atleast a screen shot of your wan, floating and forward rules.. To go along with your firewall rules if you don't feel like posting the full set..
-
There's nothing too exciting there I'm afraid.
screenshots of floating rules and wan rules as example, my floating rules goes a lot further below, mostly composed of ASN alias that you can compile checking the IBMcloud.com page doing query of the individual ASN and IP ranges (which I slightly massage, sort into an individual ASN Alias. Rule description basically is "ASN#-Nation-B(lock)-Start_BLOCKDATE.
the "shortpassblock" is currently de-activated so I can try to packet capture some data. and before you ask about 2 copies of say FB rules, I alternate them like two light switches (turn 2nd FB rule on, activate), then turn off 1st FB rule, then reload filter rules, to basically reset packet count.
There is no port forwarding, only pfB DNSBL for malicious ad blocking etc.
Last example shows a mouseover over an ASN alias, shows in the first IP description that shows the "Offender IP" that caused the ASN to be blocked.
-
Sorry, uploaded the wrong image. Here is the Floating rule one.
-
[Edit]: Please see follow-up post #24
[edit] … Learning how to packet capture deleted ...
Below is the "new firewall log entry"
Mar 28 01:11:32 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,242,25830,0,DF,6,tcp,36,168.228.167.124,[MYIPADDRESS],32068,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
-
[Edit] Removed trial packet capture info .. Please see captured Wireshark info a few posts down.
-
Dude just download the packet capture from pfsense and open it with wireshark or post up the pcap file you download.. What you have posted is just gibberish not going to help anyone figure out anything.
You can run packet capture without being logged in… just come back to it latter and stop it, etc.
-
Hi. I am posting a small progress note.
After some trial and errors, and some hit & misses, I think I have figured out a way to finally grabbing the packets to be examined, I will try to write them down so maybe if people have trouble like mine they will have something to refer to, so they can do the same.
1. I downloaded and installed latest stable version of Wireshark (for Win 7)
2. Learned how to copy a file remotely from pfSense box to local box, so I can move the packetcapture.cap to be examined by Wireshark.
https://doc.pfsense.org/index.php/HOWTO:_Access_pfSense_filesystems_remotely_with_scp
3. Learning to use tcpdump to capture the packets from pfSense box.
the most trouble for me initially was there was no predictability where the packet would be coming from, and I created a bunch of ssh windows each have:/usr/sbin/tcpdump -i re0 -p -c 100 -s 0 -w /tmp/105-212-92-243.cap ip and tcp and port 8291 and host 105.212.92.243
using previous offender IPs, and I would see no traffic from it, but a new one would come in like:
Mar 29 21:07:56 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,56,29244,0,none,6,tcp,36,155.133.83.54,[myIPAddress],40011,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
so, I finally settled on using:
/usr/sbin/tcpdump -i re0 -p -s 0 -w /tmp/port8291packetcapture.cap ip and tcp and port 8291 or 23 or 22
and was able to successful moved a copy of packet capture to examine on Wireshark.
Now that I have cast my net correctly, now I just have to wait.
With the example sample Wireshark screenshot, Once I grab the appropriate packets, what do I need to share here and How would I do it? I think I tried to upload the previous "packetcapture.cap" file but the forum would not accept it.
if you look at the screengrab posted here, what would be the relevant information to post in the next followup, once I have the data, or how would I process the data and post follow up to it?
[Edit] Small correction editing out my IP address on attached picture. The attached picture was to confirm that packets on Ports (22,23,8291) was being captured, and it seems to be working. Now I can wait for the pfSense filter log to show me another packet came in then I can correlate and post follow-up.
-
First thing to do is get a packet matched up with a log entry.
-
Got it !!!
I have absolutely no idea what this means… but here it is. Please let me know if this helps. I have the file on hand if someone needs it.
Mar 29 23:40:23 pfSense filterlog: 4294967295,,,0,re0,short,pass,in,4,0x0,,231,17518,0,DF,6,tcp,36,185.51.112.191,[myIPADDRESS],13794,23,-4,S,errormsg='[bad hdr length 20 - too long, > 16]',
-
WireShark also sees the packet as malformed and pfSense is reporting that: '[bad hdr length 20 - too long, > 16]'
When logging this same sort of traffic, I haven't seen any malformed packets at all. Given that you're not seeing any I/O errors on that interface, it's seems to me that something upstream is corrupting the packets.
You are also getting that weird rule number and the "short,pass" reason and action. That's probably just a result of the fact that the packet is corrupt but it would be good to know.
The MAC address translation in the second WireShark window says that packet came from a Cisco (maybe Linksys) device. Do you have something from one of those manufacturers in front of your pfSense box? If so, it would be interesting to see whether the other poster with this "bad hdr" error has one of those as well. Although, he wasn't hitting the "short,pass" or rule number problems.
It really doesn't look like anything is getting in to your network, despite that "pass".
It might be worth backing up your config, install a fresh copy of 2.4.3 and restoring the config to that.
-
I am feeling better today after I read your reply, because while I felt there was no unauthorized traffic coming from WAN, I could not see the underlying conditions nor the data causing the erroneous log entry, and the pfSense logger was causing me anxiety. The mysterious "rule set" number which I knew didn't exist did not help in the matter, either.
To address the "MAC address translation" question, I generate a new random MAC address and use it to "spoof" in the WAN interface periodically to obtain a different IP address once in a while from my ISP. My thinking is that if I got a new IP address, it will make me a more elusive hacking target than if I stayed at one IP address 100% of the time for years on end. I think that the "source" is the "real" MAC address of the WAN interface of my pfSense box. Would this thinking be correct?
I was thinking of updating to pfSense v2.4.3 yesterday, but, I didn't know my issue was relating to these 2 topics listed in the new release notes:
Added firewall rule tracking ID to rule list (in counter tooltip) and firewall rule edit page #8348
Fixed cases where automatic or scripted rules were not getting tracking IDs #8353Now that I have more clarity, I went ahead and did a upgrade this morning (option #13 - upgrade from console), and I will keep an eye out on the pfSense logger.
If the upgrade path doesn't seem to resolve the issue, I may do a clean install in the future. Meanwhile, I'll keep an eye out on the pfSense logger and try to continue to packet capture on those 3 ports.
If anyone have more guidance based on reading the Wireshark data, please let me know.
