Numerous ET SCAN Potential SSH Scan OUTBOUND alerts. Is Pfsense infected?



  • Hello All,

    I have been scratching my head for the past 24 hours and I can't seem to figure this out.

    I have been getting numerous ET SCAN Potential SSH Scan OUTBOUND alerts in Suricata since the last update. They all come from my Router's IP and go to random WAN addresses (in a certain order).

    I was afraid this was a device on my network (specifically a Linux machine) that may be infected and sending out ssh requests. I have turned off all devices on my network and the alerts are still coming in. Could the Pfsense install be infected? Or are these false positives? Anything I can do to test?

    Thanks in advance.



  • @lambro690
    It will help if you will post the actual alert message including the rule's SID (Signature ID).

    Are you running the IDS/IPS package on your WAN or LAN? Sounds like maybe your WAN since you say your router's IP is in all of the messages. Because of NAT and where the IDS/IPS (whether it is Snort or Suricata) sniffs packets, when you run it on the WAN all local IP addresses (from your LAN) appear with the WAN's IP address instead due to NAT. I recommend that users run the IDS/IPS on other firewall interfaces and not on the WAN for this reason. It makes it very hard to identify problematic hosts on your LAN when you run the IDS/IPS on the WAN since the only IP you will get in alerts is the public IP of your WAN.



  • Thanks for the help @bmeeks .

    I have Suricata running on both LAN and WAN interfaces. Only the WAN side is alerting about the SSH attacks. Should I disable the WAN side just to test?

    Here is a screenshot if one of the alerts (there are hundreds of them. All with different IPs.

    0_1544716636592_70dd5d0e-2227-4e71-b0c0-b5825940a34c-image.png

    The 192.168.1.3 is my WAN address (I run PfSense in the DMZ of my Verizon router. It is a long story).

    Thanks again.

    EDIT: Also, something else I noticed, I have added a rule to the WAN firewall rules to block outbound SSH and the firewall log does not alert of any attempts.



  • @lambro690

    Snort and Suricata both, with respect to the WAN, can be viewed as sitting in between the kernel stack and the physical NIC and wire. So for outbound traffic from your firewall to the Internet, firewall rules are applied before Snort or Suricata see the traffic (think of the IDS/IPS seeing the packets as they are handed off to the physical NIC). For inbound traffic from the Internet to your firewall, Snort or Suricata will see that traffic before the firewall does (again, think of the IDS/IPS as seeing the traffic as it exits the physical NIC on the way to the rest of pfSense).

    Why did you mask out the destination IP address in that alert? Was the destination one of your LAN addresses or in a DMZ? Not sure what your firewall would be doing generating a number of SSH requests. Maybe capture some of those packets and have a look at them. You can do that under the DIAGNOSTICS tab of the firewall.



  • @bmeeks

    Thank you much for the info! I have been running Suricata and previously Snort for many years and I have never seen this issue.

    I masked it out because of privacy, but I guess these days it is easy to find an IP address. Here is an example of how the alerted IPs increase as if something is scanning a list :

    0_1544726436768_26068eab-45e9-429a-ac33-17ba082a0d49-image.png

    When I turned the WAN side off, the LAN side and the VLAN I have for WiFi did not alert to anything related to SSH. This is so strange.



  • Have you been making any external connections via ssh or using sftp?

    This alert can fire if you make repeated outbound connections within a period of time:

    alert tcp $HOME_NET any -> $EXTERNAL_NET 22 (msg:"ET SCAN Potential SSH Scan OUTBOUND"; flow:to_server; flags:S,12; threshold: type threshold, track by_src, count 5, seconds 120; reference:url,en.wikipedia.org/wiki/Brute_force_attack; reference:url,doc.emergingthreats.net/2003068; classtype:attempted-recon; sid:2003068; rev:7; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
    

    So 5 attempts in 120 seconds will generate this alert.

    Do you have EVE JSON enabled in Suricata?



  • @lambro690 said in Numerous ET SCAN Potential SSH Scan OUTBOUND alerts. Is Pfsense infected?:

    Here is a screenshot if one of the alerts (there are hundreds of them. All with different IPs.

    I missed that on my first read.

    If you have hundreds of alerts on SID 2003068, you control your network (have a kid running a port scanner in the basement?), and you aren't running your own port scan on that class C, then your firewall is compromised, one of your LAN machines is, or you have a rogue wifi client.

    Are you also running Suricata on the VLAN for wifi?

    I would suspect the LAN/wifi machines before the firewall, but if you're running Suricata (I presume with the same rules?) on the LAN and VLAN, then yes, I would assume your firewall is compromised.

    If you use who from the console (or under diagnostics->command prompt), do you see any ttys?

    What output do you get from ps -aux | grep ssh ?



  • @boobletins Thank you for your response.

    I ended up reinstalling after I tried the packet capture as @bmeeks suggested.

    It seems like the firewall was compromised. This is weird because it started happening after the recent update. Not sure if it is related or not.

    After the reinstall, I powered on my servers one by one and no alerts were triggered so I think we are good. I will continue to monitor it and report back if I see anything.

    Thanks again guys!



  • @lambro690 said in Numerous ET SCAN Potential SSH Scan OUTBOUND alerts. Is Pfsense infected?:

    @boobletins Thank you for your response.

    I ended up reinstalling after I tried the packet capture as @bmeeks suggested.

    It seems like the firewall was compromised. This is weird because it started happening after the recent update. Not sure if it is related or not.

    After the reinstall, I powered on my servers one by one and no alerts were triggered so I think we are good. I will continue to monitor it and report back if I see anything.

    Thanks again guys!

    A compromised firewall would be of great concern to me! If it were me, I would want to figure out how and what compromised my firewall.



  • Indeed. Logs, pcaps, or a disk image would have been interesting.

    It would also be interesting to know how you reinstalled (not worried about rootkits? efi/bios resident malware?).

    After re-install, did you put the same config back in and just modify the password?

    All kinds of interesting questions... thread had 10/10 potential :)



  • @boobletins @bmeeks

    It would have been nice to see but I just wanted my internet back under control :)

    All good questions! Yes same config, new LastPass random password this time.

    I am a Linux admin at work so I know a few things, although not everything :p

    I reinstalled after booting up an Ubuntu instance and running chkrootkit. Not sure how effective it'll be but I made an attempt. I then formatted the drive (it was ZFS) and I have moved back to UFS.

    I installed fresh, restored backup, changed password and then sat there for about 30 mins watching the Suricata alerts just to make sure.

    Then I powered all severs/desktops back up one by one just to be sure it did not spread to them.

    Not 100% trusting just yet.



  • I'm getting the same thing, Seems to be crawling through the IP's. Really worrisome as I host a site for my wife's Co.
    Has the rebuild fixed it? Are you sure you still have the alert enabled?

    Member of the 'emerging scan rules':
    SID 2003068
    https://docs.emergingthreats.net/bin/view/Main/2003068



  • How many times per minute are you seeing this? You're sure it isn't a result of you attempting to make an ssh connection 5 times in a 2 minute window?



  • Also, could you paste the output from the command line sockstat | grep ":22"



  • 0_1546637981852_Screen Shot 2019-01-04 at 4.39.08 PM.png



  • @boobletins said in Numerous ET SCAN Potential SSH Scan OUTBOUND alerts. Is Pfsense infected?:

    sockstat | grep ":22"

    Thanks for the quick response, here is what the command yielded, what am I looking at?
    ? ? ? ? tcp4 192.168.2.1:20356 192.168.2.23:22
    ? ? ? ? tcp4 192.168.1.8:54310 192.168.1.24:22
    ? ? ? ? tcp4 192.168.1.8:54312 192.168.1.24:22



  • Those question marks are in the original output? And you're running that from the pfSense command line?

    The command should be showing you which process has open sockets on port 22. We're hoping whatever process is scanning will show up there to try to get an indication of what is going on.





  • Can you check chat for me? It will be faster.



  • chromefinch had previously had ntop-ng installed but only recently re-enabled suricata.

    sockstat | grep ":22" output from the ui did not generate helpful output.

    He re-enabled ssh access for himself and sockstat | grep ":22" generated output similar to below:

    root ntopng 15017 45 tcp4 x.x.x.x:33912 57.151.10.72:22
    

    ntop was likely scanning what it thought was an internal network for sshd servers (though I have no experience with ntop on pfsense) -- he's following up in the ntop forums.

    lambro690 -- I wonder if you had something similar going on?



  • @boobletins Thanks for your help!!!! I disabled the ntop wan interface and No more alerts!



  • @chromefinch said in Numerous ET SCAN Potential SSH Scan OUTBOUND alerts. Is Pfsense infected?:

    @boobletins Thanks for your help!!!! I disabled the ntop wan interface and No more alerts!

    So your firewall was infected -- but with ntop instead of a trojan ... ☺ . (Just kidding).



  • @boobletins Yup lol I sure did have ntop installed. Must have been a bug with the package because I haven't gotten anything since the reinstall. Now I will know what to look out for!

    Great work guys. That makes me feel a little bit better about my security :p



  • I'm having same issue with ntop .. .. seems to try ip's like 0.106.219.157...
    installed, reinstalled several times. ... anyone might know what is the issue?