Was I the target of a successful attack?
-
I have a cluster of two PFsense node in CARP (pf01 and pf02). Both are 64 bits versions, under VMware vSphere. Both nodes have two vmxnet3 NICs with various vlan interfaces. I've installed the vmxnet3 following the steps provided in the pfsense docs.
Today, the WAN vlan got knocked out of production. It stopped forwarding traffic. The CARP failover status showed pf01 and pf02 WAN VIP as master (others showed the correct master/backup states), but pf01 WAN did not forward traffic.
Through a bad NAT configuration, ssh was exposed to the internet.
I wondering a few things:
- could vmxnet3 be that unstable?
- Where should I look for more clues as to what happened?
- could someone, through an SSH exploit, kill traffic flowing through the WAN?
I'm seeting that a lot, is it worry some?
Jan 4 01:22:00 pf01 kernel: VMware memory control driver unloaded
Jan 4 01:22:00 pf01 kernel: VMware memory control driver initializedBut the most interresting part, here's the system log for the more or less 15 minutes just before the WAN crapped out:
Jan 4 00:38:54 pf01 sshd[5193]: Failed password for admin from 87.106.251.42 port 51447 ssh2
Jan 4 00:38:55 pf01 sshd[7634]: Failed password for admin from 87.106.251.42 port 51628 ssh2
Jan 4 00:38:58 pf01 check_reload_status: Syncing firewall
Jan 4 00:38:59 pf01 sshd[7811]: Invalid user debug from 87.106.251.42
Jan 4 00:38:59 pf01 sshd[7811]: Failed password for invalid user debug from 87.106.251.42 port 52592 ssh2
Jan 4 00:39:00 pf01 php: rc.filter_synchronize: Beginning XMLRPC sync to https://172.16.204.252:443.
Jan 4 00:39:01 pf01 php: rc.filter_synchronize: XMLRPC sync successfully completed with https://172.16.204.252:443.
Jan 4 00:39:03 pf01 php: rc.filter_synchronize: Filter sync successfully completed with https://172.16.204.252:443.
Jan 4 00:39:03 pf01 sshd[8099]: Invalid user ftp from 87.106.251.42
Jan 4 00:39:03 pf01 sshd[8099]: Failed password for invalid user ftp from 87.106.251.42 port 53541 ssh2
Jan 4 00:39:05 pf01 sshd[8517]: Invalid user info from 87.106.251.42
Jan 4 00:39:05 pf01 sshd[8517]: Failed password for invalid user info from 87.106.251.42 port 53954 ssh2
Jan 4 00:39:06 pf01 sshd[9120]: Invalid user karaf from 87.106.251.42
Jan 4 00:39:06 pf01 sshd[9120]: Failed password for invalid user karaf from 87.106.251.42 port 54230 ssh2
Jan 4 00:39:07 pf01 sshd[9471]: Invalid user log from 87.106.251.42
Jan 4 00:39:07 pf01 sshd[9471]: Failed password for invalid user log from 87.106.251.42 port 54406 ssh2
Jan 4 00:39:08 pf01 sshd[9641]: Invalid user marketing from 87.106.251.42
Jan 4 00:39:08 pf01 sshd[9641]: Failed password for invalid user marketing from 87.106.251.42 port 54618 ssh2
Jan 4 00:39:09 pf01 sshd[10247]: Invalid user mike from 87.106.251.42
Jan 4 00:39:09 pf01 sshlockout[37765]: Locking out 87.106.251.42 after 15 invalid attempts
Jan 4 00:39:09 pf01 sshd[10247]: Failed password for invalid user mike from 87.106.251.42 port 54834 ssh2
Jan 4 00:39:09 pf01 sshlockout[37765]: Locking out 87.106.251.42 after 15 invalid attempts
Jan 4 00:39:37 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:37 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:40 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:40 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:50 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:50 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:54 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:54 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:57 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:39:57 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:40:08 pf01 check_reload_status: Syncing firewall
Jan 4 00:40:08 pf01 check_reload_status: Reloading filter
Jan 4 00:40:11 pf01 php: rc.filter_synchronize: Beginning XMLRPC sync to https://172.16.204.252:443.
Jan 4 00:40:12 pf01 php: rc.filter_synchronize: XMLRPC sync successfully completed with https://172.16.204.252:443.
Jan 4 00:40:13 pf01 php: rc.filter_synchronize: Filter sync successfully completed with https://172.16.204.252:443.
Jan 4 00:40:24 pf01 sshd[34203]: Did not receive identification string from 222.186.56.43
Jan 4 00:41:31 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:41:31 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:41:44 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:41:44 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:42:17 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:42:17 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:42:34 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:42:34 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:04 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:04 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:11 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:11 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:14 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:14 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:24 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:43:24 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:44:21 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:44:21 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:44:47 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:44:47 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:42 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:42 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:52 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:52 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:58 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:45:58 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:08 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:08 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:15 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:15 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:46:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:06 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:06 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:16 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:16 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:21 pf01 check_reload_status: Syncing firewall
Jan 4 00:47:24 pf01 php: rc.filter_synchronize: Beginning XMLRPC sync to https://172.16.204.252:443.
Jan 4 00:47:25 pf01 php: rc.filter_synchronize: XMLRPC sync successfully completed with https://172.16.204.252:443.
Jan 4 00:47:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:27 pf01 php: rc.filter_synchronize: Filter sync successfully completed with https://172.16.204.252:443.
Jan 4 00:47:36 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:36 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:40 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:40 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:46 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:46 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:50 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:50 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:53 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:53 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:47:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:00 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:00 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:07 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:07 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:10 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:10 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:19 pf01 check_reload_status: Syncing firewall
Jan 4 00:48:20 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:20 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:21 pf01 php: rc.filter_synchronize: Beginning XMLRPC sync to https://172.16.204.252:443.
Jan 4 00:48:22 pf01 php: rc.filter_synchronize: XMLRPC sync successfully completed with https://172.16.204.252:443.
Jan 4 00:48:23 pf01 php: rc.filter_synchronize: Filter sync successfully completed with https://172.16.204.252:443.
Jan 4 00:48:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:36 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:36 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:39 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:39 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:41 pf01 check_reload_status: Syncing firewall
Jan 4 00:48:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:48:44 pf01 php: rc.filter_synchronize: Beginning XMLRPC sync to https://172.16.204.252:443.
Jan 4 00:48:45 pf01 php: rc.filter_synchronize: XMLRPC sync successfully completed with https://172.16.204.252:443.
Jan 4 00:48:46 pf01 php: rc.filter_synchronize: Filter sync successfully completed with https://172.16.204.252:443.
Jan 4 00:48:53 pf01 check_reload_status: Reloading filter
Jan 4 00:48:59 pf01 sshd[81620]: Failed password for root from 222.186.56.43 port 2842 ssh2
Jan 4 00:48:59 pf01 sshd[81620]: Failed password for root from 222.186.56.43 port 2842 ssh2
Jan 4 00:49:01 pf01 sshd[81620]: Failed password for root from 222.186.56.43 port 2842 ssh2
Jan 4 00:49:03 pf01 sshd[81620]: Failed password for root from 222.186.56.43 port 2842 ssh2
Jan 4 00:49:03 pf01 sshd[81620]: Failed password for root from 222.186.56.43 port 2842 ssh2
Jan 4 00:49:04 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:04 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:08 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:08 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:11 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:11 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:18 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:21 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:21 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:25 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:25 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:28 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:28 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:35 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:35 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:38 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:38 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:41 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:41 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:48 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:48 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:51 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:51 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:55 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:55 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:58 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:49:58 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:05 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:05 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:14 pf01 sshd[3184]: Failed password for root from 222.186.56.43 port 3942 ssh2
Jan 4 00:50:14 pf01 sshd[3184]: Failed password for root from 222.186.56.43 port 3942 ssh2
Jan 4 00:50:14 pf01 sshd[3184]: Failed password for root from 222.186.56.43 port 3942 ssh2
Jan 4 00:50:15 pf01 sshd[3184]: Failed password for root from 222.186.56.43 port 3942 ssh2
Jan 4 00:50:15 pf01 sshd[3184]: Failed password for root from 222.186.56.43 port 3942 ssh2
Jan 4 00:50:16 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:16 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:20 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:20 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:23 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:23 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:26 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:30 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:30 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:43 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:53 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:53 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:50:56 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:03 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:06 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:06 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:13 pf01 sshd[11277]: error: accept: Software caused connection abort
Jan 4 00:51:21 pf01 sshd[42291]: Failed password for root from 222.186.56.43 port 1488 ssh2
Jan 4 00:51:22 pf01 sshd[42291]: Failed password for root from 222.186.56.43 port 1488 ssh2
Jan 4 00:51:22 pf01 sshd[42291]: Failed password for root from 222.186.56.43 port 1488 ssh2
Jan 4 00:51:22 pf01 sshd[42291]: Failed password for root from 222.186.56.43 port 1488 ssh2
Jan 4 00:51:23 pf01 sshd[42291]: Failed password for root from 222.186.56.43 port 1488 ssh2
Jan 4 00:51:23 pf01 sshlockout[37765]: Locking out 222.186.56.43 after 15 invalid attemptsThe public IP addresses you see are not mine.
Thanks,
-
DDoS on SSH and it created a buffer overflow and WAN went offline??
-
Could be something like that. Are you asking a question or are you telling me that's what you think it is? :)
-
Thats a guess from here :)
-
how could I troubleshoot that further than reading from the logs in /var/log ?
-
Make your attack surface as small as possible and harden the pfsense box.
Disable SSH and predefined ports other than the outmost necessary.
Run Snort with block option enabled and released every 24hrs.
-
could it be an idea to have fail2ban as part of a pfsense installation?
fail2ban will prevent ddos attacks bringing down ssh.
-
Would be a viable option to have indeed!
-
Is it possible to add a feature request for fail2ban in pfsense? I think this is important to prevent (D)DOS attacks leading to crash of service so my hope is that it is added to pfsense core and not as a package.
-
@mir:
Is it possible to add a feature request for fail2ban in pfsense? I think this is important to prevent (D)DOS attacks leading to crash of service so my hope is that it is added to pfsense core and not as a package.
But it's already there. https://doc.pfsense.org/index.php/Sshlockout
-
Maybe not working then?? or enabled??
-
Maybe not working then?? or enabled??
Erm… read the logs posted?
Jan 4 00:51:23 pf01 sshlockout[37765]: Locking out 222.186.56.43 after 15 invalid attempts
Looks damn well working.
P.S. None of this protects against DDoS. Not possible. Won't save your WAN from crashing.
-
Yes I can see that :) Thank you.
https://doc.pfsense.org/index.php/HOWTO_enable_SSH_access
Not a way to configure it. And it seems that all though i disable it on the webgui then it doesnt get disabled in the console menu.
![secure shell.PNG_thumb](/public/imported_attachments/1/secure shell.PNG_thumb)
![secure shell.PNG](/public/imported_attachments/1/secure shell.PNG) -
The console is responsive ONLY when you enable and disable the sshd.
No can do via the gui.
-
No idea what are you trying to do with console. Serial console is not SSH.
-
I know…. but you can enable/disable it via the gui and via console.
It doesnt work disabling it via the GUI. Only via the console...
-
What's IT?!
-
SSH :)
-
Before disabling SSH via GUI:
# netstat -an | grep .22 tcp4 0 0 *.22 *.* LISTEN tcp6 0 0 *.22 *.* LISTEN
After disabling SSH via GUI:
# netstat -an | grep .22 #
Re-enabling SSH via console:
# netstat -an | grep .22 tcp4 0 0 *.22 *.* LISTEN tcp6 0 0 *.22 *.* LISTEN
and checking back the GUI:
-
Doesnt work here…