What is the biggest attack in GBPS you stopped

firewalluser · Jun 5, 2015, 7:02 AM

I think the inevitable use of Dtrace is upon us. :)

I dont know the status of FreeBSD 11, but if its at least in a Release Candidate status, is it worth trying to port pfsense onto a FreeBSD 11 build?

Although even if we could, if the problems still showed up, we'd still need Dtrace….

Is there really no debugging facility built into pfsense?

Supermule · Jun 5, 2015, 9:27 AM

I get this in the system logs as of today…

php-fpm[60486]: /interfaces.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.6 Copyright 2004-2014 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 13 leases to leases file. Can't install new lease database /var/db/dhcpd.leases.1433496021 to /var/db/dhcpd.leases: No such file or directory Listening on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Sending on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that befo

And this

php-fpm[85395]: /rc.newwanip: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.6 Copyright 2004-2014 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 13 leases to leases file. Listening on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Sending on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it bef

Comes right after this

| Jun 5 11:20:28 | check_reload_status: Reloading filter |
| Jun 5 11:20:28 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 5 11:20:28 | check_reload_status: Restarting ipsec tunnels |
| Jun 5 11:20:28 | check_reload_status: updating dyndns Yousee |
| Jun 5 11:20:28 | check_reload_status: Reloading filter |
| Jun 5 11:20:28 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 5 11:20:28 | check_reload_status: Restarting ipsec tunnels |
| Jun 5 11:20:28 | check_reload_status: updating dyndns Yousee |
| Jun 5 11:20:26 | check_reload_status: Syncing firewall |
| Jun 5 11:20:26 | check_reload_status: Syncing firewall |
| Jun 5 11:20:25 | check_reload_status: Reloading filter |
| Jun 5 11:20:25 | php-fpm[9696]: /rc.start_packages: Reloading Squid for configuration sync |
| Jun 5 11:20:25 | php-fpm[9696]: /rc.start_packages: Restarting/Starting all packages. |
| Jun 5 11:20:23 | lighttpd[33558]: (connections.c.1692) SSL (error): 5 -1 1 Operation not permitted |
| Jun 5 11:20:23 | lighttpd[33558]: (connections.c.619) connection closed: write failed on fd 20 |
| Jun 5 11:20:23 | lighttpd[33558]: (network_openssl.c.118) SSL: 5 -1 1 Operation not permitted |
| Jun 5 11:20:23 | check_reload_status: Reloading filter |
| Jun 5 11:20:23 | check_reload_status: Starting packages |
| Jun 5 11:20:23 | php-fpm[85395]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 80.197.148.74 -> 80.197.148.74 - Restarting packages. |
| Jun 5 11:20:23 | php-fpm[60486]: /interfaces.php: Removing static route for monitor 81.19.224.67 and adding a new route through 80.197.148.1 |
| Jun 5 11:20:21 | php-fpm[85395]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. |
| Jun 5 11:20:21 | check_reload_status: updating dyndns wan |

firewalluser · Jun 5, 2015, 9:37 AM

Did you set an IP address in ESXi for that nic?

Supermule · Jun 5, 2015, 9:50 AM

No it gets that from DHCP on WAN from my ISP. Hypervisor mgmt network is on the LAN.

DHCP is running on LAN issuing IP's to a Linksys wireless AP in bridge mode.

I do not run HP Jetadmin software on LAN.

Supermule · Jun 5, 2015, 10:06 AM

Disabled Apinger…

All I get in the logs is this then...

firewalluser · Jun 5, 2015, 11:02 AM

This might be an approach for getting Dtrace working on pfsense.

https://forum.pfsense.org/index.php?topic=94838.msg527131#msg527131

Different package, but might work, bit like the only way to get Rpi's working virtually is on Ubuntu with QEMU installed to emulate the ARM's, at which point it becomes alot quicker by virtual of stonking fast hw setting up freebsd and pfsense on RPi's, but I digress.

Supermule · Jun 5, 2015, 11:09 AM

Thanks man!

Over my head…. I need others to chime in on this. Spent the last mth using all my spare time to test this frigging thing.

After I did the cron job, then I can make it come alive again.

I have to settle with that right now...........

Currently migrating all the pf's in the datacenter to Mikrotik since I cant have this hosting customers. At home I dont care.

firewalluser · Jun 5, 2015, 11:12 AM

So is it worth starting a Getting Dtrace to work on pfsense thread?

We only have to ask the right questions, document everything we do so that others can learn from it as well if they have the thirst for knowledge.

There's plenty of examples online which show how to compile code onto platforms where a package doesnt exist for downloading.

There's generally parallels everywhere if you look. ;)

Supermule · Jun 5, 2015, 11:18 AM

Posted a bounty for Dtrace package.

https://forum.pfsense.org/index.php?topic=94846.0

tim.mcmanus · Jun 5, 2015, 11:25 AM

dtrace is installed already on pfSense.

Go to the FreeBSD forums now. You're not dealing with pfSense anymore when you start working with dtrace. This is the wrong place to look for that kind of guidance.

I think I'm having déjà vu….

Supermule · Jun 5, 2015, 11:27 AM

OK.

But it still leaves me with no options to monitor pfsense deep under the hood with timestamps.

https://forum.pfsense.org/index.php?topic=94260.0

Any ideas?

firewalluser · Jun 5, 2015, 11:28 AM

You need to be more specific with what you want exactly, plus programmers are very specific and you might find Dtrace does what you want, so would it be fair for anyone to just compile and get Dtrace running?

Besides when you say what you say, have you considered the other app/conversation thread here https://forum.pfsense.org/index.php?topic=94843.msg527152#msg527152 or is your browser running single tab/single core/single thread and you have not locked/mutexed this thread and swapped to the other thread yet? Or if your browser is running multitabbed/multi core/multi threaded, have you locked this thread in your browser by switching tabs to read the other thread/multi core/multi thread if you catch my drift. ;D

There's parallels everywhere if you have an abstract look on life.

firewalluser · Jun 5, 2015, 11:29 AM

@tim.mcmanus:

dtrace is installed already on pfSense.

Go to the FreeBSD forums now. You're not dealing with pfSense anymore when you start working with dtrace. This is the wrong place to look for that kind of guidance.
I think I'm having déjà vu….

Significance might not have registered. Apologies. :o

tim.mcmanus · Jun 5, 2015, 12:50 PM

dtrace stuff. Remember, dtrace is a component of the FreeBSD distribution, so you need to go there to understand it and get support from the community. These forums are more product-specific for pfSense–mostly dealing with networking versus deep coding. So you're questions are best answered in the FreeBSD forums.

https://wiki.freebsd.org/DTrace

https://wiki.freebsd.org/DTrace/Tutorial

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace.html

http://www.brendangregg.com/USEmethod/use-freebsd.html

tim.mcmanus · Jun 5, 2015, 5:39 PM

Smoking gun?

https://forums.freebsd.org/threads/freebsd-10-1-network-unaccessible-after-high-traffic.51743/

Potential fix? (synproxy state)

https://forums.freebsd.org/threads/enabling-ipv6-eventually-freezes-the-gateway.47959/#post-267820

See #4 from this link:

http://www.onlamp.com/pub/a/bsd/2004/04/15/pf_developers.html

https://forums.freebsd.org/threads/freebsd-9-0-release-syn-flood.29071/#post-163533

Notice how all of this freakin' awesome information is coming from the FreeBSD forums. I strongly suggest you consolidate your findings in this thread and move the discussion over to the FreeBSD networking forums. They have dozens of discussions on SYN floods with some excellent guidance given.

Supermule · Jun 5, 2015, 7:36 PM

Yes. It just doesnt work as expected.

Until the drop in traffic occurs. So I need to find a way to see whats initiated when the traffic drops.

tim.mcmanus · Jun 5, 2015, 8:06 PM

@Supermule:

Yes. It just doesnt work as expected.

Until the drop in traffic occurs. So I need to find a way to see whats initiated when the traffic drops.

https://forums.freebsd.org/forums/networking.7/

Supermule · Jun 6, 2015, 11:56 AM

Yes….one more forum to watch....................... :D

Supermule · Jun 7, 2015, 7:09 PM

Just a small update..

Changed NIC's to vmxnet3 in vmware and this happened.

Youtube Video

Same traffic drop but curves are different. On E1000 they follow each other. On vmxnet3 they are totally seperated. Red is 15mbit/s and black is almost zero.

Suddenly then traffic drops on incoming WAN and it begins to flow traffic and respond to ping for Google DNS.

No change what so ever to anything.

vmxnet3 drivers are much worse handling this kind of attack than the E1000 drivers.

EDIT: When Core nr 3 drops, the FW comes alive and begins to route packets. Overall load is cut in half.

EDIT EDIT: Getting 100% interrupt in the top -HSP on the console. Again the E1000 almost had zero.

Logs are flooded with this.

| Jun 7 19:54:14 | check_reload_status: Reloading filter |
| Jun 7 19:54:14 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 7 19:54:14 | check_reload_status: Restarting ipsec tunnels |
| Jun 7 19:54:14 | check_reload_status: updating dyndns Yousee |
| Jun 7 19:54:01 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:49 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:43 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:38 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:33 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:28 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:22 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:15 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:06 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:53:01 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:56 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:51 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:45 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:40 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:35 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:28 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:23 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:18 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:13 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:07 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:52:02 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:56 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:50 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:46 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:40 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:34 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:29 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:24 | check_reload_status: Reloading filter |
| Jun 7 19:51:24 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 7 19:51:24 | check_reload_status: Restarting ipsec tunnels |
| Jun 7 19:51:24 | check_reload_status: updating dyndns Yousee |
| Jun 7 19:51:22 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:16 | kernel: vmx0: watchdog timeout on queue 0 |
| Jun 7 19:51:11 | kernel: vmx0: watchdog timeout on queue 0 |

firewalluser · Jun 7, 2015, 8:08 PM

And an update from me.

Currently downloading and getting 5.42Mbps according to the dashboard, yet when you did the ddos the otherday that was only giving showing 2.42Mbps.

So why does a ddos cause my bandwidth to drop down? Could it be related to this post https://forum.pfsense.org/index.php?topic=94903.0 ie a hw issue, or is freebsd or pfsense.

I have a huwei supplied router dont know if it can go into modem only mode, but if it can I will check the speeds again with this device if you dont mind doing a Britney Spears and hit me one more time? ;)