What is the biggest attack in GBPS you stopped
-
Wait as stated is not the issue.
Moved the VM to a 6 disk/RAID10 local setup on a IBM X3650 from a 8 disk RAID10 NAS storage device.
http://youtu.be/tD5A-kElWw8
Very little difference in %WAIT
Wait state isn't the issue, correct. But it will mask the underlying kernel problem. You'll never see the IRQ interrupt storm on a VM because it's the hypervisor kernel that's managing the hardware, not the VM kernel.
-
What happens if you slow the nic speed right down to its slowest setting so the nic acts as a throttle?
I dont know if it will be worth even trying half duplex at this stage.Try the slowest speed with System:Advanced:Networking, Network Interfaces tick boxes 2,3 & 4 unticked so the nic handles more of the packet processing.
2 = disable hw checksum offload
3= disable hw tcp segmentation offload
4= disable hw large receive offload.I dont even know if these check boxes will have any effect [edit]running as a VM [/edit]in ESXi either, so it might be worth setting the nic speed in ESXi as another test.
This is just a WAG though.
-
It's not about the problem being better on bare metal. It's about reducing the number of variables in the test. A basic principle in problem isolation is to eliminate as many variables as possible. You identify the simplest configuration that demonstrates the problem, and then work with that.
VM infrastructure is a massive variable when you are trying to diagnose an under-load kernel issue.
I couldn't agree more.
Please, please, please stop wasting your time testing this issue on a hypervisor. Put pfSense on bare metal and test it there.
IT ISN'T BETTER ON BAREMETAL. Problem still exist. I tried several times on my bare metal supermicro.
Read the thread and the other threads again. You will see the history.- i am though not using pfsense anymore. So i can no more test
-
Nothing so far despite setting everything on ESXi, Switch and pfsense…
Playing around with settings on disabling offloading didnt yield anything either.
Not a blip difference.
What happens if you slow the nic speed right down to its slowest setting so the nic acts as a throttle?
I dont know if it will be worth even trying half duplex at this stage.Try the slowest speed with System:Advanced:Networking, Network Interfaces tick boxes 2,3 & 4 unticked so the nic handles more of the packet processing.
2 = disable hw checksum offload
3= disable hw tcp segmentation offload
4= disable hw large receive offload.I dont even know if these check boxes will have any effect [edit]running as a VM [/edit]in ESXi either, so it might be worth setting the nic speed in ESXi as another test.
This is just a WAG though.
-
It's not about the problem being better on bare metal. It's about reducing the number of variables in the test. A basic principle in problem isolation is to eliminate as many variables as possible. You identify the simplest configuration that demonstrates the problem, and then work with that.
VM infrastructure is a massive variable when you are trying to diagnose an under-load kernel issue.
I couldn't agree more.
Please, please, please stop wasting your time testing this issue on a hypervisor. Put pfSense on bare metal and test it there.
IT ISN'T BETTER ON BAREMETAL. Problem still exist. I tried several times on my bare metal supermicro.
Read the thread and the other threads again. You will see the history.- i am though not using pfsense anymore. So i can no more test
Ditto, ditto, ditto…
Please don't make me resurrect the KISS philosophy subject again. It doesn't seem to do well for my karma. ;)
-
I think the inevitable use of Dtrace is upon us. :)
I dont know the status of FreeBSD 11, but if its at least in a Release Candidate status, is it worth trying to port pfsense onto a FreeBSD 11 build?
Although even if we could, if the problems still showed up, we'd still need Dtrace….
Is there really no debugging facility built into pfsense?
-
I get this in the system logs as of today…
php-fpm[60486]: /interfaces.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.6 Copyright 2004-2014 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 13 leases to leases file. Can't install new lease database /var/db/dhcpd.leases.1433496021 to /var/db/dhcpd.leases: No such file or directory Listening on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Sending on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that befo
And this
php-fpm[85395]: /rc.newwanip: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.6 Copyright 2004-2014 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 13 leases to leases file. Listening on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Sending on BPF/em1/00:50:56:a5:77:03/192.168.10.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it bef
Comes right after this
| Jun 5 11:20:28 | check_reload_status: Reloading filter |
| Jun 5 11:20:28 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 5 11:20:28 | check_reload_status: Restarting ipsec tunnels |
| Jun 5 11:20:28 | check_reload_status: updating dyndns Yousee |
| Jun 5 11:20:28 | check_reload_status: Reloading filter |
| Jun 5 11:20:28 | check_reload_status: Restarting OpenVPN tunnels/interfaces |
| Jun 5 11:20:28 | check_reload_status: Restarting ipsec tunnels |
| Jun 5 11:20:28 | check_reload_status: updating dyndns Yousee |
| Jun 5 11:20:26 | check_reload_status: Syncing firewall |
| Jun 5 11:20:26 | check_reload_status: Syncing firewall |
| Jun 5 11:20:25 | check_reload_status: Reloading filter |
| Jun 5 11:20:25 | php-fpm[9696]: /rc.start_packages: Reloading Squid for configuration sync |
| Jun 5 11:20:25 | php-fpm[9696]: /rc.start_packages: Restarting/Starting all packages. |
| Jun 5 11:20:23 | lighttpd[33558]: (connections.c.1692) SSL (error): 5 -1 1 Operation not permitted |
| Jun 5 11:20:23 | lighttpd[33558]: (connections.c.619) connection closed: write failed on fd 20 |
| Jun 5 11:20:23 | lighttpd[33558]: (network_openssl.c.118) SSL: 5 -1 1 Operation not permitted |
| Jun 5 11:20:23 | check_reload_status: Reloading filter |
| Jun 5 11:20:23 | check_reload_status: Starting packages |
| Jun 5 11:20:23 | php-fpm[85395]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 80.197.148.74 -> 80.197.148.74 - Restarting packages. |
| Jun 5 11:20:23 | php-fpm[60486]: /interfaces.php: Removing static route for monitor 81.19.224.67 and adding a new route through 80.197.148.1 |
| Jun 5 11:20:21 | php-fpm[85395]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. |
| Jun 5 11:20:21 | check_reload_status: updating dyndns wan | -
Did you set an IP address in ESXi for that nic?
-
No it gets that from DHCP on WAN from my ISP. Hypervisor mgmt network is on the LAN.
DHCP is running on LAN issuing IP's to a Linksys wireless AP in bridge mode.
I do not run HP Jetadmin software on LAN.
-
Disabled Apinger…
All I get in the logs is this then...
-
This might be an approach for getting Dtrace working on pfsense.
https://forum.pfsense.org/index.php?topic=94838.msg527131#msg527131
Different package, but might work, bit like the only way to get Rpi's working virtually is on Ubuntu with QEMU installed to emulate the ARM's, at which point it becomes alot quicker by virtual of stonking fast hw setting up freebsd and pfsense on RPi's, but I digress.
-
Thanks man!
Over my head…. I need others to chime in on this. Spent the last mth using all my spare time to test this frigging thing.
After I did the cron job, then I can make it come alive again.
I have to settle with that right now...........
Currently migrating all the pf's in the datacenter to Mikrotik since I cant have this hosting customers. At home I dont care.
-
So is it worth starting a Getting Dtrace to work on pfsense thread?
We only have to ask the right questions, document everything we do so that others can learn from it as well if they have the thirst for knowledge.
There's plenty of examples online which show how to compile code onto platforms where a package doesnt exist for downloading.
There's generally parallels everywhere if you look. ;)
-
Posted a bounty for Dtrace package.
https://forum.pfsense.org/index.php?topic=94846.0
-
dtrace is installed already on pfSense.
Go to the FreeBSD forums now. You're not dealing with pfSense anymore when you start working with dtrace. This is the wrong place to look for that kind of guidance.
I think I'm having déjà vu….
-
OK.
But it still leaves me with no options to monitor pfsense deep under the hood with timestamps.
https://forum.pfsense.org/index.php?topic=94260.0
Any ideas?
-
You need to be more specific with what you want exactly, plus programmers are very specific and you might find Dtrace does what you want, so would it be fair for anyone to just compile and get Dtrace running?
Besides when you say what you say, have you considered the other app/conversation thread here https://forum.pfsense.org/index.php?topic=94843.msg527152#msg527152 or is your browser running single tab/single core/single thread and you have not locked/mutexed this thread and swapped to the other thread yet? Or if your browser is running multitabbed/multi core/multi threaded, have you locked this thread in your browser by switching tabs to read the other thread/multi core/multi thread if you catch my drift. ;D
There's parallels everywhere if you have an abstract look on life.
-
dtrace is installed already on pfSense.
Go to the FreeBSD forums now. You're not dealing with pfSense anymore when you start working with dtrace. This is the wrong place to look for that kind of guidance.
I think I'm having déjà vu….Significance might not have registered. Apologies. :o
-
dtrace stuff. Remember, dtrace is a component of the FreeBSD distribution, so you need to go there to understand it and get support from the community. These forums are more product-specific for pfSense–mostly dealing with networking versus deep coding. So you're questions are best answered in the FreeBSD forums.
https://wiki.freebsd.org/DTrace
https://wiki.freebsd.org/DTrace/Tutorial
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace.html
http://www.brendangregg.com/USEmethod/use-freebsd.html
-
Smoking gun?
https://forums.freebsd.org/threads/freebsd-10-1-network-unaccessible-after-high-traffic.51743/
Potential fix? (synproxy state)
https://forums.freebsd.org/threads/enabling-ipv6-eventually-freezes-the-gateway.47959/#post-267820
See #4 from this link:
http://www.onlamp.com/pub/a/bsd/2004/04/15/pf_developers.html
https://forums.freebsd.org/threads/freebsd-9-0-release-syn-flood.29071/#post-163533
Notice how all of this freakin' awesome information is coming from the FreeBSD forums. I strongly suggest you consolidate your findings in this thread and move the discussion over to the FreeBSD networking forums. They have dozens of discussions on SYN floods with some excellent guidance given.
