What is the biggest attack in GBPS you stopped

Supermule

Could very well be.

tim.mcmanus

So you haven't tested it? It's more of a definite maybe that this resolves the issue?

Supermule

Not yet.

@tim.mcmanus:

So you haven't tested it? It's more of a definite maybe that this resolves the issue?

firewalluser

@Supermule:

https://lists.freebsd.org/pipermail/freebsd-announce/2015-July/001655.html

Latest update.

I suppose another question to ask, is how did we miss this on our own machines, and what can we do to avoid such problems from occurring again?

Harvy66

Supermule didn't miss it. Well, possibly. If it turns out to be the same issue.

firewalluser

Spotted the behavior but nothing outputted from pfsense to observe though.

tim.mcmanus

@firewalluser:

Spotted the behavior but nothing outputted from pfsense to observe though.

Because it's not a pfSense issue. This is an FreeBSD network driver issue.

firewalluser

So there is nothing we can do then as this is just like a HW failure of sorts then?

tim.mcmanus

@firewalluser:

So there is nothing we can do then as this is just like a HW failure of sorts then?

Yes and no.

If you can code and want to help recode the FreeBSD network drivers, they could use the help. They've acknowledged that there will be more improvements in the networking layer in v11.

Because pfSense is built on top of FreeBSD, it's only as good as the underlying operating system code. So bugs/deficiencies in the OS code affect the applications running on it. So we can't expect the pfSense team to fix a problem that's not in pfSense, and they are at the mercy of the FreeBSD code releases. I know they regularly collaborate with the FreeBSD team, but they don't run that project.

So it's not hardware per se, there are two layers of code that are maintained by two separate projects.

Here are the two threads Supoermule opened up on the FreeBSD forums before they started ignoring him:

https://forums.freebsd.org/threads/dos-and-ddos-attacks.51899/

https://forums.freebsd.org/threads/freebsd-pf-and-syn-ack-flooding.51921/

With any software project, you need to provide very detailed information to the developers so they can ascertain what the root cause may be as well as any code you're using to trigger the issue. While Supermule provides a lot of data, it's not data the developers or forum users found useful, similar to what's in this thread.

So it's possible the revised FreeBSD code resolves something, but until it's tested against this use case appropriately (on bare metal) it's only a guess as best that it resolves this use case.

cmb

@Supermule:

https://lists.freebsd.org/pipermail/freebsd-announce/2015-July/001655.html

Latest update.

That has no relation to things people have been attempting here. It only applies to sessions the system itself answers (and gets all the way to LAST_ACK state, which never happens in any of these tests). No applicability with things it's routing, or NATing, or blocking, or passing but not able to complete a TCP handshake much less get to LAST_ACK.

2.2.4 snapshots have had the patch since the first build after its release, yesterday morning. Release is coming hopefully tomorrow, though not because of that specifically, as it's non-applicable for the vast majority of use cases.

Note the credit to Netflix? Probably something they ran into by coincidence on their FreeBSD CDN boxes (when you're pumping 20-40 Gbps out of a single server across a huge number of connections, you tend to run into any possible TCP bugs), then found the associated potential security impact.

jdillard

@cmb:

Note the credit to Netflix? Probably something they ran into by coincidence on their FreeBSD CDN boxes (when you're pumping 20-40 Gbps out of a single server across a huge number of connections, you tend to run into any possible TCP bugs), then found the associated potential security impact.

As an aside, you can watch Gleb Smirnoff talk about why Netflix decided to use NGINX and FreeBSD to build their own CDN: https://www.youtube.com/watch?v=KP_bKvXkoC4

KOM

So we can't expect the pfSense team to fix a problem that's not in pfSense, and they are at the mercy of the FreeBSD code releases.

While we can't expect them to, they certainly have fixed some FreeBSD bugs and submitted the patches upstream, which were accepted for inclusion by the FreeBSD team.

Supermule

Exactly. But nothing….

Let wait and see if it works.

Derelict

Why would it?

tim.mcmanus

@Supermule:

Exactly. But nothing….

Let wait and see if it works.

ivor

@tim.mcmanus:

Here are the two threads Supoermule opened up on the FreeBSD forums before they started ignoring him:

https://forums.freebsd.org/threads/dos-and-ddos-attacks.51899/

https://forums.freebsd.org/threads/freebsd-pf-and-syn-ack-flooding.51921/

I don't see their response as ignoring https://forums.freebsd.org/threads/freebsd-pf-and-syn-ack-flooding.51921/#post-291316

firewalluser

@tim.mcmanus:

Because pfSense is built on top of FreeBSD, it's only as good as the underlying operating system code. So bugs/deficiencies in the OS code affect the applications running on it.
Snip

I understand all of this.

So it's possible the revised FreeBSD code resolves something, but until it's tested against this use case appropriately (on bare metal) it's only a guess as best that it resolves this use case.

The point I'm angling at, is this, unless we have better logging facilities for all and any output from freeBSD and the pfsense elements sat on top, we wont be able to spot the problems will we?

Or am I missing something?

tim.mcmanus

@firewalluser:

The point I'm angling at, is this, unless we have better logging facilities for all and any output from freeBSD and the pfsense elements sat on top, we wont be able to spot the problems will we?

Or am I missing something?

The suggestions were made in this thread.

The underlying issue–which has yet to be determined--requires dtrace to be installed.

And since this is a FreeBSD issue, per recommendations also in this thread, one would need to install a current release of FreeBSD on bare metal with dtrace installed to capture the appropriate data from the kernel extension/module in question.

This is far beyond logging and more code debugging, hence the use of dtrace.

Additionally, Supermule has never released the code that causes the issue and adamantly refuses to, so no one can recreate the issue in a lab. So anyone who wants to legitimately troubleshoot the use case and capture data from FreeBSD can't. This stubborn refusal is one of the reasons no developer on these forums or on FreeBSD's forums will help with the issue.

firewalluser

I thought DTrace was getting nowhere because it requires ESF to include it in the version of freebsd they use with pfsense?

It is looking like v11 will need to be installed and pfsense inserted on top which is something I'm hoping to do on the rpi (as much to see the performance abilities or not)once I have a few other jobs out of the way, but I was also hoping more progress had been made on Dtrace but I think thats stalled.

Code debugging is still logging in my books, I ship all my apps in full debug mode as I've spent enough time in the past hunting down problems in code sometimes my own, sometimes 3rd party addons, made harder when some of it is black boxes and not source code provided.

I think he provided a DDOS script which was supposed to cause the problems although I havent seen it myself as my bandwidth is not enough unless GCHQ/isp have some speed restriction in place, but I cant comment on the DDOS script other than its not unlike many which can be downloaded from the web.

Harvy66

There is a difference in logging high level stuff and logging system calls. An OS can be handling thousands of calls per seconds or more. Logging kernel level stuff is much more difficult.