Buffer overruns, license violations, and bad code: FreeBSD 13’s close call

KOM

Further writeup of the Netgate-FreeBSD-Wireguard story.

At the end of the day, I just want a working implementation of Wireguard. I appreciate that Netgate stepped up to make it happen initially. I was dismayed to see them score an own-goal over the PR blowback. Looks like it's going to be awhile before Wireguard makes its way back into the kernel.

Gertjan

@kom
I'm very convinced that we will have that 'Wireguard'. After all, it already exists on other OS's.

On the positive side, I see : only some ego's are hurt - exploits have been found, but their impact has been limited if it isn't plain zero - The review process has been kick-started

I do compare all this with the first "MSDOS" or the first "Windows", or any other new functionality : it is good, great and new. Later on we knew that things needed to be redone. New versions were needed. New is never good or finished. It's just a start. From "nothing" to "something".
What counts : the first step had to be made. Now, all what's needs to be done : progressing.

kiokoman

@gertjan
yeeeah ... we can always take a saw and drill a hole in the floor of the netgate office and get them to work on it

maverickws

I had read that article and that did indeed made me wonder about a few things.

I see as inadmissible that Netgate issued public responses accusing of "irrational bias against mmacy and Netgate and irresponsible disclosure of "a number of zero-day exploits";
It is Netgate's fault the lack of concern for code quality, that they didn't properly review the code before being put to production;
It is Netgate's fault that relied on ONE person to make the port, without a proper testing and review chain;
It is Netgate's fault for issuing public announcements saying no vulnerabilities existed;
And ultimately, it is Netgate's fault for making millions of users vulnerable to exploits, by taking beta code from a pre-release candidate and putting it on pfSense 2.5.0.

One can argue about FreeBSD's code quality workflow, but Netgate is a private company with increased responsibility and should be properly prepared to ensure the best version of code is implemented onto their OS. That clearly didn't happen here, and can only leave one to wonder, how well is pfSense code reviewed overall, when so many people are moving away to other solutions.

It is quite concerning that, instead of apologising for what happened and assuming the bad decisions made, Netgate instead prefers to argue that Donenfeld was making too much fuss over nothing, and saying mmacy's code wasn't that bad. It was that bad, a public apology is due, and a visible effort to fix all the security issues should be in place.

“The difference between greatness and mediocrity is often how an individual views a mistake.”
Nelson Boswell

Cool_Corona

@maverickws said in Buffer overruns, license violations, and bad code: FreeBSD 13’s close call:

I had read that article and that did indeed made me wonder about a few things.

I see as inadmissible that Netgate issued public responses accusing of "irrational bias against mmacy and Netgate and irresponsible disclosure of "a number of zero-day exploits";
It is Netgate's fault the lack of concern for code quality, that they didn't properly review the code before being put to production;
It is Netgate's fault that relied on ONE person to make the port, without a proper testing and review chain;
It is Netgate's fault for issuing public announcements saying no vulnerabilities existed;
And ultimately, it is Netgate's fault for making millions of users vulnerable to exploits, by taking beta code from a pre-release candidate and putting it on pfSense 2.5.0.

One can argue about FreeBSD's code quality workflow, but Netgate is a private company with increased responsibility and should be properly prepared to ensure the best version of code is implemented onto their OS. That clearly didn't happen here, and can only leave one to wonder, how well is pfSense code reviewed overall, when so many people are moving away to other solutions.

It is quite concerning that, instead of apologising for what happened and assuming the bad decisions made, Netgate instead prefers to argue that Donenfeld was making too much fuss over nothing, and saying mmacy's code wasn't that bad. It was that bad, a public apology is due, and a visible effort to fix all the security issues should be in place.

“The difference between greatness and mediocrity is often how an individual views a mistake.”
Nelson Boswell

Problem is that they push IT-security software solutions.... And the trust in quality would vanish if they apologised and admittingly leveraged the fact that quality is lacking and bad code gets through scrutiny.

What would the impact be on all the clients and people that run PFsense and trust it to be secure??

Business gone over night....

maverickws

@cool_corona so, you really think its better to make sloppy announcements and trying to dodge the bullet while the whole internet and community gets a hold of what happened here?
The issues surfacing are undeniable and will only grow bigger. IMHO, denying the obvious will wear the confidence much rapidly than assuming that an error was made, identified, and all efforts are being put to mitigate and solve the issue so it doesn't happen again.

But well, that's my opinion. I don't think Netgate needs to admit it for the majority to be aware of what happened.

Cool_Corona

@maverickws Its not about pointing the finger but denying it happened is not good for business hence the fact of overwhelming evidence.

Face it, admit it, learn from it, fix it and move on.

We like Pfsense and the community is strong.

Even though Netgate is the "hub" for pfsense, it can be forked and continued elsewhere if it comes down to that.

Its just a matter of ressources and knowledge.

maverickws

I've been using pfSense for years on end and also a fan, that's not the point. And since we're at it, I don't care about Wireguard at all.

As I said, it was the behaviour adopted by Netgate that got me worried and wondering.
"Face it, admit it, learn from it, fix it and move on." that's what would be nice to see here.

When you say "the community is strong" is that like some sort of Jedi feature that will protect pfSense software and its customers? lmao. What exactly does a strong community means in terms of security when upstream issues exist?

A Former User

@maverickws said in Buffer overruns, license violations, and bad code: FreeBSD 13’s close call:

@cool_corona so, you really think its better to make sloppy announcements and trying to dodge the bullet while the whole internet and community gets a hold of what happened here?
The issues surfacing are undeniable and will only grow bigger. IMHO, denying the obvious will wear the confidence much rapidly than assuming that an error was made, identified, and all efforts are being put to mitigate and solve the issue so it doesn't happen again.

But well, that's my opinion. I don't think Netgate needs to admit it for the majority to be aware of what happened.

I'm inclined to agree with you. That said:

The number of people who are aware of the issue is small.
Of those aware the number that understand the impact is smaller yet.
Of those the number who see this as one example of a far greater issue is even smaller.

But if even a small number of people have their eyes open a bit wider, that's good.

More typically, it's Bob the IT guy at widget corp who may or may not be concerned by these events. What's he or she going to do? Run off and replace pfsense with opnsense or Vyos? Unlikely.

A MSP (managed service provider) who has pfsense deployed at some number of client sites is also unlikely to just up and abandon it. Doing so is a hard sell to clients who expect stuff to just work and who don't understand or care about the details.

For sure there is a problem. Not just with pfsense and Netgate. Complicated software systems are hard and people are people. No matter how you look at this, open or closed source, you are trusting something and the people responsible for that thing.

Those running a business are commonly obsessed with cost which has allowed this bigger issue (software quality) to go unaddressed. If Bob is cheap, Bob is good...

Me? I'm moving on. Netgate doesn't care about that because I was never going to spend significant money with them and I'm not some craptube influencer. New stuff is on the way and I'll pull the plug on pfsense by weeks end.

maverickws

@jwj I agree overall with your reply and share a similar opinion. But,
About the reach, I'm not sure: professionals and tech savvy people will sooner or later get a hold of the issue.

About the impact, I would say it depends on various factors. But I would say that for both Bob the IT guy as for the MSP who has pfSense, it may start to ring some bells. I would say the approach to the issue taken by Netgate will determine for how long the bells will be heard. If you evaluate the response as capable and responsible, probably they won't make too much noise. You already had confidence on the solution and the company. They had a response that you evaluated as appropriate, identified the issue, assumed it and dealt with it, making a commitment so it doesn't happen again. Good.

The wrong approach to the issue will keep these bells ringing. It will wear down the confidence both on the product and on the company behind it. It will open your ears to other comments and opinions on the subject, and probably, with time, will make you evaluate other solutions from companies or projects (even if that solution is OpnSense or Vyos for example). The shift may not be immediate, but will keep that door open a lot, and all because of the way the problem was dealt with.

I'm not thinking of moving away just yet, just felt like commenting on this subject.

A Former User

@maverickws said in Buffer overruns, license violations, and bad code: FreeBSD 13’s close call:

professionals and tech savvy people

Are too few and far between. These calls are made by bean counters not IT (and certainly not security) pros.

The companies with real professionals aren't playing in this space. They are running a custom IOS image because they spend huge money with Cisco (or the Juniper or HPE equivalent). Maybe they are running custom HW and SW like the big cloud providers or surveillance capitalism companies...

I'm not moving away from pfsense because I think the grass is greener elsewhere. Not at all. I move on because I can purge at least two bad people from Netgate from my world. That's admittedly a luxury, pfsense and Netgate occupy a vanishingly small space in my world. I have time for this because the pandemic has made the world temporarily smaller.

maverickws

Actually I know a few big companies, datacenter providers and ISP that have moved away from Cisco, HPE and Fortinet to pfSense (or have mixed environments) - and they did spend quite a bit on Netgate appliances.

Depends on management but also on the autonomy of tech departments to make their own choices, and being responsible for the resources allocated to each project. Bean counters define the budget, they leave the tech related decisions for someone else.
Amazon is a rather expensive solution.

I had visibility over a project of a big automotive manufacturer on Germany that decided to go for Amazon Web Services, made the developers (they did have maybe up to a thousand developers (or more even)), had them shifting a lot to there, spent many hours learning about AWS, moved a lot of resources, but when the bills started coming the advocates of AWS that proposed it as solution got scolded, quickly disinvested from Amazon and instead went back for their own DC rooms/colocation etc., solutions they were going to decommission. They didn't get everything off AWS, ofc, but AWS or alike solutions are good for some use-cases, not all - personally I perceive it as few.

A Former User

@maverickws said in Buffer overruns, license violations, and bad code: FreeBSD 13’s close call:

Actually I know a few big companies, datacenter providers and ISP that have moved away from Cisco, HPE and Fortinet to pfSense (or have mixed environments) - and they did spend quite a bit on Netgate appliances.

That certainly is what Netgate hopes to do more of. Not helping themselves at the moment...

Do you think open source (for core functionality, not just libreoffice) has gained more traction in Europe than in other regions? Generally a more open minded environment.

maverickws

@jwj honestly I haven't given a thought about that for a while. I am located in Europe, I've worked and traveled more within Europe than outside of Europe, so I would say my experience can lead me to a biased opinion on that, although being aware that are many interesting companies in the US that also invest have interest in open-source and contribute to open-source.

I would say it depends more on managerial culture, hiring requirements for the professionals that are hired for leadership on tech departments. It always derives from the reach and vision of the people in charge.
Its a complicated subject!

Cool_Corona

Europe is moving away from american software in any regard.

That is the trend here in Scandinavia.