Given up on 2.2
-
I couldn't agree more with you guys (FOM and 154218K2)
It was frustrating to upgrade my fw from 2.1.5 to 2.2 because of all the unverified packages that reside in the repo.
I know it's a huge task to test and verify all the packages, but imo thats no excuse.Snort, Squid, squidlight, havp are the most widely used packages around and they should work no matter what upgrade it is.
Thank god for the community to help me figure out some stuff and get it back up and running again !
-
I like the idea of packages having tags next to them basically stating what versions they are compatible with. If you want to provide a over ride button then that would be nice with a warning that certain packages have know compatibility issues, similar to what Windows does.
-
Where were all these package maintainers during the beta and RC cycles?
In my case, working. Porting packages to a great project does something near to zero return.
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
-
There is one now >>>> marcelloc
GET HIM!!!!
(kidding by the way - Great work you have done… For free???)
-
For free???
Most time yes. Some times somebody remembers that I need to eat and make me a donation or create a bounty.
Something that you may not have noticed is that on pfSense move from 2.0 to 2.1, packages changed from standart freebsd pkg to pcbsd pbi. So testing and compiling packages changed from as simple as using ports to a complete change on binary location and config files and the pseudo jail.
Hard work to change a lot of ported packages to keep it working.Then 2.2 jumped to freebsd 10(finally! :)) but pbi was dead on freebsd 8 or 9 (I guess). This way to keep packages on pbi, core team had to adapt it once again. Binary and config files moved(again) and started a lot of "missing libs" alerts for files that were not missing at all and all 2.1 php package changes did not worked for 2.2. Let's start checking and compiling everything again.
Once binary startup was fine without missing libs, conf files were messed up again. Some times looking on /usr/local and other times under /usr/pbi. And what about helpers and internal binaries called by squid and postfix for example. Main binary was ok but nobody to test in deep.
This is happening since november/december 2014. I have spent much more then my free time working together with Renato and available forum members that had time to test 2.2 beta and RC(cino for example).
2.3 will finally get back to freebsd packages(now on pkgng). compile and install will be much easier on developer's labs but somebody has a clue on what will happen to package gui php,xml and inc?
Will need to be fixed up again. :)
I love this project and did my best to get it even better.
Util pbis are fine on 2.2, there's nothing(or not much) maintainers can do.
-
Thats sorta kinda what I thought.
Well - What can I say. I like the project. -
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
Well, I personally don't accept that "security issues" excuse - because breaking repositories like that will actually help avoid said issues with broken/non-maintained packages in current repository.
Something tells me the reason for refusal is totally different. -
Not this thread again… every time we put out a release there's some "sky is falling" thread.
just in the 1st few days of 2.2 being released I counted over 60 forum posts about problems with 2.15 -> 2.2.
There were tens of thousands of systems upgraded via auto-update alone in the timeframe of those 60 posts. I'll make some specific stats available in the next few days.
This is the biggest jump in base OS we've ever made, which left people more exposed to hardware-specific issues or other changes in behavior.
The upgrade issues reported essentially all fall into 5 categories.
- things that would have happened just rebooting, not related to upgrading (system no longer completes POST, hard drive is dead, etc.)
- hardware-specific FreeBSD issues. Some seem to be things that people had to muck with to make FreeBSD 8.x run (probably working around some kind of problem in 8.x), which leaves 10.x unworkable. Resetting the BIOS to factory defaults fixed that. A couple cases with the Intel "fake RAID" cards that expose both the array and the underlying disks to the OS, and GEOM in 10.x was breaking boot upon seeing the underlying disks. Had to disable GEOM there. These things mostly have workarounds. Some unfortunately not as easily, like the apparent issues with HP DL360 G3 servers.
These things happen. This is one of the reasons we stress the importance of buying hardware appliances from us for any mission-critical purposes, as you know it's going to be fully validated on the hardware you have, so you're eliminating this upgrade risk.
Still, it's a small fraction of a percent that ever see such issues. - people who don't pay attention to things clearly stated in the release and upgrade notes.
- actual regressions, of which there are some, but most are covered in #3. This is actually a really small portion of anything to do with the base system.
- misconfigurations that shouldn't have worked before but happened to, and now don't. Mobile IPsec now requiring 0.0.0.0/0 as the local network on the phase 2 is the only 2.2 example I can think of there. Though every circumstance I'm aware of here is covered in the upgrade notes.
Packages, as noted in the release notes, could be full of landmines. There are very few active package maintainers, and lots of work to keep them all functional. marcelloc has put a good deal of much-appreciated work into things as a volunteer, as has Renato (as part of being employed by us), but there aren't enough package maintainers around to keep everything up to date.
Multiple dozen people have contributed packages over the years, but almost none of those are actively maintaining the packages. That's the issue with accepting community packages, everyone then expects us to be obligated to maintain anything that anyone ever submitted. If everyone who contributed packages at some point would maintain them, things would be great. That's far from the case though.
Things will improve significantly with the switch to pkgng from a maintenance perspective, for some of the reasons marcelloc noted. And we'll differentiate or classify packages in the future in some manner. Things we support and maintain ourselves, and things from a community contributor who may disappear tomorrow and whose packages may be removed if left unmaintained. People will still moan and complain if they're removed or have issues, but hopefully that at least sets people's expectations more appropriately.
The most important thing is - read the upgrade notes before you upgrade. If you're affected by something noted there (and those are continually updated post-release as things change), then make sure you take the appropriate precautions or mitigations, or in rare instances where you can't upgrade at the moment, hold off until the next release.
https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2_Upgrade_Notes -
- And frankly, "packages are now in full control of the dev team" is exactly what did NOT help.
It may not help with debugging, but from a security standpoint it's essential.
The package binary builds are done by us (and signed by us) so that assurances can be made that they are:
1. Perfectly repeatable, not requiring any manual "hocus pocus" by a dev on his personal build box or patches that aren't in the repo source for all to see
2. Originating from our build servers, and not someone's random server in their house (that could have been hacked/compromised/infected/etc)
3. Signed by us to verify their origin
4. Hosted on our servers. We've had developers servers disappear before leaving packages broken.In olden times, if someone cloned the packages and tools repo there was -no- way they could actually generate a set of binary packages from it with any hope of working. Now, if someone clones the repos and runs the package pbi build script they (in theory) can end up with the exact same set of packages we have on our servers. +/- changes in the ports tree between runs.
So it may be tougher to debug/develop "live" in the public packages repository, but overall it's better. It requires someone on our end to kick off pbi builds as needed and merge pull requests, but it's good to have those reviewed, too.
Because it's so easily repeatable, it's also better for developers to build and test locally, so they can know it will work the same for everyone once it is committed, built, and available for everyone.
PBIs on 2.2 are kind of a bust though. pkgng will be better, but by the time pkg came along, it was too late to get into 2.2.
-
@cmb:
The most important thing is - read the upgrade notes before you upgrade. If you're affected by something noted there (and those are continually updated post-release as things change), then make sure you take the appropriate precautions or mitigations, or in rare instances where you can't upgrade at the moment, hold off until the next release.
https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2_Upgrade_NotesI couldn't agree with you more! I read the release notes and planned ahead.. https://forum.pfsense.org/index.php?topic=87365.msg479666#msg479666
I've been in the IT field for too long to know upgrades can and will have issues. You have to plan for this.. Since I use a few packages and didn't want to upgrade without knowing they worked. I fired up a VM of 2.2RC and started to test the waters… If I found issues, I would try to see if I could fix it somehow and open up a bug report. In that report I would provide what is broke, logs, and how I was able to temporary correct it; in case it helps the developer make a permanent fix the issue.
Knowing who some of the community packager maintainers are (marcelloc, bbcan117, bmeeks to name a few: Thank you for all your hard work!!), I'll reached out to them to see if I could assist with testing. I'm no programmer but I sure can beat the crap out of programs and hardware. Oh, and I suggest a few bells and whistles to improve it (ask bbcan117, poor guy gets no sleep because of me).
Yeah there are days I want to bitch on the forum for something not working, or a change happen without a RFC... But in the end, it gets you no where. Open a bug report, reference that bug report on the forum and move on.. Check back in a couple of days... Not 30 minutes...
ok back to work for me
Stephen
-
I wouldn´t even considered upgrading if I had seen some of the packages didn´t work with 2.2.
Then you didn't read the release announcement, much less the upgrade notes. Blindly upgrading things isn't a good idea, please read them in the future.
- And frankly, "packages are now in full control of the dev team" is exactly what did NOT help.
Huh? No such changes were made at all. Packages must use binaries built on our build servers, but that's as simple as telling it what port and options to build. Pull requests off github accommodate that process, and have for quite some time, well before 2.2. That's for security and maintainability reasons. In the early days, we weren't as strict about that, some package maintainers had people pull binaries from their own systems. That's the only tightening of control that we've done on packages, and that happened years ago.
-
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
That was relevant to having people point their systems to package servers other than our own, to avoid situations like the lusca stupidity where people are having their package repos pointed to some free web hosting service that's continually hosting a slew of malware. Then their packages break because no one's updated what is on that server in years.
I expect we'll do something along the lines of official/unofficial classifications or similar, along the lines of what I noted in my earlier post.
-
Even though PFSense is a firewall, it also does some routing and supports some tunneling like OpenVPN, but I don't get the whole installing a ton of these packages. People try to make their "firewall" do way too much crap. Asking for the kitchen sink is asking for trouble. Firewalls are critical infrastructure. It needs to do one job and do it well.
Need a proxy? Set up a proxy server. Need an IDS? Set up a passive IDS that watches the traffic and turns off network ports when bad traffic is detected.
By putting all the responsibility on one device, you're making all of those features co-dependent on each-other. Distinctly separate services should be distinctly separate machines/vms.
As for "bugs". PFSense 2.2 had 0 open bugs the day before release and had almost no new bugs for months, and gained 40+ in 1-2 days. Obviously the people testing 2.2 beta weren't reporting bugs or weren't having bugs. Seems most of the people with crippling issues have some strange configurations, strange/old hardware, Xen doing crazy things, and a few actually being PFSense related.
1. Don't use hardware raid
2a. Why are you using VMs?
2b. It's a VM, snapshot it and test it
3. Your Firewall is not meant to be an file serving, database running, web hosting, time serving, web caching, load balancing monstrosity.I don't even want to run NTPD or OpenVPN on mine because that's outside the scope, but it is a nicety that allows me to cut corners at home and not do things correctly.
I'm not saying that PFSense can get away with some things like if OpenVPN stopped working and they were too lazy to fix it in a timely fashion, but as someone who has a background in security at all levels of computer systems, and network design, I don't blame others when things go wrong because I design my entire network to hinge on a single device that is forced to do way more than it should, and I didn't modularize my network services.
I know I took a gamble when I upgraded to 2.2, but I did my research, figured out my risks, had a back-up plan, and asked the wife when she could handle being without the Internet for an hour or two as I messed around. When I built my machine, I made sure it was all standard parts that have nothing special, but will be around. AHCI harddrives, no special drives required. Intel CPU, nothing special. Intel NIC that is recent, will be supported beyond its usefulness. Video card, integrated into the CPU and is well supported as fully opensource drives.
I leave as little to chance as possible.
-
I do agree some of your points, but in today's world nobody apart from big corporate will put a box for every other function. Pfsense as a firewall is good, but basic function like web filtering http/https is part of the utm device, which pfsense project claim to have.
It should work perfectly and without pain (not requiring a patch and workaround for everything), and administrator should take responsibility for atleast the basic function, otherwise, most of the good users will move away from this project.
I feel administrators should look into these basic function more seriously on urgent.
-
How I do upgrades in most cases follows these rules:
- Does the new release have some feature I cannot live without which the old release is missing? if no there is no need to do an imminent upgrade - wait for X.1 release
- Create a test environment - a clone of your current setup, and try the upgrade of this test environment. Work with this test environment until you are convinced that you know its ins and outs.
- If the test environment is not the same hardware as your current appliance try to make a boot on your current system from CD/USB/PXE to see that all components are recognized and functioning.
- When you finally initiate your upgrade make a full backup as part of the upgrade (The checkbox 'Make a full backup before upgrade')
- If you are using CF or some kind of limited space environment buy a similar device and clone your current setup to this. Then install the clone and upgrade on the clone
Above list should ensure minimal downtime and guaranty you always have fallback plan.
-
I do agree some of your points, but in today's world nobody apart from big corporate will put a box for every other function. Pfsense as a firewall is good, but basic function like web filtering http/https is part of the utm device, which pfsense project claim to have.
Can you share a link where ESF has claim pfSense is a UTM? I dont recall them saying that, but the community has.
