Given up on 2.2
-
Enough is enough! After a week of struggling with upgrade, fresh install, packages etc I´ve had it. The real killer blow came when a NAT change suddenly exposed my WAN IP instead of the VPN one. Had to reboot to get it working again. Seems the change completely broke the NAT and rewrote the ruleset.
Next was postfix and Lightsquid which won´t work either. I noticed some workarounds but don´t like that.
ClamAV was next and couldn´t get that to work either.
The DNS changes with forwarder as well as resolver are unclear as to how they are supposed to coexist or not and ended in me stopping them both since I noticed leaks.
The reverse proxy was another I couldn´t get to work. It just wasn´t there no matter what I did.I will leave 2.2 for now and wait for some serious fixes before I go near it again! To me it appears it wasn´t tested enough.
I would also like to add that I´ve been running 2.1.x for a long time and been very happy with it. -
How many of these issues have you reported?
-
How many of these issues have you reported?
None, never been active here except reading! I´ve done upgrades before but never had this many issues and time is limited…
-
Enough is enough! After a week of struggling with upgrade, fresh install, packages etc I´ve had it. The real killer blow came when a NAT change suddenly exposed my WAN IP instead of the VPN one. Had to reboot to get it working again. Seems the change completely broke the NAT and rewrote the ruleset.
Next was postfix and Lightsquid which won´t work either. I noticed some workarounds but don´t like that.
ClamAV was next and couldn´t get that to work either.
The DNS changes with forwarder as well as resolver are unclear as to how they are supposed to coexist or not and ended in me stopping them both since I noticed leaks.
The reverse proxy was another I couldn´t get to work. It just wasn´t there no matter what I did.I will leave 2.2 for now and wait for some serious fixes before I go near it again! To me it appears it wasn´t tested enough.
I would also like to add that I´ve been running 2.1.x for a long time and been very happy with it.154218K2 - I have to agree with you 100%. I am glad that the devs are pushing forward, but just in the 1st few days of 2.2 being released I counted over 60 forum posts about problems with 2.15 -> 2.2. I know there are always issues with new version, but this seems excessive. From some of the security news I came across (2 weeks ago'ish), I saw that there were some Openvpn, and some other TLS Security Announcements that came out which I believe applied to PFsense. I was hoping that the fix(es) were NOT going to be rolled into V2.2, because with those fixes come tons of bricks. V2.1.5 has been working pretty well. I think we could have used the security fixes, and allowed V2.2 to stay in the oven a bit longer. None the less I will be waiting until I see most of these V2.2 problems are fixed before I even think about upgrading. Just wondering if Pfsense is ever going to adopt the same FreeBSD/FreeNas Multiple Boot Volumes, so as to make it extremely easy to revert back to a different boot environment if the upgrade is crap. I think this came out in FreeBSD Version 9 something. Anyways I feel your pain. I will continue to check this thread to see how the state of 2.2 is going. Thanks
~Ash
-
Thumbs up for 2.2!
I switched our productive environment and really like it! i Just needed one workaround on ipsec and posted my fix in ipsec forum.my honest opinion on those "whiners" - forum newbies, if you would have spend some time to focus on beta, test and report problems or even post some useful debug and error reporting information on the troubles you have, you would help this project much more. i know i feed the trolls :)
154218K2: "To me it appears it wasn´t tested enough." oh, rly? your fault 154218K2 :)
pfsense team, i love pfsense, keep up the good work!
cheers
palu
-
"Next was postfix and Lightsquid which won´t work either"
Since when is it the responsibility of the pfsense developers to make sure packages work?? If you want to have a problem with someone - track down people creating/maintaining those packages. Same goes for clamav and reverse proxy..
Who said the resolver and forwarder were suppose to coexist? And what leaks did you notice?
Sorry but I feel no pain for anyone that blindly updates a production system to a brand new release, and then complains that something you use to do no longer works.. Where do you work that you could go to new release of anything without a backout plan.. If you use feature X of systems - first thing would be validate feature X works as it did before or better before moving that into production.
-
Since when is it the responsibility of the pfsense developers to make sure packages work??
I guess I'm in the minority in believing that a package that is offered via the pfSense package repository should actually work when installed without hacks and workarounds. To use his example, Lightsquid wouldn't work until you do the following:
ln -s /usr/pbi/lightsquid-amd64/local/www/lightsquid /usr/local/www/lightsquid ln -s /usr/pbi/lightsquid-amd64/local/etc/lightsquid /usr/local/etc/lightsquid pkg install perl5 pkg install p5-gd /usr/bin/perl /usr/pbi/lightsquid-amd64/www/lightsquid/lightparser.pl todayExpecting users to figure this out on their own is absurd.
-
None of this surprises me…
A release always happens
People install and works for most but there are always package issues and other issues once a large enough base has started using new release.
Then the packages get updated
Minor release with fixes of pfsense gets pushed.
And then its solid.Thats how 2.1 got to be 2.15 I think and I'm pretty sure I heard all the same sorts of complaints going into 2.1
Anyway - If something isn't working for a percentage of people, I'm sure its just a short waiting game before its fixed.
So if its a critical bug for you, roll back to last working version and wait the fixes.Thats my guess anyway.
-
I would second KOM's comment - since packages are now in full control of the dev team - they are built, hosted and toolkit access controlled by them - it becomes pfSense's team responsibility that packages work without an error with basic config on clean install. And looking through the forum, it does not seems to be the case :(
I like pfSense and I greatly respect development team's work, but really hope that they can look into packages issue…
If it would be me, installing and owning some 3rd party package through pkg_add - I wold not complain, because I always can go and do some searching, test different versions, etc... But with pfSense packages it is almost impossible... Once can try and troubleshoot and post some workaround, but there is no guarantee that anyone will be looking into implementing it in the next release... I filed bugs for packages before, no one cares fixing them. -
I hear you, Kejainshi, but Lightsquid and Sarg have been broken since I started using pfSense more than a year ago – long before 2.2. I'm not going to rant about it (again), but it doesn't look good on the project to have common packages broken on install for a long time.
-
@KOM:
I hear you, Kejainshi, but Lightsquid and Sarg have been broken since I started using pfSense more than a year ago – long before 2.2.
Yes. So, there's actually no 2.2 regression then, no? :D :D :D
-
lightsquid worked just fine for me before 2.2
I just uninstalled it and squid dansguardian and the rest because I saw no continuing need to filter my kids web when he turned 13.Thank god too… Those packages do not make the internet more reliable.
I've had to roll back once or twice in the past, wait 3 months for an update and move forward also.
I half expect it with any new release of any OS or firmware.
-
If it's important to you:
-
Document what you see as best you can.
-
Open a bug report.
-
Roll back to 2.1.5.
-
Watch redmine to see the progress of your issue.
-
Proceed back to 2.2.X when you think it's safe to do so.
-
-
The best I would hope for, is that before packages can be added that they have to be validated. So when new release comes out there are no packages until the package creators/maintainers show that it works for that release.
So when release.x comes out all packages are removed and not available to install until the makers of said package get it validated for release.x - that would for sure be a win win for everyone involved if you ask me IMHO..
My point is the developers are not coding for all the packages, they are coding for the core.. To expect them do make sure their code doesn't break any package is also absurd.
-
Where were all these package maintainers during the beta and RC cycles?
-
Where were all these package maintainers during the beta and RC cycles?
- Some stuff has not been touched for ages.
- Some maintainers are gone.
- And frankly, "packages are now in full control of the dev team" is exactly what did NOT help.
-
Where were all these package maintainers during the beta and RC cycles?
Exactly. I also realize that some of the packages are maintained by volunteers, but that's not optimal from a corporate standpoint to be reliant on Internet Joe to keep your packages validated and up to date.
-
Nothing ever changes :-) For over 20 years it´s been the same. My own developers call me a whiner when I complain about things not working or them not testing stuff properly…
My hope is that something may be learned and a new test and release procedure may help avoid this in the future. The idea to hold back unverified packages until tested is a good one. It would have saved me a lot of time and I wouldn´t even considered upgrading if I had seen some of the packages didn´t work with 2.2.
I actually did check what packages were available after the 2.2 fresh install and noticed they were all marked with 2.2 so I (naively) figured they were verified to some extent!
I also become a bit concerned by the comments that the packages are left alone and no one actually cares about them or touches them. I love PfSense but a leftover package is a huge security risk and may compromise the whole system! This may lead to demands for more secure options in a production environment. I´m also willing to bet a majority of the userbase have some packages installed and not only uses the core system.
Finally, please don´t regard this as whining, instead find a new way to make it better and keep up the good work that made me go with PfSense from the beginning!
Regards
-
Personally, I don't want 2.2 held back because of a problem with some stupid package with a long-absent maintainer that someone probably doesn't really need. They can stay on 2.1.5 until their package is supported or go to another solution.
I do like the idea of packages not showing up in available packages until they have a decent shot at working.
-
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
This way people who want to risk and test can still do that. And people, who need prod-level stability will be able to judge better before upgrading. -
I couldn't agree more with you guys (FOM and 154218K2)
It was frustrating to upgrade my fw from 2.1.5 to 2.2 because of all the unverified packages that reside in the repo.
I know it's a huge task to test and verify all the packages, but imo thats no excuse.Snort, Squid, squidlight, havp are the most widely used packages around and they should work no matter what upgrade it is.
Thank god for the community to help me figure out some stuff and get it back up and running again !
-
I like the idea of packages having tags next to them basically stating what versions they are compatible with. If you want to provide a over ride button then that would be nice with a warning that certain packages have know compatibility issues, similar to what Windows does.
-
Where were all these package maintainers during the beta and RC cycles?
In my case, working. Porting packages to a great project does something near to zero return.
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
-
There is one now >>>> marcelloc
GET HIM!!!!
(kidding by the way - Great work you have done… For free???)
-
For free???
Most time yes. Some times somebody remembers that I need to eat and make me a donation or create a bounty.
Something that you may not have noticed is that on pfSense move from 2.0 to 2.1, packages changed from standart freebsd pkg to pcbsd pbi. So testing and compiling packages changed from as simple as using ports to a complete change on binary location and config files and the pseudo jail.
Hard work to change a lot of ported packages to keep it working.Then 2.2 jumped to freebsd 10(finally! :)) but pbi was dead on freebsd 8 or 9 (I guess). This way to keep packages on pbi, core team had to adapt it once again. Binary and config files moved(again) and started a lot of "missing libs" alerts for files that were not missing at all and all 2.1 php package changes did not worked for 2.2. Let's start checking and compiling everything again.
Once binary startup was fine without missing libs, conf files were messed up again. Some times looking on /usr/local and other times under /usr/pbi. And what about helpers and internal binaries called by squid and postfix for example. Main binary was ok but nobody to test in deep.
This is happening since november/december 2014. I have spent much more then my free time working together with Renato and available forum members that had time to test 2.2 beta and RC(cino for example).
2.3 will finally get back to freebsd packages(now on pkgng). compile and install will be much easier on developer's labs but somebody has a clue on what will happen to package gui php,xml and inc?
Will need to be fixed up again. :)
I love this project and did my best to get it even better.
Util pbis are fine on 2.2, there's nothing(or not much) maintainers can do.
-
Thats sorta kinda what I thought.
Well - What can I say. I like the project. -
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
Well, I personally don't accept that "security issues" excuse - because breaking repositories like that will actually help avoid said issues with broken/non-maintained packages in current repository.
Something tells me the reason for refusal is totally different. -
Not this thread again… every time we put out a release there's some "sky is falling" thread.
just in the 1st few days of 2.2 being released I counted over 60 forum posts about problems with 2.15 -> 2.2.
There were tens of thousands of systems upgraded via auto-update alone in the timeframe of those 60 posts. I'll make some specific stats available in the next few days.
This is the biggest jump in base OS we've ever made, which left people more exposed to hardware-specific issues or other changes in behavior.
The upgrade issues reported essentially all fall into 5 categories.
- things that would have happened just rebooting, not related to upgrading (system no longer completes POST, hard drive is dead, etc.)
- hardware-specific FreeBSD issues. Some seem to be things that people had to muck with to make FreeBSD 8.x run (probably working around some kind of problem in 8.x), which leaves 10.x unworkable. Resetting the BIOS to factory defaults fixed that. A couple cases with the Intel "fake RAID" cards that expose both the array and the underlying disks to the OS, and GEOM in 10.x was breaking boot upon seeing the underlying disks. Had to disable GEOM there. These things mostly have workarounds. Some unfortunately not as easily, like the apparent issues with HP DL360 G3 servers.
These things happen. This is one of the reasons we stress the importance of buying hardware appliances from us for any mission-critical purposes, as you know it's going to be fully validated on the hardware you have, so you're eliminating this upgrade risk.
Still, it's a small fraction of a percent that ever see such issues. - people who don't pay attention to things clearly stated in the release and upgrade notes.
- actual regressions, of which there are some, but most are covered in #3. This is actually a really small portion of anything to do with the base system.
- misconfigurations that shouldn't have worked before but happened to, and now don't. Mobile IPsec now requiring 0.0.0.0/0 as the local network on the phase 2 is the only 2.2 example I can think of there. Though every circumstance I'm aware of here is covered in the upgrade notes.
Packages, as noted in the release notes, could be full of landmines. There are very few active package maintainers, and lots of work to keep them all functional. marcelloc has put a good deal of much-appreciated work into things as a volunteer, as has Renato (as part of being employed by us), but there aren't enough package maintainers around to keep everything up to date.
Multiple dozen people have contributed packages over the years, but almost none of those are actively maintaining the packages. That's the issue with accepting community packages, everyone then expects us to be obligated to maintain anything that anyone ever submitted. If everyone who contributed packages at some point would maintain them, things would be great. That's far from the case though.
Things will improve significantly with the switch to pkgng from a maintenance perspective, for some of the reasons marcelloc noted. And we'll differentiate or classify packages in the future in some manner. Things we support and maintain ourselves, and things from a community contributor who may disappear tomorrow and whose packages may be removed if left unmaintained. People will still moan and complain if they're removed or have issues, but hopefully that at least sets people's expectations more appropriately.
The most important thing is - read the upgrade notes before you upgrade. If you're affected by something noted there (and those are continually updated post-release as things change), then make sure you take the appropriate precautions or mitigations, or in rare instances where you can't upgrade at the moment, hold off until the next release.
https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2_Upgrade_Notes -
- And frankly, "packages are now in full control of the dev team" is exactly what did NOT help.
It may not help with debugging, but from a security standpoint it's essential.
The package binary builds are done by us (and signed by us) so that assurances can be made that they are:
1. Perfectly repeatable, not requiring any manual "hocus pocus" by a dev on his personal build box or patches that aren't in the repo source for all to see
2. Originating from our build servers, and not someone's random server in their house (that could have been hacked/compromised/infected/etc)
3. Signed by us to verify their origin
4. Hosted on our servers. We've had developers servers disappear before leaving packages broken.In olden times, if someone cloned the packages and tools repo there was -no- way they could actually generate a set of binary packages from it with any hope of working. Now, if someone clones the repos and runs the package pbi build script they (in theory) can end up with the exact same set of packages we have on our servers. +/- changes in the ports tree between runs.
So it may be tougher to debug/develop "live" in the public packages repository, but overall it's better. It requires someone on our end to kick off pbi builds as needed and merge pull requests, but it's good to have those reviewed, too.
Because it's so easily repeatable, it's also better for developers to build and test locally, so they can know it will work the same for everyone once it is committed, built, and available for everyone.
PBIs on 2.2 are kind of a bust though. pkgng will be better, but by the time pkg came along, it was too late to get into 2.2.
-
@cmb:
The most important thing is - read the upgrade notes before you upgrade. If you're affected by something noted there (and those are continually updated post-release as things change), then make sure you take the appropriate precautions or mitigations, or in rare instances where you can't upgrade at the moment, hold off until the next release.
https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2_Upgrade_NotesI couldn't agree with you more! I read the release notes and planned ahead.. https://forum.pfsense.org/index.php?topic=87365.msg479666#msg479666
I've been in the IT field for too long to know upgrades can and will have issues. You have to plan for this.. Since I use a few packages and didn't want to upgrade without knowing they worked. I fired up a VM of 2.2RC and started to test the waters… If I found issues, I would try to see if I could fix it somehow and open up a bug report. In that report I would provide what is broke, logs, and how I was able to temporary correct it; in case it helps the developer make a permanent fix the issue.
Knowing who some of the community packager maintainers are (marcelloc, bbcan117, bmeeks to name a few: Thank you for all your hard work!!), I'll reached out to them to see if I could assist with testing. I'm no programmer but I sure can beat the crap out of programs and hardware. Oh, and I suggest a few bells and whistles to improve it (ask bbcan117, poor guy gets no sleep because of me).
Yeah there are days I want to bitch on the forum for something not working, or a change happen without a RFC... But in the end, it gets you no where. Open a bug report, reference that bug report on the forum and move on.. Check back in a couple of days... Not 30 minutes...
ok back to work for me
Stephen
-
I wouldn´t even considered upgrading if I had seen some of the packages didn´t work with 2.2.
Then you didn't read the release announcement, much less the upgrade notes. Blindly upgrading things isn't a good idea, please read them in the future.
- And frankly, "packages are now in full control of the dev team" is exactly what did NOT help.
Huh? No such changes were made at all. Packages must use binaries built on our build servers, but that's as simple as telling it what port and options to build. Pull requests off github accommodate that process, and have for quite some time, well before 2.2. That's for security and maintainability reasons. In the early days, we weren't as strict about that, some package maintainers had people pull binaries from their own systems. That's the only tightening of control that we've done on packages, and that happened years ago.
-
Another option is to have more than one repository for packages, like stable, testing, unstable etc.
I've tried it once but before starting coding I've decided to ask core team about it and the answer was no.
That was relevant to having people point their systems to package servers other than our own, to avoid situations like the lusca stupidity where people are having their package repos pointed to some free web hosting service that's continually hosting a slew of malware. Then their packages break because no one's updated what is on that server in years.
I expect we'll do something along the lines of official/unofficial classifications or similar, along the lines of what I noted in my earlier post.
-
Even though PFSense is a firewall, it also does some routing and supports some tunneling like OpenVPN, but I don't get the whole installing a ton of these packages. People try to make their "firewall" do way too much crap. Asking for the kitchen sink is asking for trouble. Firewalls are critical infrastructure. It needs to do one job and do it well.
Need a proxy? Set up a proxy server. Need an IDS? Set up a passive IDS that watches the traffic and turns off network ports when bad traffic is detected.
By putting all the responsibility on one device, you're making all of those features co-dependent on each-other. Distinctly separate services should be distinctly separate machines/vms.
As for "bugs". PFSense 2.2 had 0 open bugs the day before release and had almost no new bugs for months, and gained 40+ in 1-2 days. Obviously the people testing 2.2 beta weren't reporting bugs or weren't having bugs. Seems most of the people with crippling issues have some strange configurations, strange/old hardware, Xen doing crazy things, and a few actually being PFSense related.
1. Don't use hardware raid
2a. Why are you using VMs?
2b. It's a VM, snapshot it and test it
3. Your Firewall is not meant to be an file serving, database running, web hosting, time serving, web caching, load balancing monstrosity.I don't even want to run NTPD or OpenVPN on mine because that's outside the scope, but it is a nicety that allows me to cut corners at home and not do things correctly.
I'm not saying that PFSense can get away with some things like if OpenVPN stopped working and they were too lazy to fix it in a timely fashion, but as someone who has a background in security at all levels of computer systems, and network design, I don't blame others when things go wrong because I design my entire network to hinge on a single device that is forced to do way more than it should, and I didn't modularize my network services.
I know I took a gamble when I upgraded to 2.2, but I did my research, figured out my risks, had a back-up plan, and asked the wife when she could handle being without the Internet for an hour or two as I messed around. When I built my machine, I made sure it was all standard parts that have nothing special, but will be around. AHCI harddrives, no special drives required. Intel CPU, nothing special. Intel NIC that is recent, will be supported beyond its usefulness. Video card, integrated into the CPU and is well supported as fully opensource drives.
I leave as little to chance as possible.
-
I do agree some of your points, but in today's world nobody apart from big corporate will put a box for every other function. Pfsense as a firewall is good, but basic function like web filtering http/https is part of the utm device, which pfsense project claim to have.
It should work perfectly and without pain (not requiring a patch and workaround for everything), and administrator should take responsibility for atleast the basic function, otherwise, most of the good users will move away from this project.
I feel administrators should look into these basic function more seriously on urgent.
-
How I do upgrades in most cases follows these rules:
- Does the new release have some feature I cannot live without which the old release is missing? if no there is no need to do an imminent upgrade - wait for X.1 release
- Create a test environment - a clone of your current setup, and try the upgrade of this test environment. Work with this test environment until you are convinced that you know its ins and outs.
- If the test environment is not the same hardware as your current appliance try to make a boot on your current system from CD/USB/PXE to see that all components are recognized and functioning.
- When you finally initiate your upgrade make a full backup as part of the upgrade (The checkbox 'Make a full backup before upgrade')
- If you are using CF or some kind of limited space environment buy a similar device and clone your current setup to this. Then install the clone and upgrade on the clone
Above list should ensure minimal downtime and guaranty you always have fallback plan.
-
I do agree some of your points, but in today's world nobody apart from big corporate will put a box for every other function. Pfsense as a firewall is good, but basic function like web filtering http/https is part of the utm device, which pfsense project claim to have.
Can you share a link where ESF has claim pfSense is a UTM? I dont recall them saying that, but the community has.
