Admin password changed itself. Twice. Yes it did.
-
_This situation is alarming and I need some serious expertise and guidance from the heavy hitters here:
I am configuring a new setup, dual HA firewalls, dual WAN, full LACP/CARP redundancy to a stacked Cisco L3 switch. Using this hardware, which came pre-installed with PFSense: https://www.amazon.ca/gp/product/B0733H75TB/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1
I upgraded (using the console, using menu item 13) to 2.4.2.
Yesterday, I successfully configured IPSec for remote access. When I returned to the system after working on something else, the admin password no longer worked. After trying the default and other user passwords (to eliminate the possibility of me having accidentally changing the password), I changed the password form the console and moved on to refine the VPN config. I ended up not making any changes, however, being called away to a meeting.
This morning the admin password has changed again. No, Caps Lock is not on. No I don't sleepwalk. Nobody else has access to the gear. I actually suspect something rather sinister is going on…
I can run paranoid scenarios with the best of them, but I want help to gather some facts to target the investigation of WTH is going on here. Please ask your questions; I will answer them as best I can. Given that this is a time-sensitive issue, I would respectfully ask that advice which is neither constructive nor expert be saved for the recrimination phase after the issue is solved!
Help._
-
well lets see the hash of the root password, and then after it changes on its own lets see the hash again.
So you can use vipw to view the hash or look in /etc/master.passwd with cat or something, etc..
-
I upgraded (using the console, using menu item 13) to 2.4.2.
That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.
I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.
-
I think you quoted the wrong section ivor ;) You prob wanted to quote the part where they state came preloaded… Which I missed on my first read thru.. nice catch
which came pre-installed with PFSense:
-
No, not in this case :) I was pointing out the issue on OP's end.
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
-
I upgraded (using the console, using menu item 13) to 2.4.2.
That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.
I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.
Ok, first of all thank you for the reply.
Second, I had no idea of the policy when I purchased hardware in good faith. I see the issue you identify now, and I understand the policy. That said, here I am and I need to resolve this.What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still?
-
Backups are just XML files - you could look through it to see exactly what in there.
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.
Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
-
What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still?OS might be compromised but we don't know if hardware is also. Considering what happened I don't see how one can trust the hardware. I am not sure you will find a BIOS for appliance board because it's a generic Aliexpress J900 board (so you can reflash it). As Johnpoz noticed, backups are in XML and one can analyze them but I would do a clean install and configure from scratch. You might want to talk to your Amazon seller about it, they're the ones responsible. I would want a refund.
-
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
[/quote]Yes, we have several build time and shipping options. http://store.netgate.com/
-
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
[/quote]Yes, we have several build time and shipping options. http://store.netgate.com/
Ok. Working on it now.
Thanks very much. Always fun to learn a lesson… see you guys around here, I'm sure.
-
You're welcome. Please feel free to contact our sales if you need assistance with your order! https://www.netgate.com/company/contact-us.html
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.
At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.
Why would anyone think it's OK to sell a device with pfsense preloaded? There is only one answer: Greed. These vendors take what we engineer, package and test and use it to sell their crapware.
In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense. That company is Netgate.
I'm still OK with members of the community building their own firewall using pfSense software.
But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?
-
@jwt:
At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.
Why would anyone think it's OK to sell a device with pfsense preloaded? There is only one answer: Greed. These vendors take what we engineer, package and test and use it to sell their crapware.
In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense. That company is Netgate.
I'm still OK with members of the community building their own firewall using pfSense software.
But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?
There seems to be something more sinister than simple greed at play in this case. Resetting the Admin password? Oh and by the way, disabling all logging - I thought I just wasn't configuring it right, but it wouldn't log anything, so I couldn't see the malicious traffic. My first clue should really have been when the "Enable IPSec" checkbox wasn't there. I assumed just a version difference between the docs and my install, but I would suggest it's more likely that there was a VPN built-in as an exploit vector, and the absence of the checkbox prevented it from being disabled.
I got lucky, by some measure at least. Lucky to know enough to set it up on a discreet WAN segment so the attack was mitigated. Lucky I didn't config it and just let it run unobserved - much less install it at the production site. Lucky the bad actors were not especially subtle or clever.
I must say, this spooked me. It's been a long time since an attack got that far in.
My order (including support) should help prop up the product environment a bit.
-
Wow!! I would be very curious to traffic it was generating.. No logging at all? And ipsec missing?
I could see creating a back door account, but why would you reset the password? Seems counter intuitive to draw attention by locking out the owner, etc.
If this is such a case of comprised box sold – You might want to work with netgate on shipping them the box to investigate further...
-
Wow!! I would be very curious to traffic it was generating.. No logging at all? And ipsec missing?
I could see creating a back door account, but why would you reset the password? Seems counter intuitive to draw attention by locking out the owner, etc.
If this is such a case of comprised box sold – You might want to work with netgate on shipping them the box to investigate further...
I reached out to jwt about sending them the devices. And yeah, wow…
-
I saw you added a negative review to that page on Amazon, mentioning the issues. If you didn't already, there should be a link on that page to report it to Amazon as well.
Alas it seems like that 3rd party vendor sells a whole array of "Pfsense" products on Amazon.ca.
I'd almost be tempted to stick it in a quarentine DMZ somewhere and see if it tries to phone home.
-
There seems to be something more sinister than simple greed at play in this case.
First, thanks for being a customer, but, to be clear, I wasn't angry with you.
To be extra clear, while I am concerned for your experience, the issue is much larger than the security of your network.
The security of everyone's network is at risk here.
pfSense is a brand. It stands for something. More, we have a registered trademark (worldwide) on pfSense, and trademarks have to be defended.
The strength and distinctiveness of many trademarks has been lost due to improper use of the marks in advertising and promotion, sometimes referred to as “genericide.”
This loss occurs if consumers perceive a trademark not as identifying a product from a single source, but rather as a mere description of the nature of the product or as an identification of a product type or product category as a whole. When a trademark no longer identifies a product from a single source, but is used to identify a category of like products, that mark is generic and available to all to use to describe their products. Some examples of common brands that are generic or come close to the generic line are ASPIRIN, ESCALATOR, KLEENEX, BAND-AID, YO-YO, THERMOS and WINDSURFER.
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
-
-
@jwt:
First, thanks for being a customer, but, to be clear, I wasn't angry with you.
I've seen angry. That wasn't it. No offense taken.
@jwt:
To be extra clear, while I am concerned for your experience, the issue is much larger than the security of your network.
The security of everyone's network is at risk here.
Yeah that's my underlying concern.
@jwt:
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
I'd be glad to discuss my thoughts. Please contact me if you'd like to.
-
-
jim at netgate dot com
-
@jwt:
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
I don't think this would change much. Those unauthorized vendors would simply use older/existing builds, maybe even modify them to report like current builds. At most you will likely have more people asking for help on outdated builds, complaining that they can't update to a newer version and/or reporting bugs that are actually fixed in the real current builds. So more chaos and maybe bad press on top.
Even large companies with non-free products, like MS for example, aren't able to stop stuff like this and they have a lot more money and man power at their disposal. It's like fighting against windmills.
Of course you can try to take on that fight, but it will probably just consume a lot of resources, time and energy from the staff without reaping any real benefits.
-
-
@jwt:
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
Really difficult to pass judgment on the Amazon seller without being able to analyze that box - I'd be really interested in the findings. I've seen PC's for sale with 'trial' versions of an OS installed, just to prove it works and all is well. At first I want to give the Amazon store the benefit of the doubt - maybe they just slapped it in to say 'see ? …working!', but the changing password is really strange and does indeed raise some alarm. I wonder if, considering they claim it'll support IPCop etc - IF it actually would support another OS. Maybe no one has tried!!
It's funny this comes up, because my brother and I were discussing something like this just this morning...about the hardware Netgate has to offer vs other systems etc, and about the trade off Netgate has to make between being attractive to consumers, but also earning revenue. So, sure, there are cheap boards out there that 'might' run pfsense, but that doesn't help to support the project.
To your questions above, obviously option 1 isn't a smart forward-looking solution.
Option 2 seems like strong reason - this smacks of what TiVo did - open source software, closed source hardware; at the risk of losing much of the community.
Option 3 would be good from a community perspective - perhaps a unique ID is generated on a new install that requires activation from pfsense ?
-
-
@Knyte:
@jwt:
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
Really difficult to pass judgment on the Amazon seller without being able to analyze that box - I'd be really interested in the findings. I've seen PC's for sale with 'trial' versions of an OS installed, just to prove it works and all is well. At first I want to give the Amazon store the benefit of the doubt - maybe they just slapped it in to say 'see ? …working!', but the changing password is really strange and does indeed raise some alarm. I wonder if, considering they claim it'll support IPCop etc - IF it actually would support another OS. Maybe no one has tried!!
It's funny this comes up, because my brother and I were discussing something like this just this morning...about the hardware Netgate has to offer vs other systems etc, and about the trade off Netgate has to make between being attractive to consumers, but also earning revenue. So, sure, there are cheap boards out there that 'might' run pfsense, but that doesn't help to support the project.
Are there any you know of less expensive than espresso.bin?
@Knyte:
To your questions above, obviously option 1 isn't a smart forward-looking solution.
Option 2 seems like strong reason - this smacks of what TiVo did - open source software, closed source hardware; at the risk of losing much of the community.
Option 3 would be good from a community perspective - perhaps a unique ID is generated on a new install that requires activation from pfsense ?I'm mulling over #3.
-
-
@jwt:
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
I don't think this would change much. Those unauthorized vendors would simply use older/existing builds, maybe even modify them to report like current builds. At most you will likely have more people asking for help on outdated builds, complaining that they can't update to a newer version and/or reporting bugs that are actually fixed in the real current builds. So more chaos and maybe bad press on top.
Even large companies with non-free products, like MS for example, aren't able to stop stuff like this and they have a lot more money and man power at their disposal. It's like fighting against windmills.
Of course you can try to take on that fight, but it will probably just consume a lot of resources, time and energy from the staff without reaping any real benefits.
So what you're saying is just quit pfSense?
-
-
@jwt:
Are there any you know of less expensive than espresso.bin?
Not at the moment, nope. Was looking at something or another on AliExpress, but then memories of BananaPi came rushing back…
-
@Knyte:
@jwt:
Are there any you know of less expensive than espresso.bin?
Not at the moment, nope. Was looking at something or another on AliExpress, but then memories of BananaPi came back…
Banana Pi routers are $65 on Aliexpress
https://www.aliexpress.com/item/Banana-PI-R1-Wireless-Router-Open-Source-Development-Board-BPI-R1-Smart-Home-Control-Plate/32811123035.htmlespresso.bin is $49 on Amazon.
https://www.amazon.com/Globalscale-Technologies-Inc-SBUD102-ESPRESSObin/dp/B06Y3V2FBK/I shouldn't have to tell you which one is faster. Hint: it's not the B-Pi router.
-
@jwt:
So what you're saying is just quit pfSense?
No that's not what I'm saying.
Look at how for example the Kodi Team handles the piracy box issues. Instead of investing time into a cat and mouse game with a DRM like approach, that will be broken/circumvented anyways, educate the community/people about what pfSense is. Why it's a bad idea to buy your security device from a shady vendor that ignores licenses. pfSense is quite well known, but the issue with those vendors isn't. Try to get coverage from IT magazines, bloggers even youtube channels to get the message out there.
Of course where it's possible use your legal rights to take down those vendors, just don't alienate your community or brake your back/company over them, they aren't worth this.
-
@jwt:
I shouldn't have to tell you which one is faster. Hint: it's not the B-Pi router.
Nope, you sure don't. I only brought that up in the sense that it looked GREAT and interesting at the time, woohoo, lets get TWO! …then, when they arrived, it became instantly clear they're completely schizophrenic. They have built in HDMI etc, but don't function very well as a media device. If one were to install a firewall OS, it'd under-perform, as you mentioned. Could install OpenWRT and turn it into an AP, but that's under-utilizing the hardware. So, they're not really great at anything...just kinda meh at everything.
Hence, it was a bad purchase, and a bad memory :) Lesson = don't impulse buy cheap hardware. Well, at least they were cheap.
I'd much rather look more closely at what Netgate has to offer, and look for ways I can support them/you.
-
@jwt:
So what you're saying is just quit pfSense?
No that's not what I'm saying.
Look at how for example the Kodi Team handles the piracy box issues. Instead of investing time into a cat and mouse game with a DRM like approach, that will be broken/circumvented anyways, educate the community/people about what pfSense is. Why it's a bad idea to buy your security device from a shady vendor that ignores licenses. pfSense is quite well known, but the issue with those vendors isn't. Try to get coverage from IT magazines, bloggers even youtube channels to get the message out there.
Of course where it's possible use your legal rights to take down those vendors, just don't alienate your community or brake your back/company over them, they aren't worth this.
Kodi has a different problem.
You won't (long) find a "Kodi box" for sale on Amazon or eBay, and you won't find a Kodi app for iPhone / iPad.
The reason why is that the content providers have been getting laws passed (for instance, in the EU). The UK has a new law, the "Digital Economy Act" that has this to say:
“A person…who infringes copyright in a work by communicating the work to the public commits an offence if [the person] knows or has reason to believe that [they are] infringing copyright in the work, and…knows or has reason to believe that communicating the work to the public will cause loss to the owner of the copyright, or will expose the owner of the copyright to a risk of loss.”
They've effectively criminalized the sale of hardware pre-loaded with Kodi. Since merely selling the box means you can be found guilty of contributory copyright infringement, and since the police in Scotland have started to equate Kodi boxes with criminal gangs, https://www.edinburghnews.scotsman.com/our-region/edinburgh/police-commit-to-crackdown-on-kodi-streaming-1-4422380, the larger retail outlets have bowed out.
-
@Knyte:
@jwt:
I shouldn't have to tell you which one is faster. Hint: it's not the B-Pi router.
Nope, you sure don't. I only brought that up in the sense that it looked GREAT and interesting at the time, woohoo, lets get TWO!
I also have two sitting in a cardboard box, somewhere.
@Knyte:
I'd much rather look more closely at what Netgate has to offer, and look for ways I can support them/you.
I'm curious. What would you pay for pfSense that could be loaded on espresso.bin?
-
@jwt:
I'm curious. What would you pay for pfSense that could be loaded on espresso.bin?
That is a tough one to answer… being in Canada the price increases quite quickly with duties and international shipping. (Canadian Partners have not helped reduce these 2 costs - it's always been cheaper to buy direct from Netgate).
When I saw your post about Espresso.Bin on reddit, I got really excited to replace my power hungry ESXi box. I priced out the board being about $107 USD shipped or ~$140CAD (that was the 2GB model with a power brick direct from GlobalScale).
Personally for my use case (at home), I could justify spending up to $200 CAD shipped. Anything above that I would start evaluating other options (mostly a lower power/cost ESXi box).
-
I’m not asking what you would pay for the hardware.
You’ve shown what it costs to get a board into Canada.The 2GB espresso.bin is $79
A power supply is ~$8.00
Total $87You’re shipped with duties at $107.
So $30 in shipping and Canada duties.
But I’m not asking what you’d pay for your router.
I’m asking what you would pay for pfSense on espresso.bin.
-
@jwt:
I’m asking what you would pay for pfSense on espresso.bin.
..which leads to the questions:
- one-time purchase or subscription modell
- buy-in for one install or multiple in my kingdom
Since I have absolutely no idea about espresso.bin's capabilities what would you compare it to - APU1/2, Atom device, … ? Comparisons to SG-1000 or Lord Vader wouldn't help (me) much either...
But, maybe, this has come quite far from "Admin password changed itself. Twice. Yes it did.", hasn't it? ;)
-
@jwt:
I’m asking what you would pay for pfSense on espresso.bin.
Doh… Of course, I misread your question as I was rushing to get out of the office.
Umm that's a good question. I can afford another $50 on top of my gold subscription.
-
@jwt:
I’m asking what you would pay for pfSense on espresso.bin.
Doh… Of course, I misread your question as I was rushing to get out of the office.
Umm that's a good question. I can afford another $50 on top of my gold subscription.
i
One answer is that it would be included in gold.
-
@jwt:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
A subscription model would preferable to closing down 'free' pfSense.
-
-
@jwt, you may want to pull out some of this discussion (the what can/should we do) into a separate topic for wider community input.
My couple of cents:
Licensing: most/all of pfSense is it BSD or BSD-ish License? If so, that gives you one set of options. Other licenses (GPL, et al) mean other things.Defend the trademark as vigorously as possible, don't just give up on it.
Make it clear on the front page of the website that the only official systems preinstalled are from NetGate. Anything else is "buyer beware"."free" pfSense: this is a similar situation to ixSystems and FreeNAS, no? There is value to providing a user downloadable version for use on their own hardware. What you need to do is differentiate it enough from the officially installed on Netgate hardware so folks will try the free version and then buy the preinstalled. I know, hard to do. You can't cripple the free version, you need to make the preinstalled version have useful bells and whistles. The free version also lets people that have the ability and desire generate patches for you.
Activation keys: I understand the reason and logic, but have never liked them. Too much like a "you paid money for this but we're going to control your ability to run it". Is it a "one and done" so I can reinstall on different hardware as much as I want? Is it tied to a specific version of pfSense on a specific hardware platform meaning I can't move it to different hardware if something dies? Do different features require different activation keys? Are the keys going to expire (yearly subscription)?
I don't know what's right for the project, I've always been fine with paying a resonable price for something I need and for my home use the SG2440 was more than I needed but it was a reasonable price.
-
This thread been mentioned on the reddit pfsense section.. One of the comments pointed out the seller of that box also sells a pc.. And states this on the specs.. If you want the link find it on the reddit thread..
"System: Free Operating System: Default installed our activated OEM cracked version(not genuine, works good)Windows English for free,
other languages can be selected among: "WTF?? This is starting to turn into a get some popcorn sort of thread ;)
-
WTF indeed.
