Admin password changed itself. Twice. Yes it did.
-
_This situation is alarming and I need some serious expertise and guidance from the heavy hitters here:
I am configuring a new setup, dual HA firewalls, dual WAN, full LACP/CARP redundancy to a stacked Cisco L3 switch. Using this hardware, which came pre-installed with PFSense: https://www.amazon.ca/gp/product/B0733H75TB/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1
I upgraded (using the console, using menu item 13) to 2.4.2.
Yesterday, I successfully configured IPSec for remote access. When I returned to the system after working on something else, the admin password no longer worked. After trying the default and other user passwords (to eliminate the possibility of me having accidentally changing the password), I changed the password form the console and moved on to refine the VPN config. I ended up not making any changes, however, being called away to a meeting.
This morning the admin password has changed again. No, Caps Lock is not on. No I don't sleepwalk. Nobody else has access to the gear. I actually suspect something rather sinister is going on…
I can run paranoid scenarios with the best of them, but I want help to gather some facts to target the investigation of WTH is going on here. Please ask your questions; I will answer them as best I can. Given that this is a time-sensitive issue, I would respectfully ask that advice which is neither constructive nor expert be saved for the recrimination phase after the issue is solved!
Help._
-
well lets see the hash of the root password, and then after it changes on its own lets see the hash again.
So you can use vipw to view the hash or look in /etc/master.passwd with cat or something, etc..
-
I upgraded (using the console, using menu item 13) to 2.4.2.
That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.
I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.
-
I think you quoted the wrong section ivor ;) You prob wanted to quote the part where they state came preloaded… Which I missed on my first read thru.. nice catch
which came pre-installed with PFSense:
-
No, not in this case :) I was pointing out the issue on OP's end.
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
-
I upgraded (using the console, using menu item 13) to 2.4.2.
That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.
I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.
Ok, first of all thank you for the reply.
Second, I had no idea of the policy when I purchased hardware in good faith. I see the issue you identify now, and I understand the policy. That said, here I am and I need to resolve this.What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still? -
Backups are just XML files - you could look through it to see exactly what in there.
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.
Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
-
What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still?OS might be compromised but we don't know if hardware is also. Considering what happened I don't see how one can trust the hardware. I am not sure you will find a BIOS for appliance board because it's a generic Aliexpress J900 board (so you can reflash it). As Johnpoz noticed, backups are in XML and one can analyze them but I would do a clean install and configure from scratch. You might want to talk to your Amazon seller about it, they're the ones responsible. I would want a refund.
-
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
[/quote]Yes, we have several build time and shipping options. http://store.netgate.com/
-
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?
[/quote]Yes, we have several build time and shipping options. http://store.netgate.com/
Ok. Working on it now.
Thanks very much. Always fun to learn a lesson… see you guys around here, I'm sure.
-
You're welcome. Please feel free to contact our sales if you need assistance with your order! https://www.netgate.com/company/contact-us.html
-
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.
At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.
Why would anyone think it's OK to sell a device with pfsense preloaded? There is only one answer: Greed. These vendors take what we engineer, package and test and use it to sell their crapware.
In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense. That company is Netgate.
I'm still OK with members of the community building their own firewall using pfSense software.
But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?
-
@jwt:
At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.
Why would anyone think it's OK to sell a device with pfsense preloaded? There is only one answer: Greed. These vendors take what we engineer, package and test and use it to sell their crapware.
In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense. That company is Netgate.
I'm still OK with members of the community building their own firewall using pfSense software.
But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?
There seems to be something more sinister than simple greed at play in this case. Resetting the Admin password? Oh and by the way, disabling all logging - I thought I just wasn't configuring it right, but it wouldn't log anything, so I couldn't see the malicious traffic. My first clue should really have been when the "Enable IPSec" checkbox wasn't there. I assumed just a version difference between the docs and my install, but I would suggest it's more likely that there was a VPN built-in as an exploit vector, and the absence of the checkbox prevented it from being disabled.
I got lucky, by some measure at least. Lucky to know enough to set it up on a discreet WAN segment so the attack was mitigated. Lucky I didn't config it and just let it run unobserved - much less install it at the production site. Lucky the bad actors were not especially subtle or clever.
I must say, this spooked me. It's been a long time since an attack got that far in.
My order (including support) should help prop up the product environment a bit.
-
Wow!! I would be very curious to traffic it was generating.. No logging at all? And ipsec missing?
I could see creating a back door account, but why would you reset the password? Seems counter intuitive to draw attention by locking out the owner, etc.
If this is such a case of comprised box sold – You might want to work with netgate on shipping them the box to investigate further...
-
Wow!! I would be very curious to traffic it was generating.. No logging at all? And ipsec missing?
I could see creating a back door account, but why would you reset the password? Seems counter intuitive to draw attention by locking out the owner, etc.
If this is such a case of comprised box sold – You might want to work with netgate on shipping them the box to investigate further...
I reached out to jwt about sending them the devices. And yeah, wow…
-
I saw you added a negative review to that page on Amazon, mentioning the issues. If you didn't already, there should be a link on that page to report it to Amazon as well.
Alas it seems like that 3rd party vendor sells a whole array of "Pfsense" products on Amazon.ca.
I'd almost be tempted to stick it in a quarentine DMZ somewhere and see if it tries to phone home.
-
There seems to be something more sinister than simple greed at play in this case.
First, thanks for being a customer, but, to be clear, I wasn't angry with you.
To be extra clear, while I am concerned for your experience, the issue is much larger than the security of your network.
The security of everyone's network is at risk here.
pfSense is a brand. It stands for something. More, we have a registered trademark (worldwide) on pfSense, and trademarks have to be defended.
The strength and distinctiveness of many trademarks has been lost due to improper use of the marks in advertising and promotion, sometimes referred to as “genericide.”
This loss occurs if consumers perceive a trademark not as identifying a product from a single source, but rather as a mere description of the nature of the product or as an identification of a product type or product category as a whole. When a trademark no longer identifies a product from a single source, but is used to identify a category of like products, that mark is generic and available to all to use to describe their products. Some examples of common brands that are generic or come close to the generic line are ASPIRIN, ESCALATOR, KLEENEX, BAND-AID, YO-YO, THERMOS and WINDSURFER.
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
-
-
@jwt:
First, thanks for being a customer, but, to be clear, I wasn't angry with you.
I've seen angry. That wasn't it. No offense taken.
@jwt:
To be extra clear, while I am concerned for your experience, the issue is much larger than the security of your network.
The security of everyone's network is at risk here.
Yeah that's my underlying concern.
@jwt:
This kind of occurrence is the type of thing that will force me to make one of three choices:
-
Ignore the problem, and continue to put the trademark at risk
-
Close down 'free" pfSense. Forever.
-
Invest the time and resources in making sure that nobody can load pfSense without authorization from Netgate
We have, I think, played more than fair to this point, but this type of thing puts the business at risk in may ways.
I'm curious what the community thinks.
I'd be glad to discuss my thoughts. Please contact me if you'd like to.
-
