pfsense forums data breach confirmed

awaranowsking

@anonymous-5132 We're investigating. Thank you for the report.

Gertjan

@anonymous-5132 said in pfsense forums data breach confirmed:

just post a screenshot

Use pastebin.com, after cleaning out all private resources like your mail IP and domain.

Btw : the spammer took the time en effort to add DKIM info, but SPF failed !

I think something is missing in the headers : No sign of TLS usage ?
Mails prefer to ~~voyage~~ =travel under TLS, and default to 'clear' if not supported.

When I receive a mail from Netgate.com, my postfix says :

Oct 22 10:11:24 mail2 postfix/smtpd[30595]: 821E83E815: client=mail-ot1-x329.google.com[2607:f8b0:4864:20::329]
Oct 22 10:11:24 mail2 postfix/cleanup[30606]: 821E83E815: message-id=<30bacfe2-b284-6f37-bfcd-fcba473ae46f@netgate.com>
Oct 22 10:11:24 mail2 postfix/qmgr[23278]: 821E83E815: from=<forum-donotreply@netgate.com>, size=17572, nrcpt=1 (queue active)
Oct 22 10:11:24 mail2 postfix/smtp[30607]: Trusted TLS connection established to mail.mydomain.me[a.b.c.d]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Oct 22 10:11:24 mail2 postfix/smtpd[30595]: disconnect from mail-ot1-x329.google.com[2607:f8b0:4864:20::329] ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
Oct 22 10:11:25 mail2 postfix/smtp[30607]: 821E83E815: to=<gertjan@mydomain.me>, relay=mail.mydomain.me[a.b.c.d]:25, delay=1.8, delays=1.1/0.02/0.16/0.52, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as C6B8363E07DF)
Oct 22 10:11:25 mail2 postfix/qmgr[23278]: 821E83E815: removed

edit :
In the headers :

....
Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329])
     (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
     key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
     client-signature RSA-PSS (2048 bits) client-digest SHA256)
     (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (not verified))
     by mail2.mydomain.fr (Postfix) with ESMTPS id 821E83E815
     for <gertjan@kroeb.me>; Fri, 22 Oct 2021 10:11:23 +0200 (CE
...

"mail2.mydomain.fr" is my mail backup MX. I wonder why the second MX was preferred above the primary.

and now we know who it is : gmail.com. forum.netgate.com uses gmail - interesting.

Note : the mail origin is MX from google ... (isn't that the biggest data collector in the world ;) )
So, even if the entire transport was under "Trusted TLS connection" there is a third party in play here. Fourth, if we take in account that the forum database of "forum.netgate.com" has our mail address.

I'm not saying that using TLS is mandatory, just advice you to put all chances on your side.
After all, not using TLS means mail send to you will travel over the net in clear, and you have to trust all 'hops'.

Bob.Dig

You joined 17 hours ago and already got spam because of that...

Looks like a troll to me. Let's find some reputable members who also got spam.

Gertjan

@bob-dig said in pfsense forums data breach confirmed:

You joined 17 hours ago

Nice catch.

AndyRH

@anonymous-5132

Did you create a new account and immediately report the spam for a different account?
Account age and message age are equal.

mer

@bob-dig I may be more disreputable than reputable, but haven't seen any ;)

Anonymous-5132

@Bob-Dig Oh? The URL to this place is forum.pfsense.org? Strange, my address bar says forum.netgate.com. I tried to log in here with the old credentials and, surprise, they don't work, probably because I hadn't used that account for over a decade.

Also, did you take the time to read the original older thread up at the top after Re: or did you immediately switch to trolling mode? If you had read the linked thread you'd have seen the very reports you want, which there likely won't be many of because most people don't go through the trouble of setting up a system so each website gets a unique email address.

Calling someone a troll because you made an incorrect assumption (that the email address I signed up here with is the email address getting spam) and clicked the wrong link (viewing my account rather than the linked forum thread) looks like a troll to me. Put down the mirror before you call someone a troll.

@AndyRH Yes, that's what I did. The message is actually older than the account by a number of hours and the older account doesn't work here. I posted here because other people were talking about it here and forum.pfsense.org redirects here.

@Gertjan The block of text at the bottom of the image contains server logs. The second line indicates that the connection is encrypted. TLSv1.2 with cipher DHE-RSA-CAMELLIA256-SHA256. Looks like my server for some reason isn't adding that info to the email headers.

mer

@anonymous-5132 said in pfsense forums data breach confirmed:

I tried to log in here with the old credentials and, surprise, they don't work, probably because I hadn't used that account for over a decade.

Well, I know that within the past few years (don't recall exactly but a year or two is close) the forum moved from a pfsense to a netgate domain and around the same time previous logins were made useless/invalid. I think there was email or messages talking about it and saying roughly "update your stuff or renew it".

So, to me, it sounds like your unique website based email address was associated with the old pfSense forums, the forums moved and then you received something based on the old email address. Is that close enough for government work?
If so, it's possible that some kind of data harvesting grabbed the email off the old website (not always a sign of a data breach), not the new forums.

I don't know what to tell you, but pretty much anytime an email address is used on anything public, there is a probability > 0 that it will get harvested and a spammer sends email to it. The email equivalent of the phone calls about "We've been trying to contact to you about your cars extended warranty".

Joolee

Can confirm. I received some spam on my "pfsenseforum@" address on October 21.

@mer Even if a spammer scraped the email addresses off the previous forum, is still a breach as that should never be allowed to happen. You might have gotten away with publicly posting email addresses once but we don't live in the 90's any more.

mer

@joolee
I'm just a user, I have no control over the forums (old or new), I was merely pointing out to the OP what I think happened with the forums.
Never once did I say or even try to imply that it was acceptable.

johnpoz

@joolee said in pfsense forums data breach confirmed:

is still a breach as that should never be allowed to happen.

And who says your mailbox was scanned by something you gave access to on your phone, etc. A email to some address you say is private does not scream "breach" to me that is for damn sure..

Yeah this is no longer the 90's agree - and I got spam on some email address that is "private" sure doesn't mean some company you posted the email to was "breached" either..

Joolee

@johnpoz I agree, there is a very slight chance that it is just a coincidence that there are two users of this forum that received the same spam e-mail, on the same date, to an address that is ONLY known by the pfsense forum and the e-mail provider. Without the source of the addresses being said forum.
There is also a chance that I guess your GPG key in one try. It's very small, but it's not zero ;)

johnpoz

@joolee Well let see I have a specific email set on my account - that has only ever been used on the forums. I have been on the forums for YEARS!! same email - specific to this forum. No spam - I just searched that email address in my mailbox, that goes back for years as well - not one spam message to that address. Sure some stuff from negate has been marked as spam - notifications, etc..

So from my "evidence' ;) what has been breached is your mailbox ;)

I have evidence going back 10+ years - hehehe

edit: What would be "evidence" of a breach to me is everyone on the forums got the same spam message. That would scream someone/thing got a hold of the db of the email addresses.

Anonymous-5132

@johnpoz said in pfsense forums data breach confirmed:

So from my "evidence' ;) what has been breached is your mailbox ;)

I have evidence going back 10+ years - hehehe

"Well I haven't received spam" is not evidence. Absence of evidence is not evidence of absence. What you have going back 10+ years is absolutely nothing, and I find it sad that you find it funny. It doesn't mean your address hasn't been leaked. It doesn't mean spam hasn't been sent to it. It just means you haven't seen spam yet for any of a number of possible reasons, most likely being a properly and normally functioning spam filter. The first report of a potential breach that I found was in June, yet it was October before at least two of us received spam. Why the delay? Again, any number of possible reasons. Perhaps the scraped list was divided up and sold off in pieces. Who knows. In the end it doesn't matter and doesn't change anything, and based on the evidence presented in this and the linked thread, you will eventually receive spam as well.

As for what evidence for a breach you would accept, you'll never see "everyone" reporting because probably at least half, and I would argue most, of the people here probably don't use a unique address for the forums or use a throwaway account and therefore would not be able to prove the source of the leak. (I'd love for an admin to comment on that. Should be a fairly simple database query to dump email addresses of active users and return only the mailbox without the domain to see how many look "unique" to the forum. I would love to be told I'm wrong and many people are using unique addresses, even though it would take some weight away from my claims of a data breach.) I would also argue there are people with unique addresses who received spam but didn't report it, further reducing the number of reports that can be expected on a breach. Your expectations don't remotely align with reality.

There are now four reports of a breach, three of them presenting hard evidence (and the fourth hopefully not too long after this post), and absolutely no hard evidence whatsoever in support of there not being a data breach. If you have evidence please post it.

@joolee There are two more reports in the thread I linked at the top of this thread. @ipfftw and @gruensfroeschli both reported spam a few months ago in that thread. I created this new thread because the forum itself suggested doing that rather than resurrecting an otherwise dead thread. Perhaps I should have posted in the old thread. Then again, perhaps not, as it would have notified more people and we'd have even more wild and baseless assumptions to wade through...

Also, if you still have the spam email, could you post the redacted headers please? I'd like to maintain our side's so far flawless victory against the naysayers who have yet to present any evidence whatsoever. If we start simply claiming we received spam without providing evidence we end up being the ones making wild and baseless assumptions. We're making a harsh claim, (data breach), so we need to back it up with solid evidence. Thanks!

To everyone else reading this: If you're going to post another "Well I haven't been spammed" post, please remember to include the evidence in support of your claims. "There's none in my mailbox" is not evidence. PROVE that you haven't received spam and a spam filter hasn't silently blocked it. I have seen commercial spam filter providers silently drop email in the past.

Here are some questions you can answer to give your claims of knowing for certain you haven't received spam a little more weight. If you don't control every link in the email chain past your MX records you can't really know for certain that you never received any spam and your claim is a baseless assumption without any evidence.

Q: Who controls the email service that your MX records point to? "Service" being the program/daemon/whatever that listens on port 25. "Controls" means who has write access to the actual config files for the service itself at a global level. (Access to a web control panel on shared hosting isn't control of the service.)

My answer: I do. It's running on a self-managed VPS to which I have root access.

Q: What mail filtering do you have? As in both spam filtering that will simply "mark" spam and rules that will completely deny delivery. (Block lists, SPF checks, DNSBLs, etc.)

My answer: Basic spamassassin configuration, manually-created block rules, SPF checks, and a small set of DNSBLs that I have found not to be overly trigger-happy. Also firewall rules to prevent connections from IPs that have been attacking the service/server, but I never use firewall rules to deal with spammers. Anything that gets blocked by the spam filter or mail service gets listed in log files.

Q: Do you have access to the low-level mail service logs on your mail server and spam filter? If so, how often do you review them?

My answer: Yes. From daily to every few days, but all logs are reviewed, mainly to see if I need to firewall an attacker or adjust fail2ban.

Q: Do you have logs of all mail delivery attempts that are blocked by the filtering mentioned in the second question? How often do you review those logs?

My answer: Yes. From daily to every few days, but all logs are reviewed to see if I need to adjust the spam filtering. (And to see if a new address has been leaked.)

Q: What email client do you use and what filtering do you have set on it?

My answer: Thunderbird in POP3 mode. I have various rules that move messages to junk, trash, or other specific folders, but nothing that auto-deletes. Every deletion is a manual action.

Q: How often do you look at your spam folders?

My answer: Once or twice a week, sometimes less often, but since nothing gets auto-deleted I won't "lose" a badly-filtered message.

As you can see, I have a system specifically designed to be less aggressive so that I can detect breaches such as what we're discussing here. It's a trade-off. More spam can get through, but in exchange I can see "unauthorized" use of specific email addresses that would indicate either a breach or unauthorized "sharing" of the address. On the other hand, it makes it possible to simply change an email address that has been leaked and set a rule to completely block further delivery to the leaked address, actually lowering spam in the long run. This system is what I have decided, for my specific use case, requires the lowest time investment for the lowest rate of false positives and false negatives.

Since I feel like continuing to assault this expired equine, I'll explain why the alternatives that have been brought up are wrong:

Alternative: Mailbox was breached (at the server side).

Why it isn't that: The last legitimate email I received to the address in question was on October 12, 2017. My current VPS hosting the email service was first activated on January 21, 2019. Prior to that the email was running through a different VPS instance. Thunderbird is configured as a POP3 client and deletes email from the server older than two weeks when it checks for new messages. (I leave messages on the sever for a short time in case my Thunderbird profile corrupts. In case of corruption I simply replace the profile with the previous night's backup and re-download all messages. Nothing gets lost that way. It has saved my bacon more than once ) I have a catch-all on the domain which sends everything to my main mailbox.

The email address in question never "appeared" on the current server, which has been in use for over two years, until the spam email was received. Therefore, we can rule out a server-side mailbox breach.

Alternative: Mailbox was breached (at the client side).

Why it isn't that: If someone got into Thunderbird, or any other application on my computer for that matter, why compromise an alias, an alias that hasn't seen email in over two years at that, and not the main address to my mailbox? Why not move laterally into userland at my user level and hit me with ransomware? Why not pivot from my main computer into other systems on my local network? Why not drop a keylogger and pivot into much more valuable targets? Basically, why stop at a single application?

The more you think about it from an attacker's point of view the less likely it becomes that my email application or breach of my local computer was the source of the leak. Therefore, we can rule out a client-side mailbox breach, and fully rule out the wild and baseless assumption about my mailbox getting breached.

Alternative: (Insert anything here that isn't some sort of data breach of the old pfsense forums.)

Why it isn't that: So you're saying that four different people, all (likely) running different email systems, with the only thing in common between them being using the leaked email address on the pfsense forum, and two of them seeming to be involved in the same spam campaign, all had that one email address compromised through some unspecified "other means" and it can't possibly be a compromise of the old forum? What's this "other means" and what extraordinary evidence do you have supporting that extraordinary claim? As I said before, "I haven't received spam" is not evidence. You simply haven't received spam yet.

I'll end this novel with the hope that it will put an end to the wild and completely baseless assumptions that have been posted so far. If not, well, I guess I'll continue to disprove more wild and baseless assumptions as they arrive. If nothing else the continued activity will perhaps attract a few more reports from people who have received spam. If you have received spam, please post the redacted headers (and if you have them, email server logs, again properly redacted) as evidence in support of your claims. Too many people are posting without evidence that it didn't happen, so we need to include evidence with our claims so we can continue to easily dismiss the naysayers.

johnpoz

@anonymous-5132 said in pfsense forums data breach confirmed:

"Well I haven't received spam" is not evidence. Absence of evidence is not evidence

Its same sort of evidence that your trying to present ;)

I have used this email (unique to these forums) for 10 some years - no spam.. So where is this breach you say happened.. ?? What these spammers said hey we just got X thousands of new email address to send our spam too - but don't send to johns

mer

I'm kind of wondering about the motivation of this thread now.
Is it simply:
Hey the old pfsense forum may have suffered a data breach, so you may want to be aware of any place you use that password/email combo and change it.

That is a good thing, a heads up to everyone.

Not sure if there is much value in posting redacted headers and such since the old forum has no way for users to change anything, all you could do is change your password on the current forum if you've used the same email/password combo.

Just my opinion feel free to ignore as you wish.

johnpoz

@mer said in pfsense forums data breach confirmed:

wondering about the motivation of this thread now.

Same here - seems like the subject is meant to draw attention, get on google, etc.. When clearly there is no such evidence of any sort.... @mer have you got any spam on your unique email - oh yours doesn't look unique never mind..

Now if we had a huge number of users saying hey I got spam xyz.. To my unique only used on this forum.. Then you might have something to talk about..

dennis_s

Our IT team has been looking into a possible breach of the Netgate forums. They have found no evidence of any breach or of users' email/passwords being compromised in any way.

All users that migrated their accounts to the new (current) forum were forced to change their password at that time, mitigating anything that may have happened prior to that.

mer

@johnpoz It's not unique, and yes "I've got spam" :) At least comcast has been training their algorithms, they do a decent job of marking them as such.

@dennis_s Thanks for the info.

Anonymous-5132

@johnpoz said in pfsense forums data breach confirmed:

@anonymous-5132 said in pfsense forums data breach confirmed:

"Well I haven't received spam" is not evidence. Absence of evidence is not evidence

Its same sort of evidence that your trying to present ;)

I've presented hard evidence in the form of server logs and email headers. You've presented absolutely nothing but your word. They are not the same in any way and any reasonable person would know that.

What if, for sake of argument, I don't believe you? I showed you the proof that I have received spam, so you don't have to take me at my word. Show me the proof that you haven't received spam so I don't have to take you at your word. If I was paranoid I might think you're hiding something...

I, and three other people, have presented evidence that the old pfsense forums suffered a data breach, while you've presented absolutely nothing that it hasn't. If not that, then what? What other explanation do you have that fits the evidence that has been posted so far? Present evidence, not words. Proof.

Why are you arguing so hard against this from such a weak position? What is it to you?

@mer said in pfsense forums data breach confirmed:

I'm kind of wondering about the motivation of this thread now.
Is it simply:
Hey the old pfsense forum may have suffered a data breach, so you may want to be aware of any place you use that password/email combo and change it.

That is a good thing, a heads up to everyone.

Something like that. "Hey, change your passwords, start using a password manager with unique passwords for all sites, and turn on 2FA. If you used a unique email, change it and block delivery to the old address if you have the ability. Also be aware that it's possible any information you provided to the old pfsense forum and anything connected to it may be in the wild now, which may include private messages."

Not sure if there is much value in posting redacted headers and such since the old forum has no way for users to change anything, all you could do is change your password on the current forum if you've used the same email/password combo.

In my opinion one should present evidence when claiming a data breach, otherwise it tends to look like simple trolling. It also gives admins a reference to look at while investigating.

@johnpoz said in pfsense forums data breach confirmed:

When clearly there is no such evidence of any sort....

The only ones not presenting evidence are you and the other naysayers. There's plenty of evidence. Have you looked at the other thread? Have you looked at the evidence I presented?

Now if we had a huge number of users saying hey I got spam xyz.. To my unique only used on this forum.. Then you might have something to talk about..

Do you know how I know you didn't read my previous post? Give it time. This thread has already attracted another confirmation, albeit without providing evidence yet. More will eventually filter in, especially if you keep keep this thread alive.

@dennis_s Thanks for the update, but this seems to be only related to the old forums. Once enough users come forward with evidence I think an official breach notification might be nice. It would also give a place to explain anything else that may have been compromised as well as listing everything that couldn't have been involved.

I have no evidence that the Netgate forum, or any other part of the Netgate website, has been breached, nor am I claiming that. If it seemed that I was claiming that the Netgate forum has suffered a data breach that was not my intent. This is all about the old pfsense forum and any software tied in to that system at that time. We know that at least email addresses have been leaked, but so far we don't know what other data may or may not have been involved.