website security problems

jc1976

this is what i mean..

i uninstalled pfblockerng, deleting all settings as well so when i reinstalled it was to it's defaults.
i also restored suricata to it's defaults.

neither are running at the moment because i can't get through to any sites. ddg.com will come up (sorta, but that's also problematic).

i also uninstalled firefox to it's entirety and reinstalled in it's default state.
I used Edge, my gf uses her mac with chrome, it's just really crappy.. if i go to speedtest.net, assuming the page even resolves, i get my full allotted speed. ping times are normal, around 12ms.

i'm wondering if it's comcast... there was an outtage a couple weeks ago when this all started happening.. they got it up and running obviously, but i wonder if there's a config issue on their end with their routing?

what do you all think? this just doesn't make any sense.. i've NEVER had pfsense be so unreliable and problematic. even with all my screwing around, it always JUST WORKS... now its basically unusable..

johnpoz

@jc1976 says right there was the problem is

That isn't a pfsense thing.

Firefox setting
https://support.mozilla.org/en-US/kb/https-only-prefs

jc1976

up until lately, i've never had a problem with it. ive had mozilla in https only mode ever since i knew the mode existed. on my pc at work, i have the same settings enabled and it works fine. also, that's just an initial screen. if i put www.amazon.com, the site resolves as it should. and this goes on for almost all sites.

the last 2 times i reinstalled pfsense from the ground, up, i use gparted to wipe out all partition tables. so, i know the install is completely fresh.

do you think it could be xfinity? a couple weeks ago we had an outtage and ever since then i've had issues.

as of now, my pfsense is doing nothing more than routing dns over tls traffic to cloudflare... THATS IT!. i disabled pfblocker, squid, suricata, got rid of watchdog.

this weekend i'm going to take a back up of the config from my other pfsense box at my family house. it's precisely the same dell optiplex 790 with an i7-2600 (with HT disabled).

Again, i REALLY appreciate your input! i absolutely LOVE pfsense! and i won't get rid of it.. i just need to figure out why i'm having so many issues with it all of a sudden. I wish i new how to read logs better..

the only other thing i question is the RAM. it's new and by crucial (16gigs).
16gigs of RAM should read 16384, however on my dashboard it reads 16201.
when i see my other pfsense box this weekend i'll be able to see if something is wrong.

jc1976

@jc1976 just to add, my other pfsense box has suricata running in inline mode whereas this one is in legacy.. always had problems with this one in inline, never in the other.. which is odd because the only difference between the two boxes is my current box has a quad port intel i340 nic, whereas my other has a dual port intel pro/1000. yes, hardware offloading has been disabled..

i dunno... i wish my GF wasn't living with me at the moment so i could focus on this when i get home from work.. LOL!

Gertjan

@jc1976

You're doing it right : re installing with bit level destroying etc.
But then you fail :
Do Not import your saved backup configuration !

Do these simple steps :
Install clean.
Assign LAN interfaces, activates DHCP where needed.
Assign a WAN and connection method.
Change the admin password.

And now the critical thing : do nothing else.
No config import - do nothing.
Done !

pfSense works out of the box.
Your issue is solved.
And you also know now what the issue was ....

nimrod

@jc1976 this was never a pfSense or Xfinity issue to begin with. The error you got was clearly a browser/plugin issue as i described in my first post in this thread. Just because you didnt have any issues before, doesnt mean that the issue was not there. Browsers and their plugins are constantly getting updated, and with those updates, their behavior changes as well. Previous versions could try to establish HTTPS traffic, and if they fail, they just continue to HTTP without you even knowing. This could be browser "security issue" so Mozilla decided to change this and warn you if HTTPS traffic is not possible, and force you to "accept the risks" before you continue.

There is no need to use gparted or any other 3rd party software to wipe partitions. pfSense installer is doing that by default. Im not sure why you removed pfBlockerNG, Squid and Suricata if you have more than capable hardware. Neither of those packages were causing the issue you had. You are just paranoid now for no reason.

You also said, and i quote:

"i just need to figure out why i'm having so many issues with it all of a sudden. I wish i new how to read logs better.."

What other issues are you having ?

As for the RAM memory, some amount of your total system memory is shared with onboard graphic card. This can be changed in BIOS. pfSense doesnt need much video RAM since its not running Xorg graphic server. Its all textual or web interface mode. Unfortunately some BIOS-es wont let you set this shared memory size less than 32, 64 and on some system even 128mb of RAM. If you want to overcome this issue, you need a dedicated video card, which is totally pointless since you have 16GB of RAM which is overkill anyway.

stephenw10

This is not a RAM issue.

It's almost certainly a browser issue. That should be easy to check though by just using a different browser.
It could be a combination of things and that browser plugin is just revealing it.

Like something is resolving those to a different page, maybe a warning or error, and your browser is showing that because it's http.

Steve

johnpoz

What is funny is his example is showing https://amazon.com in the url address bar.

If you go to amazon.com via http or https it should be sending his browser a redirection.

Resolving amazon.com (amazon.com)... 205.251.242.103, 176.32.103.205, 54.239.28.85
Connecting to amazon.com (amazon.com)|205.251.242.103|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://amazon.com/ [following]
--2021-11-11 09:29:03--  https://amazon.com/
Connecting to amazon.com (amazon.com)|205.251.242.103|:443... connected.

--2021-11-11 09:29:39--  https://amazon.com/
Resolving amazon.com (amazon.com)... 205.251.242.103, 176.32.103.205, 54.239.28.85
Connecting to amazon.com (amazon.com)|205.251.242.103|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.amazon.com/ [following]
--2021-11-11 09:29:39--  https://www.amazon.com/
Resolving www.amazon.com (www.amazon.com)... 23.11.225.174
Connecting to www.amazon.com (www.amazon.com)|23.11.225.174|:443... connected.

Notice the "301 Moved Permanently" and then the new www. url and following to that..

He could have something blocking something - and his browser is showing this error as a red herring because it doesn't know any better?

I would love to see a wget even - take the browser out of the equation.

nimrod

@stephenw10 said in website security problems:

This is not a RAM issue.

It's almost certainly a browser issue. That should be easy to check though by just using a different browser.
It could be a combination of things and that browser plugin is just revealing it.

Like something is resolving those to a different page, maybe a warning or error, and your browser is showing that because it's http.

Steve

@stephenw10 not sure if you saw, but issue is already resolved. Check the post from @johnpoz. Issue is exactly what i described in my first post. Also, i never said it was a RAM issue. @jc1976 was wondering where did the certain amount of total system RAM go, so i explained him that system RAM is shared with onboard graphics.

johnpoz

@nimrod I am not sure I would consider it "resolved" until we actually hear from the user that he has it working now.

stephenw10

@nimrod said in website security problems:

@stephenw10 not sure if you saw, but issue is already resolved. Check the post from @johnpoz. Issue is exactly what i described in my first post. Also, i never said it was a RAM issue.

I realise. But @jc1976 suspected it might be and I was ruling that out.

Yeah, is that browser setting blocking the http redirect? Seems like that should happen over https but I've never dug too deeply into that.

Steve

nimrod

@johnpoz

Im constantly getting notified that posts have been edited, i thought i saw he said issue was resolved. I scrolled back now, and i dont see it. Im sorry, my bad.

jc1976

@nimrod

Regarding trying this all out with a different browser and different computers, that was done.

i attempted the same thing with edge, and my GF is using her mac with chrome, she's having the same issues as well. She has two mac laptops (one for work, one personal), both have the same issue.

So, i'll make the change with firefox but as i said before, this wasn't an issue a few weeks ago and up until last night absolutely nothing has changed. last night i uninstalled, rebooted, reinstalled (as default) firefox and the issue is still there.

johnpoz

@jc1976 said in website security problems:

and my GF is using her mac with chrome, she's having the same issues as well.

And again pfsense has ZERO systems to make such a notification to the browser - that error is a BROWSER thing.. Now its possible your filtering the 301 redirection, but I really don't see anything in pfsense that would do that.. Maybe some sort of IPS?

How about you do a simple fetch or wget.. Now www.amazon.com is going to resolve to something different than amazon.com as you can see in my outputs from wget..

If your browser is not seeing the redirection and tries to load https://amazon.com via 443 then yeah the cert could fail and your BROWSER would warn you about it..

edit:
But when you look at my test of https://amazon.com I get the 301 redirect over 443, and the cert is trusted, etc.. Maybe you have something wrong with your systems Trusted Certs?

Here for example using debug on the wget I can see the ssl info, etc. and via that connection I get 301 that it moved to https://www.amazon.com

--2021-11-11 15:05:53--  https://amazon.com/
Resolving amazon.com (amazon.com)... 205.251.242.103, 176.32.103.205, 54.239.28.85
Caching amazon.com => 205.251.242.103 176.32.103.205 54.239.28.85
Connecting to amazon.com (amazon.com)|205.251.242.103|:443... connected.
Created socket 3.
Releasing 0x00005653b3975a30 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00005653b3975b50
certificate:
  subject: CN=*.peg.a2z.com
  issuer:  CN=DigiCert Global CA G2,O=DigiCert Inc,C=US
X509 certificate successfully verified and matches host amazon.com

---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: amazon.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 301 Moved Permanently
Server: Server
Date: Thu, 11 Nov 2021 21:05:53 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive
Location: https://www.amazon.com/
Permissions-Policy: interest-cohort=()

stephenw10

Mmm, that error you're seeing in Firefox is a specific firefox error. So what error do you see in other browsers? There are almost certainly some clues there.

Steve

johnpoz

@stephenw10 I have tried to duplicate this with firefox turning on https-only, etc. And going to http or https://amazon.com or pfsense.com etc.. all just work fine - with the 301 redirection to www.

Maybe a proxy or ips thing? Proxy maybe doing some sort of https mitm?

bPsdTZpW

@jc1976 I suspect DNS is serving incorrect addresses, or possibly some security software is acting as a MITM and is not handling amazon.com's redirect correctly. What's the output if you nslookup amazon.com , then nslookup www.amazon.com ? Is the former IP in a range owned by Amazon (lookup at https://whois.arin.net )?

FWIW, I do not see this problem when enabling HTTPS-only mode in FF (v94.0).

stephenw10

Mmm, I could certainly believe DNS shenanigans.

Try just resolving locally, stop sending your queries to Cloudflare.

jc1976

@stephenw10

Hey fellas! really, thank you SO much for your input in all this!

the issue also came up on edge and chrome. i only have firefox and edge on my home workstation. I tried it all on my laptop as well and had the same issues.

regarding resolving locally instead of cloudflare, i'm a bit uncertain what you mean by that. DNS settings are:

127.0.0.1
1.1.1.1
1.0.0.1

I don't use dnsmasq (forwarder), just unbound.

disabling https: only mode in all windows (under firefox) seems to have things behaving more normally, but again, i don't have ANY services enabled (suricata, pfblocker, squid/clamav). so who knows what will happen if i were to re-enable those.

I also don't have any rules setup, so i don't know what is causing my trading apps to hang upon startup, at least not at the moment because again, anything that would block them has been shut down..

i've even shut down my windows defender firewall to make sure that couldn't be causing the hang-up.. it's all very weird.

Johnpoz mentioned filtering 301 redirection.. i'm gonna do my research and find out how to figure that out because this whole thing feels like a filtering issue, i just don't know how to go about figuring that out (still learning).

i trying to explain to our consultant at work what goes on..
-i opened up firefox to DDG.
-searched for ad blocker test
-given a list of sites to test ad blocks.
one of which i used is "https://canyoublockit.com"

clicking on the link for https://canyoublockit.com yields nothing.. the site just doesn't open. clicking multiple times does nothing, it just sits there as if i didn't click on anything. sometimes it would give me the security error... but otherwise nothing happens.. then if i close out and come back, maybe the site would open.. it's random..

stephenw10

@jc1976 said in website security problems:

I have cloudflare setup for dns over tls

If you are doing that then you're sending DNS queries to them rather than resolving locally. Assuming it's setup correctly.

What error do you see in Edge? Or any other browser on any other client?

Steve