Single Web Site Not Working

MX5100

Hey Folks,

I am having trouble with a "Web Site" timing out and not being displayed.
I will try to explain the order of things as best I can.

The web site is "Pilotglobal.com", I was using "Ipfire" another firewall router for those not familiar. I started visiting the web site in November, looking for jobs. The website worked perfectly until about a month ago. At first I thought it may be "Ipfire", so I took that computer offline and installed a "Linksys" consumer grade router that is about 10 years old. It allowed the website to work perfectly. At about the same time Ipfire issued an update which I installed thinking it may resolve my issue. It did not. I have used Pfsense on and off since the MonoWall days, so I decided to install version 2.6 onto the Ipfire box, long story short same problem. I used the "Basic Firewall Configuration Example" straight out ot the Documentation.

I do not have any Packages Installed, I do not have "VPN" service running

I will REEMPHASIZE every web site in the world works except this one
www.pilotsglobal.com

I have gone through "network conectivity" trouble shooting.
I have gone through "trouble shooting website access"
I have done this:

Check Clear invalid DF bits instead of dropping the packets on System > Advanced, Firewall/NAT tab.

Check Disable Firewall Scrub on System > Advanced, Firewall/NAT tab.

Disable hardware checksums

I will REEMPHASIZE... If I install the old Linksys router in place of the Pfsense router the web site works.

DNS Lookup
Hostname
Results
Result Record type
143.244.163.74 A
Timings
Name server Query time
127.0.0.1 0 msec
75.75.75.75 95 msec
75.75.76.76 132 msec

PING pilotsglobal.com (143.244.163.74): 56 data bytes
64 bytes from 143.244.163.74: icmp_seq=0 ttl=34 time=69.027 ms
64 bytes from 143.244.163.74: icmp_seq=1 ttl=34 time=72.606 ms
64 bytes from 143.244.163.74: icmp_seq=2 ttl=34 time=63.600 ms

--- pilotsglobal.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 63.600/68.411/72.606/3.702 ms

Traceroute

1 96.120.15.145 6.807 ms 7.491 ms 6.132 ms
2 68.86.247.25 7.798 ms 7.756 ms 6.767 ms
3 96.108.173.205 5.890 ms 7.954 ms 6.483 ms
4 68.86.241.201 7.505 ms 7.912 ms 7.416 ms
5 162.151.118.169 14.762 ms 14.727 ms 15.770 ms
6 68.86.95.121 23.501 ms 25.941 ms
68.86.95.113 22.562 ms
7 96.110.34.94 35.376 ms 24.902 ms 23.936 ms
8 80.239.160.213 23.487 ms 24.139 ms 23.611 ms
9 62.115.136.118 24.820 ms * *
10 * * *
11 * * 62.115.137.54 40.838 ms
12 62.115.138.70 56.935 ms 55.886 ms 56.064 ms
13 62.115.141.245 62.052 ms 61.756 ms 67.292 ms
14 62.115.112.69 62.343 ms 61.012 ms 61.489 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *

I use Manjaro Linux and Firefox Browser. I have tried on my wife's Windows 10 computer using
Chrome ,Microsoft Edge and Firefox. All of the browsers time out

I am just an old retired Airline Pilot, If you folks could point me in the right direction to get this site working I would be Greatfull. I have been happily married for 25 years, that is because I have been gone for 12.5 years

Thanks

NollipfSense

A fellow pilot here as well...do you have any package install such as pfBlockerNG?

MX5100

@nollipfsense

No packages installed.

johnpoz

@mx5100 said in Single Web Site Not Working:

Pilotglobal.com

You sure your just not going to the wrong site..

https://pilotsglobal.com/

works, pilotglobal that you post first does not work. Something serving up some self signed cert for test something..

Also if you go to www.pilotsglobal.com it just sends you a 301 to https://pilotsglobal.com/

There is nothing in stock pfsense that would block some specific website from working - it works here behind my pfsense just fine..

Have you thought that maybe they are blocking your IP, changing out pfsense router with some other router would normally get you a different IP from your isp.

What exactly is happening - what error do you get in your browser? Do you get an error, just a blank page? If your using firefox I would launch the web dev tools - I can see it is loading images from s.pilotsglobal.com for example.

I would sniff on pfsense to see what is happening.. Are they sending you back a RST.. Are they just not answering at all?

Traceroutes don't always give answers for every hop, etc. While it can be a very useful tool, not getting replies doesn't always mean anything is wrong.

It loads up just fine here behind my pfsense.. It is highly unlikely there is anything in a stock pfsense that could cause an issue with one specific site.. Pfsense doesn't care it just moves the packets.. So unless you were running blocking dns, or some sort of IPS it can't really tell if your going to this site or google.com

And I remember IPfire - back when it forked from IPcop ;)

stephenw10

Well I can reach that site no problem through a few pfSense boxes so it's not something specific to pf. And you were seeing it in ipfire too.
It doesn't resolve to an IPv6 address so it's not that.

It must be something unique to your local setup, including your ISP somehow.

Traceroute to it also fails for me using the default udp but succeeds using icmp.

I remember IPCop when it forked from Smoothwall

MX5100

@stephenw10 Screenshot from 2023-05-12 16-48-27.png Screenshot from 2023-05-12 18-31-24.png

Some shots from FireFox

Would changing Pfsense Box from Resolver to Forwarder change things?

MX5100

@mx5100 Screenshot from 2023-05-12 18-59-32.png

Any thing look out of line here?

johnpoz

@mx5100 said in Single Web Site Not Working:

Any thing look out of line here?

Not not really - the only thing that "possible" comes to mind is the site using quic, which its http/https over UDP not tcp.. But just checked my state table when going to the site and shows tcp.. While quic is becoming more popular - I am not aware of any site requiring it, and to be honest not that many using it anyway. But looking at your rules and saying you had a problem connecting to somewhere, that would be the only thing I could think of.

As to resolver vs forwarder - doesn't matter as long as the fqdn returns and IP, and I see the same IP as you for the site.

;; QUESTION SECTION:                                            
;pilotsglobal.com.              IN      A                       
                                                               
;; ANSWER SECTION:                                              
pilotsglobal.com.       3574    IN      A       143.244.163.74

That sort of error could mean the site just didn't answer you, clearly you can ping it... You might get some sort of clue doing a curl to the site.. example.

So first thing would be to see if you get the 301

user@i9-win:~$ curl -v https://www.pilotsglobal.com
*   Trying 143.244.163.74:443...
* Connected to www.pilotsglobal.com (143.244.163.74) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=pilotsglobal.com
*  start date: Mar 17 21:10:01 2023 GMT
*  expire date: Jun 15 21:10:00 2023 GMT
*  subjectAltName: host "www.pilotsglobal.com" matched cert's "www.pilotsglobal.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: www.pilotsglobal.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 13 May 2023 00:52:30 GMT
< Server: Apache
< Location: https://pilotsglobal.com/
< Content-Length: 233
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://pilotsglobal.com/">here</a>.</p>
</body></html>
* Connection #0 to host www.pilotsglobal.com left intact

Then see what you get back when you go to https://pilotsglobal.com without the www.

I snipped the full output.. But the curl cmd should tell you if you get something back, maybe a failure in the tls handshake, etc..

ser@i9-win:~$ curl -v https://pilotsglobal.com
*   Trying 143.244.163.74:443...
* Connected to pilotsglobal.com (143.244.163.74) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=pilotsglobal.com
*  start date: Mar 17 21:10:01 2023 GMT
*  expire date: Jun 15 21:10:00 2023 GMT
*  subjectAltName: host "pilotsglobal.com" matched cert's "pilotsglobal.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: pilotsglobal.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 13 May 2023 00:53:07 GMT
< Server: Apache
< Set-Cookie: _pg_session=8mb21o5bocpb3q3bdku5skubte; expires=Mon, 12-Jun-2023 00:53:07 GMT; Max-Age=2592000; path=/; secure; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: _pg_v=46de-d193-9104-58b5; expires=Mon, 12-Jun-2023 00:53:07 GMT; Max-Age=2592000; path=/; domain=pilotsglobal.com; secure; HttpOnly
< Vary: Accept-Encoding
< Content-Length: 142762
< Content-Type: text/html; charset=UTF-8
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no,user-scalable=0" />
  <meta http-equiv="x-ua-compatible" content="ie=edge" />

  <meta property="og:image" content="https://s.pilotsglobal.com/c/pg-og-image.jpg" />

  <meta name="description" content="The largest pilot job network. "/>
  <title>PilotsGlobal - Pilot Jobs Worldwide</title>

MX5100

@johnpoz Thanks for all your help

Here are my two outputs after invoking "Curl"

Screenshot from 2023-05-12 20-24-25.png

johnpoz

@mx5100 yeah your just not getting an answer from that..

So sniff on your wan of pfsense when you do that command with curl.. What do you see.. This would be a normal connection -- see how my syn comes back with a syn,ack. And then more traffic site is loaded. you can see the tls handshake happening. But first step in any connection anywhere over tcp would be the SYN, and the answer back would be syn,ack..

You can do the packet capture on pfsense under diagnostics, pick your wan interface and put in the sites IP address.

You can then download the capture and view for more details in say wireshark.

If you don't even see a syn go out your wan, then pfsense didn't send on the traffic for whatever reason, and that would explain why you can not talk to that site. If you see the syn and no answer, or maybe a RST? Then the issue is upstream of pfsense. So lets see what that sniff shows.

Another test you could do - because you say you can ping the site, is what size of MTU can you ping. Normal 1500 frame size would be 1472 in this command. Without fragmenting the packet, the -f

$ ping -f -l 1472 143.244.163.74

Pinging 143.244.163.74 with 1472 bytes of data:
Reply from 143.244.163.74: bytes=1472 time=42ms TTL=53
Reply from 143.244.163.74: bytes=1472 time=41ms TTL=53

If for whatever reason you have some mtu issue in your path from you to them.. That could cause a lack of answer.. But a syn, isn't very big, etc So you should see the syn,ack atleast even if you had a mtu problem

MX5100

@johnpoz Here is what I get!

Screenshot from 2023-05-12 20-43-29.png

johnpoz

@mx5100 this

Is a response saying sorry you can not go there - that is completely out of psfsense control.. That is either the site itself or something upstream of pfsense sending you back the icmp message, hey sorry buddy you can get there.

MX5100

@johnpoz

Well we gave it the ole college try??!!!

Thanks, I will try to dig deeper into this

johnpoz

@mx5100 it could be they are blocking your IP... You say it works when you use your other router... What IP do you get then.. I have to assume its different.

What you could try - is either router the traffic through a vpn. Or am thinking your on a cable connection, or atleast get your wan public IP from dhcp... If this is the case.

You could clone the pfsense mac on its wan interface so it has the same mac as your other router.. Then it should in theory get the same IP as your other router, and since you say that works - then it should work in pfsense.

But yeah that response is from something upstream say sorry buddy, nope.

MX5100

@johnpoz

johnpoz you da man I placed the "mac" address of the Linksys router in the mac address field of the wan interface and restarted everything and IT WORKS.....

Thank a million

johnpoz

@mx5100 So your IP was different then.. My guess would be for whatever reason that website doesn't like the IP your pfsense was getting.. Maybe the guy that had the IP before was doing something the website didn't like..

edit: hmmm that is odd, I don't see any other IPs to the forums from you.. Only that one you posted in your capture.. You might want to remove that. I can edit my post of the portion of the screenshot, I should of prob removed your IP (doing that now). If you need help removing that, just let me know.

I would of thought for sure you would get a different IP.. But I only see the one in the forum info I can see as mod.. Hmmm.. The website can not see your mac, hmmm??

Guess the only thing we know for sure, is something upstream of pfsense was saying you can't go their.. If your really curious about it, you could get with your ISP... You would most likely need a level 3 guy in their support to be able to get any answers... Show him the sniff where you got the icmp unreachable, and then when you change the mac it works, etc..

edit2: My theory to why mac with the same IP works, but other does not.. Is in the isp network they have some sort of lacp sort of switching, where and physical path taken determined by mac address.. This is quite common in larger networks where something is used to determine physical path, a hash a mac etc this load shares traffic across a connection with multiple physical paths. When you take path A you can get there, but if you take this other path something is blocking it. But this would have to be happening in the isp network to be honest.

MX5100

@johnpoz

All of that stuff is above my pay grade. Thanks for the help.

Could you please edit my screen capture with an ip address

stephenw10

Do you see any blocked traffic on LAN in the firewall log when you try to open the site?

Try checking the states when you open the site:

Screenshot from 2023-05-13 12-10-27.png

The site tries to load a bunch of font and javascript etc but I have that all blocked by default in the browser and the page still opens fine.

stephenw10

Ooops I missed a bunch of replies here. I see you solved it already. Nice.