The strange story of accessing certain websites.

w0w

Ok. Looks like the problem is solved!
I just went to the WAN1 and removed MTU and MSS completely!
No It was just some cached result, whatever... not working again
Setting MSS also have no effect

w0w

The problem is isolated, and it seems that some network setting or equipment of my network is to blame because when connecting the laptop or this PC directly with a cable to the ISP and establishing a PPPoE connection, all websites are accessible and everything works fine.

Edit1: ~~I have CARP and now switched to the secondary firewall and looks like the problem is gone, this time it is isolated to the next step — only when primary firewall is active.~~ after a while, this is not true anymore, because EDGE refuses to access the site again.
Tried playing with TSO/LRO settings, compared routes, don't see much difference. Yes, the hardware on the firewalls is different, but by adding LAGG interfaces, they are brought to a "compatible" state for synchronization. The rules and settings are almost identical.
I got lost... what's next?
Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com
Edit3: after a 20 min. i can still ping micron.com but the site is no more accessible in EDGE, but firefox works.
WTF, magic?
Edit4: Same story on the secondary firewall after a while. Will re-test on the second ISP (WAN2) and will re-test on direct connection to the WAN1 ISP...

mcury

@w0w your pcap for edge is showing errors for K-Meleon.
Check packet 15 of edge_less_filtered_upload.pcap

<html><head><title>Encountered a 404 error</title>
<body>
<p>An error has been encountered in accessing this page.
<p>1. <b>Server:</b> kmeleon.sourceforge.net
<br />2. <b>URL path:</b> /update.php
<br />3. <b>Error notes:</b> NONE
<br />4. <b>Error type:</b> 404
<br />5. <b>Request method:</b> GET
<br />6. <b>Request query string:</b> NONE
<br />7. <b>Time:</b> 2024-04-06 17:30:09 UTC (1712424609)
<p><b>Reporting this problem:</b> The problem you have encountered is with a project web site hosted by SourceForge.net.  This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net).
<p><i>If this is a severe or recurring/persistent problem,</i> please do one of the following, and provide the error text (numbered 1 through 7, above):
<ol><li>Contact the project via their <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmeleon.sourceforge.net&support=1">designated support resources</a>.
<li>Contact the project administrators of this project via email (see the upper right-hand corner of the <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmele

stephenw10

Since making changes there appear to allow it to work initially did you test the direct PPPoE connection from a laptop for long enough to sure that too wouldn;t fail after some time?

w0w

@stephenw10
I just re-tested on PC, long enough, no problem.

@mcury said in The strange story of accessing certain websites.:

your pcap for edge is showing errors for K-Meleon.

I think we can ignore it. K-meleon was running in the background.

mcury

@w0w said in The strange story of accessing certain websites.:

Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com

Your pcaps don't point to any MTU problems, I don't see packet out of order, retransmissions..

w0w

It seems the issue is somehow related to IPv6. WAN1 is using IPv6, and I'm getting all the addresses normally, and even online IPv6 tests are passing without any problems. But the fact is, I've disabled the use of IPv6 now, and it seems like everything is working.

Now I don't really know what to do about it and which direction to investigate. Similarly, when connecting directly, IPv6 works perfectly fine in Windows.

mcury

@w0w check if these can help you.
Note: I just disable IPv6.

https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

stephenw10

Aha, nice catch. That would explain the hosted VMs I guess. And you just don't see the v6 traffic in the filtered pcap.

w0w

@stephenw10
Yes, hosted VM does not use IPv6
Only WAN1 utilizes IPv6. In principle, Multi-WAN is used in failover mode only for IPv4 (Tier 1 for PPPoE-WAN1 and Tier 2 for WAN2). At the moment, I don't understand what impact IPv6 has in this case.
AFAIK micron.com does not use IPv6
also

this was checked next to allow IPv6
So what can be wrong with IPv6, any thoughts?

w0w

A little update...
It seems that everything is somehow linked to the IPv6 and mpd5 settings. When I capture all PPPoE session packets on the pfSense side, I notice a high number of retransmitted IPv6 packets and duplicates.
With a direct connection, I observe smooth traffic flow without IPv6 retransmissions, and I notice that EDGE initiates the QUICK protocol. However, when using the pfSense connection, I have NEVER seen QUICK being used.

mcury

@w0w said in The strange story of accessing certain websites.:

However, when using the pfSense connection, I have NEVER seen QUICK being used.

QUIC - UDP/80 and UDP/443, allow these:
LAN_NET UDP to any 80/443 and QUIC will work.

w0w

@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea.

mcury

@w0w said in The strange story of accessing certain websites.:

@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea.

IPv6 means that the device itself will get a public IP, mostly, assuming that you are using Track interface.
What I think that is happening there is that one device is getting a public IPv6 from one of the providers, lets say primary link, and once the primary link goes down, the problem starts because the IPv6 in device is no longer valid but the device doesn't know it.

There are a few ways to circumvent that:

https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

My go-to for these situations is to just disable IPv6 in both WANs... And know what ? I don't miss it.

w0w

@mcury said in The strange story of accessing certain websites.:

What I think that is happening there is that one device is getting a public IPv6 from one of the providers, lets say primary link, and once the primary link goes down, the problem starts because the IPv6 in device is no longer valid but the device doesn't know it.

The theory is good, but it doesn't explain why IPv6 websites actually open on the same device while IPv4 ones don't at the same time. And... this strange loop that the traceroute showed, it only appears when IPv6 is active. I believe that there's some setting or bug responsible for this behavior. As for solving the problem by turning off IPv6, that's what I'll do for now, although it doesn't actually solve the underlying issue.

w0w

I've had some time to tinker with this problem a bit more. I finally figured out what's going on, at least with the browsers, and why one was working while the other wasn't.

A long time ago, I made changes to the Windows registry, specifically:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters]
"DisabledComponents"=dword:00000020

It turns out that Firefox checks and uses this key, while Edge ignores it and prioritizes IPv6.

Also tried to use HE.net tunnel… again… The main issue is gone immediately, as expected, but…
Of course, it wasn't without its issues. In fact, it has more issues than the native IPv6 from the provider. For example, many sites constantly bother with security checks from Cloudflare, and some sites don't even pass these checks. This is likely due to issues with the incorrect geolocation of the address. Well, not incorrect… reliable services identify it correctly, in the same location as the IPv4, but various lists, like those from MaxMind, for example, don't update the information. And some security thinks it's a VPN… crazy...
Youtube just always stuck somewhere in the middle of the video for no reason.
That's the situation.

Gertjan

@w0w said in The strange story of accessing certain websites.:

Of course, it wasn't without its issues. In fact, it has more issues than the native IPv6 from the provider. For example, many sites constantly bother with security checks from Cloudflare, and some sites don't even pass these checks. This is likely due to issues with the incorrect geolocation of the address. Well, not incorrect…

The Tunnel end points, and the he.tunnel IPv6 ISP, can be seen as a VPN ISP. After all, if it looks like a VPN and smells like a VPN, then it is a VPN.
And that's known as "non acceptable" for some sites. No really Huricane's fault, but that's how things go.
And worse : Hurricane offers their IPv6 for free. They shouldn't do that [ ].
I remember that Netflix didn't accepted any connection from "IPV Huricane" in the past, so pfBlockerng was equipped with a special "if this host name is visited, then block IPv6 = AAAA records" so I could deal with the exceptions.

w0w

@Gertjan
That's why I'm still digging this IPV6 black hole deeper and deeper

Gertjan

@w0w

Black hole ?
It might be very possible that these guys https://ipv6.he.net/certification/ are one of the very few that offer 'pure' non f#ck#p up IPv6 access on the entire planet.
They actually own and exploit the word wide interconnection cables, mostly on the bottom of the seas.
I tend to think they somewhat invented the IPv6 Internet (ok, not true of course) but they are one of the few that adhere to all the IPv6 RFCs.

A fact it : their access is not 'home' based, they use POPs in every country. And as such, there are some draw backs like speed and, as said above, the connection can be seen as 'VPN'.

Anyway, it us up to us right now to select the right ISP. They are all somewhat IPv4 aware these days - it took them 30 or40 years, but things got ironed out.
Now, up to us, with our choice and out wallet, to vote for the next ISP "that does things right".

For me, he.net was, in the past, a no-brain solution.
I would still using them today, if it wasn't for my (and now I should be using a shipload full of angry words) current ISP that can not 'route' (firewall) (handle) protocol 4.
Yeah, great, it does '6' = TCP and '17' = UDP, and "1" = ICMP, but not "4" .... and this "4" is needed to access he.net. As the Ipv6 packets are packed into a IPv4 packet, hence the IP-in-IP or 6to4.

I've called and written them, my ISP, Orange, and asked them : your first box supported 6in4, your second also. And version 3, and 4 and 5.
Then I got the box number 6, needed because it had the build in fiber ONT.
But no more 6in4 support.
Ok, this box supports IPv6, but it can only offer one (1 !) prefix. And you can't use it in any firewall rule ....
And this is called their 'Pro Box', because they think that companies have "just one LAN"

Ok, so be it - I negotiated a 50 % price cut, as they only do their work "half way".

w0w

@Gertjan
mean my ISP IPv6, not the HE one. HE is actually very good for a free service. I am still trying to find out what is going on and why IPv4-only (Azure or Microsoft cloud, anyway) sites cannot be accessed when my ISP IPv6 is enabled. It works without pfSense in any combination, but I am still missing something to figure out what is misconfigured.