The strange story of accessing certain websites.

dem

I've found just setting MTU can be insufficient. Try also setting MSS to the same value.

w0w

@stephenw10
edge_less_filtered_upload.pcap
Direct capture from PC, filtered unnecessary LAN traffic only.

w0w

@dem
Can't set it right now, may be tomorrow morning...

w0w

@mcury said in The strange story of accessing certain websites.:

Does this issue only happen with that website ?

Sorry for the late response. I have similar problems with some microsoft pages, like https://answers.microsoft.com/en-us and again, firefox is OK and EDGE is failing to display the page correctly like it is in text mode partially or corrupted, same I see on android phone

w0w

Ok. Looks like the problem is solved!
I just went to the WAN1 and removed MTU and MSS completely!
No It was just some cached result, whatever... not working again
Setting MSS also have no effect

w0w

The problem is isolated, and it seems that some network setting or equipment of my network is to blame because when connecting the laptop or this PC directly with a cable to the ISP and establishing a PPPoE connection, all websites are accessible and everything works fine.

Edit1: ~~I have CARP and now switched to the secondary firewall and looks like the problem is gone, this time it is isolated to the next step — only when primary firewall is active.~~ after a while, this is not true anymore, because EDGE refuses to access the site again.
Tried playing with TSO/LRO settings, compared routes, don't see much difference. Yes, the hardware on the firewalls is different, but by adding LAGG interfaces, they are brought to a "compatible" state for synchronization. The rules and settings are almost identical.
I got lost... what's next?
Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com
Edit3: after a 20 min. i can still ping micron.com but the site is no more accessible in EDGE, but firefox works.
WTF, magic?
Edit4: Same story on the secondary firewall after a while. Will re-test on the second ISP (WAN2) and will re-test on direct connection to the WAN1 ISP...

mcury

@w0w your pcap for edge is showing errors for K-Meleon.
Check packet 15 of edge_less_filtered_upload.pcap

<html><head><title>Encountered a 404 error</title>
<body>
<p>An error has been encountered in accessing this page.
<p>1. <b>Server:</b> kmeleon.sourceforge.net
<br />2. <b>URL path:</b> /update.php
<br />3. <b>Error notes:</b> NONE
<br />4. <b>Error type:</b> 404
<br />5. <b>Request method:</b> GET
<br />6. <b>Request query string:</b> NONE
<br />7. <b>Time:</b> 2024-04-06 17:30:09 UTC (1712424609)
<p><b>Reporting this problem:</b> The problem you have encountered is with a project web site hosted by SourceForge.net.  This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net).
<p><i>If this is a severe or recurring/persistent problem,</i> please do one of the following, and provide the error text (numbered 1 through 7, above):
<ol><li>Contact the project via their <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmeleon.sourceforge.net&support=1">designated support resources</a>.
<li>Contact the project administrators of this project via email (see the upper right-hand corner of the <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmele

stephenw10

Since making changes there appear to allow it to work initially did you test the direct PPPoE connection from a laptop for long enough to sure that too wouldn;t fail after some time?

w0w

@stephenw10
I just re-tested on PC, long enough, no problem.

@mcury said in The strange story of accessing certain websites.:

your pcap for edge is showing errors for K-Meleon.

I think we can ignore it. K-meleon was running in the background.

mcury

@w0w said in The strange story of accessing certain websites.:

Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com

Your pcaps don't point to any MTU problems, I don't see packet out of order, retransmissions..

w0w

It seems the issue is somehow related to IPv6. WAN1 is using IPv6, and I'm getting all the addresses normally, and even online IPv6 tests are passing without any problems. But the fact is, I've disabled the use of IPv6 now, and it seems like everything is working.

Now I don't really know what to do about it and which direction to investigate. Similarly, when connecting directly, IPv6 works perfectly fine in Windows.

mcury

@w0w check if these can help you.
Note: I just disable IPv6.

https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

stephenw10

Aha, nice catch. That would explain the hosted VMs I guess. And you just don't see the v6 traffic in the filtered pcap.

w0w

@stephenw10
Yes, hosted VM does not use IPv6
Only WAN1 utilizes IPv6. In principle, Multi-WAN is used in failover mode only for IPv4 (Tier 1 for PPPoE-WAN1 and Tier 2 for WAN2). At the moment, I don't understand what impact IPv6 has in this case.
AFAIK micron.com does not use IPv6
also

this was checked next to allow IPv6
So what can be wrong with IPv6, any thoughts?

w0w

A little update...
It seems that everything is somehow linked to the IPv6 and mpd5 settings. When I capture all PPPoE session packets on the pfSense side, I notice a high number of retransmitted IPv6 packets and duplicates.
With a direct connection, I observe smooth traffic flow without IPv6 retransmissions, and I notice that EDGE initiates the QUICK protocol. However, when using the pfSense connection, I have NEVER seen QUICK being used.

mcury

@w0w said in The strange story of accessing certain websites.:

However, when using the pfSense connection, I have NEVER seen QUICK being used.

QUIC - UDP/80 and UDP/443, allow these:
LAN_NET UDP to any 80/443 and QUIC will work.

w0w

@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea.

mcury

@w0w said in The strange story of accessing certain websites.:

@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea.

IPv6 means that the device itself will get a public IP, mostly, assuming that you are using Track interface.
What I think that is happening there is that one device is getting a public IPv6 from one of the providers, lets say primary link, and once the primary link goes down, the problem starts because the IPv6 in device is no longer valid but the device doesn't know it.

There are a few ways to circumvent that:

https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

My go-to for these situations is to just disable IPv6 in both WANs... And know what ? I don't miss it.

w0w

@mcury said in The strange story of accessing certain websites.:

What I think that is happening there is that one device is getting a public IPv6 from one of the providers, lets say primary link, and once the primary link goes down, the problem starts because the IPv6 in device is no longer valid but the device doesn't know it.

The theory is good, but it doesn't explain why IPv6 websites actually open on the same device while IPv4 ones don't at the same time. And... this strange loop that the traceroute showed, it only appears when IPv6 is active. I believe that there's some setting or bug responsible for this behavior. As for solving the problem by turning off IPv6, that's what I'll do for now, although it doesn't actually solve the underlying issue.

w0w

I've had some time to tinker with this problem a bit more. I finally figured out what's going on, at least with the browsers, and why one was working while the other wasn't.

A long time ago, I made changes to the Windows registry, specifically:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters]
"DisabledComponents"=dword:00000020

It turns out that Firefox checks and uses this key, while Edge ignores it and prioritizes IPv6.

Also tried to use HE.net tunnel… again… The main issue is gone immediately, as expected, but…
Of course, it wasn't without its issues. In fact, it has more issues than the native IPv6 from the provider. For example, many sites constantly bother with security checks from Cloudflare, and some sites don't even pass these checks. This is likely due to issues with the incorrect geolocation of the address. Well, not incorrect… reliable services identify it correctly, in the same location as the IPv4, but various lists, like those from MaxMind, for example, don't update the information. And some security thinks it's a VPN… crazy...
Youtube just always stuck somewhere in the middle of the video for no reason.
That's the situation.