The strange story of accessing certain websites.
-
@stephenw10 said in The strange story of accessing certain websites.:
@mcury said in The strange story of accessing certain websites.:
server is answering with a fin ack directly
The client is sending the fin-ack. But the fin is not shown. The client must have seen it though.
so, if its not MTU, or a loop, and work with some browsers and not others, it must be something with the website/browser compatibility.
this wouldn't be the first time that this site presents problems with browsers, but the weird part is that in the past, it was Firefox that had problems with it.. at that time, a banner would show up showing "outdated browser detected".
-
@stephenw10 said in The strange story of accessing certain websites.:
Were those pcap filtered?
Yes, but both filtered the same way. ip.addr == 13.0.0.0/8
It was captured on pfSense. I'll try it again on the PC side@stephenw10 said in The strange story of accessing certain websites.:
. Could there be some route symmetry somehow? Loadbalancing between the WANs? You do still occasionally see sites that can't handle that.
No load-balancing currently. I have tried to disable WAN2 interface completely, same behavior exists.
-
I've found just setting MTU can be insufficient. Try also setting MSS to the same value.
-
@stephenw10
edge_less_filtered_upload.pcap
Direct capture from PC, filtered unnecessary LAN traffic only. -
@dem
Can't set it right now, may be tomorrow morning... -
@mcury said in The strange story of accessing certain websites.:
Does this issue only happen with that website ?
Sorry for the late response. I have similar problems with some microsoft pages, like https://answers.microsoft.com/en-us and again, firefox is OK and EDGE is failing to display the page correctly like it is in text mode partially or corrupted, same I see on android phone
-
Ok. Looks like the problem is solved!
I just went to the WAN1 and removed MTU and MSS completely!
No
It was just some cached result, whatever... not working again
Setting MSS also have no effect -
The problem is isolated, and it seems that some network setting or equipment of my network is to blame because when connecting the laptop or this PC directly with a cable to the ISP and establishing a PPPoE connection, all websites are accessible and everything works fine.
Edit1:
I have CARP and now switched to the secondary firewall and looks like the problem is gone, this time it is isolated to the next step — only when primary firewall is active.after a while, this is not true anymore, because EDGE refuses to access the site again.
Tried playing with TSO/LRO settings, compared routes, don't see much difference. Yes, the hardware on the firewalls is different, but by adding LAGG interfaces, they are brought to a "compatible" state for synchronization. The rules and settings are almost identical.
I got lost... what's next?
Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com
Edit3: after a 20 min. i can still ping micron.com but the site is no more accessible in EDGE, but firefox works.
WTF, magic?
Edit4: Same story on the secondary firewall after a while. Will re-test on the second ISP (WAN2) and will re-test on direct connection to the WAN1 ISP... -
@w0w your pcap for edge is showing errors for K-Meleon.
Check packet 15 of edge_less_filtered_upload.pcap<html><head><title>Encountered a 404 error</title> <body> <p>An error has been encountered in accessing this page. <p>1. <b>Server:</b> kmeleon.sourceforge.net <br />2. <b>URL path:</b> /update.php <br />3. <b>Error notes:</b> NONE <br />4. <b>Error type:</b> 404 <br />5. <b>Request method:</b> GET <br />6. <b>Request query string:</b> NONE <br />7. <b>Time:</b> 2024-04-06 17:30:09 UTC (1712424609) <p><b>Reporting this problem:</b> The problem you have encountered is with a project web site hosted by SourceForge.net. This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net). <p><i>If this is a severe or recurring/persistent problem,</i> please do one of the following, and provide the error text (numbered 1 through 7, above): <ol><li>Contact the project via their <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmeleon.sourceforge.net&support=1">designated support resources</a>. <li>Contact the project administrators of this project via email (see the upper right-hand corner of the <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmele -
Since making changes there appear to allow it to work initially did you test the direct PPPoE connection from a laptop for long enough to sure that too wouldn;t fail after some time?
-
@stephenw10
I just re-tested on PC, long enough, no problem.@mcury said in The strange story of accessing certain websites.:
your pcap for edge is showing errors for K-Meleon.
I think we can ignore it. K-meleon was running in the background.
-
@w0w said in The strange story of accessing certain websites.:
Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com
Your pcaps don't point to any MTU problems, I don't see packet out of order, retransmissions..
-
It seems the issue is somehow related to IPv6. WAN1 is using IPv6, and I'm getting all the addresses normally, and even online IPv6 tests are passing without any problems. But the fact is, I've disabled the use of IPv6 now, and it seems like everything is working.
Now I don't really know what to do about it and which direction to investigate. Similarly, when connecting directly, IPv6 works perfectly fine in Windows. -
@w0w check if these can help you.
Note: I just disable IPv6.https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html -
Aha, nice catch. That would explain the hosted VMs I guess. And you just don't see the v6 traffic in the filtered pcap.
-
@stephenw10
Yes, hosted VM does not use IPv6
Only WAN1 utilizes IPv6. In principle, Multi-WAN is used in failover mode only for IPv4 (Tier 1 for PPPoE-WAN1 and Tier 2 for WAN2). At the moment, I don't understand what impact IPv6 has in this case.
AFAIK micron.com does not use IPv6
also
this was checked next to allow IPv6
So what can be wrong with IPv6, any thoughts? -
A little update...
It seems that everything is somehow linked to the IPv6 and mpd5 settings. When I capture all PPPoE session packets on the pfSense side, I notice a high number of retransmitted IPv6 packets and duplicates.
With a direct connection, I observe smooth traffic flow without IPv6 retransmissions, and I notice that EDGE initiates the QUICK protocol. However, when using the pfSense connection, I have NEVER seen QUICK being used. -
@w0w said in The strange story of accessing certain websites.:
However, when using the pfSense connection, I have NEVER seen QUICK being used.
QUIC - UDP/80 and UDP/443, allow these:
LAN_NET UDP to any 80/443 and QUIC will work. -
@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea. -
@w0w said in The strange story of accessing certain websites.:
@mcury
I apologize for the confusion; I may have misspoken. I believe that, if anything, the issue might be related to the QUICK protocol, but specifically in connection with the micron.com website for some unknown reason. Some found online QUICK protocol tests proceed without any issues. I think the functionality of the QUICK protocol itself is not the problem.
What is definitely happening, when I try to access micron.com with Ipv6 enabled is some attempt to communicate with Microsoft servers via IPv6, and something seems to be going wrong in that process, why it is going through IPv6 I have no idea.IPv6 means that the device itself will get a public IP, mostly, assuming that you are using Track interface.
What I think that is happening there is that one device is getting a public IPv6 from one of the providers, lets say primary link, and once the primary link goes down, the problem starts because the IPv6 in device is no longer valid but the device doesn't know it.There are a few ways to circumvent that:
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.htmlMy go-to for these situations is to just disable IPv6 in both WANs... And know what ? I don't miss it.
