The strange story of accessing certain websites.

w0w

@stephenw10
Yep, maybe, but this traceroute was performed in the VM, where I can access this site at least in IE, Chrome, and Firefox. Those two PCAPs are from this PC, with Firefox accessing this site and Edge not.
nonworking_edge.pcap
working_firefox.pcap
So what is the magic?

stephenw10

Run a pcap, look for differences. I'd check the TTL first. Thought it looks like the loop is exhausting the TTL.

Edit: I see you did!

mcury

@stephenw10 said in The strange story of accessing certain websites.:

Edit: I see you did!

I don't think its a loop anymore, server is answering with a fin ack directly

stephenw10

Were those pcap filtered?

The non-working one does have the initial syn / syn-ack / ack handshake.

It appears to initially connect to something via http and get's redirected to https. That can see the browser sending the request.

In the failed connection we are not seeing all the traffic in that pcap. Could there be some route symmetry somehow? Loadbalancing between the WANs? You do still occasionally see sites that can't handle that.

stephenw10

@mcury said in The strange story of accessing certain websites.:

server is answering with a fin ack directly

The client is sending the fin-ack. But the fin is not shown. The client must have seen it though.

mcury

@stephenw10 said in The strange story of accessing certain websites.:

@mcury said in The strange story of accessing certain websites.:

server is answering with a fin ack directly

The client is sending the fin-ack. But the fin is not shown. The client must have seen it though.

so, if its not MTU, or a loop, and work with some browsers and not others, it must be something with the website/browser compatibility.

this wouldn't be the first time that this site presents problems with browsers, but the weird part is that in the past, it was Firefox that had problems with it.. at that time, a banner would show up showing "outdated browser detected".

https://github.com/webcompat/web-bugs/issues/101995

w0w

@stephenw10

@stephenw10 said in The strange story of accessing certain websites.:

Were those pcap filtered?

Yes, but both filtered the same way. ip.addr == 13.0.0.0/8
It was captured on pfSense. I'll try it again on the PC side

@stephenw10 said in The strange story of accessing certain websites.:

. Could there be some route symmetry somehow? Loadbalancing between the WANs? You do still occasionally see sites that can't handle that.

No load-balancing currently. I have tried to disable WAN2 interface completely, same behavior exists.

dem

I've found just setting MTU can be insufficient. Try also setting MSS to the same value.

w0w

@stephenw10
edge_less_filtered_upload.pcap
Direct capture from PC, filtered unnecessary LAN traffic only.

w0w

@dem
Can't set it right now, may be tomorrow morning...

w0w

@mcury said in The strange story of accessing certain websites.:

Does this issue only happen with that website ?

Sorry for the late response. I have similar problems with some microsoft pages, like https://answers.microsoft.com/en-us and again, firefox is OK and EDGE is failing to display the page correctly like it is in text mode partially or corrupted, same I see on android phone

w0w

Ok. Looks like the problem is solved!
I just went to the WAN1 and removed MTU and MSS completely!
No It was just some cached result, whatever... not working again
Setting MSS also have no effect

w0w

The problem is isolated, and it seems that some network setting or equipment of my network is to blame because when connecting the laptop or this PC directly with a cable to the ISP and establishing a PPPoE connection, all websites are accessible and everything works fine.

Edit1: ~~I have CARP and now switched to the secondary firewall and looks like the problem is gone, this time it is isolated to the next step — only when primary firewall is active.~~ after a while, this is not true anymore, because EDGE refuses to access the site again.
Tried playing with TSO/LRO settings, compared routes, don't see much difference. Yes, the hardware on the firewalls is different, but by adding LAGG interfaces, they are brought to a "compatible" state for synchronization. The rules and settings are almost identical.
I got lost... what's next?
Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com
Edit3: after a 20 min. i can still ping micron.com but the site is no more accessible in EDGE, but firefox works.
WTF, magic?
Edit4: Same story on the secondary firewall after a while. Will re-test on the second ISP (WAN2) and will re-test on direct connection to the WAN1 ISP...

mcury

@w0w your pcap for edge is showing errors for K-Meleon.
Check packet 15 of edge_less_filtered_upload.pcap

<html><head><title>Encountered a 404 error</title>
<body>
<p>An error has been encountered in accessing this page.
<p>1. <b>Server:</b> kmeleon.sourceforge.net
<br />2. <b>URL path:</b> /update.php
<br />3. <b>Error notes:</b> NONE
<br />4. <b>Error type:</b> 404
<br />5. <b>Request method:</b> GET
<br />6. <b>Request query string:</b> NONE
<br />7. <b>Time:</b> 2024-04-06 17:30:09 UTC (1712424609)
<p><b>Reporting this problem:</b> The problem you have encountered is with a project web site hosted by SourceForge.net.  This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net).
<p><i>If this is a severe or recurring/persistent problem,</i> please do one of the following, and provide the error text (numbered 1 through 7, above):
<ol><li>Contact the project via their <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmeleon.sourceforge.net&support=1">designated support resources</a>.
<li>Contact the project administrators of this project via email (see the upper right-hand corner of the <a href="https://sourceforge.net/support/prweb-lookup.php?host=kmele

stephenw10

Since making changes there appear to allow it to work initially did you test the direct PPPoE connection from a laptop for long enough to sure that too wouldn;t fail after some time?

w0w

@stephenw10
I just re-tested on PC, long enough, no problem.

@mcury said in The strange story of accessing certain websites.:

your pcap for edge is showing errors for K-Meleon.

I think we can ignore it. K-meleon was running in the background.

mcury

@w0w said in The strange story of accessing certain websites.:

Edit2: I just went a bit crazy and set the MTU to 1200 on the primary firewall... The result is that I can now access and ping micron.com

Your pcaps don't point to any MTU problems, I don't see packet out of order, retransmissions..

w0w

It seems the issue is somehow related to IPv6. WAN1 is using IPv6, and I'm getting all the addresses normally, and even online IPv6 tests are passing without any problems. But the fact is, I've disabled the use of IPv6 now, and it seems like everything is working.

Now I don't really know what to do about it and which direction to investigate. Similarly, when connecting directly, IPv6 works perfectly fine in Windows.

mcury

@w0w check if these can help you.
Note: I just disable IPv6.

https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

stephenw10

Aha, nice catch. That would explain the hosted VMs I guess. And you just don't see the v6 traffic in the filtered pcap.