he.net problem tunnel, one works the other does not
-
@gertjan thanks for the reply.
I'm certain that the pppoe tunnel worked years ago when I set it up as I only had one WAN at the time. When I added the second WAN it was 5x faster so the new WAN became the default gateway and the pppoe WAN became the backup and barely used.
I have checked that the client IPv4 address is correct for both tunnels. I'm using the dynamic DNS method to keep those IPv4 addresses updated. If the IPv4 address is wrong the gateway status goes offline. What I forgot to mention in the original post is that the gateway status for the problem tunnel is online, which makes sense because pings are working.
I have checked the GIF status multiple times thinking something has to be wrong, but they're both setup the same. At first, I thought the problem had to be something wrong in the GIF settings, but I can't see anything wrong there. I will check again.
If I bring the working tunnel down, or unplug the ethernet cable from the cable modem, then pfsense will switch the IPv6 gateway to the pppoe tunnel as expected, because it thinks it's online. When this happens the behavior doesn't change, pings and tracert work but http/https does not. In this can the IPv4 gateway switches over and works fine.
Sorry, I should have mentioned the curl command was being run from a Windows machine, and yes using policy routing. If I run the curl command directly on the pfsense box's command line it just hangs until I get a 504 gateway time-out, curl --interface gif0 ip6.me/api/. If I run the curl command on gif1 is returns the result almost immediately.
Thanks again.
-
@wbond said in he.net problem tunnel, one works the other does not:
If I bring the working tunnel down
Not related to your question, but, in this case, your tunnel will fail, right ? The GIF setup is 'hard coded' to an interface name, probably your main '5x faster' WAN.
@wbond said in he.net problem tunnel, one works the other does not:
pfsense box's command line it just hangs until I get a 504 gateway time-out, curl --interface gif0 ip6.me/api/
There you have your error : the GIF tunnel doesn't work.
I have :
[2.5.2-RELEASE][root@pfsense.right-here.net]/root: curl --interface gif0 ip6.me/api/ IPv6,2001:470:1f12:xxx::2,Remaining fields reserved for future use,,,where 2001:470:1f12:xxx::2 is my local tunnel IP.
-
If I bring the cable tunnel (GIF1) down the pppoe tunnel (GIF0) becomes the default gateway, because pings are working pfsense thinks the pppoe tunnel is working ok. In this case I lose IPv6 http/https connectivity, but IPv6 pings and tracert still work. When I bring the cable tunnel back up it becomes the default gateway again and everything works normally. Not sure if this answers your question.
Yes, the error is the GIF0 tunnel doesn't work with http/https, but pings/tracert do work. The part I haven't figured out is why is it not working for http/https? The GIF1 tunnel works as expected and passes any IPv6 tests I've thrown at it.
thanks
-
Below is a packet capture from the gif0 interface when trying to run "curl --interface gif0 ip6.me/api/" on the webgui diagnostics > command prompt. The 2001:470 address is the client end of the IPv6 tunnel. This creates the connection, and creates a pair of states on the firewall, but then the connection times out with a "504 gateway time-out".
All the while dpinger is able to ping a google ipv6 address (2001:4860:4860::8844) through the same interface, so the gateway shows online. The same curl command on gif1 completes without any error.
I'm not good at reading packet captures, but don't see any reason as to why it fails. It has something to do with the pppoe connection, I think, but don't understand why. If I create a completely new tunnel it will fail on the same way on the same connection. The isp is Centurylink in case it matters. The firewall is running on a Supermicro motherboard with 4 igb ports, so it's fairly decent hardware.
Any ideas of what else I could check would be most welcome.
17:36:23.962116 AF IPv6 (28), length 84: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 40) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [S], cksum 0x6f27 (correct), seq 1841097079, win 65228, options [mss 1220,nop,wscale 7,sackOK,TS val 419389964 ecr 0], length 0
17:36:24.016422 AF IPv6 (28), length 84: (flowlabel 0x04ecc, hlim 54, next-header TCP (6) payload length: 40) 2001:4838:0:1b::201.80 > 2001:470:xxxx:yyyy::2.30098: Flags [S.], cksum 0x3a7c (correct), seq 2521845452, ack 1841097080, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 737747318 ecr 419389964], length 0
17:36:24.016478 AF IPv6 (28), length 76: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 32) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [.], cksum 0x66fa (correct), seq 1, ack 1, win 515, options [nop,nop,TS val 419390018 ecr 737747318], length 0
17:36:24.016619 AF IPv6 (28), length 150: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 106) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [P.], cksum 0xa1d0 (correct), seq 1:75, ack 1, win 515, options [nop,nop,TS val 419390018 ecr 737747318], length 74: HTTP, length: 74
GET /api/ HTTP/1.1
Host: ip6.me
User-Agent: curl/7.76.1
Accept: /17:36:24.380784 AF IPv6 (28), length 150: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 106) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [P.], cksum 0xa063 (correct), seq 1:75, ack 1, win 515, options [nop,nop,TS val 419390383 ecr 737747318], length 74: HTTP, length: 74
GET /api/ HTTP/1.1
Host: ip6.me
User-Agent: curl/7.76.1
Accept: /17:36:24.910777 AF IPv6 (28), length 150: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 106) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [P.], cksum 0x9e51 (correct), seq 1:75, ack 1, win 515, options [nop,nop,TS val 419390913 ecr 737747318], length 74: HTTP, length: 74
GET /api/ HTTP/1.1
Host: ip6.me
User-Agent: curl/7.76.1
Accept: /17:36:25.025399 AF IPv6 (28), length 84: (flowlabel 0x04ecc, hlim 54, next-header TCP (6) payload length: 40) 2001:4838:0:1b::201.80 > 2001:470:xxxx:yyyy::2.30098: Flags [S.], cksum 0x368c (correct), seq 2521845452, ack 1841097080, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 737748326 ecr 419389964], length 0
17:36:25.025444 AF IPv6 (28), length 76: (hlim 64, next-header TCP (6) payload length: 32) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [.], cksum 0x62c0 (correct), seq 75, ack 1, win 514, options [nop,nop,TS val 419391027 ecr 737747318], length 0
17:36:25.770775 AF IPv6 (28), length 150: (flowlabel 0x4bdcc, hlim 64, next-header TCP (6) payload length: 106) 2001:470:xxxx:yyyy::2.30098 > 2001:4838:0:1b::201.80: Flags [P.], cksum 0x9af5 (correct), seq 1:75, ack 1, win 515, options [nop,nop,TS val 419391773 ecr 737747318], length 74: HTTP, length: 74
GET /api/ HTTP/1.1
Host: ip6.me
User-Agent: curl/7.76.1
Accept: / -
I'm revisiting this because I haven't found a solution. Previous posts left out that I'm using pfsense 2.5.2.
In the packet capture I think the 3 way handshake and state creation rule out asymmetric routing. Is that true? What else could prevent a response to the curl request from coming back? The packet size of a response that works on the other GIF connection is only 357 which is well below the MSS of 1220, so I think that rules out a MTU problem on the pppoe WAN.
Both asymmetric routing and MTU problems have seemingly been causes of others having similar problems so I'm trying to rule those out. I'm not sure if the problem is something in my pfsense configuration, but that seems to me the most likely problem.
-
@wbond I don't have a 2nd IP to test with and he will only let you bring up a 2nd tunnel from another IP.
Are you trying to bring up a 2nd tunnel with the same /48 or a different /48? Are you going to the same HE pop?
-
@johnpoz thanks for the reply.
They are two different /48's on different HE pops, Denver and Fremont. You are correct in that you need a 2nd IP to bring up a second tunnel. I tried creating a new tunnel in another pop and had the same result. The problem follows the pppoe / DSL connection. I don't think it could be an HE.net problem, but don't really see how it could be a Centurylink problem either. This leads me back to my pfsense configuration being the likely problem, but nothing I've tried changes the behavior.
I have a backup firewall box and think I'll try starting from scratch with a single WAN on the pppoe connection and see if that behaves correctly.
-
I have a backup firewall box and think I'll try starting from scratch with a single WAN on the pppoe connection and see if that behaves correctly.
On my backup firewall I deleted all references to the second WAN interface, 2nd GIF, extra gateways, etc., leaving only the pppoe connection and one HE net connection. The behavior is the same, pings and traceroute work but http/https does not. I think this rules out the multi WAN configuration as being the problem, but still no closer to figuring out what the problem is.
-
@wbond
Just one WAN and still issues ?
Most of us have one WAN, and it works.
I was using pppoe myself in the past, but isn't possible any more, as my ISP forces me us to use their xDSL router.
Can you switch your modem to router mode and see if the issue disappears, thus related to 'pppoe' ?No firewall rules on WAN are needed to make this work.
Exception : your real WAN IPv4 must reply on pings from the tunnel server to get your WAN IPv4 accepted as a tunnel endpoint. -
@gertjan yes, issues even with just the one WAN.
I currently have the DSL modem in bridge mode. I had not considered switching it, but that is worth trying, thanks for the idea.
Yes, the only related rule on real WAN is the rule allowing pings from HE.net.
-
@wbond
Ok, great.
The dull thing about "ISP router in in router mode" is : it should work, as my he IPv6 tunnels are all up right now @work and also the one @home.
I'm using the he.net POP in Paris.Note : he.net is supplying me with IPv6, as my ISP doesn't know what that is (@work) - or, @home, they just supply on /64. so none are available for the LAN's.
Btw : always check the tunnel status. And if doubt, the forum on he.net.
