IPv6 traceroute not showing first hop (pfSense)



  • I am trying to resolve a problem which I believe is caused by my ISP. In the process of testing, I'm running traceroute to www.yahoo.com. On IPv4, I see pfSense as the first hop. However, on IPv6, I only get an asterisk, which indicates no response from the router. Is this default on pfSense? Can it be fixed?



  • I just checked with Wireshark and don't even see a time exceeded message from pfSense on IPv6, but I do on IPv4. Is pfSense discarding IPv6 traceroute? On IPv4, I see both the UDP packet going out and ICMP timeout coming back, but on IPv6, I only see the UDP packet going out, without even an single timeout packet coming back.


  • LAYER 8 Global Moderator

    Your going to have to give us more to work with here... I see pfsense as my first hop in a ipv6 trace.. Nothing special here..
    0_1547756413260_ipv6trace.png

    That was from windows, here is from linux on a different ipv6 vlan even
    0_1547756421934_linuxipv6trace.png

    Windows normally does a trace via icmp, while linux udp.



  • $ traceroute -6 ipv6.google.com
    traceroute to ipv6.google.com (2607:f8b0:400b:808::200e), 30 hops max, 80 byte packets
    1 * * *
    2 * * *
    3 * * *
    4 * * *
    5 * * *
    6 * * *
    7 * * *
    8 * * *
    9 * * *

    As you can see, no response, not even from pfSense.


  • LAYER 8 Global Moderator

    Well you got something else going on then.. What are you rules... You doing that stupid ULA shit you love?



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    You doing that stupid ULA shit you love?

    The packets show global addresses, not ULA. They are received at pfSense, as shown in Packet Capture. They are also leaving pfSense and out to the Internet, as shown with Wireshark, between pfSense computer and modem. I'm just not getting any response from pfSense.


  • LAYER 8 Global Moderator

    what packet capture? I don't see any capture..



  • Here's the file. However, I was just mentioning that valid addresses were shown in the captures.
    0_1547757089735_packetcapture.pcapng

    Curious, this site wouldn't accept the Packet Capture .cap file. I had to use Wireshark to save as .pcapng.


  • LAYER 8 Global Moderator

    I have a meeting I have to run to, take a look see later - do a sniff my own trace, etc. And compare. Again what are you rules? Are you bridging? Your also a big fan of that ;)



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    Are you bridging? Your also a big fan of that ;)

    The modem is in bridge mode. It's the only way to use a /56 prefix.

    0_1547758173077_aae594de-033d-4d50-8399-30890305f85b-image.png



  • @jknott said in IPv6 traceroute not showing first hop (pfSense):

    Curious, this site wouldn't accept the Packet Capture .cap file. I had to use Wireshark to save as .pcapng.

    Yeah, I found the same, just renamed it to .*pcap
    https://forum.netgate.com/topic/138124/posting-to-a-forum-issue/8


  • LAYER 8 Global Moderator

    What do you mean your modem is in bridge mode? So your gateway is NOT pfsense?

    001700 ARRIS Group, Inc.

    From your cap that is where your sending the UDP traffic with a TTL of 1, I assume that is your "modem" and not pfsense... So if that is the case why would you think pfsense should answer back as your first hop?

    Trace being sent to ipv6.google.com - the mac is pfsense, not my cable modem..

    0_1548068082469_traceroute.png

    And you notice pfsense sends back answer since the TTL has expired..



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    I have a meeting I have to run to, take a look see later

    WOW!! That was a long meeting!!! 😉

    @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    What do you mean your modem is in bridge mode? So your gateway is NOT pfsense?

    Traceroute was run on a computer behind my firewall/router, so pfSense is the first hop

    001700 ARRIS Group, Inc.

    From your cap that is where your sending the UDP traffic with a TTL of 1, I assume that is your "modem" and not pfsense... So if that is the case why would you think pfsense should answer back as your first hop?

    As I mentioned above, it is the first hop. The modem is in bridge mode, so it should be transparent. The capture is done on a computer that's behind pfSense.

    Trace being sent to ipv6.google.com - the mac is pfsense, not my cable modem..

    <image removed>

    And you notice pfsense sends back answer since the TTL has expired..

    That's the whole issue. I'm not getting a response from pfSense on IPv6, though I do on IPv4.

    BTW, what led to this issue is a problem I have with my ISP. I noticed I was having some performance issues and found IPv6 was not working from my local network. For example pinging & traceroute to Google or Yahoo, with IPv6 failed. However, I could do both from pfSense. In addition, host lookup on the host name for my pfSense firewall shows two different IPv6 addresses, with the last segment of the prefix differing, as well as the entire 64 bit suffix. This leads me to believe there's a routing error on the return path, possibly related to the two addresses. I just spent over an hour on the phone with my ISP's tech support, including 2nd level. They agree there's likely a problem in their network causing this issue.


  • LAYER 8 Global Moderator

    Why would pfsense answer you - you didn't send the traffic to pfsense you sent it to your casa mac address - why would pfsense answer that.. Its not sent to is interface... Look at your pcap - your send it to a 00:17:10:X:X:X via mac which is NOT pfsense is it?? Unless your pfsense is using an interface made by casa?? ;)

    You can not expect pfsense to send you back an answer to something that was not SENT to it..

    Look again at your pcap...

    Are you saying this is pfsense mac address of its interface on your lan side?
    0_1548085902983_macaddress.png

    that 00:17:10 mac
    0_1548086361906_casa.png

    Even if your ISP was not answering trace - you would still see the first hop from pfsense... But I find it hard to believe that 00:17:10 mac is pfsense lan side interface.. Since I show it being casa (my bad read it as 00:17:00 (arris) before... They make modems... So that is the mac of your ISP device?



  • @johnpoz

    Sorry, I must have uploaded the wrong capture. I just ran Wireshark again and here's the capture.

    0_1548087760571_capture.pcapng

    Even if your ISP was not answering trace - you would still see the first hop from pfsense...

    That is what I expect too. Here again is what happens:

    IPv6 - No response from hops beyond pfSense expected due to ISP problem.
    $ traceroute -6 www.yahoo.com
    traceroute to www.yahoo.com (2001:4998:58:1836::11), 30 hops max, 80 byte packets
    1 * * * < I should see the pfSense firewall here
    2 * * *
    3 * * *
    4 * * *
    5 * * *
    6 * * *
    7 * * *
    8 * * *
    9 * * *

    IPv4
    $ traceroute -4 www.yahoo.com
    traceroute to www.yahoo.com (72.30.35.10), 30 hops max, 60 byte packets
    1 [host name removed to protect the guilty] (172.16.0.1) 0.247 ms 0.231 ms 0.219 ms
    2 * * *
    3 24.156.150.217 (24.156.150.217) 19.862 ms 20.057 ms 20.276 ms
    4 0-5-0-6-cgw01.wlfdle.rmgt.net.rogers.com (209.148.233.169) 19.217 ms 0-4-0-6-cgw01.wlfdle.rmgt.net.rogers.com (209.148.233.165) 18.898 ms 0-5-0-6-cgw01.wlfdle.rmgt.net.rogers.com (209.148.233.169) 19.604 ms
    5 209.148.237.5 (209.148.237.5) 39.950 ms 209.148.230.26 (209.148.230.26) 39.410 ms 39.699 ms
    6 * * *
    7 UNKNOWN-216-115-110-X.yahoo.com (216.115.110.238) 50.284 ms ae-4-0.pat1.nyc.yahoo.com (216.115.104.121) 38.405 ms UNKNOWN-216-115-110-X.yahoo.com (216.115.110.236) 40.179 ms
    8 ae-0.pat2.bfw.yahoo.com (216.115.111.30) 74.575 ms ae-1.pat1.bfw.yahoo.com (216.115.111.28) 48.381 ms 48.484 ms
    9 et-1-0-0.msr2.bf1.yahoo.com (74.6.227.45) 44.879 ms^C

    Here's what I get when I run traceroute6 on pfSense
    raceroute6 www.yahoo.com
    traceroute6: Warning: atsv2-fp-shed.wg1.b.yahoo.com has multiple addresses; using 2001:4998:58:1836::11
    traceroute6 to atsv2-fp-shed.wg1.b.yahoo.com (2001:4998:58:1836::11) from 2607:f798:804:90:75f6:4cc0:abcd:xyz, 64 hops max, 20 byte packets
    1 * * *
    2 2607:f798:10:10d2:0:241:5615:217 12.761 ms 12.572 ms 11.274 ms
    3 2607:f798:10:31a:0:2091:4823:3165 19.792 ms
    2607:f798:10:349:0:2091:4823:5109 12.531 ms
    2607:f798:10:31b:0:2091:4823:3169 20.734 ms
    4 2607:f798:10:370:0:2091:4823:7005 26.660 ms
    2607:f798:10:d6:0:640:7124:1110 32.538 ms
    2607:f798:10:10cf:0:2091:4823:3106 26.588 ms
    5 2607:f798:14:2::310 27.666 ms 32.706 ms 24.959 ms
    6 2001:4998:f003:224:: 28.924 ms


  • LAYER 8 Global Moderator

    Ok I take it that is your pfsense interface at the 00:16:17 mac..

    So did you edit your icmp redirects in tunables? Should be a 1

    0_1548088868142_didyousetthis0.png

    If you set that for 0 for IPv6, then that would explain why you get them for IPv4 and not for IPv6



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    So did you edit your icmp redirects in tunables? Should be a 1

    It's set to 1. However, wouldn't that setting affect redirects, when a packet is not supposed to pass through a router? Traceroute is supposed to receive an ICMP message, when the hop limit decrements to 0, which has nothing to do with redirects.


  • LAYER 8 Global Moderator

    When the TTL does not allow it to be forwarded, it sends you a ICMP does it not. I guess I could reboot mine changing it to 0 and see if causes the problem. But that was the only guess I had at the time which could cause that problem..

    There might be some other tunable that could cause it not to send the ICMP message I guess. Out of the box this should just work... If its sending the traffic to pfsense, out of the box pfsense should send the ICMP v6 message when TTL on does not allow it to forward.



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    Out of the box this should just work.

    That is my expectation too. I should at least see a response from pfSense. I just ran Wireshark again and do not see any response at all on IPv6, but see all the TTL exceeded messages on IPv4. As I mentioned above, I'm not expecting response from anything beyond pfSense on IPv6, due to the ISP problem. I'll have to try again after that's been resolved. However, I'd be very surprised if that problem caused pfSense to not respond.


  • LAYER 8 Global Moderator

    So do you have any rules in say floating that would stop the udp... Did you try with icmp vs udp?

    Where exactly are you sniff at... The client machine or pfsense interface?



  • That capture was between my desktop computer and pfSense. I just ran Wireshark again, filtering on ICMP6, and still do not see any ICMP6 TTL exceeded messages. I do see other ICMP6 traffic.


  • LAYER 8 Global Moderator

    So pfsense is actually seeing this traffic? Sniff on pfsense.. Setup packet capture on pfsense, then run your trace on your client... Pfsense actually sees the trace?



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    So pfsense is actually seeing this traffic? Sniff on pfsense.. Setup packet capture on pfsense, then run your trace on your client... Pfsense actually sees the trace?

    Given that I can see the outgoing UDP between pfSense and modem, it has to pass through pfSense. As mentioned earlier, the problem is not pfSense passing the traceroute. The problem is that it doesn't respond to packets that die with hop limit of 1. I've attached the Packet Capture on pfSense of the outgoing UDP. Packet Capture, filtering on ICMP6, does not show any TTL exceeded messages.

    0_1548098773747_packetcapture.pcap


  • LAYER 8 Global Moderator

    Where are you sniffing at? And 00:16:17:a7:f2:d3 is PFSENSE mac address?

    "Given that I can see the outgoing UDP between pfSense and modem,"

    And how exactly are you seeing that?

    If pfsense is not a HOP on your way to get where your going, then it will not respond if your "bridging" at pfsense..

    Are you doing any policy routing or multi wan setup for IPv6?
    https://www.netgate.com/docs/pfsense/routing/troubleshooting-traceroute-output.html



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    Where are you sniffing at? And 00:16:17:a7:f2:d3 is PFSENSE mac address?

    "Given that I can see the outgoing UDP between pfSense and modem,"

    And how exactly are you seeing that?

    If pfsense is not a HOP on your way to get where your going, then it will not respond if your "bridging" at pfsense..

    Are you doing any policy routing or multi wan setup for IPv6?
    https://www.netgate.com/docs/pfsense/routing/troubleshooting-traceroute-output.html

    That was done with the pfSense Packet Capture, to show you that the UDP packets are heading out through pfSense. That MAC is for my desktop computer. I do not have any policy routing or multiwan setup. As I mentioned a few times, when I run Wireshark between pfSense and modem, I can see the outgoing UDP packets, but not seeing any returned ICMP6 TTL exceeded messages. This indicates that traceroute is leaving the desktop computer, passing through pfSense and out to the Internet. I don't understand why you're asking about things like policies, when it's obvious pfSense passing those packets out to the Internet. I mentioned that 4 days ago when I said: "They are also leaving pfSense and out to the Internet, as shown with Wireshark, between pfSense computer and modem. I'm just not getting any response from pfSense". At the moment, I'm not worried about ICMP6 TTL exceeded messages from any point beyond pfSense, as I have that ISP problem I mentioned earlier. That file I uploaded earlier, with the "CASA" MAC, shows the UDP packets leaving pfSense.


  • LAYER 8 Global Moderator

    @jknott said in IPv6 traceroute not showing first hop (pfSense):

    That MAC is for my desktop computer.

    And how is that???

    0_1548100863064_tracertoute.png

    Where is pfsense mac in this trace... If pfsense is not a HOP, and doesn't lower the TTL then no it wouldn't respond with icmp..

    You see from my above sniff... That mac is pfsense interface.. and my raspberry pi sending the trace.. How is the dest mac in your sniff your PC? When it should be the mac address of your pfsense interface that is the gateway for your client doing the trace.

    I ask about policy routing because if you read that link, when doing that pfsense is not a hop in the path, and therefor will not send back icmp on your 0 TTL hop..



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    @jknott said in IPv6 traceroute not showing first hop (pfSense):

    That MAC is for my desktop computer.

    And how is that???

    Where is pfsense mac in this trace... If pfsense is not a HOP, and doesn't lower the TTL then no it wouldn't respond with icmp..

    You see from my above sniff... That mac is pfsense interface.. and my raspberry pi sending the trace.. How is the dest mac in your sniff your PC? When it should be the mac address of your pfsense interface that is the gateway for your client doing the trace.

    On the desktop computer:

    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 74:d4:35:5b:f5:fa brd ff:ff:ff:ff:ff:ff

    And pfSense firewall:

    bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
    ether 00:16:17:a7:f2:d3
    hwaddr 00:16:17:a7:f2:d3

    Whether captured on pfSense or desktop computer, the packets between them should show the same MAC addresses. The WAN link will have the pfSense & ISP MACs. The capture I posted about 41 minutes ago was on the LAN side, caputerd in Wireshark on the desktop. The one 4 days ago was on the WAN link, captured with Packet Capture on pfSense. Regardless, whether I have packet captures or not, pfSense is not responding to the traceroute time outs, as I showed with the command line capture earlier. There is only one path from my local network to the Internet and that is via pfSense and cable modem.


  • LAYER 8 Global Moderator

    Ok now that we have cleared that up... Are you doing any policy routing? Are you doing any port forwarding... Have you modified any tunables?

    I can not duplicate this problem.. Pfsense should answer these out of the box. So have you tried icmp traceroute?



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    Ok now that we have cleared that up... Are you doing any policy routing? Are you doing any port forwarding... Have you modified any tunables?

    No to all the above. Also, given that pfSense is the first hop, why should routing or port forwarding make a difference? An IPv6 packet with a hop limit of 1 should always trigger a TTL timeout without exception.

    I can not duplicate this problem.. Pfsense should answer these out of the box. So have you tried icmp traceroute?

    Yes and same thing.



  • @jknott

    I've just noticed something else. IPv6 pings to the WAN interface also fail. IPv4 pings do work.



  • On my system, for both ipv4 and ipv6, the first hop is my ISP. All hops give an address and most of the will resolve. The result is similar for both udp and icmp.



  • @bimmerdriver said in IPv6 traceroute not showing first hop (pfSense):

    On my system, for both ipv4 and ipv6, the first hop is my ISP. All hops give an address and most of the will resolve. The result is similar for both udp and icmp.

    When I do a traceroute, on IPv4, from a computer behind my pfSense firewall, pfSense is the first hop and the first one beyond doesn't show an address. At the moment, there's a problem with my ISP providing IPv6, so I'll have to wait for that to be fixed before seeing what happens with IPv6.



  • @jknott said in IPv6 traceroute not showing first hop (pfSense):

    @bimmerdriver said in IPv6 traceroute not showing first hop (pfSense):

    On my system, for both ipv4 and ipv6, the first hop is my ISP. All hops give an address and most of the will resolve. The result is similar for both udp and icmp.

    When I do a traceroute, on IPv4, from a computer behind my pfSense firewall, pfSense is the first hop and the first one beyond doesn't show an address. At the moment, there's a problem with my ISP providing IPv6, so I'll have to wait for that to be fixed before seeing what happens with IPv6.

    Very strange. I'm doing the same thing, but getting a different result.


  • LAYER 8 Global Moderator

    So your getting the results I get, where it just works out of the box @bimmerdriver



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    So your getting the results I get, where it just works out of the box @bimmerdriver

    It works fine for me on IPv4, but not IPv6. As I mentioned above, my WAN port is not responding to pings on IPv6, but does on IPv4.



  • @johnpoz said in IPv6 traceroute not showing first hop (pfSense):

    So your getting the results I get, where it just works out of the box @bimmerdriver

    FWIW, I have rule to pass ipv4 and ipv6 echoreq. Nothing else. I get 20/20 on ipv6-test.com (when it works) and 10/10 on test-ipv6.com.


Log in to reply