Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Traceroute with pfSense and 2wire possibly MUSH

    Scheduled Pinned Locked Moved General pfSense Questions
    6 Posts 2 Posters 2.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dannjr
      last edited by

      I have a weird thing happening with traceroute from a windows machine and PFsense 2.0.1
      NAT with 4 CARP Virtual IP's… Give some history first
      This is a AT&T U-Verse connection and with CARP our Static IP's work perfect The HEX is assigned and its picked up by the 2wire modem/NAT router
      The 2wire requires MAC address tables not just the IP's assigned by Pfsense.
      It's also built around FreeBSD 4.4 according to the info in the 2wire router/nat.
      In order to use the Static IP's assigned by AT&T they have us open the Firewall in what they call DMZ_plus which is another word for them using what use to be called Bridged_LZ
      OK that said.. If I connect to the 2wire RG direct with a laptop or use the Trace-route in the 2wire I can get the tracert to run and show all.
      From behind the pfsense it shows the Windohs Tracert and PfSense the same

      Sample from Windohs
      C:>tracert yahoo.com

      Tracing route to yahoo.com [209.191.122.70]
      over a maximum of 30 hops:

      1    2 ms    2 ms    2 ms  192.168.0.4
        2    3 ms    2 ms    2 ms  xx-xx-xx-xx.lightspeed.cicril.sbcglobal.net
      [xxx.xxx.xxx.xxx]
        3    *        *        *    Request timed out.
        4    *        *        *    Request timed out.
        5    *        *        *    Request timed out.
        6    *        *        *    Request timed out.
        7    *        *        *    Request timed out.
        8    *        *        *    Request timed out.
        9    *        *        *    Request timed out.
      10    *        *        *    Request timed out.
      11    *        *        *    Request timed out.
      12    *        *        *    Request timed out.
      13    *        *        *    Request timed out.
      14    *        *        *    Request timed out.
      15    49 ms    50 ms    51 ms  ir1.fp.vip.mud.yahoo.com [209.191.122.70]

      Trace complete.

      Trace from PfSense is the same minus the first hop

      We're able to ping without a problem and ICMP is enabled in Firewall Rules (to any)
      We use NAT but not NAT 1.1
      Running one main WAN IP with 4 CARP virtual IP's All master on separate vhid..

      I also tried making a firewall rule based on probably weird info
      UDP traceroute uses ports 33434 through 33434+<max number="" of="" hops="">-1. Note that for the firewall to respond with a TTL expired ICMP reply, you will need to allow ICMP 11 outbound from the firewall. The standard Shorewall sample configurations all set this up for you automatically since those sample configurations enable all ICMP packet types originating on the firewall itself.
      After thinking about it and the way they had it AND the fact it didnt work. I removed the rule

      This is the only strange item happening.. We're running 2 pfSense with static IP's to different ISP's
      This one is using CARP virtual IP's and the other is Using Alias IP's
      NOTE WE Can't fully Bridge the 2wire and being the 2wire is built on FreeBSD is the only Clue I might have to go on besides possibly CARP
      Not even sure adding a route would work…

      I MUSHed out and really need some direction on this..
      Just getting the IP's to work with the 2wire over U-verse vDSL or there version of vDSL over ethernet was fun in itself...
      BUT Getting U-verse to work with pfSense was a much better option then what others are trying..

      The answers probably simple and right in front of me.. to go over the top technically might expose whats left in brain cells
      Thank you.
      Dannjr AKA Mush-headed-out</max>

      1 Reply Last reply Reply Quote 0
      • D
        dannjr
        last edited by

        forgot to add
        Intel 2.8
        re0 enabled
        re1 enabled
        re2 disabled
        using realtek gigabyte cards we had them sitting around
        1 gig memory
        80Gig HD
        Forgot to also add we're using Squid and pfblocker which dosnt seem to effect our other pfsense box

        0 connection errors
        since march 30th over 100 gig transfered
        FTP enabled. AND we're using this as a production machine

        Thanks again

        1 Reply Last reply Reply Quote 0
        • C
          cmb
          last edited by

          That's one thing that REALLY irks me with the 2wire Uverse RGs. No matter what you configure on them, they think TTL expired messages are evil and block them - but just on static publics. Can't traceroute out of them no matter what you do. Google it, lots of people complain about it on AT&T's forum. When you "disable" the firewall on them for your public IP range, it doesn't actually disable it, it leaves hidden crap you can't get rid of. It only affects your static public IPs, hosts getting a private IP behind it can traceroute, and the built in traceroute works.

          1 Reply Last reply Reply Quote 0
          • D
            dannjr
            last edited by

            @cmb:

            That's one thing that REALLY irks me with the 2wire Uverse RGs. No matter what you configure on them, they think TTL expired messages are evil and block them - but just on static publics. Can't traceroute out of them no matter what you do. Google it, lots of people complain about it on AT&T's forum. When you "disable" the firewall on them for your public IP range, it doesn't actually disable it, it leaves hidden crap you can't get rid of. It only affects your static public IPs, hosts getting a private IP behind it can traceroute, and the built in traceroute works.

            Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8

            Since we use Apache and reverse lookups it made it even more difficult
            My solution for now till we all come up with a solution was to do the scrub info from this post
            http://forum.pfsense.org/index.php/topic,27206.0.html

            What thats done is shortened up the tracert from the Servers to one hop
            its getting a answer immediately and reverse lookup comes up twice as fast through Apache.
            Im sure pfblocker and squid shouldnt be effected..

            I personally wish I left the 3rd NIC enabled.. I was orignally using dual connect with the public and the 2wire internal LAN IP
            Latency was close to half at that point..

            I know your pain with the Staic IP's and I Still have one more time on the phone with ATT getting the rest of the DNS assigned to the IP's
            On a plus note The speed and lower latency has helped get my work dont earlier when its not something to do with Weird.

            Tracerts.. I tried a few things Set up different routes and that didnt help
            Tried enabling RIP.. Carp didn't seem to like that the filters didn't want to load all the way

            A complete Tracert for me would be a plus.. But more important Revers lookup for apache FTP which works well.. and ping Stats
            Tables and such is a little more important..
            I just get into trying to find the answer and sometimes forget the big picture

            I assume the ICMP is different rules then Tracroute.. If so is there a way to port it threw NAT OR am I missing something with CARP.. I have CARP using the Same NAT rules for all IP's

            I also last night Ran Ubuntu direct connect to ATT 2wire and the 2wire thought I was a router.. I changed DHCP settings in Ubuntu and was able to Tracert..
            So I'm At a loss other then to think ports or permissions…. AND yes still MUSH

            1 Reply Last reply Reply Quote 0
            • C
              cmb
              last edited by

              @dannjr:

              Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8

              No, has nothing whatsoever to do with what you're running, it blocks the replies necessary for traceroute to function on the public IPs, and does so for everyone regardless of OS. The reason the last hop works is because it actually replies, it doesn't send back the TTL expired message that the 2wire drops, which is what traceroute times on intermediate hops.

              All you're doing by dropping the TTL to 1 is ensuring the traceroute never traverses the modem. The modem's internal IP responds with TTL expired, it's just when something upstream of it responds, it blocks it. You'll see it in your firewall log on the modem.

              Reverse DNS lookups have no relation to traceroute.

              1 Reply Last reply Reply Quote 0
              • D
                dannjr
                last edited by

                @cmb:

                @dannjr:

                Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8

                No, has nothing whatsoever to do with what you're running, it blocks the replies necessary for traceroute to function on the public IPs, and does so for everyone regardless of OS. The reason the last hop works is because it actually replies, it doesn't send back the TTL expired message that the 2wire drops, which is what traceroute times on intermediate hops.

                All you're doing by dropping the TTL to 1 is ensuring the traceroute never traverses the modem. The modem's internal IP responds with TTL expired, it's just when something upstream of it responds, it blocks it. You'll see it in your firewall log on the modem.

                Reverse DNS lookups have no relation to traceroute.

                Thanks for that answer.. I looked in the 2wire logs before but didn't notice it right away cause it only writes one small line about it..
                As for setting TTL to 1 in the long run that will work out with our resident goof here so we're not waisting additional time.. It's just gonna advirt a ping TTL of 254 and that could be insain since 90% of all ATT is about a TTL of 116 New meaning..

                I think its time for me to see IF I CAN do a work around with TOS
                We use to lie in Windows for the TOS to 92 which bypassed some of the ISP info with TOS 254 and some other settings would be would be proper..
                Best explanation for TOS I can think of is here
                http://www.dslnuts.com/discussion/index.php/topic,1878.msg9712.html#msg9712
                Anything is possible till they catch it.. and that depends on if it can work..

                In any event
                If nothing else no matter how many routes we setup with the2wire in front it can't respond to the Traceroute because someone at AT&T thinks its a security risk..
                There's allot of things I like about the 2wire when they left it be.. But now I guess they want to make sure QoS for the TV and the phones don't get put in the same layer..
                I tested that with our Satilite dich(directv) and did a speedtest while using the DVR to download and it cut the download speed by 5Mb.. Since directv has its own way of doing things.. Even though I set it up to have a static IP for the 2wire it was also getting an assigned IP (DHCP) from the 2wire as well.. I decided to put the Directv behind the pfSense and its still working well without the second IP..

                Latency threw the 2wire threw the Pfsense has an average 18ms So I cant complain about the speed..
                I'm also using STATIC IP's threw the 2wire not what some people think are sticky IPs.. It was just getting the MAC tables to take.. Which I've cleared out of the 2wire several times and they're listed re-immediately

                So other then this trace issue all is well.. All that's really left is to get a hold of ATT to set our rDNS records which is a pain and dont even mention that to Teir 1 they'll ask what email client your using. You have to ask for the Static IP dept after several transfers you might get through

                SO after all this we need to get a few thousand people together and bust ATT on trace routes…..

                cmb Thank you for that Quick reply and info… I can't say what I'm thinking about the AT&T engineers right now!

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.