Traceroute with pfSense and 2wire possibly MUSH
-
I have a weird thing happening with traceroute from a windows machine and PFsense 2.0.1
NAT with 4 CARP Virtual IP's… Give some history first
This is a AT&T U-Verse connection and with CARP our Static IP's work perfect The HEX is assigned and its picked up by the 2wire modem/NAT router
The 2wire requires MAC address tables not just the IP's assigned by Pfsense.
It's also built around FreeBSD 4.4 according to the info in the 2wire router/nat.
In order to use the Static IP's assigned by AT&T they have us open the Firewall in what they call DMZ_plus which is another word for them using what use to be called Bridged_LZ
OK that said.. If I connect to the 2wire RG direct with a laptop or use the Trace-route in the 2wire I can get the tracert to run and show all.
From behind the pfsense it shows the Windohs Tracert and PfSense the sameSample from Windohs
C:>tracert yahoo.comTracing route to yahoo.com [209.191.122.70]
over a maximum of 30 hops:1 2 ms 2 ms 2 ms 192.168.0.4
2 3 ms 2 ms 2 ms xx-xx-xx-xx.lightspeed.cicril.sbcglobal.net
[xxx.xxx.xxx.xxx]
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 49 ms 50 ms 51 ms ir1.fp.vip.mud.yahoo.com [209.191.122.70]Trace complete.
Trace from PfSense is the same minus the first hop
We're able to ping without a problem and ICMP is enabled in Firewall Rules (to any)
We use NAT but not NAT 1.1
Running one main WAN IP with 4 CARP virtual IP's All master on separate vhid..I also tried making a firewall rule based on probably weird info
UDP traceroute uses ports 33434 through 33434+<max number="" of="" hops="">-1. Note that for the firewall to respond with a TTL expired ICMP reply, you will need to allow ICMP 11 outbound from the firewall. The standard Shorewall sample configurations all set this up for you automatically since those sample configurations enable all ICMP packet types originating on the firewall itself.
After thinking about it and the way they had it AND the fact it didnt work. I removed the ruleThis is the only strange item happening.. We're running 2 pfSense with static IP's to different ISP's
This one is using CARP virtual IP's and the other is Using Alias IP's
NOTE WE Can't fully Bridge the 2wire and being the 2wire is built on FreeBSD is the only Clue I might have to go on besides possibly CARP
Not even sure adding a route would work…I MUSHed out and really need some direction on this..
Just getting the IP's to work with the 2wire over U-verse vDSL or there version of vDSL over ethernet was fun in itself...
BUT Getting U-verse to work with pfSense was a much better option then what others are trying..The answers probably simple and right in front of me.. to go over the top technically might expose whats left in brain cells
Thank you.
Dannjr AKA Mush-headed-out</max>
-
forgot to add
Intel 2.8
re0 enabled
re1 enabled
re2 disabled
using realtek gigabyte cards we had them sitting around
1 gig memory
80Gig HD
Forgot to also add we're using Squid and pfblocker which dosnt seem to effect our other pfsense box0 connection errors
since march 30th over 100 gig transfered
FTP enabled. AND we're using this as a production machineThanks again
-
That's one thing that REALLY irks me with the 2wire Uverse RGs. No matter what you configure on them, they think TTL expired messages are evil and block them - but just on static publics. Can't traceroute out of them no matter what you do. Google it, lots of people complain about it on AT&T's forum. When you "disable" the firewall on them for your public IP range, it doesn't actually disable it, it leaves hidden crap you can't get rid of. It only affects your static public IPs, hosts getting a private IP behind it can traceroute, and the built in traceroute works.
-
@cmb:
That's one thing that REALLY irks me with the 2wire Uverse RGs. No matter what you configure on them, they think TTL expired messages are evil and block them - but just on static publics. Can't traceroute out of them no matter what you do. Google it, lots of people complain about it on AT&T's forum. When you "disable" the firewall on them for your public IP range, it doesn't actually disable it, it leaves hidden crap you can't get rid of. It only affects your static public IPs, hosts getting a private IP behind it can traceroute, and the built in traceroute works.
Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8
Since we use Apache and reverse lookups it made it even more difficult
My solution for now till we all come up with a solution was to do the scrub info from this post
http://forum.pfsense.org/index.php/topic,27206.0.htmlWhat thats done is shortened up the tracert from the Servers to one hop
its getting a answer immediately and reverse lookup comes up twice as fast through Apache.
Im sure pfblocker and squid shouldnt be effected..I personally wish I left the 3rd NIC enabled.. I was orignally using dual connect with the public and the 2wire internal LAN IP
Latency was close to half at that point..I know your pain with the Staic IP's and I Still have one more time on the phone with ATT getting the rest of the DNS assigned to the IP's
On a plus note The speed and lower latency has helped get my work dont earlier when its not something to do with Weird.Tracerts.. I tried a few things Set up different routes and that didnt help
Tried enabling RIP.. Carp didn't seem to like that the filters didn't want to load all the wayA complete Tracert for me would be a plus.. But more important Revers lookup for apache FTP which works well.. and ping Stats
Tables and such is a little more important..
I just get into trying to find the answer and sometimes forget the big pictureI assume the ICMP is different rules then Tracroute.. If so is there a way to port it threw NAT OR am I missing something with CARP.. I have CARP using the Same NAT rules for all IP's
I also last night Ran Ubuntu direct connect to ATT 2wire and the 2wire thought I was a router.. I changed DHCP settings in Ubuntu and was able to Tracert..
So I'm At a loss other then to think ports or permissions…. AND yes still MUSH
-
Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8
No, has nothing whatsoever to do with what you're running, it blocks the replies necessary for traceroute to function on the public IPs, and does so for everyone regardless of OS. The reason the last hop works is because it actually replies, it doesn't send back the TTL expired message that the 2wire drops, which is what traceroute times on intermediate hops.
All you're doing by dropping the TTL to 1 is ensuring the traceroute never traverses the modem. The modem's internal IP responds with TTL expired, it's just when something upstream of it responds, it blocks it. You'll see it in your firewall log on the modem.
Reverse DNS lookups have no relation to traceroute.
-
@cmb:
Actually I think theres a issue between the FreeBSD 4.4 on the 2Wire and the FreeBSD 8
No, has nothing whatsoever to do with what you're running, it blocks the replies necessary for traceroute to function on the public IPs, and does so for everyone regardless of OS. The reason the last hop works is because it actually replies, it doesn't send back the TTL expired message that the 2wire drops, which is what traceroute times on intermediate hops.
All you're doing by dropping the TTL to 1 is ensuring the traceroute never traverses the modem. The modem's internal IP responds with TTL expired, it's just when something upstream of it responds, it blocks it. You'll see it in your firewall log on the modem.
Reverse DNS lookups have no relation to traceroute.
Thanks for that answer.. I looked in the 2wire logs before but didn't notice it right away cause it only writes one small line about it..
As for setting TTL to 1 in the long run that will work out with our resident goof here so we're not waisting additional time.. It's just gonna advirt a ping TTL of 254 and that could be insain since 90% of all ATT is about a TTL of 116 New meaning..I think its time for me to see IF I CAN do a work around with TOS
We use to lie in Windows for the TOS to 92 which bypassed some of the ISP info with TOS 254 and some other settings would be would be proper..
Best explanation for TOS I can think of is here
http://www.dslnuts.com/discussion/index.php/topic,1878.msg9712.html#msg9712
Anything is possible till they catch it.. and that depends on if it can work..In any event
If nothing else no matter how many routes we setup with the2wire in front it can't respond to the Traceroute because someone at AT&T thinks its a security risk..
There's allot of things I like about the 2wire when they left it be.. But now I guess they want to make sure QoS for the TV and the phones don't get put in the same layer..
I tested that with our Satilite dich(directv) and did a speedtest while using the DVR to download and it cut the download speed by 5Mb.. Since directv has its own way of doing things.. Even though I set it up to have a static IP for the 2wire it was also getting an assigned IP (DHCP) from the 2wire as well.. I decided to put the Directv behind the pfSense and its still working well without the second IP..Latency threw the 2wire threw the Pfsense has an average 18ms So I cant complain about the speed..
I'm also using STATIC IP's threw the 2wire not what some people think are sticky IPs.. It was just getting the MAC tables to take.. Which I've cleared out of the 2wire several times and they're listed re-immediatelySo other then this trace issue all is well.. All that's really left is to get a hold of ATT to set our rDNS records which is a pain and dont even mention that to Teir 1 they'll ask what email client your using. You have to ask for the Static IP dept after several transfers you might get through
SO after all this we need to get a few thousand people together and bust ATT on trace routes…..
cmb Thank you for that Quick reply and info… I can't say what I'm thinking about the AT&T engineers right now!