Multi-WAN vs Whatever this is...
-
I had pfSense with a standard Comcast / Docsis 3.0 modem for a while and then switched to AT&T fiber which necessitates the retention of their BGW320-500 customer gateway configured for "IP-Passthrough" which is definitely not a simple bridge. I don't completely understand it, but regardless for 2-3 days after I'd configured it for pfSense, I was not getting "internet" connectivity. I put "internet" in quotes because I WAS able to see / communicate with any other hosts on the /23 ip block that my public IP is a part of. Then, in the middle of the night, I was connected to internet at large and without any change to my config, it "fixed itself". This was very bothersome but I had things to do and so I didn't dwell on it too much.
Fast forward to last night I had shown a friend zenmap (nmap gui frontend) and issued a traceroute to a larger block of my neighboring IP addresses (I think /20). That is when I saw the following topology generated.
The green area is my LAN. The red area is what I am confused at seeing. First off, the "upstream gateway" that AT&T specifies is never the hop after 192.168.0.254. I am wondering if that is because with IP Passthrough, the bgw320-500 essentially adopts that address routing. Anyways, the typical next hop for all outbound traffic is the 107 host. Hence why that region is red. I opened a new window of Zenmap and re-issued the traceroute command to make sure it was not the case the typical next hop was dropping icmp due to load or whatever. Results (even weirder) below...
So, can someone clue me in... What is going on here?
-
I ran another traceroute against that group of hosts from pfSense, and here is the graphic via zenmap. (the nmap command is shown in full)
[2.5.2-RELEASE][admin@pf.oak.crib]/root: netstat -r Routing tables Internet: Destination Gateway Flags Netif Expire default 23-118-48-1.lights UGS igb0 dns.google 23-118-48-1.lights UGHS igb0 10.44.35.0/24 link#12 U lagg0.35 10.44.35.1 link#12 UHS lo0 10.44.44.0/24 link#15 U lagg0.44 10.44.44.1 link#15 UHS lo0 10.44.68.0/24 link#10 U tun_wg1 10.44.68.254 link#10 UHS lo0 10.44.69.0/24 link#9 U tun_wg0 pf link#9 UHS lo0 23.118.48.0/23 link#1 U igb0 23-118-48-1.lights 00:0d:b9:51:9b:a4 UHS igb0 23-118-48-xxx.ligh link#1 UHS lo0 localhost link#6 UH lo0 172.16.0.0/24 link#14 U lagg0.17 172.16.0.1 link#14 UHS lo0 192.168.1.0/24 link#13 U lagg0.19 192.168.1.1 link#13 UHS lo0 -
@boethius Information on how you have configured pfsense may help such as
- Screen shot of pfsense -> Dashboard -> interfaces
- Screen shot of pfsense -> Interfaces -> Interface Assignments
- Output of pfsense -> Status -> Interfaces
-
-
@boethius My knowledge of routing with multiple Wan & VPN tunnels is limited however I assumed the Wireguard tunnel through pfsense, the BGW320-500 modem, the ISP gateway etc will not see that part of the route (as in is in the tunnel). Which may confuse zenmap or at least the interpretation of the resultant graphs.
The graphs generated as going to depend on the routing policy you have set up in pfsense to determine which gateway is used.
-
@patch I have no routing policy other than that which is shown via netstat -r.
I do not have multiwan.
The tunnels you see are not in use and both are "remote access" tunnels which require the app be run clientside, which wasn't the case with any of those tests.
I started to get an idea regarding the single-hop-proximity host at 23.118.49.106.
I learned a while ago that the BGW320-500 will respond to ARP requests for any IP within it's subnet (23.118.48.0/23). That is to say the ARP table (after running nmap with default ARP host discovery for same-subnet hosts) will populate with entries for every IP address mapping to it's own MAC.
Enter an option: 8 Command history storage is enabled. Clear history with: history -c; history -S. [2.5.2-RELEASE][admin@pf.oak.crib]/root: traceroute 23.118.49.106 traceroute to 23.118.49.106 (23.118.49.106), 64 hops max, 40 byte packets 1 192.168.0.254 (192.168.0.254) 0.896 ms 0.253 ms 0.491 ms 2 107-208-8-1.lightspeed.sntcca.sbcglobal.net (107.208.8.1) 1.593 ms 0.859 m s 1.029 ms 3 * * * ^C [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute 23.118.49.106 -vv Starting Nmap 7.91 ( https://nmap.org ) at 2022-06-12 00:30 PDT Initiating ARP Ping Scan at 00:30 Scanning 23.118.49.106 [1 port] Completed ARP Ping Scan at 00:30, 0.00s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 00:30 Completed Parallel DNS resolution of 1 host. at 00:30, 0.03s elapsed Nmap scan report for 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.10 6) Host is up, received arp-response (0.00083s latency). MAC Address: CC:AB:2C:6C:2B:D1 (Humax) TRACEROUTE HOP RTT ADDRESS 1 0.83 ms 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.106) Read data files from: /usr/local/share/nmap Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds Raw packets sent: 1 (28B) | Rcvd: 1 (28B) [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute --no-arp 23.118.4 9.106 -vv nmap: unrecognized option `--no-arp' See the output of nmap -h for a summary of options. [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute --disable-arp 23. 118.49.106 -vv Starting Nmap 7.91 ( https://nmap.org ) at 2022-06-12 00:31 PDT Initiating Ping Scan at 00:31 Scanning 23.118.49.106 [4 ports] sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 23.118.49.106, 16) => Perm ission denied Offending packet: TCP 23.118.48.246:43581 > 23.118.49.106:80 A ttl=58 id=17725 i plen=40 seq=0 win=1024 Completed Ping Scan at 00:31, 0.00s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 00:31 Completed Parallel DNS resolution of 1 host. at 00:31, 0.03s elapsed Nmap scan report for 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.10 6) Host is up, received timestamp-reply ttl 61 (0.0022s latency). MAC Address: CC:AB:2C:6C:2B:D1 (Humax) TRACEROUTE HOP RTT ADDRESS 1 2.22 ms 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.106) Read data files from: /usr/local/share/nmap Nmap done: 1 IP address (1 host up) scanned in 0.27 seconds Raw packets sent: 3 (112B) | Rcvd: 1 (40B)I don't quite know what happened here. The permission send-to errors I've encountered in the past with no explanation. And in re-running those same traceroute commands as earlier, am now getting different results. Any input or further troubleshooting welcomed. I don't have an explanation or answer but this has been over a year of strange and disruptive problems with no answers for all but small pieces of the overall picture.
-
@boethius So what is confusing you is that in your traceroute your not seeing what is listed as your gateway?
This is not uncommon to see. Example My gateway is 1 address, but in a traceroute comes back as another address.
To understand this you need to understand how traceroute works, and how a router can use say a loopback address in the answer to the traceroute.
Or other possibility is a VIP and physical setup on the router. Where when you send the traffic it would be going to the VIP, but your answer comes from the physical IP on the interface.
I would find it highly likely that an ISP wouldn't be using multiple routers in their setup in an HA.. Ie using VRRP or HSRP, etc.
Notice the 10 address at my 3rd hop, this is still inside the isp network, and there is nothing saying they can not use 10 addresses in their internal routing of traffic, etc.
As to not seeing gateway when you do other IPs on your wan network - this is also common since your wan is in that nework, so why would it send traffic to the gateway to get to an IP in its own network, etc.
The first hop there 9.253 is my pfsense lan IP.. As I am doing this trace from a client on my local network. in the 192.168.9/24 network.
-
What is confusing me is the section in red, which is neither to my own network, nor the internet at large except via an alternate route.
I've rerun the traceroutes just now, the same command issued at both my laptop and pfSense. Green is me understanding, red is me not understanding.
Laptop
pfSense
I don't see how there is some other host / router at these points... I have been seeing signs of asymmetric routing but can't figure out where / how. https://forum.netgate.com/topic/168975/permission-errors-running-commands-as-admin/14 is where I started troubleshooting that.
Or other possibility is a VIP and physical setup on the router. Where when you send the traffic it would be going to the VIP, but your answer comes from the physical IP on the interface.-|
That makes sense, I'd guess that's what's going on here, I think perhaps the bgw500-320 is using the assigned upstream gateways VIP and responses come back from the physical interface 192.168.0.254..Notice the 10 address at my 3rd hop, this is still inside the isp network, and there is nothing saying they can not use 10 addresses in their internal routing of traffic, etc.
That's also clarifying. This is not CGNAT, as there is/are publicly routable hops in between... there's no problem with this because if you were blocking bogons on WAN, it doesn't matter, that hop is only a hop to other ISP routers, never a destination. Thank you for pointing that out. -
@boethius That red is the network of your wan.. As I stated you don't need to talk to gateway to talk to stuff on your own network.
There is nothing your showing that is odd at all.
My take is the graph is confusing representation of what is going on for you.
You understand pfsense has its own local IP, and the wan IP - so to it, 192.168.0.254 can directly talk to any IP that is in the network 23.whater your wan interface is, which is also pfsense.
Your wan mask is what? a /23 keep in mind that you might also be able to talk to other ISP IPs on that interface because many an ISP run multiple layer 3 networks on the same layer 2..
Here look at some of the networks seeing arp from on my wan
Not all of those are listed as my /23 on my wan. But they are on the same L2 network as my wan..
Also that topology map can get confusing if you do not flush out previous other scans, etc. Also just doing something simple can be confusing on that topology map
example here is a scan of couple of my local networks that hang off my pfsense. So pfsense IP is 192.168.9.253.. As you can see to get to hosts in in those networks the topology shows going through pfsense. But also look at 192.168.2.253 and 192.168.3.253 which are pfsense IPs on those networks. From that graph they look like my pc doing the scan from 192.168.9.100 is directly attached to them and not going through pfsense to get there.
But if look at the traceroute it does make sense because its really only 1 hop away, its an IP on pfsense, which is my gateway.
If you are not intimately aware of the nuances of what the topology graph is trying to represent, then sure it could be very odd looking..
-
@johnpoz First and foremost, thank you for taking your time to not only explain but show me with examples from your own setup. While I'm still unclear , you've helped already in my understanding of the broader picture regarding routing & ISP innerworkings.
Here's what I am still not understanding.
This scan was run on pfSense. pfSense WAN is connected via 192.168.0.254. I don't understand how it is reaching these hosts, that's why I said it was like multi-wan. All other traffic flows through 192.168.0.254. How could there be another route? Using what?
-
Sorry to go off topic, but the last time I went to install zenmap, I found it was tied to a deprecated version of python. Did someone finally decide to fix this, or did you have to load python libraries from many years ago?
-
@dotdash I'm using Windows here. I did have to go through that process on my linux hosts. Don't think it's fixed yet.
-
@boethius via your rings those are multiple hops away..
Why don't you look at the traceroute for those specific IPs
So your ran the command on pfsense - how exactly are you getting the gui picture when you ran it on pfsense?
What are the interfaces on pfsense? If your using IP pass through as you mention then pfsense should have a public IP on its wan..
Per your routing table it does
default 23-118-48-1.lights UGS igb0So your topology there with showing 192.168.0.254 makes less sense.. In what scenario would pfsense send traffic to that IP, its not in your routing table. How is it doing the IP passthru exactly.. What would make sense is its routing the traffic and its answering you via its loopback address.
What I would suggest vis trying to do discovery for the internet and trying to understand how nmaps topology map works, and can be confusing when things respond with different IPs (we went over this already).. Why not do a simple just traceroute.. to single IP on the internet if your trying to understand the path of traffic.
From your routing table you posted.. Pfsense will send ALL traffic that is not directly attached or has another router for to that IP.. Be it that IP answer back via a traceroute with that same IP we went over. Be it that IP some VIP on a HA setup in your isp makes no matter.
I am having a hard time understanding what your looking to understand. Pfsense is going to send ALL traffic to that IP - period, there are no other routes listed in your routing table. All that topology stuff is doing is confusing you because of how traceroute works, and icmp info sent back when a ttl expires.. That IP that icmp info is sent back from doesn't have to be the IP you show as your gateway..
edit
I'm using Windows here.
Then how are you running it on pfsense like you said??
This scan was run on pfSense
-
@johnpoz said in Multi-WAN vs Whatever this is...:
So your ran the command on pfsense - how exactly are you getting the gui picture when you ran it on pfsense?
I used
nmap -oA <filename>which will output, among others, an .xml which can then be opened in zenmap and viewed.What are the interfaces on pfsense? If your using IP pass through as you mention then pfsense should have a public IP on its wan..
I have interfaces in reply to Patch, 3rd message of thread.
So your topology there with showing 192.168.0.254 makes less sense.. In what scenario would pfsense send traffic to that IP, its not in your routing table. How is it doing the IP passthru exactly.. What would make sense is its routing the traffic and its answering you via its loopback address.
I don't know how IP passthrough works exactly. It's weird. Here's a pcap of arps on WAN.
I am having a hard time understanding what your looking to understand. Pfsense is going to send ALL traffic to that IP - period, there are no other routes listed in your routing table. All that topology stuff is doing is confusing you because of how traceroute works, and icmp info sent back when a ttl expires.. That IP that icmp info is sent back from doesn't have to be the IP you show as your gateway..
When I switched from comcast to AT&T, it took 3 days before it worked. It was a simple config, I didn't change anything from the first attempt, just poured over stuff trying to figure out what in the heck was happening, as I wasn't able to reach the internet. Just this 23.118.48.0/23 block. Overnight, it magically connected to the internet. I made no changes, nothing explains it.
-
@boethius said in Multi-WAN vs Whatever this is...:
t, it magically connected to the internet. I made no changes, nothing explains it.
Your isp finally turned it on, you finally got a dhcp address.. Nothing magic about it - why did you not call your isp about it on day 1? I would of been calling them first hour, etc.
So yeah your connected to that network on L2.. Still not getting what your trying to glean from running a discovery on the internet or your wan network, etc.
All traffic will be sent to your gateway IP - that is how it works.. what is confusing you is how traceroute works, and how the answers can be interpreted by not too bright software trying to draw a map based on icmp responses.
-
In comparing nmaps's traceroute with TCP on port 80, to regular traceroute utility on pfSense, well, they give different results.
-
@johnpoz said in Multi-WAN vs Whatever this is...:
All traffic will be sent to your gateway IP - that is how it works.. what is confusing you is how traceroute works, and how the answers can be interpreted by not too bright software trying to draw a map based on icmp responses.
I think nmap is very bright software.. the map makes sense..
I think nmap does traceroute in reverse essentially, so it will find how many hops away the target is, then it will decrement TTL. If our path to some hosts has 192.168.0.254 as the first hop, and it responds to ICMP time exceeded, and we get no response for the TTL 1 packet, then it was a different path.
And this is it as far as other hosts on the internet that I could connect to for those few days. Weird, right?
-
@boethius not that software doesn't do what it designed to do.. My point is interpretation of what it draws when the info is not what you would expect.
In a perfect world when I traceroute through a router at IP address 1.2.3.4 that is where the response would come from... But when it comes from 1.2.4.5 how do you graph that were it actually makes sense.. Since your gateway is 1.2.3.4
As to getting back 192.168.0.254, again that is how traceroute works.. That your isp gateway responds back with that IP is odd and exactly what I am talking about for where something like nmap has a hard time drawing that on a chart.
-
I just discovered this tool -- record route on ping. Look at these results... (This is a good example of how it works) https://flylib.com/books/en/3.223.1.87/1/)
So my traffic is actually going OUT via 192.168.254.254.
Coming back in on 192.168.0.254.And I have no idea what this 192.168.20 stuff is.
-
@boethius How many times do we have to go over how this works... That works the same freaking way - it only knows what devices respond from..
Why are you pinging 23.118.48.241 ???
What part are you not understanding about loopback answers and other interfaces on routers?
So in your RECORD_ROUTE test to 23.118.48.241 its a hop in its own path? And 192.168.20.11 is used twice? ;)
so is 107.208.8.1 ;)
Your confusing yourself with info that doesn't matter - again these tools don't always work how you think they do ;)
