Multi-WAN vs Whatever this is...
-
@patch I have no routing policy other than that which is shown via netstat -r.
I do not have multiwan.
The tunnels you see are not in use and both are "remote access" tunnels which require the app be run clientside, which wasn't the case with any of those tests.
I started to get an idea regarding the single-hop-proximity host at 23.118.49.106.
I learned a while ago that the BGW320-500 will respond to ARP requests for any IP within it's subnet (23.118.48.0/23). That is to say the ARP table (after running nmap with default ARP host discovery for same-subnet hosts) will populate with entries for every IP address mapping to it's own MAC.
Enter an option: 8 Command history storage is enabled. Clear history with: history -c; history -S. [2.5.2-RELEASE][admin@pf.oak.crib]/root: traceroute 23.118.49.106 traceroute to 23.118.49.106 (23.118.49.106), 64 hops max, 40 byte packets 1 192.168.0.254 (192.168.0.254) 0.896 ms 0.253 ms 0.491 ms 2 107-208-8-1.lightspeed.sntcca.sbcglobal.net (107.208.8.1) 1.593 ms 0.859 m s 1.029 ms 3 * * * ^C [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute 23.118.49.106 -vv Starting Nmap 7.91 ( https://nmap.org ) at 2022-06-12 00:30 PDT Initiating ARP Ping Scan at 00:30 Scanning 23.118.49.106 [1 port] Completed ARP Ping Scan at 00:30, 0.00s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 00:30 Completed Parallel DNS resolution of 1 host. at 00:30, 0.03s elapsed Nmap scan report for 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.10 6) Host is up, received arp-response (0.00083s latency). MAC Address: CC:AB:2C:6C:2B:D1 (Humax) TRACEROUTE HOP RTT ADDRESS 1 0.83 ms 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.106) Read data files from: /usr/local/share/nmap Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds Raw packets sent: 1 (28B) | Rcvd: 1 (28B) [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute --no-arp 23.118.4 9.106 -vv nmap: unrecognized option `--no-arp' See the output of nmap -h for a summary of options. [2.5.2-RELEASE][admin@pf.oak.crib]/root: nmap -sn --traceroute --disable-arp 23. 118.49.106 -vv Starting Nmap 7.91 ( https://nmap.org ) at 2022-06-12 00:31 PDT Initiating Ping Scan at 00:31 Scanning 23.118.49.106 [4 ports] sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 23.118.49.106, 16) => Perm ission denied Offending packet: TCP 23.118.48.246:43581 > 23.118.49.106:80 A ttl=58 id=17725 i plen=40 seq=0 win=1024 Completed Ping Scan at 00:31, 0.00s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 00:31 Completed Parallel DNS resolution of 1 host. at 00:31, 0.03s elapsed Nmap scan report for 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.10 6) Host is up, received timestamp-reply ttl 61 (0.0022s latency). MAC Address: CC:AB:2C:6C:2B:D1 (Humax) TRACEROUTE HOP RTT ADDRESS 1 2.22 ms 23-118-49-106.lightspeed.sntcca.sbcglobal.net (23.118.49.106) Read data files from: /usr/local/share/nmap Nmap done: 1 IP address (1 host up) scanned in 0.27 seconds Raw packets sent: 3 (112B) | Rcvd: 1 (40B)I don't quite know what happened here. The permission send-to errors I've encountered in the past with no explanation. And in re-running those same traceroute commands as earlier, am now getting different results. Any input or further troubleshooting welcomed. I don't have an explanation or answer but this has been over a year of strange and disruptive problems with no answers for all but small pieces of the overall picture.
-
@boethius So what is confusing you is that in your traceroute your not seeing what is listed as your gateway?
This is not uncommon to see. Example My gateway is 1 address, but in a traceroute comes back as another address.
To understand this you need to understand how traceroute works, and how a router can use say a loopback address in the answer to the traceroute.
Or other possibility is a VIP and physical setup on the router. Where when you send the traffic it would be going to the VIP, but your answer comes from the physical IP on the interface.
I would find it highly likely that an ISP wouldn't be using multiple routers in their setup in an HA.. Ie using VRRP or HSRP, etc.
Notice the 10 address at my 3rd hop, this is still inside the isp network, and there is nothing saying they can not use 10 addresses in their internal routing of traffic, etc.
As to not seeing gateway when you do other IPs on your wan network - this is also common since your wan is in that nework, so why would it send traffic to the gateway to get to an IP in its own network, etc.
The first hop there 9.253 is my pfsense lan IP.. As I am doing this trace from a client on my local network. in the 192.168.9/24 network.
-
What is confusing me is the section in red, which is neither to my own network, nor the internet at large except via an alternate route.
I've rerun the traceroutes just now, the same command issued at both my laptop and pfSense. Green is me understanding, red is me not understanding.
Laptop
pfSense
I don't see how there is some other host / router at these points... I have been seeing signs of asymmetric routing but can't figure out where / how. https://forum.netgate.com/topic/168975/permission-errors-running-commands-as-admin/14 is where I started troubleshooting that.
Or other possibility is a VIP and physical setup on the router. Where when you send the traffic it would be going to the VIP, but your answer comes from the physical IP on the interface.-|
That makes sense, I'd guess that's what's going on here, I think perhaps the bgw500-320 is using the assigned upstream gateways VIP and responses come back from the physical interface 192.168.0.254..Notice the 10 address at my 3rd hop, this is still inside the isp network, and there is nothing saying they can not use 10 addresses in their internal routing of traffic, etc.
That's also clarifying. This is not CGNAT, as there is/are publicly routable hops in between... there's no problem with this because if you were blocking bogons on WAN, it doesn't matter, that hop is only a hop to other ISP routers, never a destination. Thank you for pointing that out. -
@boethius That red is the network of your wan.. As I stated you don't need to talk to gateway to talk to stuff on your own network.
There is nothing your showing that is odd at all.
My take is the graph is confusing representation of what is going on for you.
You understand pfsense has its own local IP, and the wan IP - so to it, 192.168.0.254 can directly talk to any IP that is in the network 23.whater your wan interface is, which is also pfsense.
Your wan mask is what? a /23 keep in mind that you might also be able to talk to other ISP IPs on that interface because many an ISP run multiple layer 3 networks on the same layer 2..
Here look at some of the networks seeing arp from on my wan
Not all of those are listed as my /23 on my wan. But they are on the same L2 network as my wan..
Also that topology map can get confusing if you do not flush out previous other scans, etc. Also just doing something simple can be confusing on that topology map
example here is a scan of couple of my local networks that hang off my pfsense. So pfsense IP is 192.168.9.253.. As you can see to get to hosts in in those networks the topology shows going through pfsense. But also look at 192.168.2.253 and 192.168.3.253 which are pfsense IPs on those networks. From that graph they look like my pc doing the scan from 192.168.9.100 is directly attached to them and not going through pfsense to get there.
But if look at the traceroute it does make sense because its really only 1 hop away, its an IP on pfsense, which is my gateway.
If you are not intimately aware of the nuances of what the topology graph is trying to represent, then sure it could be very odd looking..
-
@johnpoz First and foremost, thank you for taking your time to not only explain but show me with examples from your own setup. While I'm still unclear , you've helped already in my understanding of the broader picture regarding routing & ISP innerworkings.
Here's what I am still not understanding.
This scan was run on pfSense. pfSense WAN is connected via 192.168.0.254. I don't understand how it is reaching these hosts, that's why I said it was like multi-wan. All other traffic flows through 192.168.0.254. How could there be another route? Using what?
-
Sorry to go off topic, but the last time I went to install zenmap, I found it was tied to a deprecated version of python. Did someone finally decide to fix this, or did you have to load python libraries from many years ago?
-
@dotdash I'm using Windows here. I did have to go through that process on my linux hosts. Don't think it's fixed yet.
-
@boethius via your rings those are multiple hops away..
Why don't you look at the traceroute for those specific IPs
So your ran the command on pfsense - how exactly are you getting the gui picture when you ran it on pfsense?
What are the interfaces on pfsense? If your using IP pass through as you mention then pfsense should have a public IP on its wan..
Per your routing table it does
default 23-118-48-1.lights UGS igb0So your topology there with showing 192.168.0.254 makes less sense.. In what scenario would pfsense send traffic to that IP, its not in your routing table. How is it doing the IP passthru exactly.. What would make sense is its routing the traffic and its answering you via its loopback address.
What I would suggest vis trying to do discovery for the internet and trying to understand how nmaps topology map works, and can be confusing when things respond with different IPs (we went over this already).. Why not do a simple just traceroute.. to single IP on the internet if your trying to understand the path of traffic.
From your routing table you posted.. Pfsense will send ALL traffic that is not directly attached or has another router for to that IP.. Be it that IP answer back via a traceroute with that same IP we went over. Be it that IP some VIP on a HA setup in your isp makes no matter.
I am having a hard time understanding what your looking to understand. Pfsense is going to send ALL traffic to that IP - period, there are no other routes listed in your routing table. All that topology stuff is doing is confusing you because of how traceroute works, and icmp info sent back when a ttl expires.. That IP that icmp info is sent back from doesn't have to be the IP you show as your gateway..
edit
I'm using Windows here.
Then how are you running it on pfsense like you said??
This scan was run on pfSense
-
@johnpoz said in Multi-WAN vs Whatever this is...:
So your ran the command on pfsense - how exactly are you getting the gui picture when you ran it on pfsense?
I used
nmap -oA <filename>which will output, among others, an .xml which can then be opened in zenmap and viewed.What are the interfaces on pfsense? If your using IP pass through as you mention then pfsense should have a public IP on its wan..
I have interfaces in reply to Patch, 3rd message of thread.
So your topology there with showing 192.168.0.254 makes less sense.. In what scenario would pfsense send traffic to that IP, its not in your routing table. How is it doing the IP passthru exactly.. What would make sense is its routing the traffic and its answering you via its loopback address.
I don't know how IP passthrough works exactly. It's weird. Here's a pcap of arps on WAN.
I am having a hard time understanding what your looking to understand. Pfsense is going to send ALL traffic to that IP - period, there are no other routes listed in your routing table. All that topology stuff is doing is confusing you because of how traceroute works, and icmp info sent back when a ttl expires.. That IP that icmp info is sent back from doesn't have to be the IP you show as your gateway..
When I switched from comcast to AT&T, it took 3 days before it worked. It was a simple config, I didn't change anything from the first attempt, just poured over stuff trying to figure out what in the heck was happening, as I wasn't able to reach the internet. Just this 23.118.48.0/23 block. Overnight, it magically connected to the internet. I made no changes, nothing explains it.
-
@boethius said in Multi-WAN vs Whatever this is...:
t, it magically connected to the internet. I made no changes, nothing explains it.
Your isp finally turned it on, you finally got a dhcp address.. Nothing magic about it - why did you not call your isp about it on day 1? I would of been calling them first hour, etc.
So yeah your connected to that network on L2.. Still not getting what your trying to glean from running a discovery on the internet or your wan network, etc.
All traffic will be sent to your gateway IP - that is how it works.. what is confusing you is how traceroute works, and how the answers can be interpreted by not too bright software trying to draw a map based on icmp responses.
-
In comparing nmaps's traceroute with TCP on port 80, to regular traceroute utility on pfSense, well, they give different results.
-
@johnpoz said in Multi-WAN vs Whatever this is...:
All traffic will be sent to your gateway IP - that is how it works.. what is confusing you is how traceroute works, and how the answers can be interpreted by not too bright software trying to draw a map based on icmp responses.
I think nmap is very bright software.. the map makes sense..
I think nmap does traceroute in reverse essentially, so it will find how many hops away the target is, then it will decrement TTL. If our path to some hosts has 192.168.0.254 as the first hop, and it responds to ICMP time exceeded, and we get no response for the TTL 1 packet, then it was a different path.
And this is it as far as other hosts on the internet that I could connect to for those few days. Weird, right?
-
@boethius not that software doesn't do what it designed to do.. My point is interpretation of what it draws when the info is not what you would expect.
In a perfect world when I traceroute through a router at IP address 1.2.3.4 that is where the response would come from... But when it comes from 1.2.4.5 how do you graph that were it actually makes sense.. Since your gateway is 1.2.3.4
As to getting back 192.168.0.254, again that is how traceroute works.. That your isp gateway responds back with that IP is odd and exactly what I am talking about for where something like nmap has a hard time drawing that on a chart.
-
I just discovered this tool -- record route on ping. Look at these results... (This is a good example of how it works) https://flylib.com/books/en/3.223.1.87/1/)
So my traffic is actually going OUT via 192.168.254.254.
Coming back in on 192.168.0.254.And I have no idea what this 192.168.20 stuff is.
-
@boethius How many times do we have to go over how this works... That works the same freaking way - it only knows what devices respond from..
Why are you pinging 23.118.48.241 ???
What part are you not understanding about loopback answers and other interfaces on routers?
So in your RECORD_ROUTE test to 23.118.48.241 its a hop in its own path? And 192.168.20.11 is used twice? ;)
so is 107.208.8.1 ;)
Your confusing yourself with info that doesn't matter - again these tools don't always work how you think they do ;)
-
How many times do we have to go over how this works... That works the same freaking way - it only knows what devices respond from..
It doesn't work the exact same way, obviously, as it produces different results. It shows what interfaces are used going outbound toward gateway. I only saw the outbound interfaces/what the devices respond from TOWARD me. Now I see that 192.168.254.254 is used for traffic going toward the gateway.
Now I understand why I wasn't seeing the gateway in the hops to anywhere on the internet. Traffic was being routed to the gateway, but the traceroute responses were from another interface. So don't think I am confusing myself pointlessly, I am feeling much more clear on routing / interfaces / etc. And I appreciate your help. I can be slow sometimes but I do try to have a full picture of what's going on.
and I am wondering what to make of that result to 23.118.48.241.
-
@boethius sorry but it doesn't work that way either.
Sniff on your traffic, what is the destination of the traffic mac address. This is obtained by arping for the mac. So if your gateway is 23.118.48.1 then you have to be able to arp for it.
If there was a router at 192.168.254.254 in your path - you wouldn't be able to arp for 23.118.48.1
A device when the IP is not local, arps for its gateway, or the gateway in its routing table for that network. Then sends the traffic to that mac address with the destination IP address.
Maybe your isp device is doing proxy arp for for every IP?
I would suggest you sniff while you do you traceroute - what is the mac address of where your sending and what is the destination. What is the mac address when you ping other IPs on your same wan network..
Also notice the ttl on your pings their with -R when you ping your actual gateway IP the 48.1 you have a ttl of 63, so this is only 1 hop away. But your showing 3 IPs their in your route path.. when you ping 48.241 your getting back ttl of 61..
I am afraid you are going to draw some odd conclusion on how routing works.. When your isp could be doing odd things with bridges and proxy arp, etc. etc..
Routing is pretty simple. If IP is not on my network, send it to gateway be that default or one in my routing table. How do I do that, arp for that IP. Send to that mac..
Now what your isp might be doing in its network to support its customers spread over a wide area and wanting to put on the same L3 network with bridges and proxy arp and extended L2 for an area.. And what you might get back with a trace from loopback addresses and vips, etc. in such a network.
But routing is basic.. What is the IP of my gateway, ok arp for that - send to that mac with my destination IP. Just not understanding what your trying to figure out other than the isp network is prob doing all kinds of odd stuff. They don't have just your device via a cable to a switch to their single router with 1 IP on it.
-
@boethius said in Multi-WAN vs Whatever this is...:
So your topology there with showing 192.168.0.254 makes less sense.. In what scenario would pfsense send traffic to that IP, its not in your routing table. How is it doing the IP passthru exactly.. What would make sense is its routing the traffic and its answering you via its loopback address.
I don't know how IP passthrough works exactly. It's weird. Here's a pcap of arps on WAN.
@johnpozPCEngine is my pfsense box. HUMAX is the AT&T BGW.
If I use nmap with ARP (default) for discovery and traceroute to my /23 public subnet, it fills with every address in that space showing online, 1 hop away. Because the BGW is answering for them all. In fact, I believe I could set my gateway to any address there, say, 23.118.48.69, or 23.118.49.255 even, I bet it would work all the same.
In my first post in creating this thread, I said
First off, the "upstream gateway" that AT&T specifies is never the hop after 192.168.0.254. I am wondering if that is because with IP Passthrough, the bgw320-500 essentially adopts that address routing
It didn't seem like you were quite oriented when you started to help me out, maybe overlooking the oddities of this IP Passthrough thing, but you've also overlooked insight I've had. You asked twice about how it was pfSense shown in the GUI screenshot. I thought maybe you were doubting me entirely, that I was so clueless as to think I was running those commands from pfSense when I was on a windows host. While I do make mistakes, I'm more able than that.
Indeed routing is fairly simple. This isn't that, as I've been saying. I am trying to understand my network / ISP's network and routing of my traffic. Beyond a simple curiosity, other reasons I am interested I've noted already. My network has behaved (very) strangely in the past. A previous discussion on here with another knowledgeable member of the administration had me finding more questions and few answers in the course of investigating. https://forums.netgate.com/topic/168975/permission-errors-running-commands-as-admin/14 I was also seeing these errors while troubleshooting this, as shown here
It seems that the command worked... so I don't know what's causing this to happen. Is it possible for a permission denied error to be seen in pfSense due to some form of rejection that occurs with the BGW? I wouldn't think so. -
I tried to edit the message to have your
@johnpozreply outside of the quote text region and was seeing "Post content flagged as spam by askimet.com"
