Can't reach a specific IP address

danswartz

I am confused too. I can ping 216.251.231.64 but not 216.251.231.61.

MTHead

Yes, it was a typo. Sorry for the red herring; I should just have said "numeric IP". The correct IP address is 216.251.231.64.

"CMD prompt": I meant the "DOS box" on my Windows 7 laptop: literally the command prompt you get when you run CMD.exe; I'm sorry I wasn't clear. I will refer to it as "DOS" from now on; although that's not correct, I suspect it's less prone to misinterpretation. In any case, the Windows ping defaults to three tries and out, and that's how I was running it.

So let me try again. On my (Windows) laptop, I open a Putty session to the pfSense shell and a (Windows) command prompt. I run tcpdump in the Putty session and ping in the DOS box. When I try to ping using the domain name, tcpdump shows the DNS transactions and then nothing more. When I try to ping using the numeric IP (see what I did there?), tcpdump shows no activity at all. (In both cases, I mean "no activity related to what I'm doing"; people were still surfing the web, checking email, etc.)

Regarding which interface the packet goes out: just for giggles, I ran a tcpdump on the LAN interface (LAN is fxp0; WAN is fxp1). As you might expect, there was a flurry of traffic between my laptop and pfSense, but I did not see any packets addressed to 216.251.231.64. If pfSense is routing packets addressed to 216.251.231.64 through a different interface (but not packets addressed to 216.251.231.63, which I am able to reach), shouldn't there be a file or setting somewhere to specify that? Essentially, that was my initial question, and it's still what I'm puzzling over.

Regarding traceroute: When I run it from DOS, the first line shows that I've reached the pfSense box; after that, only "* * " until it times out. When I run it from either the pfSense shell, or the Diagnostics menu in the WebGUI, I get nothing but " * *". Just now, running it from DOS at home, the trace completed in 21 hops (including my own router).

wallabybob

@MTHead:

Regarding which interface the packet goes out: just for giggles, I ran a tcpdump on the LAN interface (LAN is fxp0; WAN is fxp1). As you might expect, there was a flurry of traffic between my laptop and pfSense, but I did not see any packets addressed to 216.251.231.64.

Do you realise the significance of "did not see any packets addressed to 216.251.231.64"?

You have just said you did not see any (that would include incoming!) packets addressed to 216.251.231.64. That implies your laptop is not sending them OR the physical connection is seriously broken!

If the ping works when your laptop is "directly connected" to the T1 how does the laptop get its IP address? How does the laptop gets its IP address when its connected to pfSense? Do both the laptop and pfSense think they are on the same subnet in the latter case? (If not, your configuration is broken.) How will the laptop know to route 216.251.231.64 to pSense? If you are using static IPs anywhere did you restart your laptop when you changed what it was connected to? (If you didn't, how can you be sure it wasn't using stale network information?)

You say you have disabled checksum offloading at some stage. If you aren't running with checksum offloading you should do so until further notice. (There is a known problem in the FreeBSD fxp driver that it erroneously thinks some fxps have checksum offload capability.)

MTHead

OK, I think I finally might be on to something, but how it happened or how to fix it are still unanswered questions…

The office is closed today, so I'm working from home; as such, I don't have physical access to the LAN interface. So as an experiment, I opened two SSH sessions in separate windows; I presume that in this case all traffic between my machine and pfSense would be over the WAN interface? In any case, in the first session I ran tcpdump on the LAN interface, and in the second I tried to ping Palmetto. Result? In the tcpdump I get lots and lots of entries like this:


10:46:29.182937 arp who-has 216.251.231.64 tell 192.168.33.1
10:46:30.183472 arp who-has 216.251.231.64 tell 192.168.33.1

192.168.33.0 is the net I reserved for OpenVPN "road warriors". I changed the OpenVPN net to 192.168.35.0 and tried the same experiment; the result was pretty much the same:


11:13:18.106462 arp who-has 216.251.231.64 tell 192.168.35.1
11:13:19.107454 arp who-has 216.251.231.64 tell 192.168.35.1

I then disabled the OpenVPN tunnel and tried again - all packets went through! No traffic was generated on the LAN interface (as I expected - why should it be?), and a tcpdump on the WAN interface looks normal - at least as far as I'm able to recognize "normal". I re-enabled OpenVPN, and once again Palmetto is unreachable. I'm leaving it that way for now, because the doctors need to use the system over the holiday but the billing department doesn't need to reach Palmetto until Monday…

My next experiment, I suppose, would be to delete the OpenVPN tunnel entirely and create a new one. I'm reluctant to do that (unless I know that it will be a permanent fix) because I would need to re-generate and re-distribute certificates.

Assumption: Somewhere in the bowels of pfSense there is a setting that says "route all packets intended for Palmetto over OpenVPN".
Questions: Where would I find this setting? Why would it have spontaneously changed over Xmas weekend?
If anyone has any suggestions as to where I might look, I would be much obliged.

ARPTable.jpg_thumb

MTHead

@wallabybob:

If the ping works when your laptop is "directly connected" to the T1 how does the laptop get its IP address? How does the laptop gets its IP address when its connected to pfSense? Do both the laptop and pfSense think they are on the same subnet in the latter case? (If not, your configuration is broken.) How will the laptop know to route 216.251.231.64 to pSense? If you are using static IPs anywhere did you restart your laptop when you changed what it was connected to? (If you didn't, how can you be sure it wasn't using stale network information?)

You say you have disabled checksum offloading at some stage. If you aren't running with checksum offloading you should do so until further notice. (There is a known problem in the FreeBSD fxp driver that it erroneously thinks some fxps have checksum offload capability.)

Direct connection: When I attached directly to the T1, I configured my IP/netmask/gateway/DNS manually, to the same settings as the pfSense WAN interface. When I attach to the LAN, I use DHCP. Really, it ain't rocket surgery.

Checksum offloading: What I was trying to say was that I had tried both checking and unchecking BEFORE upgrading to 1.2.3Release, and it had not made a difference either way; and then AFTER the upgrade I had tried it both ways, with again no difference. However, I did read the release notes and have left it checked, as recommended. In any case, I'm pretty sure that this issue didn't have anything to do with checksum offloading…

danswartz

Well, this is the first we are hearing that openvpn is involved. One smoking gun is a host on the LAN subnet trying to ARP for a remote host. That is most likely the root of the problem. As to why, dunno.

wallabybob

@MTHead:

Assumption: Somewhere in the bowels of pfSense there is a setting that says "route all packets intended for Palmetto over OpenVPN".
Questions: Where would I find this setting? Why would it have spontaneously changed over Xmas weekend?
If anyone has any suggestions as to where I might look, I would be much obliged.

If there is such a setting its because you activated it through your own configuration setting.

Based on the evidence you have given you have come to the wrong conclusion. In particular you say that when you try to ping Palmetto from the laptop there are no packets with the Palmetto address in the tcpdump. This means pfSense isn't receiving the packets destined for Palmetto so of course it isn't forwarding them!
The routing is broken on the laptop.

The VPN adds another factor to the problem bust since you haven't given any information about it other than to mention there is a VPN I can't take it into account. I think you should really to try to understand why the VPN is in the configuration before you attempt to recreate it.

Wild speculation: When connected directly to the internet the laptop is able to create a VPN that enables it to get to Palmetto. When connected to pfSense the laptop can't establish the VPN so "falls back" to attempting to connect with Palmetto over the only operating interface - the LAN.

MTHead

@danswartz:

Well, this is the first we are hearing that openvpn is involved. One smoking gun is a host on the LAN subnet trying to ARP for a remote host. That is most likely the root of the problem. As to why, dunno.

That's because it's the first time I had any idea that OpenVPN had anything to do with it. I use OpenVPN on every pfSense box I set up - including my home router, from behind which I'm typing this. In fact, I use the same configuration (except for certificates, of course) for all my clients (except the ones who need to use PPTP from multiple hosts, in which case I use Endian.) I've never seen anything like this before, nor heard of it.

Any ideas on how I could track down which host is "volunteering" to ARP? Of course I can go to the office and unplug machines from the network one by one, but if there's a more sophisticated way to find the answer…

MTHead

@wallabybob:

@MTHead:

Assumption: Somewhere in the bowels of pfSense there is a setting that says "route all packets intended for Palmetto over OpenVPN".
Questions: Where would I find this setting? Why would it have spontaneously changed over Xmas weekend?
If anyone has any suggestions as to where I might look, I would be much obliged.

If there is such a setting its because you activated it through your own configuration setting.

Based on the evidence you have given you have come to the wrong conclusion. In particular you say that when you try to ping Palmetto from the laptop there are no packets with the Palmetto address in the tcpdump. This means pfSense isn't receiving the packets destined for Palmetto so of course it isn't forwarding them!
The routing is broken on the laptop.

The VPN adds another factor to the problem bust since you haven't given any information about it other than to mention there is a VPN I can't take it into account. I think you should really to try to understand why the VPN is in the configuration before you attempt to recreate it.

Wild speculation: When connected directly to the internet the laptop is able to create a VPN that enables it to get to Palmetto. When connected to pfSense the laptop can't establish the VPN so "falls back" to attempting to connect with Palmetto over the only operating interface - the LAN.

It seems to me that you answer my posts without reading them. It seems only fair that I should read yours, and not answer it.

MTHead

Just to follow up in case anyone else ever has a similar problem: I added a static route, thusly:

Interface Network Gateway Description

WAN 216.251.231.64/32 (our gateway) Palmetto

and now my users can reach the Palmetto website. This static route is the same as the default route, so I don't really understand why it's necessary… but it works.