Weird setup issue
-
I've just set up a satellite office at work and bought a Netgate M1n1wall pre-loaded with pfSense to ship there. I've been using pfSense both on Netgates as well as virtualized for a few years now but this is the first time that I've seen this.
I configured the unit before shipping it out. I gave it a static IP from my fiber provider, configured my rules/NAT/etc, and build my IPSEC tunnel. Everything worked just fine. Just before I shipped it out, I changed the static IP and gateway to those given to me by the ISP for the remote site. The remote site has Ethernet over Copper supplied by Cox Communications in California. Now that the unit has arrived, one of the staff members down there connected the Netgate to the EoC equipment and the LAN. They were able to access the WebGUI and confirm that the WAN port is up but are unable to ping the ISP gateway address via the WAN from the WebGUI.
I walked the user through disconnecting the Netgate, reconfiguring a local desktop with the same static information supplied by the ISP, and testing. The desktop connected just fine using all of the same settings. Putting the Netgate back in place shows the same thing as before. I get connectivity indication that the WAN port is "up" and physically connected to the EoC equipment, but I have no Internet access. I've walked the user through confirming the IP, Gateway, and CIDR mask over and over and everything looks just fine.
I don't understand what I'm missing. Can anyone offer any suggestions as to what is different now that this unit is connected at my remote site as compared to when it was here at my site?
-
Is the gateway at the remote site able to respond to pings? Some are not and in that case pfSense will see the connection as down shortly after it's connected. You would see that reported by apinger in the logs though. The solution to that is to choose a different monitor IP.
IS the remote EoC equipment locked to a MAC address cable modem style? Perhaps a simple power cycle of that could help. Maybe the ISP monitors IPs against MACs and won't talk to one that changes without some sort of authorisation. That doesn't explain why the desktop machine was able to connect without a problem though. What hardware is the m1n1wall replacing?
Steve
-
Is the gateway at the remote site able to respond to pings?
Yes, I'm able to ping the gateway for the remote site from my office. I was also able to ping that gateway once I had the user put his Win7 PC in place of pfSense.
IS the remote EoC equipment locked to a MAC address cable modem style? Perhaps a simple power cycle of that could help. Maybe the ISP monitors IPs against MACs and won't talk to one that changes without some sort of authorisation. That doesn't explain why the desktop machine was able to connect without a problem though. What hardware is the m1n1wall replacing?
It's my understanding that the EoC setup isn't locked to a MAC. When pfSense didn't connect I then swapped in the PC before putting the Netgate back in place. I would expect the Netgate to have been the first MAC that the EoC equipment saw, unless there is some reason that it can't see pfSense, and then the desktop wouldn't have worked.
This is a new installation so I don't have any older hardware to fall back on. My EoC has only been in a couple of days and my gear just arrived at the new location yesterday.
-
Is the gateway outside the WAN subnet?
That is something that has caught out a few people. It's a configuration that should never exist because it breaks the rules. Since it's outside the IP specification FreeBSD doesn't support it but Windows has some sort of cludge that allows it to work.It's beyond my memory but since it's not the first time there may be a workaround if that's the case.
Steve
E.g.: http://forum.pfsense.org/index.php?topic=37301.0
-
Nope, the gateway is part of our /29 block.
IP Block: XXX.XXX.XXX.168/29
Gateway: XXX.XXX.XXX.169
Subnet: 255.255.255.248
First Usable: XXX.XXX.XXX.170
Last Usable: XXX.XXX.XXX.174 -
Hmm, Do you have access to the logs? Anything in them? :-\
Steve
-
I'm waiting for the onsite guy to turn up for work so I can have another look. With the time difference between here and California, I won't have access to my remote hands for another hour. :)
It's frustrating, having set it all up and tested it before shipping I anticipated an easy peasy setup without needing the onsite guy to do anything other than plug it together for me. I've done this numerous times without any issue. I can't for the life of me figure out what's different this time.
-
I can't say as I can see anything out of the ordinary in the logs. Below are the log entries from the time my local guy got onsite this morning. I had him restart the ISP equipment first thing but that didn't get us anywhere.
Sep 4 08:38:13 check_reload_status: Linkup starting vr1
Sep 4 08:38:13 kernel: vr1: link state changed to DOWN
Sep 4 08:38:16 php: :Hotplug event detected for wan but ignoring since interface is configured with static IP
Sep 4 08:41:52 check_reload_status: Linkup starting vr1
Sep 4 08:41:52 kernel: vr1: link state changed to UP
Sep 4 08:41:55 php: :Hotplug event detected for wan but ignoring since interface is configured with static IP
Sep 4 08:41:56 check_reload_status: rc.newwanip starting vr1
Sep 4 08:42:00 php: :rc.newwanip: Informational is starting vr1.
Sep 4 08:42:00 php: :rc.newwanip: on (IP address: {MyStaticIPHere}) (interface: wan) (real interface: vr1)
Sep 4 08:42:00 php: ROUTING: setting default route to MyISPGatewayIP
Sep 4 08:42:00 apinger: Exiting on signal 15
Sep 4 08:42:01 apinger: Starting Alarm Pinger, apinger(48687)
Sep 4 08:42:01 check_reload_status: Reloading filter
Sep 4 08:42:11 apinger: ALARM: COXGW (MyISPGatewayIP) down
Sep 4 08:42:21 check_reload_status: Reloading filter
Sep 4 08:49:18 dnsmasq[23098]: reading /etc/resolv.conf
Sep 4 08:49:18 dnsmasq[23098]: using nameserver MyDNS#53
Sep 4 08:49:18 dnsmasq[23098]: using nameserver MyISPDNS1
Sep 4 08:49:18 dnsmasq[23098]: using nameserver MyISPDNS1
Sep 4 09:01:18 php: /index.php: Successful webConfigurator login for user 'ANTech' from 10.2.100.2 -
Apinger is showing the gateway as down. Even if you can ping it remotely I would try changing the monitor IP to, say, 8.8.8.8. Can your man ping the gateway from the Win7 box?
Steve
-
We were unable to ping both the ISP gateway and Google's DNS from the webConfigurator.
When the Win7 box was put in place of pfSense, both of the above were pingable.
-
I assume you are using 2.0.3 32bit? Some people have had some odd IPv6 routing issues recently with 2.1RC.
Other than that I out of suggestions. :( Other than contacting Negate who may have some insight specific to your ISP. I'm the wrong side of the pond for that. ;)
Steve
-
Perhaps this is something more fundamental. The Netgate box has be proven when you configured it initially (assuming it wasn't damaged in transit). The EoC box has been proven by connecting the Win7 box.
The interface reports being UP but is it really? So far you have seen no traffic at all from vr1, yes?
This is the sort of thing that can be caused by some rare hardware mismatch. Is the Netgate box connecting 100Mbps full duplex? Can you try putting a switch between the Netgate box and EoC equipment?Steve
-
I've asked my onsite guy to pull everything out of the site switch, connect it in between the Netgate and Cox and then configure a static in his PC and plug it into the LAN port of the Netgate to check on that.
I've also purchased support and opened a ticket with the pfSense gang. I suspect one way or another this will be resolved shortly. Support is so rarely required with pfSense that I generally only purchase it as a last resort. Sounds like we're there. ;)
-
You could have your man take a laptop and broadcast wifi from his phone and with the laptop ethernet port connect to pfsense and wifi to phone.
You could then use his laptop to see whats up with pfsense via teamviewer yourself.
I suspect you are dealing with a fat fingered typo in settings or something very simple like that.
(I did this for two of the forum members recently - Fat finger their settings I mean… :P )
-
Unfortunately we don't have any laptops onsite and my local guy hasn't been in the U.S. long enough to have re-purchased the basic amenities for himself. :)
He's actually surprisingly good. There's no way I could have achieved this with any of our other warehouse managers. If I was going to have an issue like this, I'm glad that it happened with this site. I expected a typo as well but he helped rule that out very quickly (several times…just to be sure). The only things that were changed from when it was in a working state were the static IP and the gateway address really so it was a short list of things to confirm.
I sent a copy of my config as well as some basic command line results to pfSense support yesterday and they confirmed that all was well. So far, they appear to be as stumped as I've been but are narrowing the options down.
We pulled everything apart yesterday and put a switch in between pfSense and the ISP equipment but that didn't do anything either.
-
I've set one of these up with on cable before. What version of pfsense are you running and is it 64 or 32 bit?
-
I've used pfSense quite successfully on Cable, DSL, and Fiber in the past. While Ethernet over Copper isn't hugely different, it's definitely not a cable line. The ISP's hardware does still present me with a modem-like device that sits behind the EoC bonding device.
I'm currently running pfSense 2.03. While I didn't think to look before I shipped it out, the ALIX board has an AMD Geode LX800 which I believe should mean that it's likely running the amd64 NanoBSD build but I don't know that for certain. I'll have the local guy confirm that when he gets on site today. I have a number of ALIX boards identical to this one running on both DSL and Cable installs but this is my first on EoC. Given how easily a PC connects with a static IP, I can't imagine the differences are significant…and yet here we are. :) If I hadn't already tested for it so many times already, I would feel inclined to say that it's a bad cable based upon the behaviour that I'm seeing.
-
Hmmmm - Maybe try backing up your install configuration and then installing the same box with the 32 bit version of 2.1?
See if results are better. Once, when 2.03 didn't work for me with lots of IPs 2.1 did.
-
That's certainly an option but I'll probably leave it as a last resort. Given that I have several implementations of 2.03 in the field that have been trouble free and given that this one is such a basic setup (and so far away) I'm hoping to avoid deploying an RC in production unless I have to.
-
I agree its very strange - I can only imagine a few reasons for this to happen.
There was a router of some sort previously connected and you should clone its MAC to get an IP. (This is some BS I encounter occasionally)
There is something different about this set up than your others.
I've actually not had much luck with 2.03 and more than 4 IPs. (Others maybe have. My experience with it is limited)Lastly, maybe its not a router problem at all. Maybe the ISP made an error either in what it allocated you or the info they provided you?
