Phantom Route Appearing

ajh

Greetings!

I'm definitely new here! I'm assisting a local business with their IT setup after losing their internal guy. I've inherited what appears to be a hastily configured pfsense box. Most everything is working great, except for one peculiar hiccup. They had one server (an important one) that couldn't get to the Internet and could not ping the firewall. Upon some closer inspection, I discovered a route for this specific host pointing to the upstream gateway (ISP). Not being able to find it anywhere in the web interface, I removed it manually using:

route del -host 10.1.1.2

Problem solved. Right? Temporarily at least. Everything works swimmingly until at random times the route re-appears. I figure it's hard coded somewhere and something is triggering it. It's not at any specific recurring time. It literally is random times. Sometimes once a day, sometimes 3 times a day.

This is my first experience with pfsense, although I have a 15 year background with Linux firewalls. (ipfwadm, ipchains, iptables) I'm open to ANY suggestions on places to look. I'd list the places I've looked, but near as I can tell I have literally click through every screen. Even did a grep through the file system looking for the IP address and didn't come up with anything useful.

Thanks!

phil.davis

/conf/config.xml contains all the things that are setup through the web interface.
Diagnostics->Edit File, and search for the IP address. That might give a clue.

johnpoz

So there a host specific route on pfsense, where this host is on your lan segment say 10.1.1.0/24 and the route said to get to 10.1.1.2 go talk to your isp gateway??

ajh

@phil.davis:

/conf/config.xml contains all the things that are setup through the web interface.
Diagnostics->Edit File, and search for the IP address. That might give a clue.

I found that file the other day. Here are the occurences of the that IP address. Consider the "…" a "yadda yadda yadda"

(this one seem innocent enough)
<system>...
<dnsserver>10.1.1.2</dnsserver>
...</system>

(not sure what this one is for)
<dnsmasq><enable><domainoverrides><domain>neitx.local</domain>
<ip>10.1.1.2</ip></domainoverrides></enable></dnsmasq>

(not sure why this even exists.. its from machines on the inside to a server on the inside)
<rule><id><type>pass</type>
<interface>lan</interface>
<ipprotocol>inet</ipprotocol>
<tag><tagged><max><max-src-nodes><max-src-conn><max-src-states><statetimeout><statetype>keep state</statetype>
<os><protocol>tcp</protocol>
<source>

<created><time>1417636661</time>
<username>Easy Rule</username></created>
<updated><time>1417636677</time>
<username>admin@10.1.1.29</username></updated>
<tracker>1422273434</tracker></os></statetimeout></max-src-states></max-src-conn></max-src-nodes></max></tagged></tag></id></rule>

(this was apparently for a vpn setup that I'm not sure anyone is even using)
<openvpn-server>…
<dns_server1>10.1.1.2</dns_server1>
...</openvpn-server>

Thanks for your help. Let me know if you see anything there that could cause a problem!

ajh

@johnpoz:

So there a host specific route on pfsense, where this host is on your lan segment say 10.1.1.0/24 and the route said to get to 10.1.1.2 go talk to your isp gateway??

Yes and no. The internal segment is 10.1.1.0/24. When the route appears, it's listed as

Destination Gateway Netif
10.1.1.2 65.36.44.129 (same as default route) re0 (outside interface)

as compared to
10.1.1.0/24 link#2 sk0

The next time it appears I'll grab a screen shot of it. I've always been in such a hurry to ditch it that I can't remember what the flags or Mtu are set to.

Thanks!

johnpoz

Well I get a specific route to a host that goes out my wan gateway - see attached. But that is my gif I setup for my HE tunnel.

So there prob something setup somewhere that is causing that, from your previous guy. Maybe related to some vpn he was trying to setup?

routetoisp.png_thumb

ajh

I considered the VPN issue. I'm still waiting for it to show back up. I recall it being routed out the same interface as the WAN.

Here's the current table. When it appears it pops in right below the 10.1.1.1 lo0 entry. As soon as it reappears, I'll grab a shot of it and post it. Thanks for all the suggestions so far! It's much appreciated!

routes.png_thumb

ajh

New data to share! Thanks in advance for any suggestions!

OK. So it happened again. I've got a valid screen shot of the route table with the extra entry. I've also got a log section that may be relevant. I had to keep checking the status periodically to have a narrower window of when it happens to know what might have value. It looks as through a link problem may be triggering whatever is happening by causing the internal sk0 to drop and come back. Now, if that's the case I at least have the trigger action. I just need to figure out what's being triggered.

Possible relevant log:

Feb 18 20:07:03 check_reload_status: Linkup starting sk0
Feb 18 20:07:03 kernel: sk0: link state changed to DOWN
Feb 18 20:07:03 kernel: sk0_vlan1: link state changed to DOWN
Feb 18 20:07:03 kernel: sk0: link state changed to UP
Feb 18 20:07:03 kernel: sk0_vlan1: link state changed to UP
Feb 18 20:07:03 check_reload_status: Linkup starting sk0_vlan1
Feb 18 20:07:03 check_reload_status: Linkup starting sk0
Feb 18 20:07:03 check_reload_status: Linkup starting sk0_vlan1
Feb 18 20:07:04 php-fpm[21785]: /rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.1.1.1 )
Feb 18 20:07:04 php-fpm[21785]: /rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.1.1.1 )
Feb 18 20:07:04 check_reload_status: rc.newwanip starting sk0
Feb 18 20:07:05 php-fpm[21785]: /rc.newwanip: rc.newwanip: Info: starting on sk0.
Feb 18 20:07:05 php-fpm[21785]: /rc.newwanip: rc.newwanip: on (IP address: 10.1.1.1) (interface: LAN[lan]) (real interface: sk0).
Feb 18 20:07:05 check_reload_status: Reloading filter

Routes with extra host route pointing outside:

routes-broken.png_thumb

phil.davis

 <system>...
   <dnsserver>10.1.1.2</dnsserver>
   ...</system>

Also look for the section near that which specifies the gateway for each DNS server (if a gateway is specified), like:

<dns1gw>OPT1GW</dns1gw>
<dns2gw>WANGW</dns2gw>
<dns3gw>OPT1GW</dns3gw>
<dns4gw>none</dns4gw>

If there is a gateway specified for 10.1.1.2 DNS server then the system will make a specific route for that. That could be the issue.

 <dnsmasq><enable><domainoverrides><domain>neitx.local</domain>
                    <ip>10.1.1.2</ip></domainoverrides></enable></dnsmasq>

You will find that in DNS Forwarder, Domain Overrides. Any requests for names inside neitx.local will be referred to 10.1.1.2 - which hopefully has a DNS server that knows about that domain.

The rule is a bit odd, somebody somehow managed to see something in the firewall log and add a rule to pass it. That should not even have been seen by pfSense, as you say. It should not do any harm.

The OpenVPN is giving 10.1.1.2 as DNS server for any "road warriors" connecting. You should be able to see that in VPN->OpenVPN, Servers.

ajh

@phil.davis:

You will find that in DNS Forwarder, Domain Overrides. Any requests for names inside neitx.local will be referred to 10.1.1.2 - which hopefully has a DNS server that knows about that domain.

That's a correct assumption. In fact it was their DNS server not functioning that pointed me to the communication issue in the first place. It couldn't resolve names, so no one could get out.

@phil.davis:

The rule is a bit odd, somebody somehow managed to see something in the firewall log and add a rule to pass it. That should not even have been seen by pfSense, as you say. It should not do any harm.

The OpenVPN is giving 10.1.1.2 as DNS server for any "road warriors" connecting. You should be able to see that in VPN->OpenVPN, Servers.

I came to the same conclusion on both of those. Unfortunately, neither of those should be doing any harm. As far as I can tell.

Hence my problem.

I can't find anything that screams "what are you doing there?" or "A-Ha! There's the culprit!"

Thanks!

ajh.

phil.davis

And what about the DNS Server entry in System->General Setup.
Does that have a gateway selected?
(It should not - since that DNS server is local)

ajh

@phil.davis:

And what about the DNS Server entry in System->General Setup.
Does that have a gateway selected?
(It should not - since that DNS server is local)

Phil,

It is with great admiration that I post this message and image. In skimming all of the pages looking for anything route based, I never once noticed that section of the screen. The DNS settings is not where I would expect a route option to be placed. I can safely say that you have found my problem.

You Da Man.

Many Thanks!

routes-solution.png_thumb

phil.davis

Always happy to help, and to find that every problem eventually succumbs to analysis :)