Asymetric routing, ISP environment

lambert

I have two Internet connections in towns which are separated by about 100 miles.

I have wireless links using over a dozen towers between the two towns.

I've been using pfSense as a NAT gateway in the remote town to provide a little extra bandwidth locally. It has been setup as a NAT router since the ISP there wouldn't route my /20 for me without going BGP; we have several old style customers depending on the one large NAT setup used by the previous owners, and, we didn't have the towers linked all the way through until this month.

We've been using the OpenOSPFd package to talk to the other routers at the towers in between. It has worked okay, but sometimes, when one of the bridge links to another tower goes down, I have to```
killall ospfd && /usr/local/etc/rc.d/ospfd.sh


Now I've added OpenBGPd to the mix.    During testing and turn up of the BGP association, the provider took the association down to reconfigure their advertisement filters and when they did, OpenBGPd took out the statically configured default route.  Annoying, but easy enough to fix the first time.  The second time, it took me a few minutes to notice and OpenOSPFd had inserted it's learned default.  Grrr, but same GUI fix. 

We take a default from the provider and advertise a subset of our space up there.  Well, we did for about a day before we got a handle on the user complaints.  I had thought I had the rules setup to not need to keep state on the traffic to and from my advertised space, but it doesn't seem to have worked very well. When traffic from that space left the network at our primary location, and came back via the remote connection, the packets got dropped as they came in.  Of course, traceroute and ping worked just fine.  Next time, I test with "telnet mailserver 110".

So, I disable BGP and get all the return traffic to come in through the primary location again, I did a "killall bgpd" since there doesn't seem to be a way to get that to happen through the GUI.  Then I re-enabled the default route.  I also changed the config such that I thought it would be unable to successfully create an association if pfSense restarted the connection.  I messed that up too.  We had a power outage and it came back up.  Grrr.  When I killed it, I was in a hurry and failed to check the default route immediately.  OpenOSPF replaced it.  I fixed it in the GUI and went home. 

Apparently OpenOSPFd had taken ownership of that route and, a couple of hours later, noticed that it had changed.  It "corrected" the default route.  GRRR.

I rebooted to get it to quit overriding the default route.

Has anyone successfully used pfSense in this kind of environment, without entirely disabling the NAT capabilities?  If they have, I'm guessing the did it with Quagga.  I'd like to hear any success stories, but I'm replacing the pfSense box with a spare ImageStream router on Tuesday.  Nothing against pfSense, I just think I'm trying to use it outside the scope of its core design as a stateful packet filtering machine. 

I was hoping I could manipulate pfSense into handling this to have a web GUI router for my less technical co-workers so I can take vacation, but at this point, I'm going back to the old ways for this application.