OSPF, routing, route vs interface, chicken and egg

[Vetal]

I have a 3 node network with OSPF Quagga running on each of them.

Each node connected with p2p, OpenVPN:

pfSense has two OpenVPN clients, with ip 10.0.1.2 and 10.0.2.2, going to server 1 and 2 respectively.

Now, connection C (10.0.2.1 - 10.0.2.2) dies. What happens is, network is adapted and pfSence receives OSPF update that (10.0.2.1 - 10.0.2.2) is accessible via [B->A] route, instead of just [C]
pfSense has information about 10.0.2.1 - 10.0.2.2 link and knows it exists 2 hops away.

Result is, when OpenVPN is trying to restart, it can't. Since it can't assign ip address which is in routing table:

/sbin/ifconfig ovpnc1 10.0.2.1 - 10.0.2.2 mtu 1500 netmask 255.255.255.255 up
FreeBSD ifconfig failed: external program exited with error status: 1

Manual call tells

ifconfig: ioctl (SIOCAIFADDR): Address already in use

Which way I can prevent this? Can I probe link somehow?

I already tried link-detect, it shows link as up on Server 2 (listening sed) anyway:

interface tun1
 description VPS link C
 link-detect
 ipv6 nd suppress-ra
!

Shall it be separated with zones, e.g. A - backbone, ? Or is there a tcp-probe inset to quagga to tell if link is pingable?

If there is a probe, it will help to deal with the case like on the picture, if segment A is down. Traffic will be rerouted via pfSense and stuck there. Since it is disabled (intentionally) by pfSense.

I am new to dynamic routing and believe there is a standard way to deal with this loop

[rubic]

@Vetal:

when OpenVPN is trying to restart, it can't. Since it can't assign ip address which is in routing table

/etc/inc/openvpn.inc:

function openvpn_restart($mode, $settings) {
	...
	...
	...

	openvpn_clear_route($mode, $settings);
	mwexec_bg("/usr/local/sbin/openvpn --config {$fpath}");

	...
}

function openvpn_clear_route($mode, $settings) {
	if (empty($settings['tunnel_network']))
		return;
	list($ip, $cidr) = explode('/', $settings['tunnel_network']);
	$mask = gen_subnet_mask($cidr);
	$clear_route = false;

	switch($settings['mode']) {
		case 'shared_key':
			$clear_route = true;
			break;
		case 'p2p_tls':
		case 'p2p_shared_key':
			if ($cidr == 30)
				$clear_route = true;
			break;
	}

	if ($clear_route && !empty($ip) && !empty($mask)) {
		list($ip1, $ip2) = openvpn_get_interface_ip($ip, $mask);
		$ip_to_clear = ($mode == "server") ? $ip1 : $ip2;
		mwexec("/sbin/route -q delete {$ip_to_clear}");
	}
}

It seems that in p2p mode the obstructing route won't be deleted if your tunnel network prefix length is not "/30".

[heper]

you'd need to prevent the distribution of the vpn tunnel networks.
goto quagga-ospf –> interface settings --> accept filter . check this box & try again.

sometimes its not enough (depending on whether or not you have assigned interfaces to your ovpn setup).
if you have interfaces, you might still get hit by the /32 route of each node.
==> you can manually disable acceptance of the /32 routes in the quagga config .... but you would have todo it on all nodes. (servers get the .1/32 , clients the .2/32)

i hope this makes sense somehow

[Vetal]

@heper:

you'd need to prevent the distribution of the vpn tunnel networks.
goto quagga-ospf –> interface settings --> accept filter . check this box & try again.

sometimes its not enough (depending on whether or not you have assigned interfaces to your ovpn setup).
if you have interfaces, you might still get hit by the /32 route of each node.

Thanks a lot that's did the trick !!!
Rule of the thumb, on each pfSense (OVPN-client) box:

for each VPN-ospf endpoint, disable acceptance for "my(endpoint) IP/32". So, it won't come from other end.

It makes perfect sense after all. With my 1-week Quagga/OSPF experience, I've been trying to close /24 P2P part everywhere.
Obviously, it didn't work, being advertized anyway with ospf statement  "network …./30 area 0.0.0.0"

Now I never get my endpoint from OSPF and VPN links restart easily.

[heper]

;D

[Vetal]

Decided to add metric to my routes. based on latency, since I am not bandwidth bound, but latency bound (Remote desktop experience). And half way through the process I realized there is another issue:

Quagga doesn't know there is no path in pfSense router between p2p VPN links (red arrow-line):

Leading to some nasty effects, where ping won't go to the interface on the same machine, but going back through the seemingly direct path. Even when I can "align" all net, there is still feeling of things might go wrong once "shorter" link fails: non-passable area must be declared as such.

Is there an easy way to disable traffic from being routed through the pfSense router, while allowing to go to the router. To reach the to stub LANs?

Got some materials to read, though still not sure if it is relevant:
http://blog.ine.com/2009/08/17/ospf-route-filtering-demystified/
http://www.informit.com/library/content.aspx?b=CCIE_Practical_Studies_I&seqNum=138

[heper]

is there a good reason your are making it more complicated by firewalling the "path" through the pfsense?

so basically if "link-A' goes down, you don't want server2 to be able to reach server1 through the pfsense ?

what do you mean by this?

where ping won't go to the interface on the same machine, but going back through the seemingly direct path.

[Vetal]

@heper:

is there a good reason your are making it more complicated by firewalling the "path" through the pfsense?

I am transitioning multi-site company, something like this, star + redundant path topology:

First reason, once I've done it goes to the local supporter. Which might firewall, block or screw it up by mistake. So I assume, it is already screwed :) And network must resist to such things from VPS A and B prospective

Another reason, all this schema assume that traffic won't pass through the branch offices. It doesn't make sense anyway - kind of switching terminals in Frankfurt via Chicago

@heper:

so basically if "link-A' goes down, you don't want server2 to be able to reach server1 through the pfsense ?

what do you mean by this?

where ping won't go to the interface on the same machine, but going back through the seemingly direct path.

Let's assume following fragment, where I set metrics on Company C, VPS A and VPS B. And trying to ping some interface (not shown) on VPS B, where central DNS server reside. :

Let's assume,
tun0, 10.2.0.1 is this non P2P interface on VPS B
tun1, is P2P interface on VPS,  [VPS B to VPS A P2P segment, path for ping red arrow 2]

If I ping from PC on Company C, ip 10.2.0.1. Ping (request part) goes via tun1 on VPS B (Red arrow 2). And it is last point where it is seen. I assume, #3 is ping suppressed by loop prevention code

If I disable Company A (which doesn't have metric set anywhere yet). Leaving net with VPS A, VPS B, and Company C, ping goes to 10.2.0.1.

Moreover, on VPS B,

ping 10.2.0.1 -I tun1

will drive ping away, as it assume path via Company A as more direct one. Obviously, ping never reach same machine interface, even it is less than 1ms away

Yes, metric is not set up everywhere. Even if Company A maintainer won't screw up, it can be the case for non reachable part.

VPS A to VPS B, metric 50
Company A => VPS 1, metric 10 (very fast line)
Company A => VPS 2, metric 10 (very fast line)

So, Company A will attract traffic to go via it's fast lines just to firewall it.

P.S. What is the editor network used here  to quick draw diagrams, local or online? Coming from development field, I don't have any tool set worked out for it yet.