Routing between two pfsense firewalls connected by fiber

Jeroen

Hi,

I have a "little" issues with 2 pfsense boxes.
Both boxes are at version 2.1.3.

Each box is at a different site.
Each site has a fiber connection out.
There is also a fixed vpn connection between the two sites.
I need to be able to route between the sites on some interfaces, but that doesn't seem to be working.

Site A:

eth0 = wan
eth1 = lan1 (192.168.1.0/24, interface/gateway ip 192.168.1.254)
eth2 = lan2 (192.168.2.0/24, interface/gateway ip 192.168.2.254)
eth3 = lan3 (10.0.0.0/24, interface/gateway ip 10.0.0.1)
eth5 = vpn (interface ip 192.168.50.2)

gateway 192.168.50.2 as route to Site B
static route to 192.168.10.0/24 using gateway 192.168.50.2
statis route to 192.168.11.0/24 using gateway 192.168.50.2

Site B:

eth0 = wan
eth1 = lan1 (192.168.10.0/24, interface/gateway ip 192.168.10.1)
eth2 = lan2 (192.168.11.0/24, interface/gateway ip 192.168.11.1)
eth5 = vpn (interface ip 192.168.50.1)

gateway 192.168.50.1 as route to Site A
static route to 192.168.0.0/24 using gateway 192.168.50.1
statis route to 192.168.1.0/24 using gateway 192.168.50.1

Each pfsense box has more interfaces in use for wlan, but they don't need to be routed between the sites.

I can ping the 192.168.50.1 from the 192.168.50.2 and vise versa.
routing between inferface within a pfsense box works also (so from 192.168.10.0 to 192.168.11.0)

i just can't seem to be able to reach 192.168.10.0 and 192.168.11.0 from Site A and 192.168.1.0 and 192.168.2.0 from Site B.

i also have rules set like source = lan1 net, destination = 192.168.10.0/24 using gateway 192.168.50.2.
and also have allow rules on the 192.168.10.0 net from 192.168.1.0 net.

I just can't seem to be able to reach anything on the other site.
Any one see what i am doing wrong?

georgeman

What type of VPN are you using? If it is IPsec, routing is not handled by the static routes. To make this work, you need to add the appropriate Phase2's on both boxes, with the source and destination subnets according to what you want to route.

Jeroen

Let me rephrase the vpn :)

it's a fixed tunnel between the two sites over fiber.
Both ends have a lightning edge for fiber to utp.

there is no internet traffic, just a tunnel between the two sites.
At both sides connected to eth5.
at site A the eth5 has 192.168.50.2 as ip and at site B eth5 has 192.168.50.1 as ip.
I can ping both from each site.
So the tunnel is working.

So it's not a vpn tunnel setup on either pfsense boxes.

divsys

Have you allowed traffic from VPN to LAN1/LAN2, ie (on Site A) 192.168.50.x to 192.168.1.x/192.168.2.x and similarly on Site B?

Do the logs show anything when you try and ping between sites?

Jeroen

On both sites i allow:

vpn net to lan1 net
lan1 net to vpn net
lan1 to 192.168.10.0 network (and the 1.0/2.0/11.0)
192.168.10.0 network (and the 1.0/2.0/11.0) to lan1 (lan2 etc)

i have rules allowing vpn to lanx and lanx to lany etc on any protocol.

logs show nothing.
I have the lan1 to 192.168.x.0/24 network set to logging.
just when i ping, nothing (not blocked or passed) shows in the log.
i do use the ping util in pfsense to ping the other site.

so what i basicly allow is:

Site A: eth1 (lan1) to eth5 (tunnel) to Site B: eth5 (tunnel) to eth1 (lan1 remote network).
i also allow eth1 (lan1) to eth1 (lan1 remote network) in a rule.

heper

i believe your static routes are flawed…. i would think your gateway, is the ~~host~~ pfsense on the other end.

try this:
site-a

eth5 = vpn (interface ip 192.168.50.2)

gateway ~~192.168.50.2~~ as route to Site B
static route to 192.168.10.0/24 using gateway ~~192.168.50.2~~
statis route to 192.168.11.0/24 using gateway ~~192.168.50.2~~

change "192.168.50.2' –> '192.168.50.1'

site-b:

eth5 = vpn (interface ip 192.168.50.1)

gateway ~~192.168.50.1~~ as route to Site A
static route to 192.168.0.0/24 using gateway ~~192.168.50.1~~
statis route to 192.168.1.0/24 using gateway ~~192.168.50.1~~

change "192.168.50.1' –> '192.168.50.2'

hope it helps

mvg
jeroen

Jeroen

i tried to change the ip's for the gateways, still no luck.

i did notice just yet (by testing the change to the gateway ips) that if i use the diagnostic ping from pfsense and i choose destination 192.168.10.1 and set the source to lan1 net.
I get the following reply:

PING 192.168.10.1 (192.168.10.1) from 192.168.1.254: 56 data bytes

it looks like it's using the 192.168.1.254 as gateway for the 192.168.10.0 network instead of the 192.168.50.2 (or is this correct and am i seeing thing wrong :))
When i select the vpn tunnel net as source i can't ping the 192.168.10.1 addres either, but is show the correct gateway: PING 192.168.10.1 (192.168.10.1) from 192.168.50.2: 56 data bytes

it seems i can also ping the 192.168.50.1 from the 192.168.50.2 (and the other way around)
I can't ping the 50.x adresses from any other interface on the other site.

Is the routing not using the right gateway here?

heper

did you flush states after changing the static routes ?

~~are you combining gateway with static routes for some reason ?~~
what i was trying to ask, is if you are using policy-routing in your firewall-rules, that contradict the "natural flow' of your static routes? ;)

Jeroen

i did reset states.

There is one default gateway (wan side)
Then there is a gateway for the route.

on site A, lan1 there is a rule:

protocal = any
source = lan1 net
destination = 192.168.10.0/24 network

At the bottom the gateway for this rule is the extra gateway (192.168.50.2 for site A)

So at Site A there is a second gateway called ToSiteB and the gateway adres is 192.168.50.2
There is a route network 192.168.10.0/24 using gateway ToSiteB (192.168.50.2) on interface eth5 (the tunnel nic)

At Site B there is a second gateway called ToSiteA and the gateway adres is 192.168.50.1
There is a route network 192.168.1.0/24 using gateway ToSiteA (192.168.50.1) on interface eth5 (the tunnel nic).

obviously there is something wrong or missing (or the tunnel isn't working like it should although i can ping eth5 ip's from either site).
to me right now all config and rules seem fine though :)

heper

as i tried to explain in the previous post:

(site-A: 192.168.50.2)–---------------------------------------------------------------------(site-B:192.168.50.1)
| |
LAN 192.168.1.x LAN 192.168.10.x
LAN 192.168.2.x LAN 192.168.11.x

on site-A:
modify/create the gateway for the fiber interface from 192.168.50.2 TO 192.168.50.1

on site-B
modify/create the gateway for the fiber interface from 192.168.50.1 TO 192.168.50.5

also remove any policy-routing the firewall rules (the gateway-selection at the advanced section)

georgeman

Be careful with the firewall rules, the allow rules should be on the interfaces that serve as a link.

For troubleshooting purposes I suggest to place an allow all rule on those interfaces

Jeroen

@Heper

I've changed that and now i can ping the interface (eg 192.168.10.1) on the other pfsense.
I can't however ping any client in that lan (eg 192.168.10.10).

heper

so to summerize:

client_LAN-A:
-ping to 192.168.50.1 = OK
-ping to 192.168.10.1 = OK
-ping to 192.168.11.1 = OK
-ping to 192.168.10.10 = Not OK
-ping to 192.168.11.10 = Not OK

client_LAN-B:
-ping to 192.168.50.2 = OK
-ping to 192.168.1.1 = OK
-ping to 192.168.2.1 = OK
-ping to 192.168.1.10 = Not OK
-ping to 192.168.2.10 = Not OK

Is this the current situation?

-Is there firewalling-software running on the clients?
-Do the clients have pfSense set as their default gateway in their network settings?
-Try running a "packet capture" on one of your LAN interfaces (specify a lan_client as host-address). Try to figure out what is happening to the pings.

I'm guessing the pings arrive at the destination, but the reply gets lost somewhere.

georgeman

@heper:

-Is there firewalling-software running on the clients?

I bet this is the issue. By default, the Windows firewall allows ICMP only from within its own subnet