Routing between two LANs

ebdjimenez

We have two buildings on the same lot, each with a unique subnet and internet circuit. The routers on both sides are pfsense 2.2.4. Each side has a domain controller on the same domain.

In order to replace the existing IpsecVPN (which is very slow), we recently had a direct Ethernet line installed between the two buildings and would like to be able to route traffic between the two sites while maintaining independent internet service at each site. Here is a crude diagram:

Site A Router Bridge Router Site B Router
LAN LAN OPT1 LAN
192.168.1.253 –------------------------------- 192.168.1.3 / 192.168.2.3 --------------------------- 192.168.2.253
| |
| |
| |
Internet Internet

I figured this could be accomplished using a 3rd pfsense box with a dual nic (I call this the bridge router). I setup this bridge router with LAN and OPT1 interfaces for each subnet. I then created static routes at each site that points traffic destined for the other site to the bridge router. Also, created firewall rules that allow any traffic from the LAN and OPT1 interfaces

I was able to log into all three pfsense boxes from either side and I was able to ping from each side to the other including printers and servers. However, nothing else appeared to work. I could not use RDP to get to the servers at the other building nor could I access the file shares via IP addresses.

I'm a networking novice and new to pfsense so any help or advice would be appreciated. Would this even be the best way to accomplish routing between the two subnets while maintaining their own internet service?

Derelict

You don't need a third router. You need a third interface in each router and a transit network.

Your problem is that the hosts on each network don't know where to send traffic to the other site. They send it to the default gateway instead.

![Transit Network.png_thumb](/public/imported_attachments/1/Transit Network.png_thumb)
![Transit Network.png](/public/imported_attachments/1/Transit Network.png)

johnpoz

^ exactly how you would do it.

These sorts of statements always get my curiosity up
"I'm a networking novice"

Why are you working on such a project then.. Are you the IT guy by default because there is nobody else?? Your the help desk guy and boss threw you into the deep end to see if you could swim to see if can move you to the networking team?

No offense but this is basic 101 networking.

I'm a networking novice and new to pfsense so any help or advice would be appreciated.

Building a connection between two buildings in a wise or common way would be of the need of fibreglass
or fiber optics cable because of the electric potential equalization of the two buildings!!! No insurance will
pay after both buildings were burned down! And using fiber optics is there fore the most common way to
surround this!

Would this even be the best way to accomplish routing between the two subnets while maintaining
their own internet service?

It is one of them but not the best one as I see it right. If there is no need for a greater throughput
and redundancy I would more tend to use a smaller MikroTik router instead of a third pfSense firewall.
But there are other ways to get it right done, likes;

Using smaller MikroTik router or switch with one or more SFP port(s)
Using switches with one or more SFP Ports
Using one or more SFP ports directly at the pfSense appliance
pending on the Switches it will be also able to set up a geo stack or switch stack using
two switches that are supporting stacking over the SFP Ports
Using L3/L2 Switches on both sides with more then one SFP Port and building a LAG (LACP) with redundancy
Using one or more SFP+ Ports to get more throughput and/or redundancy between both sides

You see there are many more options that will be point this project more to the safe side.
If you are not willing or not able to use fiber optics or cables I really want not to realize this
project upon the potential dangerous for both buildings.

The Best and also common way would be using two switches, one on each side and building a
LAG (LACP) or geo stack over the two switches. this is safe and redundant then.

Derelict

Or, these days, wireless can PTP between two buildings for not a lot of money. A couple small dishes from Ubiquiti can probably be had for about $200. At that price buy a couple spares.

@Derelict:

Or, these days, wireless can PTP between two buildings for not a lot of money. A couple small dishes from Ubiquiti can probably be had for about $200. At that price buy a couple spares.

Yep this would be also a really easy way, UBNT is offering some PtP sets (master-client), starting at
~70 € till ~250 € UBNT Power Beam

ebdjimenez

Thanks for all the responses!

@Derelict:

You don't need a third router. You need a third interface in each router and a transit network.

Your problem is that the hosts on each network don't know where to send traffic to the other site. They send it to the default gateway instead.

I agree. The problem here is that the routers in place do not have 3 interfaces or additional card slots and the owners are very "frugal" to say the least. We figured it would be more expensive to replace them then it would be to add a cheap PFS box. I assumed a static route (i.e. 192.168.2.0/24 to 192.168.1.3) on each GW would take care of the problem with hosts sending traffic to their default GW. And traffic does get routed between the two buildings as I can ping. However, I took a look at the firewall logs and it appears that traffic between the two subnets is being blocked by a "Default deny rule IPv4". I've gone over the firewalls rules (there aren't many) and do not see anything that would cause such blocked traffic. Attached is a picture of the Firewall logs.

@johnpoz:

These sorts of statements always get my curiosity up
"I'm a networking novice"

Why are you working on such a project then.. Are you the IT guy by default because there is nobody else?? Your the help desk guy and boss threw you into the deep end to see if you could swim to see if can move you to the networking team?

I'm an entry level tech for small IT firm and I'm working on this project for a small and very cheap company that doesn't have an IT department. The existing infrastructure was already in place before we acquired them as a client, including the site-to-site Ethernet. It just hadn't been utilized or even punched down.

@BlueKobold:

Building a connection between two buildings in a wise or common way would be of the need of fibreglass
or fiber optics cable because of the electric potential equalization of the two buildings!!! No insurance will
pay after both buildings were burned down! And using fiber optics is there fore the most common way to
surround this!

Wow that is frightening. I have no knowledge of electric potential equalization and I did not consider it at all. I didn't think low voltage cabling could be such a risk. Would this come into place during an electrical storm? Like if one of the buildings was struck by lightning? If that's the case then I will try to convince ownership to transition to Ubiquity dishes, as I have some experience with those.

Thanks again for all the advice.

Capture.JPG_thumb

Derelict

@ebdjimenez:

I agree. The problem here is that the routers in place do not have 3 interfaces or additional card slots and the owners are very "frugal" to say the least. We figured it would be more expensive to replace them then it would be to add a cheap PFS box.

Add two cheap pfSense nodes with three interfaces and ditch the gear that doesn't have the interfaces to do what you need to do. Or investigate a couple cheap managed switches and use VLANs for LAN and TRANSIT.

I assumed a static route (i.e. 192.168.2.0/24 to 192.168.1.3) on each GW would take care of the problem with hosts sending traffic to their default GW. And traffic does get routed between the two buildings as I can ping. However, I took a look at the firewall logs and it appears that traffic between the two subnets is being blocked by a "Default deny rule IPv4". I've gone over the firewalls rules (there aren't many) and do not see anything that would cause such blocked traffic. Attached is a picture of the Firewall logs.

Unsound design. Route your traffic properly, not hairpinning in then back out of the same interface.

If you simply MUST do it that way:

System > Advanced > Firewall/NAT tab

Enable this: Static route filtering - Bypass firewall rules for traffic on the same interface

ETA - That will have to be done using whatever mechanisms the two existing routers have. It is unclear whether those are pfSense.

If that's the case then I will try to convince ownership to transition to Ubiquity dishes,
as I have some experience with those.

This would be the right way as I see it right, there fore nothing can be going false.
UBNT is offering Point-to-Point bridges easy to configure with 150 MBit/s, 300 MBit/s and 450 MBit/s throughput.

johnpoz

I don't know what country these people are in talking about equal electric potential in the building from running a ethernet cable between them.. I think they are high on something to be honest, have never in my life heard of such a thing.. Please point to sources where this is an issue with fire harzard or insurance.. Clearly this should be plainly document everywhere if that was the case.. You know how many buildings have network connections between them!!

How exactly did you connect these 2 buildings. I would assume if part of the same company and area they share electrical connections and water connections, etc. etc.. Or is the cable laying on the ground? Who installed this cable? I would assume you get a certified installer, etc. Or did you run it?

If you can not get a 3rd nic in each pfsense - so there are no slots? nics can be had for $10 for gosh sake.. Dual port nics can be had for like < $50 to replace the nic in there now so you have an extra port. Then run vlan off your current pfsense lan interfaces and use that as your transit network.

There is no freaking way your putting in a unifi point to point bridge for cheaper than adding a couple of nics ports. Who is going to install them for you correctly - that sure is not FREE ;) Unless your going to do it yourself and not charge the customer.. Them being so cheap and all. Even if you went for the ptp 450mbps setup – you could buy 2 new screaming direct from pfsense routers with enough interfaces so that you could do it for less than the cost of that setup and now you would have gig between the building..

You already have the wire run right - how long is this run? Since you say ethernet and not fiber it has to be short less than 100m

I don't know what country these people are in talking about equal electric potential in the building from running a ethernet cable between them..

This is no special network knowledge, but more common electric knowledge over this electric potential equalization or shielding and if both sites get connected to the entire electric shielding of the buildings
(16 mm min.) this theme could be really easy sorted or right handled. I am from Germany and if those
work is not done by peoples who learned this job ended with a given certification, insurances in Germany
would not pay for problems generated by or based on this cabling and I am pretty sure that in many
other European countries the same situation would be exactly like that!

I think they are high on something to be honest, have never in my life heard of such a thing.

But this can not be our problem that you never heard from something!

Please point to sources where this is an issue with fire harzard or insurance..

Easily call any German insurance and ask them, for this special case; that a non learnt electrician
is doing a cabling between two buildings, if they would pay for any problems resulting on this cabling.

Clearly this should be plainly document everywhere if that was the case.. You know how many buildings have network connections between them!!

I really don´t know your country but if I am using Google.de for sure I am a naturally german speaking
person and I get something around ~19.200 hits over this theme it might be that you get also some hits
over your Google.xyz and in your language! One of the best links I found and also I am often linking to
is this one 2 buildings with LAN connection

Search words in german were: "Potentialausgleich zwischen Gebäuden Netzwerk" this is in english like
"electric potential equalization between buildings network"

By the way with the search words "electric potential equalization between to buildings" I got
3.950.000 so nearly ~4 million hits on this theme over Google.co.uk and with the search words
"electric potential equalization network between to buildings" are nearly ~600.000 hits will be
shown on this theme.

So documents are out there and if an electric weather strike hits a coper cable that is lightening
this power in one or more buildings might be not coming true every day and yes this building must
not burn really down, but that there is a potential danger that this is able to come true should not be
discussed in a firewall forum as I see it right.

There is no freaking way your putting in a unifi point to point bridge for cheaper than adding a couple of nics ports.

coper cable
fiber cable
WLAN
VPN

So if the coper is not earthed or grounded proper, fiber cable is to high in price, vpn is to lame or slow
the or one accurate choice will be WLAN for sure why not?

jahonix

With different ground potentials between buildings you will create hum loops, leading to slow or unreliable data transmissions. This is physically given, not an option.
And that's when isolation by fibre or air-waves is to be used.

My home has an old and a new building, both with their own power distro. (Don't ask, was done before I married in).
Uplinks between switches in different parts of the building are isolated by fibre because of this very reason.

ebdjimenez

@Derelict:

@ebdjimenez:

I agree. The problem here is that the routers in place do not have 3 interfaces or additional card slots and the owners are very "frugal" to say the least. We figured it would be more expensive to replace them then it would be to add a cheap PFS box.

Add two cheap pfSense nodes with three interfaces and ditch the gear that doesn't have the interfaces to do what you need to do. Or investigate a couple cheap managed switches and use VLANs for LAN and TRANSIT.

I assumed a static route (i.e. 192.168.2.0/24 to 192.168.1.3) on each GW would take care of the problem with hosts sending traffic to their default GW. And traffic does get routed between the two buildings as I can ping. However, I took a look at the firewall logs and it appears that traffic between the two subnets is being blocked by a "Default deny rule IPv4". I've gone over the firewalls rules (there aren't many) and do not see anything that would cause such blocked traffic. Attached is a picture of the Firewall logs.

Unsound design. Route your traffic properly, not hairpinning in then back out of the same interface.

If you simply MUST do it that way:

System > Advanced > Firewall/NAT tab

Enable this: Static route filtering - Bypass firewall rules for traffic on the same interface

ETA - That will have to be done using whatever mechanisms the two existing routers have. It is unclear whether those are pfSense.

Thanks for all the replies and guidance. This solved my problem temporarily. I've learned quite a bit while researching the comments in this thread.

We are going to replace the two routers with ones that have 3 interfaces, we would only need 1 more at this point. I've spoken with ownership and they are not concerned about the possible electrical issues, as the lines were installed underground by Verizon. (Can't say I didn't warn them)

Thanks again.

jahonix

@ebdjimenez:

I've spoken with ownership and they are not concerned about the possible electrical issues, as the lines were installed underground by Verizon.

That's funny because owners usually are not technically savvy to a point where they may decide this - because it's beyond their knowledge.

Anyway, if a company like Verizon installed the cables then chances are that they grounded both ends properly. You should examine this.
Code says you must use one common ground per building and only one.
Good luck with nearby lightning, maybe you should have a spare NIC handy…

chpalmer

Please point to sources where this is an issue with fire harzard or insurance..

A lightning strike nearby might change your position on this.

Look up Motorola R56 standards. Its stated there for my line of work.

Derelict

Motorola. What do they know.

Great document. Thanks. Filed so it'll come up in spotlight searches.

jahonix

Thanks, chpalmer, great document to read and keep as reference.
I am born, raised and based in Germany so not everything applies here. But the basics are always right.