Two firewall accessing each other when gateway is down
-
@Farh well, maybe I'm missing something :).
Could you do simple drawing of the network layout?
If the gateway is up, how can you access the other servers WAN ip? Ping or SSH or can you even route to the other LAN?
-
@patient0
No. You don't miss anything.
I get your point
Yes. I created port forward rules to allow http and https on port 80 and 443.
Both of them work fine world wide and locally when gateway is up.
Sorry I don't have access to pc right now to draw diagram but it's very simple:
LAN1 <----> pfsense <----> WAN1<---> unmanaged switch <----> ISP Radio
LAN2 <----> opnsense <----> WAN2<---> unmanaged switch <----> ISP Radio
Two WANs are connected to one Unmanaged switch that mentioned on both path and i believe ISP radio must be the gateway. It's a point to point connection.
I access via both http and https.
Until today i expected to access same wan subnet on layer 2 when gateway is down but today when my internet goes down i lose the connection.
Let's say i believed that when two IP address is on same subnet must at least try to access each other with layer 2 base on each other MAC address without using layer 3 routing tables.
Is it wrong ? Or i need a configuration ? -
@Farh it's simple enough you are right.
And again you are right in that since you are on the same link/layer 2 you shouldn't need a gateway to access the other *sense.
When you set the WAN interfaces with a /28 subnet you should see the other *sense in the ARP table (Diagnostics > ARP table). Is the other *sense listed there?
-
@patient0 No. It's not in ARP table.
-
@Farh said in Two firewall accessing each other when gateway is down:
LAN1 <----> pfsense <----> WAN1<---> unmanaged switch <----> ISP Radio
LAN2 <----> opnsense <----> WAN2<---> unmanaged switch <----> ISP Radio
Until today i expected to access same wan subnet on layer 2 when gateway is down but today when my internet goes down i lose the connection.Which connection are you loosing?
WAN1 <> WAN2?
LAN1 <> LAN2?
LAN1 <> WAN2 or vv? -
@viragomann
LAN1 to service that is in LAN2 with port forward and NAT but i think no connection between WAN1and WAN2 too. -
@Farh said in Two firewall accessing each other when gateway is down:
i think no connection between WAN1and WAN2 too.
That's what we were talking about. I expect, that WAN <> WAN should work at least if you initiate the traffic on the firewalls themself.
-
@viragomann No unfortunately
I even connect a linux pc directly to firewall 2 WAN (opnsense). Set the public ip of forewall 1 on linux pc with no luck to telnet port 80 or 443. -
@Farh
Both firewalls give you diagnostic tools like ping or port probe. There is no need to attach an additional device the check this out.How are your WANs configured? Do they have static IP or DHCP?
-
@viragomann yes of course you're right but i can not take internet down on that time and i have no choice other than use another device.
It's static IP address with one wan ip and 4 virtual IP on pfsense and one static IP on opnsense
-
@Farh said in Two firewall accessing each other when gateway is down:
Until today i expected to access same wan subnet on layer 2 when gateway is down but today when my internet goes down i lose the connection.
No. It's not in ARP table.
No ARP, no layer 2 communication.
Seems you have an L2 issue.
Is there even an entry of the respective other WAN in the ARP table if the gateway is up?If not, to investigate the issue, go on one of the firewalls and start a packets capture of ARP. Then try to ping the other box.
Check after, what you got in the capture. -
@viragomann thanks for your guidance
pinging other side added it to ARP table.
is ICMP most effective rather than TCP for adding to ARP table or it's even required ??
But unfortunately even after both firewalls have each other MAC address in ARP table issue still persist. No gateway no connection.
Add other WANs to ARP table still need to pinging it. -
@Farh
No, the protocol doesn't matter. If an IP within the subnet is requested, it does an ARP resolution.
So my assumption is, that the other WAN isn't requested at all, when you try to access it from inside the LAN.Do you policy route the LAN traffic by any chance?
-
@viragomann said in Two firewall accessing each other when gateway is down:
So my assumption is, that the other WAN isn't requested at all, when you try to access it from inside the LAN.
It's impossible because my DNS return my WAN2 IP addresses and there is no other path to access.
Also traceroute result on LAN1 shows the LAN IP of pfsense but then it shows timeout.@viragomann said in Two firewall accessing each other when gateway is down:
Do you policy route the LAN traffic by any chance?
No. I don't think so. Which kind of police do you mean. It's outbound NAT
-
@Farh
Policy routing doesn't care about DNS.
And it has nothing to do with NAT.In you WAN rules for allowing upstream traffic, did you state a gateway?
If you're unsure, please show your rules. -
@viragomann if I'm getting correct you mean choosing a gateway on firewall > rules.
In my case every rule gateway on both LAN and WAN is set to * -
First of all i want to thanks everyone reply to this topic and help me find the problem
I do the following to resolve my problem. Maybe somebody else face this problem and may it helps:
1- first i check the ARP table on both firewall and discover none of them have each other MAC addresses. It's so strange for me because i connected to them with https and http but pinging other firewall inside each other even without response that because of firewall rule added it to ARP table.
2- step one doesn't resolve the problem but i believe it was required. Secondly i try many thing s that doesn't work. After several hours of try and error i discovered that base on MAC addresses pfsense send packages directly to opnsense but opnsense reply through defaut gateway to pfsense. After searching on internet i found this link:
https://forum.opnsense.org/index.php?topic=5615.0
Some of opnsense guys says enabling "disable reply-to" option in opensense may resolve the problem.
3- surprisingly enabling " disable reply-to " resolve the problem.
I don't disconnecting the internet yet but i believe problem is resolved.
I'm glad if anyone can explain why this option worked ? Because I'm confused a little bit.
Thanks -
@Farh
Disabling reply-to on the accessed node - yeah, this could be a reason.
When enabled, replies are directed to the gateway, which is stated in the interface settings.Disabling reply-to could lead into issues with multi-WAN setup, however.
To avoid this, you can add pass rules to the top of the WAN rule set only for the source of the WAN subnet and disable reply-to in the advanced options.
