my openvpn site to site i cant seem to ping or access other site doesnt stay stable

comet424

so i have my site to site pfsense to pfsense openvpn connected and i had followed some videos but i not sure if my PIA vpn is conflicting or not

so i can make a connection and the connection stays from site A to Site B

but when you ping a computer on the other network or the other pfsnese box it will work and if u stop and try again it cant ping it.. you stop and start say 10 times and it will ping 4 5 times out of 10 tries.. i find accessing server gui webpages dont always like to load..

si this NAT issue, compression issue under openvpn tunnel.. could be a gate way issue?
and what should a Nat for site to site look like

and i also was googling and seen about wireguard.. is wireguard better for site to site? as my ip is dynamic changes from the DSL modem reboots etc. and i read wireguard needs static ip..

but ya thats what i trying to figure out why my ip pinging and accessing messed.. its like when you have 2 computers on the network with the same IP address and you know it will work sometimes then not cuz there is 2 comps with the same ip.. and it doesnt know which way to go.. so its acting similar to that

i saw in another video they did a NAT setting as other videos ddont do NAT but i did this kinda works better... but the section translation i set to WAN Address in the video it shows network interface

and should it be WAN address? or network & alias?

and the network ip 172.168.0.0 thats the tunnel address

stephenw10

You shouldn't need NAT here, assuming the two networks on each side are using different subnets.

Seeing alternate connections working seems more like you could be policy routing via a load-balanced gateway.

What firewall rules do you have passing traffic on the local LAN interface?

comet424

@stephenw10
and can i delete any of the NATs probably dont do anything? do you see any that are not needed etc? like this 1::128 or whatever they are what is the 127 one do i need all those what does the 128 do to specific network?
trying to make it run as smooth as possible
pfsense 1.png pfsense 2.png

i found if i didnt haven the openvpn it didnt wanna communicate across the networks and i found some articles have nat setting some dont.. so thats why i confused and whats the proper way
i was having openvpn nat for 192.168.1.0 instead of the 172.168.0.0 and that also caused glitching

so my network is 192.168.0.0 and 192.168.1.0 for both pfsense boxes

as for my firewall rules these are on the 1 server.. the other server is similar but ill have to get pics later if you need them

comet424

the site A main pfsense the LAN firewalls page is

stephenw10

You can certainly remove that OpenVPN oubound NAT rule. It's NATing to the WAN address on the OpenVPN interface which is always wrong!

The two localhost addresses (127/8 and 1/128) are not always needed. pfSense itself will usually use the WAN address directly.

It looks like you have switched OBN to manual mode? Better to use hybrid for most situations.

How exactly are you testing here? From what IP to what remote IP?

It could be the remote device blocking traffic itself. Some OSes (windows) will block traffic from a different subnet by default.

comet424

@stephenw10
ok i removed the openvpn interface to wan address NAT so what was that basiclly doing you mentioned it doesnt work.. and how come there is no openvpn interface in the NAT... so i removed it and seems to be working alot better.. reason i added it was like video from this guy
https://www.youtube.com/watch?v=SVUE6tcznM4
at 11min mark he does the NAT for openvpn is it something that was needed in the past? as it is a 4 year old video.. i have seen this in a couple videos

and reason i use Manual NAT is due to when i had NordVPN and when i switched to PIA VPN they both require manual mode....

what are the benefits of hybrid vs the Manual?

and what does that 127/8 and the 1::128 do like what happenes on the network? so should i just remove the wan too.. or leave it..

as for testing i was pinging from my Unraid Box on the network. a VM ubuntu desktop. VM windows Desktop and my main desktop and i was pinging like 192.168.1.1 and any ip on the 1.x network

and to use wireguard site to site you need static ips right ? cant use dynamic from the dsl modem i get

stephenw10

So you're pinging directly by IP address (not hostname) between hosts on either end of the tunnel? And those subnets are 192.168.1.0/24 and 192.168.0.0/24?

And that's working now?

I prefer to use hybrid mode for OBN because it keeps the automatic rules. If you add or change some internal subnet the rules will be updated to allow connectivity.

If you want to intentional prevent traffic going out of he WAN dircetly you can just add a 'do not NAT' rule.

comet424

@stephenw10 sorry delay ive had power out big storm saturday in canada and i just got my power back

what is OBN? and i can switch i found an issue that is not working for me. and will this fix it with the hybrid?

so on my unraid box i have a 10Gig nick and broken into some bridges

br0 = LAN 192.168.0.0
br0.10 = Vlan Cameras 192.168.10.0
br0.20 = Vlan IOT 192.168.20.0

now i cant seem to ping the 192.168.1.0 network
i noticed the issue as i cant access the remote Home Assistant from my main home assistant

on my home assistant i have
enp1s0 192.168.0.12 gateway 192.168.0.1 dns 192.168.0.1
enp2s0 192.168.20.12 gateway 192.168.20.1 dns 192.168.20.1
enp3s0 192.168.10.12 gateway 192.168.10.1 dns 192.168.10.1

now i try to ping 192.168.1.12 remote HA or 192.168.1.1 other pfsense
and it cant
even though in the rules i say IOT can have access to 192.168.1.12 same for camera

i tried
ping -I enp1s0 192.168.1.1
ping -I enp1s0 192.168.1.12
those work

but if i try
ping -I enp2s0 192.168.1.1
ping -I enp2s0 192.168.1.1
ping -I enp3s0 192.168.1.1
ping -I enp3s0 192.168.1.1
it doesnt work

when i try the pinging in unraid terminal with the
ping -I br0.10 192.168.1.1 or 192.168.10.1 it doesnt wanna work is there a rule or is that NAT outboard that is blocking it?

reason i noticed Home assistant uses 1 ethernet bridge as the main one and i noticed sometimes it changes but id trying to ping from any of those bridges to go out the OPENVPN connection or the network
is something conflicting ? i know when i just have 1 LAN network not issues but when i switched to having vlans to break up my network some i come across it..

maybe complicated? maybe i just doing something wrong.. and if you need any pics let me know as i wanna be able to ping the 192.168.1.12 from the termainal and it should go out any of the vlan lan ports or be able to pick it off

here is the IOT and Camera Rules as the LAN one pings ... maybe its outside of the scope or maybe i need a better video to watch to setup?
and sorry if i confusing my dislyexia gets best of me sounds fine to me but i may not explain it right

stephenw10

@comet424 said in my openvpn site to site i cant seem to ping or access other site doesnt stay stable:

what is OBN?

OutBound Nat.

@comet424 said in my openvpn site to site i cant seem to ping or access other site doesnt stay stable:

when i try the pinging in unraid terminal with the
ping -I br0.10 192.168.1.1 or 192.168.10.1 it doesnt wanna work

192.168.10.1 is in he br0.10 subnet so that would go directly if it's actually using the correct source IP. In which case pfSense never sees it and it must be something in the virtual infrastructure or the taret host blocking it.

When you have multi-homed hosts like that it's common to see asymmetric routing issues. Check the firewall logs for blocked traffic.

comet424

@stephenw10
so i did screen capture the camera logs.. where you see its pinging from 192.168.10.12 which is the home assistant ip for Cameras.. and trying to reach 192.168.1.12

firewall 1.png

so
192.168.0.1 LAN pfsense ip
192.168.10.1 pfsense Camera VLAN
192.168.20.1 pfsense IOT VLAN

unraid
192.168.0.3 BR0 LAN network
192.168.10.3 BR0.10 Camera VLAN
192.168.20.3 BR0.20 IOT Vlan

home assistant
192.168.0.12 LAN network
192.168.10.12 Camera VLAN
192.168.20.12 IOT Vlan

now i found it would work if it static ip them
but set it like this
192.168.0.12 Gateway 192.168.0.1 DNS 192.168.0.1
192.168.10.12 Gateway 192.168.0.1 DNS 192.168.0.1
192.168.20.12 Gateway 192.168.0.1 DNS 192.168.0.1

that made it work but it doesnt work if you specify it in the DHCP Server for doing the dhcp part it doesnt like you setting the gateway 192.168.0.1 for the camera or IOT says its out of the range

but it made HA at least work..
so how do you fix it to work properly

and the

192.168.0.1 LAN pfsense ip
192.168.10.1 pfsense Camera VLAN
192.168.20.1 pfsense IOT VLAN

they are all from the single pfsense box so its not seperate machines just the one

comet424

oh and i also have a
192.168.30.x vlan i made a management port from ym onboard network cards

i found i had to disable the gateway on the desktop and servers.. i found that it wasnt using the 10gig network card 192.168.0.1 but was going through the 192.168.30.x and id have no internet
even though i blocked it in the rules and left it 192.168.30.1 as a gateway it always tried to go through it and not the 10gig so i ended up removing the gateway in windows adapter for the onboard.. that was frustrating to figure out

stephenw10

@comet424 said in my openvpn site to site i cant seem to ping or access other site doesnt stay stable:

now i found it would work if it static ip them
but set it like this
192.168.0.12 Gateway 192.168.0.1 DNS 192.168.0.1
192.168.10.12 Gateway 192.168.0.1 DNS 192.168.0.1
192.168.20.12 Gateway 192.168.0.1 DNS 192.168.0.1

Mmm, see that's the problem with multi-homed hosts. When you do that it sends all traffic via the LAN gateway on the LAN subnet so bypasses the other firewall rules entirely.

So I assume the 192.168.1.0/24 subnet is at the other end of the tunnel?

There are no hits in the firewall logs for 192.168.1.12. So if you're trying to ping that it's probably passing. You have specific rules to allow that and one of them has bytes recorded on it.

Check the states for that IP when you're pinging.

comet424

@stephenw10
so is there an easy way to do mult-home? so like i dont seem to have issues when its 1 nic to a device

just when i have 3 network lans going into the home assistant or the unraid
as i have to ping -I <network interfrace>

how come mult home is hard to do? that its all going to the same pfsense box etc?

and ya on the openvpn tunnel the other side is the 192.168.1.0/24 and the remote home assistant is 192.168.1.12
and with HA when you reboot sometimes it changes which is the main network card and it has option to only use 1 for multicast.. but its like it rotates it sometimes 192.168.0.0 or 192.168.20.0 or 192.168.10.0 and it can never find ips on the network of the other 2 lans unless i choose the 3
here is it screen shot of HA the 3 networks.. the default will rotate from time to time from reboots.. so i want to be able to whichever its on when you just type ping 192.168.1.12 to test instead you gotta type ping -I enp1s0 192.168.1.12 as example cuz connecting doesnt work unless it can connect...

i guess the Hybrid NAT cant fix this issue?
so i rebooted the pfsense and re reset the HA nics to
192.168.0.12 Gateway 192.168.0.1 DNS 192.168.0.1
192.168.10.12 Gateway 192.168.10.1 DNS 192.168.10.1
192.168.20.12 Gateway 192.168.20.1 DNS 192.168.20.1

and back to cant ping or connect to the tunnel..

as for checking the states of the ip of 192.168.1.12 not sure i did click on the the bytes and came up the states page.. i did type in 192.168.1.12
but its showing nothing even when i doing pinging i probably doing that wrong

i guess its not a simple solution?

i did google when you mentioned "multi-home" to try to learn if others done it but i finding 0 hits really 1.. but the person solved it and didnt show how he did it and others is about multiwan but ill also keep trying to look up more on multi-home

i guess the multi home issue i have for the desktop and my servers too right
where i have
10gig nic 192.168.0.0
onboard nic i use as Management port 192.168.30.0
and only way i got it to work was block WAN address and subnets for the 192.168.30.0 and i had to remove gateway ip address from the server and desktop remove the 192.168.30.1 cuz thats where my 10g copying was going through the 1gb management ... guess its a learning curve but once you know how to do it its simple?

stephenw10

Don't filter by ruleID just look for all states with that IP address. You must be actively pinging at the time because those states expire quickly.

The only safe way to use multi-home is don't use multi-home!

The problem is that other hosts from different VLANs may try to connect to it on an IP that must be routed through pfSense. But the HA host has direct access to all 3 subnets so it doesn't need to send replies back to pfSense to be routed, it just sends them dircetly. But doing so create an asymmetric route where the firewall only sees half of the conversation and because of that it blocks the unexpected traffic.

However, whilst that's bad, it should not have any affect on the ability to ping across the VPN so I would look at that for now.

comet424

@stephenw10
ok so i tried the pinging and i got these screen shots

so when you say HA creates asymmetric route
like for cameras or IOT devices it doesnt do

HA --> PFsense --> IOT Device 192.168.20.x

instead what its doing is

HA --> IOT Device

skipping PF Sense directly

but if i pinging 192.168.1.12 its neither on any of those 3 subnet interfaces so shouldnt it try to directly go out the pfsense?

and each time i do a reboot of HA a different interface becomes i guess the MASTER so on a reboot if 192.168.0.12 gets the default (master) it can ping 192.168.1.1 or 192.168.1.12
but when a reboot and if Cameras or IOT Interface gets the master i cant ping them..

so i dunno if this is doing anything i tried to make a NAT for cameras and IOT to use the openvpn connection now its probably totally wrong but i trying...

so if the idea is not to use multi home why do it? and isnt multi home same as like VPN_FAil over? so i have vpn fail over group i have 2 PIAs set to Teir 1 or its 1 and then 2 i forget.. and if one fails it goes out the other direction... but in esstance i doing the reverse using the 2 vpn fail overs and going out the 1.. using vpn fail over as like a camera and iot vlan just as example of trying to understand things

oh and if multihome is bad right how do you then use 1 interface.. in HA but push the 3 networks LAN, IOT,CAMERAS from pfsense split there forced into 1 interface and HA splits it back into 3 sections basiclly like a Hour glass
3 interfaces on PFSEnese goes to 1 interface on HA and HA re splits that 1 interface into 3 seperate networks

comet424

so with the HA defaulting to Camera network i ran the pinging and these were the results
and the NAT settings i have for Openvpn for iot and cameras did nothing so that didnt help my issue it was a try.. or is it working cuz its established but i cant ping

comet424

so it seems ya that i needed the NAT i tested for Cameras network. it is the Master right now..

here is Before that NAT Camera

and after i enabled the NAT the openvpn to Camera source and Camera Address
then it started working. well not the pinging but the connection to the other Home Assistant

and what does the NAT Address do.. and do you always keep it the same as the Source
so Camera subnets Camera Address
IOT Subnets IOT Address?

and i having troubles posting this thing saying i got spam in it wish they would say what is spam by the akismet.com

comet424

and for the IOT when its the master

Before
before 2.png

and after i enable the NAT openvpn iot subnet iot address
after 2.png

what i find weird is i dont have camera nat enable yet its established... but when its master i have to enable the NAT as it wont access untill i enable that NAT route.. its weird.. but that seems to work.. and you will say that shouldnt work like that im sure

stephenw10

@comet424 said in my openvpn site to site i cant seem to ping or access other site doesnt stay stable:

so when you say HA creates asymmetric route
like for cameras or IOT devices it doesnt do

HA --> PFsense --> IOT Device 192.168.20.x

instead what its doing is

HA --> IOT Device

skipping PF Sense directly

but if i pinging 192.168.1.12 its neither on any of those 3 subnet interfaces so shouldnt it try to directly go out the pfsense?

Yes exactly. You cannot have asymmetry to the remote subnet as the traffic must pass the firewall both ways.

But, yes, it could still try to send from any of those interfaces. However there should be a way to set one of those as priority in some way so it always uses 192.168.0.12 as the source for routed connections.

That last view showing it using the cameras subnet as source is still filtered by rule 137 so it's only showing states opened by that. You need to remove that so you see the outbound state it's using If it;s correctly opening a state on the openvpn interface then it's almost certainly blocked at the other end.

comet424

@stephenw10
ah ok sorry about that it always defaults to 137 whatever that means... here is a new screen shot..
after 3.png

ill have to see if i can help from the Unraid group... there is boot order for the VM i guess so you can load the nics in order. as right now it just randomly picks whatever first.. it gets frustrating i tell ya..

i know on my host openvpn of the site to site 192.168.1.1 i added the openvpn connection for remote local i did
192.168.0.0/24,192.168.10.0/24,192.168.20.0/24

but here is a pic from the 192.168.1.1 pfsense the states

i guess sometimes its almost a guessing game? like the openvpn NAT i dont need one for LAN connection but i needed to add one for CAMERAS and IOT to get it to connect..