Survey: Who has successfully set up peer-to-peer network?
-
I am wondering, who in this forum has successfully set up an OpenVPN peer-to-peer network between to boxes running pfSense.
-
I have used it several times in the past, but not got it configured anymore.
Plenty of guides on the interweb, as well as in Netgate documents.
-
@pwood999: The reason I am asking is that I am banging my head against the wall with my setup not working. It’s gotten to the point, where I am wondering, whether my problems are due to bugs in the OpenVPN implementation in pfSense being buggy.
-
Well, it might help if you break down the problem a bit. Where is it failing? Is it failing to connect? Failing to get elsewhere on the pfSense box? On your local network?
-
Post your server & client config (obcure the certs when posting), plus firewall rules.
-
I have connections up and running, like so:
Pings to both the clients’ virtual and remote real address fail with 100% packet loss.
What led me to blame it on bugs is that it briefly worked, when I fixed the client-specific override with the help of @viragomann and his suggestion to include the remote networks there. Subsequently, however, with no other change (no apparent one, at least), my ability to access the remote network went away.
Cf. Still no reliable peer-to-peer connection, but progress madeThere are log entries with the word “disconnect” in it. However, not having a clear understanding of what is logged and how it is expressed in the log it was not clear to me, whether it was a problem, or not.
Cf. Still issues with peer-to-peer network.This is where I am at now.
-
If you're trying to ping the server WAN that will fail unless you specifically allow it.
The Status page in that other thread shows two clients with 192.168.7.2 as their virtual IP. surely that will cause ARP & Route issues ?
Do site-b & site-c need to talk to each other ? If so you will need routes via site-a.
-
@DominikHoffmann Unless you've got a special use case, just stick with tried and true IPSEC VPN tunnels for peer-to-peer networking as are way easier to manage. All the big boys and cloud providers till use them today.
-
@pwood999 said in Survey: Who has successfully set up peer-to-peer network?:
The Status page in that other thread shows two clients with 192.168.7.2 as their virtual IP. surely that will cause ARP & Route issues ?
That was an issue with the client-specific overrides for those two clients, which has since been fixed.
-
@pwood999—you mean something like the following? What are you going to check for?
Server config:
<vpnid>2</vpnid> <dco>disabled</dco> <mode>p2p_tls</mode> <protocol>UDP4</protocol> <dev_mode>tun</dev_mode> <interface>wan</interface> <ipaddr></ipaddr> <local_port>1194</local_port> <description><![CDATA[Client site-to-site VPN server]]></description> <custom_options></custom_options> <tls>************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************</tls> <tls_type>auth</tls_type> <tlsauth_keydir>default</tlsauth_keydir> <caref>*************</caref> <crlref></crlref> <ocspurl></ocspurl> <certref>*************</certref> <dh_length>2048</dh_length> <ecdh_curve>none</ecdh_curve> <cert_depth>1</cert_depth> <digest>SHA256</digest> <tunnel_network>192.168.7.0/24</tunnel_network> <tunnel_networkv6></tunnel_networkv6> <remote_network>192.168.34.0/24,192.168.42.0/24,192.168.45.0/24,192.168.48.0/24,192.168.51.0/24,192.168.54.0/24</remote_network> <remote_networkv6></remote_networkv6> <gwredir></gwredir> <gwredir6></gwredir6> <local_network>192.168.7.0/24</local_network> <local_networkv6></local_networkv6> <maxclients>15</maxclients> <connlimit></connlimit> <allow_compression>no</allow_compression> <compression>none</compression> <compression_push>yes</compression_push> <passtos></passtos> <client2client>yes</client2client> <dynamic_ip>yes</dynamic_ip> <topology>subnet</topology> <serverbridge_dhcp></serverbridge_dhcp> <serverbridge_interface>none</serverbridge_interface> <serverbridge_routegateway></serverbridge_routegateway> <serverbridge_dhcp_start></serverbridge_dhcp_start> <serverbridge_dhcp_end></serverbridge_dhcp_end> <dns_domain>****.****.com</dns_domain> <dns_server1>192.168.4.1</dns_server1> <dns_server2></dns_server2> <dns_server3></dns_server3> <dns_server4></dns_server4> <username_as_common_name><![CDATA[enabled]]></username_as_common_name> <exit_notify>none</exit_notify> <sndrcvbuf></sndrcvbuf> <netbios_enable></netbios_enable> <create_gw>v4only</create_gw> <verbosity_level>1</verbosity_level> <data_ciphers>AES-128-GCM,AES-256-CBC</data_ciphers> <data_ciphers_fallback>AES-256-CBC</data_ciphers_fallback> <ping_method>keepalive</ping_method> <keepalive_interval>10</keepalive_interval> <keepalive_timeout>60</keepalive_timeout> <ping_seconds>10</ping_seconds> <ping_push></ping_push> <ping_action>ping_restart</ping_action> <ping_action_seconds>60</ping_action_seconds> <ping_action_push></ping_action_push> <inactive_seconds>0</inactive_seconds> </openvpn-server>
Client-specific override:
<openvpn-csc> <server_list>2</server_list> <custom_options></custom_options> <common_name><![CDATA[********]]></common_name> <block></block> <description><![CDATA[******** site-to-site OpenVPN network]]></description> <tunnel_network>192.168.7.4/24</tunnel_network> <tunnel_networkv6></tunnel_networkv6> <local_network></local_network> <local_networkv6></local_networkv6> <remote_network>192.168.45.0/24,192.168.46.0/24,192.168.47.0/24</remote_network> <remote_networkv6></remote_networkv6> <gwredir></gwredir> <push_reset></push_reset> <remove_route></remove_route> <netbios_enable></netbios_enable> </openvpn-csc>
Client configuration:
<openvpn-client> <auth_user></auth_user> <auth_pass></auth_pass> <proxy_user></proxy_user> <proxy_passwd></proxy_passwd> <vpnid>1</vpnid> <dco>disabled</dco> <protocol>UDP4</protocol> <dev_mode>tun</dev_mode> <interface>wan</interface> <ipaddr></ipaddr> <local_port></local_port> <server_addr>****.****.com</server_addr> <server_port>1194</server_port> <proxy_addr></proxy_addr> <proxy_port></proxy_port> <proxy_authtype>none</proxy_authtype> <description><![CDATA[pfSense-UDP4-1194-benningtons-config_OpenVPN]]></description> <mode>p2p_tls</mode> <topology>subnet</topology> <custom_options>verify-x509-name "server" name</custom_options> <caref>*************</caref> <certref>*************</certref> <crlref></crlref> <tls>************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************</tls> <tls_type>auth</tls_type> <tlsauth_keydir>1</tlsauth_keydir> <digest>SHA256</digest> <tunnel_network></tunnel_network> <tunnel_networkv6></tunnel_networkv6> <remote_network></remote_network> <remote_networkv6></remote_networkv6> <use_shaper></use_shaper> <allow_compression>asym</allow_compression> <compression></compression> <auth-retry-none></auth-retry-none> <passtos></passtos> <udp_fast_io></udp_fast_io> <exit_notify>none</exit_notify> <sndrcvbuf></sndrcvbuf> <route_no_pull></route_no_pull> <route_no_exec></route_no_exec> <dns_add></dns_add> <verbosity_level>1</verbosity_level> <create_gw>both</create_gw> <data_ciphers>AES-128-GCM,AES-256-CBC</data_ciphers> <data_ciphers_fallback>AES-256-CBC</data_ciphers_fallback> <ping_method>keepalive</ping_method> <keepalive_interval>10</keepalive_interval> <keepalive_timeout>60</keepalive_timeout> <ping_seconds>10</ping_seconds> <ping_action>ping_restart</ping_action> <ping_action_seconds>60</ping_action_seconds> <inactive_seconds>0</inactive_seconds> </openvpn-client>
-
@DominikHoffmann yup, used OpenVPN and Wireguard, both work as expected on pfSense.
-
Right upfront : sorry, me neither, never connected two remote sites together using VPN.
But I have a proposition for you.
The very first on : set up an pfSense OpenVPN server. As half the planet was using this setup during 2019-2020-2021, this part should be feasible.
I'm using this 'remote warrior' setup nearly every day, using iPhone, iPad, and my home PCs nearly every day.
Test this OpenVPN server. And keep it, as it permits you to admin your pfSense from everywhere on the planet. I've set up mine so when I'm connected to it, all traffic, all Internet traffic, goes to my OpenVPN server pfSense, so when connected my WAN IP will be the IP of my company.
Again, this works well for more then 10 years.
I always used the OpenVPN Connect "app" on all my devices. I did not used the binaries that are build into pfSense, I keep them up to date from the source : https://openvpn.net/client/For the next step, you need an inexpensive old desktop PC, and an extra network card.
It would be even better if you had a second Internet connection next to your pfSense main server setup, and if not : or some other location not far away.
A good solution would be : some neighbor company. Just bring along this old PC, attach it to their LAN, a PC or your device behind the PC (your future client VPN).
Get pfSense 2.7.2.
Use the client OpenVPN, and make the connection to your pfSense server openVPN.
If no neighbors, by default : test this from home. No need to redo the home network, just connect the new WAN PC-pfSense plug into your home LAN.
No ports to open, nothing.
Connect your PC or device you test with to the pfSense LAN.What matters now is : you have access to both side !
You can se the openvpn logs on both sides !!
The second step is : make pfSense client talk and connect to pfSense server.Some hard to detect, but important criteria come into play now, like :
You are using two ISPs. Are they both transparent ? I mean : the UDP or TCP traffic used by OpenVPN isn't filtered in any way ? Do you use a classic connection like cable, fibre or ADSL ? Or is there a less then standard connection in play here ? Even if it might work very well, Starlink, 3G/4G/5G et isn't standard (for me).Final step : as soon as the connection is there, you have an interface on both side with an IP address, using a third IP network as the tunnel.
Now it comes down to : define routing rules : according to your needs : tell site A how it can reach the network of site B, and, if applicable, site B to site A.
And no, you will never be able to see in Windows Explorer on site B the windows PC's running on site A. That's not how it works.
If you declare DNS names on both side, site A and site B with their IP addresses, you can use DNS names to talk to the devices 'on the other side'.Again, I never went this far, but it can be done.
-
@DominikHoffmann said in Survey: Who has successfully set up peer-to-peer network?:
Pings to both the clients’ virtual and remote real address fail with 100% packet loss.
The clients real address = his public address?
Verify that pings are allowed on the clients WAN.What led me to blame it on bugs is that it briefly worked, when I fixed the client-specific override
however, with no other change (no apparent one, at least), my ability to access the remote network went away.I'm not aware of any bug in the pfSense OpenVPN implementation.
If there is an issue it's almost due to a configuration failure and you have to find out, where it is.
It can be the routes on both sites or the firewalls. If you have trouble to access a device behind a VPN endpoint also consider that there might a firewall running on the device itself, which possibly blocks access from outside of its subnet, while it allows access from inside.So check the routes on both sites, then check the firewall rules. For investigating enable the logging of the default block rule as well as of related pass rules and check the logs after trying to access the remote device.
For further investigation use the packet capture tool on pfSense to see if the packets even arrive on the remote site and are forwarded on the internal interface.
-
@viragomann said in Survey: Who has successfully set up peer-to-peer network?:
I'm not aware of any bug in the pfSense OpenVPN implementation.
Me neither.
And I know that the OpenVPN server is used a lot, as it's needed for remote admin, for for worker to conect to their company while working from home.The OpenVPN client works also, as it used a lot by people that don't trust their ISP. So they took Shark, Nord, Express, or whatever VPN supplier so they can continue to download their Disneys without being assigned to court.
Important detail : the client connects to an 'unknown' OpenVPN server... and they route some or the entire LAN over to this connection. This to ensure that their very private, encrypted TLS traffic is encrypted a second time.So, it's easy to proclaim : "OpenVPN" works.
It's all a question of how to set it up, as OpenVPN isn't really a click and run solution. -
@Gertjan: I will try and set this up with a second Netgate 1100 I have. Thanks very much for the suggestion!
Given that I only have to be able to see Site B and Site C from Site A, but not Site B from Site C or vice versa, do I need to set up explicit routes? Am I right in thinking that the requisite routes are set up by the client-server connection automatically?
Will I have to set up firewall rules? Will the firewalls at Site B and C have to allow connections from Site A? The reason I have not pursued this is that there was a day or so, when I was able to access Site B, Site C, Site D, but not Site E and Site F from Site A, without such rules. Now Site B, C and D behave like Site E and F did from the get-go—which is why I am banging down the door of this forum for help.
If the answer to that question is yes, what would I set as the source of the connection to be allowed? Will I have to set up a new interface for the client OpenVPN connection, or even one for each Site’s client connection?
-
@viragomann said in Survey: Who has successfully set up peer-to-peer network?:
So check the routes on both sites, then check the firewall rules. For investigating enable the logging of the default block rule as well as of related pass rules and check the logs after trying to access the remote device.
For further investigation use the packet capture tool on pfSense to see if the packets even arrive on the remote site and are forwarded on the internal interface.
I will do that, when I have configured the lab setup @Gertjan suggested. Thanks very much for the suggestion!
-
So, I found a problem in the server configuration. I had to include the remote networks in the IPv4 Local network(s) section. Now it says:
192.168.1.0/24,192.168.34.0/24,192.168.42.0/24,192.168.45.0/24,192.168.48.0/24,192.168.51.0/24,192.168.54.0/24
and it works for some of those, but only for some.
I compared the two client configurations on 192.168.34.1 and 192.168.54.1:
<openvpn> <openvpn-client> <auth_user></auth_user> <auth_pass></auth_pass> <proxy_user></proxy_user> <proxy_passwd></proxy_passwd> <vpnid>1</vpnid> <dco>disabled</dco> <protocol>UDP4</protocol> <dev_mode>tun</dev_mode> <interface>wan</interface> <ipaddr></ipaddr> <local_port></local_port> <server_addr>hoffmann.homeunix.net</server_addr> <server_port>1194</server_port> <proxy_addr></proxy_addr> <proxy_port></proxy_port> <proxy_authtype>none</proxy_authtype> <description><![CDATA[Ipheion Solutions management interface]]></description> <mode>p2p_tls</mode> <topology>subnet</topology> <custom_options>verify-x509-name "server" name</custom_options> <caref>66d22c7d1f7c9</caref> <certref>66d22c7d9e38e</certref> <crlref></crlref> <tls>********</tls> <tls_type>auth</tls_type> <tlsauth_keydir>1</tlsauth_keydir> <digest>SHA256</digest> <tunnel_network></tunnel_network> <tunnel_networkv6></tunnel_networkv6> <remote_network>192.168.54.0/24</remote_network> <remote_networkv6></remote_networkv6> <use_shaper></use_shaper> <allow_compression>asym</allow_compression> <compression></compression> <auth-retry-none></auth-retry-none> <passtos></passtos> <udp_fast_io></udp_fast_io> <exit_notify>none</exit_notify> <sndrcvbuf></sndrcvbuf> <route_no_pull></route_no_pull> <route_no_exec></route_no_exec> <dns_add></dns_add> <verbosity_level>1</verbosity_level> <create_gw>both</create_gw> <data_ciphers>AES-128-GCM,AES-256-CBC</data_ciphers> <data_ciphers_fallback>AES-256-CBC</data_ciphers_fallback> <ping_method>keepalive</ping_method> <keepalive_interval>10</keepalive_interval> <keepalive_timeout>60</keepalive_timeout> <ping_seconds>10</ping_seconds> <ping_action>ping_restart</ping_action> <ping_action_seconds>60</ping_action_seconds> <inactive_seconds>0</inactive_seconds> </openvpn-client> </openvpn>
and
<openvpn> <openvpn-client> <auth_user></auth_user> <auth_pass></auth_pass> <proxy_user></proxy_user> <proxy_passwd></proxy_passwd> <vpnid>1</vpnid> <dco>disabled</dco> <protocol>UDP4</protocol> <dev_mode>tun</dev_mode> <interface>wan</interface> <ipaddr></ipaddr> <local_port></local_port> <server_addr>hoffmann.homeunix.net</server_addr> <server_port>1194</server_port> <proxy_addr></proxy_addr> <proxy_port></proxy_port> <proxy_authtype>none</proxy_authtype> <description><![CDATA[pfSense-UDP4-1194-millers-config_OpenVPN]]></description> <mode>p2p_tls</mode> <topology>subnet</topology> <custom_options>verify-x509-name "server" name</custom_options> <caref>66bd527e46839</caref> <certref>66bd527ec81c0</certref> <crlref></crlref> <tls>********</tls> <tls_type>auth</tls_type> <tlsauth_keydir>1</tlsauth_keydir> <digest>SHA256</digest> <tunnel_network></tunnel_network> <tunnel_networkv6></tunnel_networkv6> <remote_network></remote_network> <remote_networkv6></remote_networkv6> <use_shaper></use_shaper> <allow_compression>asym</allow_compression> <compression></compression> <auth-retry-none></auth-retry-none> <passtos></passtos> <udp_fast_io></udp_fast_io> <exit_notify>none</exit_notify> <sndrcvbuf></sndrcvbuf> <route_no_pull></route_no_pull> <route_no_exec></route_no_exec> <dns_add></dns_add> <verbosity_level>1</verbosity_level> <create_gw>both</create_gw> <data_ciphers>AES-128-GCM,AES-256-CBC</data_ciphers> <data_ciphers_fallback>AES-256-CBC</data_ciphers_fallback> <ping_method>keepalive</ping_method> <keepalive_interval>10</keepalive_interval> <keepalive_timeout>60</keepalive_timeout> <ping_seconds>10</ping_seconds> <ping_action>ping_restart</ping_action> <ping_action_seconds>60</ping_action_seconds> <inactive_seconds>0</inactive_seconds> </openvpn-client> </openvpn>
The only real differences are in
- the description,
- the CA reference,
- the cert reference, and
- that the second has remote network specified, client side.