Routing Problem on Bridged network - Clients ignore route
-
First to say, I do not need any comments like "Why bridged?" or "Just make different subnets" as I have a crappy piece of software that needs broadcasts. My comuters MUST be all in the same subnet.
I have now sucessfully setup two ALIX.2D13 in a test environment. The Internet is simulated my my current LAN 192.168.2.0/24
My LAN for this setup is 172.17.172.0/24One OpenVPN Server is setup on Port 1194 for a site-to-site bridged network. It is working fine. The clients on the other ALIX.2D13 can see the clients on first one.
So this is the working one:
dev ovpns1 verb 1 dev-type tap dev-node /dev/tap1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-256-CBC auth SHA512 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.2.109 engine cryptodev lport 1194 management /var/etc/openvpn/server1.sock unix secret /var/etc/openvpn/server1.secret comp-lzo adaptive passtos
The second one (failing) is running on port 1195 is for two mobile users:
dev ovpns2 verb 1 dev-type tap dev-node /dev/tap2 writepid /var/run/openvpn_server2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-256-CBC auth RSA-SHA512 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.2.109 engine cryptodev tls-server server-bridge 172.17.172.1 255.255.255.0 172.17.172.140 172.17.172.150 client-config-dir /var/etc/openvpn-csc tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'pfsensegel.jkf-domain.de' 1" lport 1195 management /var/etc/openvpn/server2.sock unix max-clients 10 #push "route 172.17.172.0 255.255.255.0" #push "register-dns" client-to-client duplicate-cn ca /var/etc/openvpn/server2.ca cert /var/etc/openvpn/server2.cert key /var/etc/openvpn/server2.key dh /etc/dh-parameters.2048 crl-verify /var/etc/openvpn/server2.crl-verify tls-auth /var/etc/openvpn/server2.tls-auth 0 comp-lzo adaptive passtos persist-remote-ip float #push "route 172.17.172.0 255.255.255.0 vpn_gateway"
My client config (Authentification with Aladdin USB stick, Ceritficate + password):
dev tap persist-tun persist-key cipher AES-256-CBC auth RSA-SHA512 tls-client client resolv-retry infinite remote 192.168.2.109 1195 udp lport 0 verify-x509-name "pfsensegel.jkf-domain.de" name comp-lzo adaptive passtos # keys and certificates ns-cert-type server ca OpenVPN_CA.crt tls-auth pfSenseGel-udp-1195-vpnuser1-tls.key 1 pkcs11-providers opensc-pkcs11.dll pkcs11-id 'OpenSC\x20Project/PKCS\x2315/28818C11221C/OpenSC\x20Card\x20\x28vpnuser1\x29/1A2ED950DD6084D70D19C3B085514FA85DFFB7DC' route 172.17.172.0 255.255.255.0
It took me two days as I knowed all the stuff in theory and even worked in troubleshooting on similar systems but I never had to setup it by my own.
My last open issue is, that the client gets an IP from the DHCP Server on my pfsense, but the route to the tap network is failing.
I tried a lot of different route commands on client and on server side …. always the same problem. A tracert 172.17.172.1 (pfsense) from client 172.17.172.140 goes to the default gateway from the notebook and from there to internet :-\
route print on my win7 notebook shows me currently two routes for the same network:
target / mask / gateway / device
172.17.172.0 255.255.255.0 172.17.172.1 172.17.172.140
172.17.172.0 255.255.255.0 172.17.172.1 192.168.2.101
and additional this one:
172.17.172.140 255.255.255.0 blank 172.17.172.140My Firewall on pfsense (internal wide open) even shows me, that the client 172.17.172.140 is sending IGMP packages to 244.0.0.22
As I get an IP from pfsenses DHCP and firewall allows these IGMP packages, I think my connection is fine. Also the interfaces and bridge there should be fine.So where's my routing issue?
-
IGMP has TTL=1 and will not get forwarded anywhere. You might want to use the IGMP proxy.
-
This will not change anything to the issue, that I cant get any TCP or UDP packages through the tunnel. Also no ICMP ….
I first want to have my network running ..... but thanks for the comment.
What I later need are broadcasts in my LAN subnet. The IGMP is not required for me. I only used it as an example that the line is up in general.
-
The IGMP is not required for me. I only used it as an example that the line is up in general.
Was one hell of sucky example… How about firewall rules screenshot? Firewall logs? Some real info like packet captures? Your clients ignore routing? Why's this even pfSense issue?
-
tracert 172.17.172.1 on my client produces a route to the internet
192.168.2.1
212.x.x.x
timed outThe firewall rules on ALL interfaces are very simple at the moment:
IPv4+6 Pass * * **
Except the WAN Interface, where I allow only traffic outgoing, except incoming traffic to Ports 1194, 1995, 433 target: this firewallSo it seems not a firewall issue.
The rules are by the way identical to the rules for the first working VPN server. So, no , not a firewall issue.
Your clients ignore routing? Why's this even pfSense issue?
I posted it in OpenVPN section, becasue it is a OpenVPN issue if my client gets an IP from my pfsense DHCP across the VPN tunnel, but then he has no route to the VPN Server.
A packet capture of the connecting client on interface ovpns2:
12:16:27.089535 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.129593 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87 12:16:27.167962 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28 12:16:27.168759 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:27.169470 IP6 :: > ff02::1:ff48:8c9c: ICMP6, neighbor solicitation, who has fe80::1431:31b7:ae48:8c9c, length 24 12:16:27.170233 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16 12:16:27.170987 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48 12:16:27.193296 IP6 fe80::1431:31b7:ae48:8c9c.65175 > ff02::1:3.5355: UDP, length 25 12:16:27.194112 IP 172.17.172.140.63999 > 224.0.0.252.5355: UDP, length 25 12:16:27.203386 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:27.239396 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.274433 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.275954 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.276635 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:27.393601 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.394494 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:27.395175 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:27.395864 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:27.494367 IP6 fe80::1431:31b7:ae48:8c9c.56991 > ff02::1:3.5355: UDP, length 25 12:16:27.495201 IP 172.17.172.140.55777 > 224.0.0.252.5355: UDP, length 25 12:16:27.668053 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:27.668784 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 3 group record(s), length 68 12:16:28.130166 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87 12:16:28.169751 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28 12:16:28.170482 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:28.171135 IP6 fe80::1431:31b7:ae48:8c9c > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::1431:31b7:ae48:8c9c, length 32 12:16:28.171982 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:28.172659 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:28.203401 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:28.204109 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:28.305541 IP6 fe80::1431:31b7:ae48:8c9c.49981 > ff02::1:3.5355: UDP, length 25 12:16:28.306891 IP 172.17.172.140.61320 > 224.0.0.252.5355: UDP, length 25 12:16:28.668059 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:28.668782 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:29.167993 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28 12:16:29.168547 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:29.325846 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:29.436618 IP6 fe80::1431:31b7:ae48:8c9c.58109 > ff02::1:3.5355: UDP, length 24 12:16:29.437393 IP 172.17.172.140.61708 > 224.0.0.252.5355: UDP, length 24 12:16:29.634017 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70 12:16:29.667922 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:29.744311 IP6 fe80::1431:31b7:ae48:8c9c.52097 > ff02::1:3.5355: UDP, length 22 12:16:29.755052 IP 172.17.172.140.58167 > 224.0.0.252.5355: UDP, length 22 12:16:29.812257 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110 12:16:30.076220 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110 12:16:30.129912 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87 12:16:30.167880 ARP, Request who-has 172.17.172.140 tell 172.17.172.140, length 28 12:16:30.182757 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:30.183417 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:30.201260 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:30.201906 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:30.234115 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:30.302394 IP6 fe80::1431:31b7:ae48:8c9c.50310 > ff02::1:3.5355: UDP, length 25 12:16:30.303167 IP 172.17.172.140.50496 > 224.0.0.252.5355: UDP, length 25 12:16:30.341144 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110 12:16:30.591435 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277 12:16:30.668100 IP 172.17.172.140 > 224.0.0.22: igmp 12:16:30.668716 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 12:16:30.710116 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70 12:16:31.168112 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:31.168665 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16 12:16:31.663584 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277 12:16:32.041550 IP6 fe80::1431:31b7:ae48:8c9c.65532 > ff02::1:3.5355: UDP, length 24 12:16:32.042327 IP 172.17.172.140.54912 > 224.0.0.252.5355: UDP, length 24 12:16:32.168117 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:32.727674 IP6 fe80::1431:31b7:ae48:8c9c.63303 > ff02::1:3.5355: UDP, length 22 12:16:32.728473 IP 172.17.172.140.59970 > 224.0.0.252.5355: UDP, length 22 12:16:33.254573 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:33.619657 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277 12:16:33.731311 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70 12:16:34.130549 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87 12:16:34.168073 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:35.168495 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:16:35.169043 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16 12:16:36.168476 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
Another one if I ping from client the 172.17.172.1:
12:18:50.171033 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28 12:18:51.170148 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
-
Solved it mostly …
looks like it was almost everything regarding the interface bridges.
I have now on first pfsense:
The site-to-site server on interface tap1
The remote clients server on interface tap2
A bridge which contains LAN, tap1 and tap2The trick was to set the remoteclients server to the "Bridge interface" bridge0
Now I can ping across all three locations. The only thing I can't ping is my remote client?!? It isn't important for now, but it would interrest me why.
Server site-to-site:
/var/etc/openvpn/server1.confdev ovpns1 verb 1 dev-type tap dev-node /dev/tap1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-256-CBC auth SHA512 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.2.109 engine cryptodev lport 1194 management /var/etc/openvpn/server1.sock unix secret /var/etc/openvpn/server1.secret comp-lzo adaptive passtos
The remote site:
/var/etc/openvpn/client1.confdev ovpnc1 verb 1 dev-type tap dev-node /dev/tap1 writepid /var/run/openvpn_client1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-256-CBC auth SHA512 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.2.108 engine cryptodev lport 0 management /var/etc/openvpn/client1.sock unix remote 192.168.2.109 1194 secret /var/etc/openvpn/client1.secret comp-lzo adaptive passtos resolv-retry infinite
Server for remote clients:
/var/etc/openvpn/server2.confdev ovpns2 verb 1 dev-type tap dev-node /dev/tap2 writepid /var/run/openvpn_server2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-256-CBC auth RSA-SHA512 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.2.109 engine cryptodev tls-server mode server tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'pfsensegel.jkf-domain.de' 1" lport 1195 management /var/etc/openvpn/server2.sock unix max-clients 10 push "route 172.17.172.0 255.255.255.0" push "dhcp-option DOMAIN jkf-domain.de" push "dhcp-option DNS 172.17.172.1" push "dhcp-option DNS 8.8.8.8" push "register-dns" client-to-client duplicate-cn ca /var/etc/openvpn/server2.ca cert /var/etc/openvpn/server2.cert key /var/etc/openvpn/server2.key dh /etc/dh-parameters.2048 crl-verify /var/etc/openvpn/server2.crl-verify tls-auth /var/etc/openvpn/server2.tls-auth 0 comp-lzo adaptive passtos persist-remote-ip float
Configuration on Win7 client (64Bit) with 32Bit OpenVPN and 32Bit OpenSC for the aladdin usb-stick
dev tap persist-tun persist-key cipher AES-256-CBC auth RSA-SHA512 tls-client client resolv-retry infinite remote 192.168.2.109 1195 udp lport 0 verify-x509-name "pfsensegel.jkf-domain.de" name comp-lzo adaptive passtos # keys and certificates ns-cert-type server ca OpenVPN_CA.crt tls-auth pfSenseGel-udp-1195-vpnuser1-tls.key 1 pkcs11-providers opensc-pkcs11.dll pkcs11-id 'OpenSC\x20Project/PKCS\x2315/28818C11221C/OpenSC\x20Card\x20\x28vpnuser1\x29/1A2ED950DD6084D70D19C3B085514FA85DFFB7DC'
-
Could it be as simple as: The remote client firewall is blocking pings from an "unknown" network?
This is a common Windows PC issue.
If it's only pings (and you've got a rule to allow ICMP), then the problem is often at the remote PC end not in pfSense.Glad to hear you've got most of it solved.
-
Thank you! Windows Firewall really blocked the ICMP packages. ??? How I made ithis beginners failure to not check this? :-[
Now everything what I need is working. There is still an Item I just can't understand. I can ping now from all clients all other clients in all locations, but my clients coming from outside over OpenVPN can't ping one of the Openvpn servers?!? Also the frontend of pfsense is not accessible from these notebooks if I choose the internal IPs. If I go over WAN, everything is fine?!?
Packet captures show, that the echo and the reply is visible in two devices only: The TAP of the server for remoteclients and the server for remoteclients itsself. Nothing in LAN or in the bridge interface…. It is really funny, that a ping goes over two VPN tunnels and two servers without any trouble, but the servers themselfes are stealh devices ;D