Routing Problem on Bridged network - Clients ignore route

Bonsai · May 22, 2015, 9:05 AM

First to say, I do not need any comments like "Why bridged?" or "Just make different subnets" as I have a crappy piece of software that needs broadcasts. My comuters MUST be all in the same subnet.

I have now sucessfully setup two ALIX.2D13 in a test environment. The Internet is simulated my my current LAN 192.168.2.0/24
My LAN for this setup is 172.17.172.0/24

One OpenVPN Server is setup on Port 1194 for a site-to-site bridged network. It is working fine. The clients on the other ALIX.2D13 can see the clients on first one.

So this is the working one:

dev ovpns1
verb 1
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CBC
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.2.109
engine cryptodev
lport 1194
management /var/etc/openvpn/server1.sock unix
secret /var/etc/openvpn/server1.secret 
comp-lzo adaptive
passtos

The second one (failing) is running on port 1195 is for two mobile users:

dev ovpns2
verb 1
dev-type tap
dev-node /dev/tap2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CBC
auth RSA-SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.2.109
engine cryptodev
tls-server
server-bridge 172.17.172.1 255.255.255.0 172.17.172.140 172.17.172.150
client-config-dir /var/etc/openvpn-csc
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'pfsensegel.jkf-domain.de' 1"
lport 1195
management /var/etc/openvpn/server2.sock unix
max-clients 10
#push "route 172.17.172.0 255.255.255.0"
#push "register-dns"
client-to-client
duplicate-cn
ca /var/etc/openvpn/server2.ca 
cert /var/etc/openvpn/server2.cert 
key /var/etc/openvpn/server2.key 
dh /etc/dh-parameters.2048
crl-verify /var/etc/openvpn/server2.crl-verify 
tls-auth /var/etc/openvpn/server2.tls-auth 0
comp-lzo adaptive
passtos
persist-remote-ip
float
#push "route 172.17.172.0 255.255.255.0 vpn_gateway"

My client config (Authentification with Aladdin USB stick, Ceritficate + password):

dev tap
persist-tun
persist-key
cipher AES-256-CBC
auth RSA-SHA512
tls-client
client
resolv-retry infinite
remote 192.168.2.109 1195 udp
lport 0
verify-x509-name "pfsensegel.jkf-domain.de" name
comp-lzo adaptive
passtos

# keys and certificates
ns-cert-type server
ca OpenVPN_CA.crt
tls-auth pfSenseGel-udp-1195-vpnuser1-tls.key 1
pkcs11-providers opensc-pkcs11.dll
pkcs11-id 'OpenSC\x20Project/PKCS\x2315/28818C11221C/OpenSC\x20Card\x20\x28vpnuser1\x29/1A2ED950DD6084D70D19C3B085514FA85DFFB7DC'

route 172.17.172.0 255.255.255.0

It took me two days as I knowed all the stuff in theory and even worked in troubleshooting on similar systems but I never had to setup it by my own.

My last open issue is, that the client gets an IP from the DHCP Server on my pfsense, but the route to the tap network is failing.

I tried a lot of different route commands on client and on server side …. always the same problem. A tracert 172.17.172.1 (pfsense) from client 172.17.172.140 goes to the default gateway from the notebook and from there to internet :-\

route print on my win7 notebook shows me currently two routes for the same network:
target / mask / gateway / device
172.17.172.0 255.255.255.0 172.17.172.1 172.17.172.140
172.17.172.0 255.255.255.0 172.17.172.1 192.168.2.101
and additional this one:
172.17.172.140 255.255.255.0 blank 172.17.172.140

My Firewall on pfsense (internal wide open) even shows me, that the client 172.17.172.140 is sending IGMP packages to 244.0.0.22
As I get an IP from pfsenses DHCP and firewall allows these IGMP packages, I think my connection is fine. Also the interfaces and bridge there should be fine.

So where's my routing issue?

doktornotor · May 22, 2015, 9:22 AM

IGMP has TTL=1 and will not get forwarded anywhere. You might want to use the IGMP proxy.

Bonsai · May 22, 2015, 9:47 AM

This will not change anything to the issue, that I cant get any TCP or UDP packages through the tunnel. Also no ICMP ….

I first want to have my network running ..... but thanks for the comment.

What I later need are broadcasts in my LAN subnet. The IGMP is not required for me. I only used it as an example that the line is up in general.

doktornotor · May 22, 2015, 9:52 AM

@Bonsai:

The IGMP is not required for me. I only used it as an example that the line is up in general.

Was one hell of sucky example… How about firewall rules screenshot? Firewall logs? Some real info like packet captures? Your clients ignore routing? Why's this even pfSense issue?

Bonsai · May 22, 2015, 10:22 AM

tracert 172.17.172.1 on my client produces a route to the internet
192.168.2.1
212.x.x.x
timed out

The firewall rules on ALL interfaces are very simple at the moment:
IPv4+6 Pass * * **
Except the WAN Interface, where I allow only traffic outgoing, except incoming traffic to Ports 1194, 1995, 433 target: this firewall

So it seems not a firewall issue.

The rules are by the way identical to the rules for the first working VPN server. So, no , not a firewall issue.

Your clients ignore routing? Why's this even pfSense issue?

I posted it in OpenVPN section, becasue it is a OpenVPN issue if my client gets an IP from my pfsense DHCP across the VPN tunnel, but then he has no route to the VPN Server.

A packet capture of the connecting client on interface ovpns2:

12:16:27.089535 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.129593 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87
12:16:27.167962 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28
12:16:27.168759 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:27.169470 IP6 :: > ff02::1:ff48:8c9c: ICMP6, neighbor solicitation, who has fe80::1431:31b7:ae48:8c9c, length 24
12:16:27.170233 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16
12:16:27.170987 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
12:16:27.193296 IP6 fe80::1431:31b7:ae48:8c9c.65175 > ff02::1:3.5355: UDP, length 25
12:16:27.194112 IP 172.17.172.140.63999 > 224.0.0.252.5355: UDP, length 25
12:16:27.203386 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:27.239396 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.274433 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.275954 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.276635 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:27.393601 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.394494 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:27.395175 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:27.395864 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:27.494367 IP6 fe80::1431:31b7:ae48:8c9c.56991 > ff02::1:3.5355: UDP, length 25
12:16:27.495201 IP 172.17.172.140.55777 > 224.0.0.252.5355: UDP, length 25
12:16:27.668053 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:27.668784 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 3 group record(s), length 68
12:16:28.130166 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87
12:16:28.169751 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28
12:16:28.170482 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:28.171135 IP6 fe80::1431:31b7:ae48:8c9c > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::1431:31b7:ae48:8c9c, length 32
12:16:28.171982 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:28.172659 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:28.203401 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:28.204109 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:28.305541 IP6 fe80::1431:31b7:ae48:8c9c.49981 > ff02::1:3.5355: UDP, length 25
12:16:28.306891 IP 172.17.172.140.61320 > 224.0.0.252.5355: UDP, length 25
12:16:28.668059 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:28.668782 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:29.167993 ARP, Request who-has 172.17.172.140 tell 0.0.0.0, length 28
12:16:29.168547 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:29.325846 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:29.436618 IP6 fe80::1431:31b7:ae48:8c9c.58109 > ff02::1:3.5355: UDP, length 24
12:16:29.437393 IP 172.17.172.140.61708 > 224.0.0.252.5355: UDP, length 24
12:16:29.634017 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70
12:16:29.667922 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:29.744311 IP6 fe80::1431:31b7:ae48:8c9c.52097 > ff02::1:3.5355: UDP, length 22
12:16:29.755052 IP 172.17.172.140.58167 > 224.0.0.252.5355: UDP, length 22
12:16:29.812257 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110
12:16:30.076220 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110
12:16:30.129912 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87
12:16:30.167880 ARP, Request who-has 172.17.172.140 tell 172.17.172.140, length 28
12:16:30.182757 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:30.183417 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:30.201260 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:30.201906 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:30.234115 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:30.302394 IP6 fe80::1431:31b7:ae48:8c9c.50310 > ff02::1:3.5355: UDP, length 25
12:16:30.303167 IP 172.17.172.140.50496 > 224.0.0.252.5355: UDP, length 25
12:16:30.341144 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 110
12:16:30.591435 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277
12:16:30.668100 IP 172.17.172.140 > 224.0.0.22: igmp
12:16:30.668716 IP6 fe80::1431:31b7:ae48:8c9c > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
12:16:30.710116 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70
12:16:31.168112 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:31.168665 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16
12:16:31.663584 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277
12:16:32.041550 IP6 fe80::1431:31b7:ae48:8c9c.65532 > ff02::1:3.5355: UDP, length 24
12:16:32.042327 IP 172.17.172.140.54912 > 224.0.0.252.5355: UDP, length 24
12:16:32.168117 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:32.727674 IP6 fe80::1431:31b7:ae48:8c9c.63303 > ff02::1:3.5355: UDP, length 22
12:16:32.728473 IP 172.17.172.140.59970 > 224.0.0.252.5355: UDP, length 22
12:16:33.254573 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:33.619657 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 277
12:16:33.731311 IP 172.17.172.140.5353 > 224.0.0.251.5353: UDP, length 70
12:16:34.130549 IP6 fe80::1431:31b7:ae48:8c9c.546 > ff02::1:2.547: UDP, length 87
12:16:34.168073 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:35.168495 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:16:35.169043 IP6 fe80::1431:31b7:ae48:8c9c > ff02::2: ICMP6, router solicitation, length 16
12:16:36.168476 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28

Another one if I ping from client the 172.17.172.1:

12:18:50.171033 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28
12:18:51.170148 ARP, Request who-has 172.17.172.1 tell 172.17.172.140, length 28

Bonsai · May 22, 2015, 3:01 PM

Solved it mostly …

looks like it was almost everything regarding the interface bridges.

I have now on first pfsense:
The site-to-site server on interface tap1
The remote clients server on interface tap2
A bridge which contains LAN, tap1 and tap2

The trick was to set the remoteclients server to the "Bridge interface" bridge0

Now I can ping across all three locations. The only thing I can't ping is my remote client?!? It isn't important for now, but it would interrest me why.

Server site-to-site:
/var/etc/openvpn/server1.conf

dev ovpns1
verb 1
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CBC
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.2.109
engine cryptodev
lport 1194
management /var/etc/openvpn/server1.sock unix
secret /var/etc/openvpn/server1.secret 
comp-lzo adaptive
passtos

The remote site:
/var/etc/openvpn/client1.conf

dev ovpnc1
verb 1
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CBC
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.2.108
engine cryptodev
lport 0
management /var/etc/openvpn/client1.sock unix
remote 192.168.2.109 1194
secret /var/etc/openvpn/client1.secret 
comp-lzo adaptive
passtos
resolv-retry infinite

Server for remote clients:
/var/etc/openvpn/server2.conf

dev ovpns2
verb 1
dev-type tap
dev-node /dev/tap2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CBC
auth RSA-SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.2.109
engine cryptodev
tls-server
mode server
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'pfsensegel.jkf-domain.de' 1"
lport 1195
management /var/etc/openvpn/server2.sock unix
max-clients 10
push "route 172.17.172.0 255.255.255.0"
push "dhcp-option DOMAIN jkf-domain.de"
push "dhcp-option DNS 172.17.172.1"
push "dhcp-option DNS 8.8.8.8"
push "register-dns"
client-to-client
duplicate-cn
ca /var/etc/openvpn/server2.ca 
cert /var/etc/openvpn/server2.cert 
key /var/etc/openvpn/server2.key 
dh /etc/dh-parameters.2048
crl-verify /var/etc/openvpn/server2.crl-verify 
tls-auth /var/etc/openvpn/server2.tls-auth 0
comp-lzo adaptive
passtos
persist-remote-ip
float

Configuration on Win7 client (64Bit) with 32Bit OpenVPN and 32Bit OpenSC for the aladdin usb-stick

dev tap
persist-tun
persist-key
cipher AES-256-CBC
auth RSA-SHA512
tls-client
client
resolv-retry infinite
remote 192.168.2.109 1195 udp
lport 0
verify-x509-name "pfsensegel.jkf-domain.de" name
comp-lzo adaptive
passtos

# keys and certificates
ns-cert-type server
ca OpenVPN_CA.crt
tls-auth pfSenseGel-udp-1195-vpnuser1-tls.key 1
pkcs11-providers opensc-pkcs11.dll
pkcs11-id 'OpenSC\x20Project/PKCS\x2315/28818C11221C/OpenSC\x20Card\x20\x28vpnuser1\x29/1A2ED950DD6084D70D19C3B085514FA85DFFB7DC'

divsys · May 22, 2015, 3:12 PM

Could it be as simple as: The remote client firewall is blocking pings from an "unknown" network?
This is a common Windows PC issue.
If it's only pings (and you've got a rule to allow ICMP), then the problem is often at the remote PC end not in pfSense.

Glad to hear you've got most of it solved.

Bonsai · May 26, 2015, 1:01 PM

Thank you! Windows Firewall really blocked the ICMP packages. ??? How I made ithis beginners failure to not check this? :-[

Now everything what I need is working. There is still an Item I just can't understand. I can ping now from all clients all other clients in all locations, but my clients coming from outside over OpenVPN can't ping one of the Openvpn servers?!? Also the frontend of pfsense is not accessible from these notebooks if I choose the internal IPs. If I go over WAN, everything is fine?!?

Packet captures show, that the echo and the reply is visible in two devices only: The TAP of the server for remoteclients and the server for remoteclients itsself. Nothing in LAN or in the bridge interface…. It is really funny, that a ping goes over two VPN tunnels and two servers without any trouble, but the servers themselfes are stealh devices ;D