Some clients can ping lan some can't.

careymichael · Sep 20, 2019, 8:00 PM

@Rico I also created a new user and I was able to log in with that user/cert (User1). I tried to login using a different user / cert (User2) on a different machine and that user couldn't ping anything. It seems for the most part that it can't have more than one user. Both are connected. Only one can pass traffic.

If I ping a live host behind the vpn and run a packet capture on on the openvpn interface on the pfsense.

User1 I see 10.200.4.3 -> 10.200.0.5 10.200.4.3 <- 10.200.0.5
User2 I only see 10.200.4.4 -> 10.200.0.5

User2 also can't ping 10.200.4.1 the gateway on the vpn network.

It also seems that if I log out of both and login with user2 then user1, user 2 is still broken user1 is not.

If I restart openvpn on the pfsense and I login with user2 then user1, user2 works and user1 doesn't.

Concurrent users is set to 75

Strict user cn matching is unchecked.

Rico · Sep 22, 2019, 5:30 PM

Please share your OpenVPN Settings and Firewall Rules via Screenshot.

-Rico

careymichael · Sep 23, 2019, 1:41 PM

Rico · Sep 23, 2019, 1:57 PM

What size is this Local network 10.200.0.0 ? It's cut off in your screenshot.
Asking because for example 10.200.0.0/21 would overlap your tunnel network.

-Rico

careymichael · Sep 23, 2019, 3:15 PM

/22

careymichael · Sep 23, 2019, 3:32 PM

@Rico This is the routes and IP info on a machine that can't ping anything.

IPv4 Route Table

Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.70.0.1 10.70.0.206 35
10.50.0.0 255.255.254.0 10.200.4.1 10.200.4.3 291
10.70.0.0 255.255.255.0 On-link 10.70.0.206 291
10.70.0.206 255.255.255.255 On-link 10.70.0.206 291
10.70.0.255 255.255.255.255 On-link 10.70.0.206 291
10.200.0.0 255.255.252.0 10.200.4.1 10.200.4.3 291
10.200.4.0 255.255.255.0 On-link 10.200.4.3 291
10.200.4.3 255.255.255.255 On-link 10.200.4.3 291
10.200.4.255 255.255.255.255 On-link 10.200.4.3 291
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
127.0.0.1 255.255.255.255 On-link 127.0.0.1 331
127.255.255.255 255.255.255.255 On-link 127.0.0.1 331
172.18.0.0 255.255.248.0 10.200.4.1 10.200.4.3 291
192.168.4.0 255.255.252.0 10.200.4.1 10.200.4.3 291
198.18.0.0 255.255.0.0 10.200.4.1 10.200.4.3 291
224.0.0.0 240.0.0.0 On-link 127.0.0.1 331
224.0.0.0 240.0.0.0 On-link 10.70.0.206 291
224.0.0.0 240.0.0.0 On-link 10.200.4.3 291
255.255.255.255 255.255.255.255 On-link 127.0.0.1 331
255.255.255.255 255.255.255.255 On-link 10.70.0.206 291
255.255.255.255 255.255.255.255 On-link 10.200.4.3 291

C:\Users\carey>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : DESKTOP-5KT6HBN
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : XXXXXXXXXXXX
hsd1.ma.comcast.net

Wireless LAN adapter Local Area Connection* 1:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft Wi-Fi Direct Virtual Adapter
Physical Address. . . . . . . . . : 2A-3A-4D-0B-B9-AD
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes

Wireless LAN adapter Local Area Connection* 2:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft Wi-Fi Direct Virtual Adapter #2
Physical Address. . . . . . . . . : 28-3A-4D-0B-B9-AD
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes

Ethernet adapter Ethernet 2:

Connection-specific DNS Suffix . : XXXXXXXXXXXXXX
Description . . . . . . . . . . . : TAP-Windows Adapter V9
Physical Address. . . . . . . . . : 00-FF-74-AB-C9-A7
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::b918:8599:1acc:32a7%12(Preferred)
IPv4 Address. . . . . . . . . . . : 10.200.4.3(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Monday, September 23, 2019 11:26:00 AM
Lease Expires . . . . . . . . . . : Tuesday, September 22, 2020 11:25:59 AM
Default Gateway . . . . . . . . . :
DHCP Server . . . . . . . . . . . : 10.200.4.254
DHCPv6 IAID . . . . . . . . . . . : 637599604
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-23-9C-1D-16-00-00-10-00-1E-C8
DNS Servers . . . . . . . . . . . : 172.18.2.6
10.200.0.1
NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 4:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : TAP-Windows Adapter V9 #2
Physical Address. . . . . . . . . : 00-FF-E2-9E-2E-02
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes

Wireless LAN adapter Wi-Fi:

Connection-specific DNS Suffix . : hsd1.ma.comcast.net
Description . . . . . . . . . . . : Realtek 8822BE Wireless LAN 802.11ac PCI-E NIC
Physical Address. . . . . . . . . : 28-3A-4D-0B-B9-AD
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : 2603:3005:7bd:3000::d79(Preferred)
Lease Obtained. . . . . . . . . . : Thursday, September 19, 2019 11:06:37 AM
Lease Expires . . . . . . . . . . : Monday, September 30, 2019 9:22:13 AM
IPv6 Address. . . . . . . . . . . : 2603:3005:7bd:3000:ed52:da17:5edd:e097(Preferred)
Temporary IPv6 Address. . . . . . : 2603:3005:7bd:3000:d423:16bd:2578:daa7(Preferred)
Link-local IPv6 Address . . . . . : fe80::ed52:da17:5edd:e097%17(Preferred)
IPv4 Address. . . . . . . . . . . : 10.70.0.206(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Thursday, September 19, 2019 11:06:37 AM
Lease Expires . . . . . . . . . . : Monday, September 30, 2019 11:25:51 AM
Default Gateway . . . . . . . . . : fe80::623d:26ff:fe68:45c0%17
10.70.0.1
DHCP Server . . . . . . . . . . . : 10.70.0.1
DHCPv6 IAID . . . . . . . . . . . : 170408525
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-23-9C-1D-16-00-00-10-00-1E-C8
DNS Servers . . . . . . . . . . . : 2001:558:feed::1
2001:558:feed::2
10.70.0.1
2603:3005:7bd:3000:623d:26ff:fe68:45c0
2001:558:feed::1
2001:558:feed::2
NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Bluetooth Network Connection:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Bluetooth Device (Personal Area Network)
Physical Address. . . . . . . . . : 28-3A-4D-0B-B9-AE
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes

careymichael · Sep 23, 2019, 3:35 PM

@Rico This is the IP and route info on a machine that can connect.

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
XHC0: flags=0<> mtu 0
XHC20: flags=0<> mtu 0
XHC1: flags=0<> mtu 0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether f0:18:98:08:1b:47
inet 10.70.0.74 netmask 0xffffff00 broadcast 10.70.0.255
media: autoselect
status: active
en3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether b2:00:e8:71:41:01
media: autoselect <full-duplex>
status: inactive
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether b2:00:e8:71:41:00
media: autoselect <full-duplex>
status: inactive
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether b2:00:e8:71:41:05
media: autoselect <full-duplex>
status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether b2:00:e8:71:41:04
media: autoselect <full-duplex>
status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether b2:00:e8:71:41:00
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 11 priority 0 path cost 0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 13 priority 0 path cost 0
member: en3 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 10 priority 0 path cost 0
member: en4 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 12 priority 0 path cost 0
media: <unknown type>
status: inactive
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
ether 02:18:98:08:1b:47
media: autoselect
status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
ether b2:00:b0:b6:35:c9
inet6 fe80::b000:b0ff:feb6:35c9%awdl0 prefixlen 64 scopeid 0x10
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
inet6 fe80::5366:e0a3:af1d:ab20%utun0 prefixlen 64 scopeid 0x11
nd6 options=201<PERFORMNUD,DAD>
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=4<VLAN_MTU>
ether a0:ce:c8:31:17:f3
media: autoselect (none)
status: inactive
en7: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether ac48:00:11:22
inet6 fe80::aede:48ff:fe00:1122%en7 prefixlen 64 scopeid 0x8
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet 10.200.4.2 --> 10.200.4.2 netmask 0xffffff00

Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.70.0.1 UGSc 110 0 en0
10.50/23 10.200.4.1 UGSc 0 0 utun1
10.70/24 link#9 UCS 27 0 en0 !
10.70.0.1/32 link#9 UCS 1 0 en0 !
10.70.0.1 60:3d:26:68:45:c0 UHLWIir 35 2859 en0 1198
10.70.0.7 e4:e7:49:42:ec:d5 UHLWI 0 0 en0 1094
10.70.0.8 64:5d:86:cf:e9:9b UHLWIi 1 154 en0 626
10.70.0.9 80:c5:f2:84:56:ca UHLWI 0 6 en0 1069
10.70.0.10 d0:c5:d3:4c:d:4d UHLWI 0 6 en0 1101
10.70.0.24 90:e1:7b:9a:fe:9a UHLWI 0 0 en0 984
10.70.0.28 b0:70:2d:d1:8d:aa UHLWI 0 0 en0 531
10.70.0.31 38:f9:d3:31:1b:8d UHLWI 0 0 en0 732
10.70.0.32 64:c7:53:c0:7d:9a UHLWI 0 0 en0 602
10.70.0.34 a4:83:e7:50:eb:3f UHLWI 0 0 en0 1145
10.70.0.36 78:4f:43:4c:f4:c0 UHLWI 0 0 en0 724
10.70.0.74/32 link#9 UCS 0 0 en0 !
10.70.0.91 70:2a:d5:64:1a:36 UHLWI 0 10 en0 1200
10.70.0.94 a4:83:e7:50:e4:ec UHLWI 0 0 en0 1199
10.70.0.121 f8:6f:c1:89:59:25 UHLWI 0 0 en0 210
10.70.0.128 8c:85:90:3c:59:e8 UHLWI 0 0 en0 848
10.70.0.138 38:f9:d3:71:97:fc UHLWI 0 0 en0 1145
10.70.0.141 74:70:fd:76:15:88 UHLWI 0 0 en0 515
10.70.0.152 8c:86:1e:38:6b:51 UHLWI 0 0 en0 471
10.70.0.160 f4:5c:89:a2:f0:5 UHLWI 0 0 en0 1200
10.70.0.177 3c:15:c2:c5:69:e4 UHLWI 0 0 en0 1065
10.70.0.178 4c:56:9d:d:a9:be UHLWI 0 0 en0 983
10.70.0.180 f0:18:98:74:5b:89 UHLWI 0 0 en0 1200
10.70.0.183 78:4f:43:5c:bf:58 UHLWI 0 0 en0 1199
10.70.0.197 64:5a:ed:92:0:e9 UHLWI 0 0 en0 191
10.70.0.204 94:bf:2d:45:88:b3 UHLWI 0 0 en0 552
10.70.0.207 3c:2e:f9:12:92:30 UHLWI 0 0 en0 305
10.70.0.226 a4:83:e7:93:51:88 UHLWI 0 0 en0 912
10.70.0.235 70:2a:d5:63:f9:d8 UHLWI 0 10 en0 1198
10.200/22 10.200.4.1 UGSc 1 83 utun1
10.200.4/24 10.200.4.2 UGSc 7 0 utun1
10.200.4.2 10.200.4.2 UH 1 0 utun1
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 4 116765 lo0
169.254 link#9 UCS 0 0 en0 !
172.18/21 10.200.4.1 UGSc 1 0 utun1
192.168.4/22 10.200.4.1 UGSc 0 0 utun1
198.18.0/16 10.200.4.1 UGSc 0 0 utun1
224.0.0/4 link#9 UmCS 2 0 en0 !
224.0.0.251 1:0:5e:0:0:fb UHmLWI 0 0 en0
239.255.255.250 1:0:5e:7f:ff:fa UHmLWI 0 142 en0
255.255.255.255/32 link#9 UCS 0 0 en0 !

kiokoman · Sep 23, 2019, 4:00 PM

could it be related to this? https://forum.netgate.com/post/830456

careymichael · Sep 23, 2019, 4:24 PM

@kiokoman Yes this seems to be exactly the same issue. I get ERROR: FreeBSD route add command failed: external program exited with error status:1

Is there another way to fix this rather than using an external script? I'm running a new version of code 2.4.4-RELEASE-p3.

kiokoman · Sep 23, 2019, 7:54 PM

i realy don't know, i was searching for a solution for you and i found that post
@jimp do you already have/do you need a bug report for this ? or are you aware of a possible cause and solution ?

jimp · Sep 23, 2019, 7:57 PM

I doubt there is a bug here. Probably something in the setup/config but please don't ping me or other devs directly for this kind of thing.

kiokoman · Sep 23, 2019, 8:14 PM

sorry I'm probably putting too much enthusiasm into it :)

careymichael · Sep 23, 2019, 8:15 PM

@kiokoman Some more info.

If I restart openvpn the following routes are added automatically with out any connections to vpn.

10.200.4.1 link#26 UHS lo0
10.200.4.2 link#26 UH ovpns2

When the first client that connects it gets 10.200.4.2. Which is the only one that works because the route is already there.

When the second client connects it gets 10.200.4.3. The route add fails on the pfsense. If I manually add the route using route add -host 10.200.4.2 10.200.4.1. The client still can't connect to internal systems. Using packet capture I can see the traffic going inbound only on the openvpn interface. On the Lan when I add the route I can see both the sent packet and the return packet. It seems that the lan isn't routing the packet back to the openvpn interface.

The first client can ping it self and the vpn gateway 10.200.4.1.
The second client can't ping it self or the vpn gateway.

Is there a way to see why the route add is failing? It just says returned a status of 1.

kiokoman · Sep 23, 2019, 8:28 PM

if i understand it right ..
route add -host 10.200.4.3 -interface ovpns2
before connecting the second cllient

careymichael · Sep 23, 2019, 8:32 PM

@kiokoman Hehe I was just googleing that. That works. I can now ping internal hosts. Oddly enough I can now ping myself 10.200.4.3 and the gateway 10.200.4.. This I find odd because that should be L2 from my host.

Is there some way to get more logs regarding why the automatic route add is failing?

kiokoman · Sep 23, 2019, 8:49 PM

increase Verbosity level option for openvpn

careymichael · Sep 23, 2019, 9:12 PM

@kiokoman More Info.

Interesting I get the following.

Sep 23 16:43:52 openvpn 19893 ERROR: FreeBSD route add command failed: external program exited with error status: 1
Sep 23 16:43:52 openvpn 19893 /sbin/route add -net 10.200.4.0 10.200.4.2 255.255.255.0

This makes sense because 10.200.4.2 exists (not sure why). This is created automatically when openvpn is re/started on the pfsense. I haven't been able to see the error when .3 is given out. The verbosity is high and I can only see 2000 entries. So finding the route is kind of tough.

kiokoman · Sep 23, 2019, 9:15 PM

restart openvpn connect the first client, clear the log
connect the second client and see what is logged

careymichael · Sep 24, 2019, 9:02 PM

@kiokoman Setup a syslog server and turned up logging to 11. Found some interesting things.

When the server openvpn service is started I see the following entries.

ifconfig_local = '10.200.4.1'
ifconfig_remote_netmask = '255.255.255.0'
route_script = '[UNDEF]'
route_default_gateway = '10.200.4.2'
'route_default_metric = 0
route_noexec = DISABLED
route_delay = 0
route_delay_window = 30
route_delay_defined = DISABLED
route_nopull = DISABLED
route_gateway_via_dhcp = DISABLED
allow_pull_fqdn = DISABLED

server_network = 10.200.4.0
server_netmask = 255.255.255.0
push_entry = 'route-gateway 10.200.4.1'
push_entry = 'topology subnet'

ifconfig_pool_defined = ENABLED
ifconfig_pool_start = 10.200.4.2
ifconfig_pool_end = 10.200.4.253
ifconfig_pool_netmask = 255.255.255.0
ifconfig_pool_persist_filename = '[UNDEF]'
ifconfig_pool_persist_refresh_freq = 600
ifconfig_ipv6_pool_defined = DISABLED
ifconfig_ipv6_pool_base = ::
ifconfig_ipv6_pool_netbits = 0

/sbin/ifconfig ovpns2 10.200.4.1 10.200.4.2 mtu 1500 netmask 255.255.255.0 up
/sbin/route add -net 10.200.4.0 10.200.4.2 255.255.255.0
IFCONFIG POOL: base=10.200.4.2 size=252, ipv6=0

1st Client Connecting getting 10.200.0.2 (Note that is set as the default gateway of the server above)

MULTI_sva: pool returned IPv4=10.200.4.2, IPv6=(Not enabled)
MULTI: Learn: 10.200.4.2 -> user1/XXX.XXX.XXX.XXX:1194
MULTI: primary virtual IP for user1/XXX.XXX.XXX.XXX:1194: 10.200.4.2

I don't see a route add on the server side at all. I grep for it and only find this for the client connect

user1/XXX.XXX.XXX.XXX:1194 SENT CONTROL [michael.carey]: 'PUSH_REPLY,route 172.18.0.0 255.255.248.0,route 198.18.0.0 255.255.0.0,route 192.168.4.0 255.255.252.0,route 10.50.0.0 255.255.254.0,route 10.200.0.0 255.255.252.0,dhcp-option DOMAIN xxxxxx.com,dhcp-option DNS 172.18.2.6,dhcp-option DNS 10.200.0.1route-gateway 10.200.4.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.200.4.2 255.255.255.0,peer-id 0' (status=1)

2nd client connecting

MULTI_sva: pool returned IPv4=10.200.4.3, IPv6=(Not enabled)
MULTI: Learn: 10.200.4.3 -> user2/50.208.131.246:5238
MULTI: primary virtual IP for user2/50.208.131.246:5238: 10.200.4.3
Again no route on the server side

only the push

user/XXX.XXX.XXX.XXX:5238 SENT CONTROL [mike]: 'PUSH_REPLY,route 172.18.0.0 255.255.248.0,route 198.18.0.0 255.255.0.0,route 192.168.4.0 255.255.252.0,route 10.50.0.0 255.255.254.0,route 10.200.0.0 255.255.252.0,dhcp-option DOMAIN xxxxx.com,dhcp-option DNS 172.18.2.6,dhcp-option DNS 10.200.0.1,route-gateway 10.200.4.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.200.4.3 255.255.255.0,peer-id 1' (status=1)

kiokoman · Sep 25, 2019, 1:09 PM

Nothing related to "FreeBSD route add command failed: external program exited with error status: 1" ?
The first route already exist so the question is .. why the other are not created if it is needed? I will try to test it on my lab but i first need some hours of sleep