One external IP is being (wrongly) routed to OpenVPN

MTHead

I had originally posted this question in "General Questions" (http://forum.pfsense.org/index.php/topic,21658.0.html), but after some investigation I think it might be more of an OpenVPN question.

One of my clients was suddenly unable to access a certain website (www.palmettogba.com) on the Monday after Xmas. They had used it every day for months; my other clients in the building (all of whom are also using pfSense!) could still reach it; pinging or tracerouting directly from the pfSense box failed, but if I configured my laptop with the office's static IP and plugged directly into the T1, I could reach it.

Cutting to the chase: I ran tcpdump on the LAN interface in one shell session, and pinged the numeric IP of the Palmetto site in another shell session. I received the following in the tcpdump:

10:46:29.182937 arp who-has 216.251.231.64 tell 192.168.33.1
10:46:30.183472 arp who-has 216.251.231.64 tell 192.168.33.1

Meanwhile, each ping simply timed out with no response.

192.168.33.0/24 was the Address Pool I had reserved for the "road warrior" OpenVPN tunnel. As an experiment, I changed the Address Pool to 192.168.35.0/24, and tried ping/tcpdump again. This time, the response was:

11:13:18.106462 arp who-has 216.251.231.64 tell 192.168.35.1
11:13:19.107454 arp who-has 216.251.231.64 tell 192.168.35.1

I then tried disabling the tunnel, and suddenly I was able to ping/surf to Palmetto! However, as soon as I re-enabled the tunnel, Palmetto disappeared again.

So my question is: where is this set? I've looked through all the configuration files/locations I can think of, and don't see anything that specifically routes 216.251.231.64 through OpenVPN; by the way, 216.254.231.63 IS reachable (but I don't need it.) And how/why would this have changed spontaneously over the weekend? I'm the only person who has (or wants) authorized access, and I don't see any evidence of other tampering; I don't think this is a hack.

Before anyone asks - yes, this is persistent across reboots.

If you'd like me to post any specific files/screenshots, let me know. Thank you!

jimp

The output of "netstat -rn" before and during an OpenVPN connection would be nice, as well as the output of "ipconfig -a" before and during.

It may also help to have that same (or equivalent output) from the client machine. If it's windows, this would be the output of "route print" and also "ipconfig /all"

MTHead

@jimp:

The output of "netstat -rn" before and during an OpenVPN connection would be nice, as well as the output of "ipconfig -a" before and during.

It may also help to have that same (or equivalent output) from the client machine. If it's windows, this would be the output of "route print" and also "ipconfig /all"

Here ya go… I'm afraid I don't think you'll find this very helpful, though...
(I've taken the liberty of obfuscating the client's public IP info.)
Before connecting:


# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            OurPublicIP        UGS         0   811675   fxp1
OurPublicNet/29    link#2             UC          0        0   fxp1
OurPublicGW        00:a0:c8:41:75:f3  UHLW        2     9190   fxp1    860
127.0.0.1          127.0.0.1          UH          0        0    lo0
192.168.35.0&0xc0a82302 link#9             UC          0        0   tap0
192.168.254.0/24   link#1             UC          0        0   fxp0
192.168.254.5      00:10:5a:62:f0:a9  UHLW        1      207   fxp0    209
192.168.254.18     00:11:11:97:e8:44  UHLW        1    46270   fxp0   1176
192.168.254.32     00:0d:56:9b:4e:fc  UHLW        1    10776   fxp0   1184
192.168.254.48     00:15:f2:92:40:cc  UHLW        1     1607   fxp0   1037
192.168.254.49     00:15:f2:92:41:3b  UHLW        1      427   fxp0   1173
192.168.254.50     00:15:f2:92:41:4b  UHLW        1      897   fxp0   1087
192.168.254.101    00:15:f2:92:d0:60  UHLW        1      124   fxp0   1131
192.168.254.103    00:15:f2:92:3d:d7  UHLW        1     2480   fxp0    726
192.168.254.112    00:15:f2:92:d0:78  UHLW        1    10343   fxp0   1182
192.168.254.113    00:15:f2:92:41:23  UHLW        1      210   fxp0   1122
192.168.254.114    00:15:f2:92:d0:7a  UHLW        1   269024   fxp0    993
192.168.254.116    00:0d:56:9b:4f:ee  UHLW        1    31277   fxp0    996
192.168.254.132    00:26:18:30:13:8f  UHLW        1    67652   fxp0    867
192.168.254.172    00:13:20:96:c5:33  UHLW        1    36753   fxp0   1157
192.168.254.193    00:80:77:ed:c4:79  UHLW        1        0   fxp0   1039
192.168.254.254    00:50:8b:68:d6:8a  UHLW        1    96982    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%fxp0/64                    link#1                        UC         fxp0
fe80::250:8bff:fe68:d68a%fxp0     00:50:8b:68:d6:8a             UHL         lo0
fe80::%fxp1/64                    link#2                        UC         fxp1
fe80::250:8bff:fe68:d68b%fxp1     00:50:8b:68:d6:8b             UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#4                        UHL         lo0
fe80::2bd:e1ff:fe25:0%tap0        00:bd:e1:25:00:00             UHL         lo0
ff01:1::/32                       link#1                        UC         fxp0
ff01:2::/32                       link#2                        UC         fxp1
ff01:4::/32                       ::1                           UC          lo0
ff01:9::/32                       link#9                        UC         tap0
ff02::%fxp0/32                    link#1                        UC         fxp0
ff02::%fxp1/32                    link#2                        UC         fxp1
ff02::%lo0/32                     ::1                           UC          lo0
ff02::%tap0/32                    link#9                        UC         tap0

After connecting:


# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            OurPublicIP        UGS         0   811953   fxp1
OurPublicNet/29    link#2             UC          0        0   fxp1
OurPublicGW        00:a0:c8:41:75:f3  UHLW        2     9200   fxp1    703
127.0.0.1          127.0.0.1          UH          0        0    lo0
192.168.35.0&0xc0a82302 link#9             UC          0        0   tap0
192.168.254.0/24   link#1             UC          0        0   fxp0
192.168.254.5      00:10:5a:62:f0:a9  UHLW        1      207   fxp0   1078
192.168.254.18     00:11:11:97:e8:44  UHLW        1    46270   fxp0   1019
192.168.254.32     00:0d:56:9b:4e:fc  UHLW        1    10776   fxp0   1199
192.168.254.48     00:15:f2:92:40:cc  UHLW        1     1607   fxp0    880
192.168.254.49     00:15:f2:92:41:3b  UHLW        1      429   fxp0   1145
192.168.254.50     00:15:f2:92:41:4b  UHLW        1      897   fxp0    930
192.168.254.101    00:15:f2:92:d0:60  UHLW        1      124   fxp0    974
192.168.254.103    00:15:f2:92:3d:d7  UHLW        1     2480   fxp0    569
192.168.254.112    00:15:f2:92:d0:78  UHLW        1    10343   fxp0   1053
192.168.254.113    00:15:f2:92:41:23  UHLW        1      210   fxp0   1195
192.168.254.114    00:15:f2:92:d0:7a  UHLW        1   269024   fxp0   1168
192.168.254.116    00:0d:56:9b:4f:ee  UHLW        1    31277   fxp0   1142
192.168.254.132    00:26:18:30:13:8f  UHLW        1    67652   fxp0    710
192.168.254.150    00:ff:f0:09:36:d9  UHLW        1        1   fxp0   1194
192.168.254.172    00:13:20:96:c5:33  UHLW        1    36753   fxp0   1000
192.168.254.193    00:80:77:ed:c4:79  UHLW        1        0   fxp0    882
192.168.254.254    00:50:8b:68:d6:8a  UHLW        1    96982    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%fxp0/64                    link#1                        UC         fxp0
fe80::250:8bff:fe68:d68a%fxp0     00:50:8b:68:d6:8a             UHL         lo0
fe80::%fxp1/64                    link#2                        UC         fxp1
fe80::250:8bff:fe68:d68b%fxp1     00:50:8b:68:d6:8b             UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#4                        UHL         lo0
fe80::2bd:e1ff:fe25:0%tap0        00:bd:e1:25:00:00             UHL         lo0
ff01:1::/32                       link#1                        UC         fxp0
ff01:2::/32                       link#2                        UC         fxp1
ff01:4::/32                       ::1                           UC          lo0
ff01:9::/32                       link#9                        UC         tap0
ff02::%fxp0/32                    link#1                        UC         fxp0
ff02::%fxp1/32                    link#2                        UC         fxp1
ff02::%lo0/32                     ::1                           UC          lo0
ff02::%tap0/32                    link#9                        UC         tap0

Windows settings, after connecting:


C:\Users\Marc>route print
===========================================================================
Interface List
 16...00 ff ec f4 7e 6f ......TAP-Win32 Adapter V9 #2
 15...00 ff f0 09 36 d9 ......TAP-Win32 Adapter V9
 12...00 24 2b d3 13 13 ......Atheros AR5007 802.11b/g WiFi Adapter
 11...00 23 5a 37 ed ee ......Realtek RTL8102E/RTL8103E Family PCI-E Fast Ethern
et NIC (NDIS 6.20)
 21...08 00 27 00 54 4b ......VirtualBox Host-Only Ethernet Adapter
  1...........................Software Loopback Interface 1
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 13...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
 24...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 25...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
 23...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #4
 22...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #5
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.0.1    192.168.0.144     25
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
      192.168.0.0    255.255.255.0         On-link     192.168.0.144    281
    192.168.0.144  255.255.255.255         On-link     192.168.0.144    281
    192.168.0.255  255.255.255.255         On-link     192.168.0.144    281
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    276
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    276
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    276
    192.168.254.0    255.255.255.0         On-link   192.168.254.150    286
    192.168.254.0    255.255.255.0  192.168.254.254  192.168.254.150     30
  192.168.254.150  255.255.255.255         On-link   192.168.254.150    286
  192.168.254.255  255.255.255.255         On-link   192.168.254.150    286
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    276
        224.0.0.0        240.0.0.0         On-link     192.168.0.144    281
        224.0.0.0        240.0.0.0         On-link   192.168.254.150    286
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    276
  255.255.255.255  255.255.255.255         On-link     192.168.0.144    281
  255.255.255.255  255.255.255.255         On-link   192.168.254.150    286
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
 21    276 fe80::/64                On-link
 15    286 fe80::/64                On-link
 15    286 fe80::c8d3:8856:771e:c54/128
                                    On-link
 21    276 fe80::dd17:67ce:1898:494e/128
                                    On-link
  1    306 ff00::/8                 On-link
 21    276 ff00::/8                 On-link
 15    286 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

C:\Users\Marc>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : Marc-HP
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : lbms.local

Ethernet adapter OpenVPN-CCMG:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : TAP-Win32 Adapter V9 #2
   Physical Address. . . . . . . . . : 00-FF-EC-F4-7E-6F
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Ethernet adapter OpenVPN:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : TAP-Win32 Adapter V9
   Physical Address. . . . . . . . . : 00-FF-F0-09-36-D9
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::c8d3:8856:771e:c54%15(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.254.150(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Thursday, December 31, 2009 8:07:43 PM
   Lease Expires . . . . . . . . . . : Friday, December 31, 2010 8:07:42 PM
   Default Gateway . . . . . . . . . :
   DHCP Server . . . . . . . . . . . : 192.168.254.0
   DHCPv6 IAID . . . . . . . . . . . : 402718704
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-92-34-EA-00-23-5A-37-ED-EE

   DNS Servers . . . . . . . . . . . : 192.168.254.254
   NetBIOS over Tcpip. . . . . . . . : Enabled

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . : lbms.local
   Description . . . . . . . . . . . : Atheros AR5007 802.11b/g WiFi Adapter
   Physical Address. . . . . . . . . : 00-24-2B-D3-13-13
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.0.144(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Thursday, December 31, 2009 7:32:50 PM
   Lease Expires . . . . . . . . . . : Thursday, December 31, 2009 9:32:50 PM
   Default Gateway . . . . . . . . . : 192.168.0.1
   DHCP Server . . . . . . . . . . . : 192.168.0.1
   DNS Servers . . . . . . . . . . . : 192.168.0.1
                                       208.67.222.222
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Local Area Connection:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Realtek RTL8102E/RTL8103E Family PCI-E Fa
st Ethernet NIC (NDIS 6.20)
   Physical Address. . . . . . . . . : 00-23-5A-37-ED-EE
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : VirtualBox Host-Only Ethernet Adapter
   Physical Address. . . . . . . . . : 08-00-27-00-54-4B
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::dd17:67ce:1898:494e%21(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.56.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 705167399
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-92-34-EA-00-23-5A-37-ED-EE

   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.lbms.local:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : lbms.local
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{ECF47E6F-D4B5-4F85-AE89-A70F744115A6}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{6AFFB67E-9A26-47F7-A49F-EA070A9B7F01}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{F00936D9-FBAD-4401-9B43-4848C8B3BA98}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #4
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{DBBED88B-A4B8-4D6E-98AF-E01EBC1C62BB}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #5
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

MTHead

I should give a little extra background: I only discovered the OpenVPN tie-in after barking up many wrong trees. I don't understand the connection at all…

The client is a doctor's office, and Palmetto is the Medicare carrier for our region, so the office accesses the website every day. This has always worked, up until Monday morning. Also, I set up an OpenVPN road-warrior tunnel at the same time that I installed pfSense (about six months ago), and there hadn't been any previous problems. Suddenly, on Monday the office could not reach the Palmetto site; besides being unable to browse the site, I could not ping from the client machines, from the pfSense shell (via Putty), or from Diagnostics/Ping in the WebGUI. I have several other clients in the same medical building who also use pfSense, and I was able to ping Palmetto from their pfSense boxes, so initially I blamed the ISP. The ISP's tech support instructed me to bypass the router and plug my laptop directly into the T1… and imagine how foolish I felt when it worked! However, I still couldn't find anything in the pfSense configuration that would cause such a result - all packets destined for Palmetto seemed to vanish into thin air. (And ONLY Palmetto - all other sites seem to be just fine, and even an IP address that's off-by-one from the Palmetto address works.)

Finally this morning I logged in via Putty from home, and got the result that I posted earlier. It hadn't even crossed my mind earlier that OpenVPN might be involved, as this seemed to be a strictly internal issue (and since I hadn't made any changes recently.)
It makes no difference whether anyone is connected via OpenVPN or not; if the tunnel is enabled, "arp who-has Palmetto" gets the answer "OpenVPN has it" - and presumably, the packet then gets sent through OpenVPN and into a black hole (especially if no-one is connected at the time!) If I disable the tunnel, packets intended for Palmetto are routed through the WAN interface as they should be, and the ping succeeds. I could delete the tunnel and re-create it, but I would rather not (unless I was sure that it would fix the problem) because I dread the thought of re-creating and re-distributing certificates...

In any case, I think I've looked in all the usual places. I was hoping that someone would have some ideas/experience of UNusual places to look?

danswartz

I am confused by the client arping for an address not on its subnet and getting a reply. This may be some openvpn oddity, though. Have you looked at the openvpn config (/conf/config.xml) to see if that address shows up?

jimp

Yes, the contents of the OpenVPN client (on windows) config and the server on pfsense (/var/etc/openvpn_*.conf) might also help shed some light on the situation.

MTHead

@jimp:

Yes, the contents of the OpenVPN client (on windows) config and the server on pfsense (/var/etc/openvpn_*.conf) might also help shed some light on the situation.

Windows config file (tz.ovpn):


client
dev tap
dev-node OpenVPN
proto udp
remote OurPublicIP 1194
resolv-retry infinite
nobind
persist-key
persist-tun
mute-replay-warnings

ca tz-ca.crt
cert tz-marc-hp.crt
key tz-marc-hp.key

ns-cert-type server
cipher BF-CBC
comp-lzo
verb 3

cat openvpn_server0.conf


writepid /var/run/openvpn_server0.pid
#user nobody
#group nobody
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
dev tun
proto udp
cipher BF-CBC
up /etc/rc.filter_configure
down /etc/rc.filter_configure
tls-server
ifconfig 192.168.35.1 192.168.35.2
push "route 192.168.254.0 255.255.255.0"
lport 1194
push "dhcp-option DNS 192.168.254.254"
push "dhcp-option NBT 4"
ca /var/etc/openvpn_server0.ca
cert /var/etc/openvpn_server0.cert
key /var/etc/openvpn_server0.key
dh /var/etc/openvpn_server0.dh
comp-lzo
dev tap0
server-bridge 192.168.254.254 255.255.255.0 192.168.254.150 192.168.254.160

MTHead

@danswartz:

I am confused by the client arping for an address not on its subnet and getting a reply. This may be some openvpn oddity, though.

I'm not sure what you mean by "the client" in this context. I was pinging from the pfSense shell… I do wonder where the reply came from. Is there any switch I can use with tcpdump (or another tool) to find out?

@danswartz:

Have you looked at the openvpn config (/conf/config.xml) to see if that address shows up?

Yes I have, and no it doesn't. In fact, even the first octet doesn't appear anywhere in the file. I can post it if you'd like, but it would obviously require some obfuscation.

I think the answer lies in the fact that somebody somewhere on the network is answering that ARP who-has, and if I can find out who it is (and stop it) the problem will be solved. Perhaps I'm in for a day of good old-fashioned unplugging and re-plugging… I'm hoping for a more modern answer, though.

danswartz

sorry misread who was doing the arp. you could try 'tcpdump -lnvvv' instead. what i think must be going on is that the local IP (192.168.x.x) would not be arping for the palmetto address normally, much less getting a reply, so this must be coming to/from the tunnel. What does 'clog /var/log/openvpn.log' show?

jimp

Why are you using "dev tap" on the client? That is probably your problem. And why the server-bridge line on the server side?

iirc, tap devices are more for bridging than routing. You probably want that to be tun.

If you are trying to bridge, you want that to be tap on both sides, but I'm not familiar with a bridged setup.

danswartz

Good point about tap vs tun.

MTHead

@jimp:

Why are you using "dev tap" on the client? That is probably your problem. And why the server-bridge line on the server side?

iirc, tap devices are more for bridging than routing. You probably want that to be tun.

If you are trying to bridge, you want that to be tap on both sides, but I'm not familiar with a bridged setup.

I'm using tap intentionally because I do use bridging. (It's not necessary at this office, but I have been using a standard configuration across all of my installations - much less hassle that way!) One of my offices has a permanent (or, at least, we WISH it were permanent) IPSec tunnel to a hosting company. The hosting company's road-warrior IPSec VPN (from a company that rhymes with Crisco) is much less reliable than the site-to-site, so I decided to have my road warriors connect to the office network and access the hosted network via bridging. It's been very stable and very easy to add/remove clients, as opposed to dealing with the hosting people… Once I got that set up and working, I standardized my OpenVPN setup; this particular office doesn't currently need bridging, but it's been working smoothly.

I can change the OpenVPN setup to use tun instead; again, I'd prefer not to unless I'm sure it will fix this problem, because it means modifying all the client machines (most of which are at the doctors'/employees' houses.) I have this same setup in nearly a dozen offices, running for over two years now, and this is the first time anything like this has come up. Perhaps this is a hidden danger of using bridging, in which case I should expect to get hit with it at my other sites sooner or later; meanwhile, I'd like to get to the bottom of why it happened here.

It's persistent across reboots.
It doesn't matter whether any OpenVPN clients are connected or not.
As far as I can tell, it's only this one IP address that's affected. (Google.com and forum.pfsense.org, for instance, work just fine.)

These things make me think that pfSense has stored this (bad) routing information somewhere. At this point it's more of an intellectual curiosity to me to find out where. Should I perhaps ask on a FreeBSD forum?

MTHead

@danswartz:

sorry misread who was doing the arp. you could try 'tcpdump -lnvvv' instead. what i think must be going on is that the local IP (192.168.x.x) would not be arping for the palmetto address normally, much less getting a reply, so this must be coming to/from the tunnel. What does 'clog /var/log/openvpn.log' show?


# clog /var/log/openvpn.log
Jan  1 22:55:25 pfsense openvpn[345]: OpenVPN 2.0.6 i386-portbld-freebsd7.2 [SSL] [LZO] built on Dec  4 2009
Jan  1 22:55:25 pfsense openvpn[345]: WARNING: file '/var/etc/openvpn_server0.key' is group or others accessible
Jan  1 22:55:25 pfsense openvpn[345]: WARNING: Since you are using --dev tap, the second argument to --ifconfig must be a netmask, for example something like 255.255.255.0\. (silence this warning with --ifconfig-nowarn)
Jan  1 22:55:25 pfsense openvpn[345]: TUN/TAP device /dev/tap0 opened
Jan  1 22:55:25 pfsense openvpn[345]: /sbin/ifconfig tap0 192.168.35.1 netmask 192.168.35.2 mtu 1500 up
Jan  1 22:55:25 pfsense openvpn[345]: /etc/rc.filter_configure tap0 1500 1574 192.168.35.1 192.168.35.2 init
Jan  1 22:55:25 pfsense openvpn[352]: UDPv4 link local (bound): [undef]:1194
Jan  1 22:55:25 pfsense openvpn[352]: UDPv4 link remote: [undef]
Jan  1 22:55:25 pfsense openvpn[352]: Initialization Sequence Completed

MTHead

Just thought I'd post the eventual solution, in case anyone else ever has the same problem. I added a static route:

Interface Network Gateway Description

WAN 216.251.231.64/32 (our gateway) Palmetto

in other words, I added an explicit rule to reinforce what should be happening anyway. And now it works. What caused the original problem, I don't know…