One external IP is being (wrongly) routed to OpenVPN



  • I had originally posted this question in "General Questions" (http://forum.pfsense.org/index.php/topic,21658.0.html), but after some investigation I think it might be more of an OpenVPN question.

    One of my clients was suddenly unable to access a certain website (www.palmettogba.com) on the Monday after Xmas.  They had used it every day for months; my other clients in the building (all of whom are also using pfSense!) could still reach it; pinging or tracerouting directly from the pfSense box failed, but if I configured my laptop with the office's static IP and plugged directly into the T1, I could reach it.

    Cutting to the chase: I ran tcpdump on the LAN interface in one shell session, and pinged the numeric IP of the Palmetto site in another shell session.  I received the following in the tcpdump:

    10:46:29.182937 arp who-has 216.251.231.64 tell 192.168.33.1
    10:46:30.183472 arp who-has 216.251.231.64 tell 192.168.33.1
    

    Meanwhile, each ping simply timed out with no response.

    192.168.33.0/24 was the Address Pool I had reserved for the "road warrior" OpenVPN tunnel.  As an experiment, I changed the Address Pool to 192.168.35.0/24, and tried ping/tcpdump again.  This time, the response was:

    11:13:18.106462 arp who-has 216.251.231.64 tell 192.168.35.1
    11:13:19.107454 arp who-has 216.251.231.64 tell 192.168.35.1
    

    I then tried disabling the tunnel, and suddenly I was able to ping/surf to Palmetto!  However, as soon as I re-enabled the tunnel, Palmetto disappeared again.

    So my question is: where is this set?  I've looked through all the configuration files/locations I can think of, and don't see anything that specifically routes 216.251.231.64 through OpenVPN; by the way, 216.254.231.63 IS reachable (but I don't need it.)  And how/why would this have changed spontaneously over the weekend?  I'm the only person who has (or wants) authorized access, and I don't see any evidence of other tampering; I don't think this is a hack.

    Before anyone asks - yes, this is persistent across reboots.

    If you'd like me to post any specific files/screenshots, let me know.  Thank you!


  • Rebel Alliance Developer Netgate

    The output of "netstat -rn" before and during an OpenVPN connection would be nice, as well as the output of "ipconfig -a" before and during.

    It may also help to have that same (or equivalent output) from the client machine. If it's windows, this would be the output of "route print" and also "ipconfig /all"



  • @jimp:

    The output of "netstat -rn" before and during an OpenVPN connection would be nice, as well as the output of "ipconfig -a" before and during.

    It may also help to have that same (or equivalent output) from the client machine. If it's windows, this would be the output of "route print" and also "ipconfig /all"

    Here ya go… I'm afraid I don't think you'll find this very helpful, though...
    (I've taken the liberty of obfuscating the client's public IP info.)
    Before connecting:

    
    # netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            OurPublicIP        UGS         0   811675   fxp1
    OurPublicNet/29    link#2             UC          0        0   fxp1
    OurPublicGW        00:a0:c8:41:75:f3  UHLW        2     9190   fxp1    860
    127.0.0.1          127.0.0.1          UH          0        0    lo0
    192.168.35.0&0xc0a82302 link#9             UC          0        0   tap0
    192.168.254.0/24   link#1             UC          0        0   fxp0
    192.168.254.5      00:10:5a:62:f0:a9  UHLW        1      207   fxp0    209
    192.168.254.18     00:11:11:97:e8:44  UHLW        1    46270   fxp0   1176
    192.168.254.32     00:0d:56:9b:4e:fc  UHLW        1    10776   fxp0   1184
    192.168.254.48     00:15:f2:92:40:cc  UHLW        1     1607   fxp0   1037
    192.168.254.49     00:15:f2:92:41:3b  UHLW        1      427   fxp0   1173
    192.168.254.50     00:15:f2:92:41:4b  UHLW        1      897   fxp0   1087
    192.168.254.101    00:15:f2:92:d0:60  UHLW        1      124   fxp0   1131
    192.168.254.103    00:15:f2:92:3d:d7  UHLW        1     2480   fxp0    726
    192.168.254.112    00:15:f2:92:d0:78  UHLW        1    10343   fxp0   1182
    192.168.254.113    00:15:f2:92:41:23  UHLW        1      210   fxp0   1122
    192.168.254.114    00:15:f2:92:d0:7a  UHLW        1   269024   fxp0    993
    192.168.254.116    00:0d:56:9b:4f:ee  UHLW        1    31277   fxp0    996
    192.168.254.132    00:26:18:30:13:8f  UHLW        1    67652   fxp0    867
    192.168.254.172    00:13:20:96:c5:33  UHLW        1    36753   fxp0   1157
    192.168.254.193    00:80:77:ed:c4:79  UHLW        1        0   fxp0   1039
    192.168.254.254    00:50:8b:68:d6:8a  UHLW        1    96982    lo0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               ::1                           UHL         lo0
    fe80::%fxp0/64                    link#1                        UC         fxp0
    fe80::250:8bff:fe68:d68a%fxp0     00:50:8b:68:d6:8a             UHL         lo0
    fe80::%fxp1/64                    link#2                        UC         fxp1
    fe80::250:8bff:fe68:d68b%fxp1     00:50:8b:68:d6:8b             UHL         lo0
    fe80::%lo0/64                     fe80::1%lo0                   U           lo0
    fe80::1%lo0                       link#4                        UHL         lo0
    fe80::2bd:e1ff:fe25:0%tap0        00:bd:e1:25:00:00             UHL         lo0
    ff01:1::/32                       link#1                        UC         fxp0
    ff01:2::/32                       link#2                        UC         fxp1
    ff01:4::/32                       ::1                           UC          lo0
    ff01:9::/32                       link#9                        UC         tap0
    ff02::%fxp0/32                    link#1                        UC         fxp0
    ff02::%fxp1/32                    link#2                        UC         fxp1
    ff02::%lo0/32                     ::1                           UC          lo0
    ff02::%tap0/32                    link#9                        UC         tap0
    
    

    After connecting:

    
    # netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            OurPublicIP        UGS         0   811953   fxp1
    OurPublicNet/29    link#2             UC          0        0   fxp1
    OurPublicGW        00:a0:c8:41:75:f3  UHLW        2     9200   fxp1    703
    127.0.0.1          127.0.0.1          UH          0        0    lo0
    192.168.35.0&0xc0a82302 link#9             UC          0        0   tap0
    192.168.254.0/24   link#1             UC          0        0   fxp0
    192.168.254.5      00:10:5a:62:f0:a9  UHLW        1      207   fxp0   1078
    192.168.254.18     00:11:11:97:e8:44  UHLW        1    46270   fxp0   1019
    192.168.254.32     00:0d:56:9b:4e:fc  UHLW        1    10776   fxp0   1199
    192.168.254.48     00:15:f2:92:40:cc  UHLW        1     1607   fxp0    880
    192.168.254.49     00:15:f2:92:41:3b  UHLW        1      429   fxp0   1145
    192.168.254.50     00:15:f2:92:41:4b  UHLW        1      897   fxp0    930
    192.168.254.101    00:15:f2:92:d0:60  UHLW        1      124   fxp0    974
    192.168.254.103    00:15:f2:92:3d:d7  UHLW        1     2480   fxp0    569
    192.168.254.112    00:15:f2:92:d0:78  UHLW        1    10343   fxp0   1053
    192.168.254.113    00:15:f2:92:41:23  UHLW        1      210   fxp0   1195
    192.168.254.114    00:15:f2:92:d0:7a  UHLW        1   269024   fxp0   1168
    192.168.254.116    00:0d:56:9b:4f:ee  UHLW        1    31277   fxp0   1142
    192.168.254.132    00:26:18:30:13:8f  UHLW        1    67652   fxp0    710
    192.168.254.150    00:ff:f0:09:36:d9  UHLW        1        1   fxp0   1194
    192.168.254.172    00:13:20:96:c5:33  UHLW        1    36753   fxp0   1000
    192.168.254.193    00:80:77:ed:c4:79  UHLW        1        0   fxp0    882
    192.168.254.254    00:50:8b:68:d6:8a  UHLW        1    96982    lo0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               ::1                           UHL         lo0
    fe80::%fxp0/64                    link#1                        UC         fxp0
    fe80::250:8bff:fe68:d68a%fxp0     00:50:8b:68:d6:8a             UHL         lo0
    fe80::%fxp1/64                    link#2                        UC         fxp1
    fe80::250:8bff:fe68:d68b%fxp1     00:50:8b:68:d6:8b             UHL         lo0
    fe80::%lo0/64                     fe80::1%lo0                   U           lo0
    fe80::1%lo0                       link#4                        UHL         lo0
    fe80::2bd:e1ff:fe25:0%tap0        00:bd:e1:25:00:00             UHL         lo0
    ff01:1::/32                       link#1                        UC         fxp0
    ff01:2::/32                       link#2                        UC         fxp1
    ff01:4::/32                       ::1                           UC          lo0
    ff01:9::/32                       link#9                        UC         tap0
    ff02::%fxp0/32                    link#1                        UC         fxp0
    ff02::%fxp1/32                    link#2                        UC         fxp1
    ff02::%lo0/32                     ::1                           UC          lo0
    ff02::%tap0/32                    link#9                        UC         tap0
    
    

    Windows settings, after connecting:

    
    C:\Users\Marc>route print
    ===========================================================================
    Interface List
     16...00 ff ec f4 7e 6f ......TAP-Win32 Adapter V9 #2
     15...00 ff f0 09 36 d9 ......TAP-Win32 Adapter V9
     12...00 24 2b d3 13 13 ......Atheros AR5007 802.11b/g WiFi Adapter
     11...00 23 5a 37 ed ee ......Realtek RTL8102E/RTL8103E Family PCI-E Fast Ethern
    et NIC (NDIS 6.20)
     21...08 00 27 00 54 4b ......VirtualBox Host-Only Ethernet Adapter
      1...........................Software Loopback Interface 1
     14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
     13...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
     24...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
     25...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
     23...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #4
     22...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #5
    ===========================================================================
    
    IPv4 Route Table
    ===========================================================================
    Active Routes:
    Network Destination        Netmask          Gateway       Interface  Metric
              0.0.0.0          0.0.0.0      192.168.0.1    192.168.0.144     25
            127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
            127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
      127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
          192.168.0.0    255.255.255.0         On-link     192.168.0.144    281
        192.168.0.144  255.255.255.255         On-link     192.168.0.144    281
        192.168.0.255  255.255.255.255         On-link     192.168.0.144    281
         192.168.56.0    255.255.255.0         On-link      192.168.56.1    276
         192.168.56.1  255.255.255.255         On-link      192.168.56.1    276
       192.168.56.255  255.255.255.255         On-link      192.168.56.1    276
        192.168.254.0    255.255.255.0         On-link   192.168.254.150    286
        192.168.254.0    255.255.255.0  192.168.254.254  192.168.254.150     30
      192.168.254.150  255.255.255.255         On-link   192.168.254.150    286
      192.168.254.255  255.255.255.255         On-link   192.168.254.150    286
            224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
            224.0.0.0        240.0.0.0         On-link      192.168.56.1    276
            224.0.0.0        240.0.0.0         On-link     192.168.0.144    281
            224.0.0.0        240.0.0.0         On-link   192.168.254.150    286
      255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
      255.255.255.255  255.255.255.255         On-link      192.168.56.1    276
      255.255.255.255  255.255.255.255         On-link     192.168.0.144    281
      255.255.255.255  255.255.255.255         On-link   192.168.254.150    286
    ===========================================================================
    Persistent Routes:
      None
    
    IPv6 Route Table
    ===========================================================================
    Active Routes:
     If Metric Network Destination      Gateway
      1    306 ::1/128                  On-link
     21    276 fe80::/64                On-link
     15    286 fe80::/64                On-link
     15    286 fe80::c8d3:8856:771e:c54/128
                                        On-link
     21    276 fe80::dd17:67ce:1898:494e/128
                                        On-link
      1    306 ff00::/8                 On-link
     21    276 ff00::/8                 On-link
     15    286 ff00::/8                 On-link
    ===========================================================================
    Persistent Routes:
      None
    
    C:\Users\Marc>ipconfig /all
    
    Windows IP Configuration
    
       Host Name . . . . . . . . . . . . : Marc-HP
       Primary Dns Suffix  . . . . . . . :
       Node Type . . . . . . . . . . . . : Mixed
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : lbms.local
    
    Ethernet adapter OpenVPN-CCMG:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : TAP-Win32 Adapter V9 #2
       Physical Address. . . . . . . . . : 00-FF-EC-F4-7E-6F
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
    
    Ethernet adapter OpenVPN:
    
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : TAP-Win32 Adapter V9
       Physical Address. . . . . . . . . : 00-FF-F0-09-36-D9
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
       Link-local IPv6 Address . . . . . : fe80::c8d3:8856:771e:c54%15(Preferred)
       IPv4 Address. . . . . . . . . . . : 192.168.254.150(Preferred)
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Lease Obtained. . . . . . . . . . : Thursday, December 31, 2009 8:07:43 PM
       Lease Expires . . . . . . . . . . : Friday, December 31, 2010 8:07:42 PM
       Default Gateway . . . . . . . . . :
       DHCP Server . . . . . . . . . . . : 192.168.254.0
       DHCPv6 IAID . . . . . . . . . . . : 402718704
       DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-92-34-EA-00-23-5A-37-ED-EE
    
       DNS Servers . . . . . . . . . . . : 192.168.254.254
       NetBIOS over Tcpip. . . . . . . . : Enabled
    
    Wireless LAN adapter Wireless Network Connection:
    
       Connection-specific DNS Suffix  . : lbms.local
       Description . . . . . . . . . . . : Atheros AR5007 802.11b/g WiFi Adapter
       Physical Address. . . . . . . . . : 00-24-2B-D3-13-13
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 192.168.0.144(Preferred)
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Lease Obtained. . . . . . . . . . : Thursday, December 31, 2009 7:32:50 PM
       Lease Expires . . . . . . . . . . : Thursday, December 31, 2009 9:32:50 PM
       Default Gateway . . . . . . . . . : 192.168.0.1
       DHCP Server . . . . . . . . . . . : 192.168.0.1
       DNS Servers . . . . . . . . . . . : 192.168.0.1
                                           208.67.222.222
       NetBIOS over Tcpip. . . . . . . . : Enabled
    
    Ethernet adapter Local Area Connection:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Realtek RTL8102E/RTL8103E Family PCI-E Fa
    st Ethernet NIC (NDIS 6.20)
       Physical Address. . . . . . . . . : 00-23-5A-37-ED-EE
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
    
    Ethernet adapter Local Area Connection 2:
    
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : VirtualBox Host-Only Ethernet Adapter
       Physical Address. . . . . . . . . : 08-00-27-00-54-4B
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       Link-local IPv6 Address . . . . . : fe80::dd17:67ce:1898:494e%21(Preferred)
       IPv4 Address. . . . . . . . . . . : 192.168.56.1(Preferred)
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . :
       DHCPv6 IAID . . . . . . . . . . . : 705167399
       DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-92-34-EA-00-23-5A-37-ED-EE
    
       DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                           fec0:0:0:ffff::2%1
                                           fec0:0:0:ffff::3%1
       NetBIOS over Tcpip. . . . . . . . : Enabled
    
    Tunnel adapter isatap.lbms.local:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . : lbms.local
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    Tunnel adapter Teredo Tunneling Pseudo-Interface:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    Tunnel adapter isatap.{ECF47E6F-D4B5-4F85-AE89-A70F744115A6}:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    Tunnel adapter isatap.{6AFFB67E-9A26-47F7-A49F-EA070A9B7F01}:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    Tunnel adapter isatap.{F00936D9-FBAD-4401-9B43-4848C8B3BA98}:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter #4
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    Tunnel adapter isatap.{DBBED88B-A4B8-4D6E-98AF-E01EBC1C62BB}:
    
       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter #5
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
    
    


  • I should give a little extra background: I only discovered the OpenVPN tie-in after barking up many wrong trees.  I don't understand the connection at all…

    The client is a doctor's office, and Palmetto is the Medicare carrier for our region, so the office accesses the website every day.  This has always worked, up until Monday morning.  Also, I set up an OpenVPN road-warrior tunnel at the same time that I installed pfSense (about six months ago), and there hadn't been any previous problems.  Suddenly, on Monday the office could not reach the Palmetto site; besides being unable to browse the site, I could not ping from the client machines, from the pfSense shell (via Putty), or from Diagnostics/Ping in the WebGUI.  I have several other clients in the same medical building who also use pfSense, and I was able to ping Palmetto from their pfSense boxes, so initially I blamed the ISP.  The ISP's tech support instructed me to bypass the router and plug my laptop directly into the T1… and imagine how foolish I felt when it worked!  However, I still couldn't find anything in the pfSense configuration that would cause such a result - all packets destined for Palmetto seemed to vanish into thin air.  (And ONLY Palmetto - all other sites seem to be just fine, and even an IP address that's off-by-one from the Palmetto address works.)

    Finally this morning I logged in via Putty from home, and got the result that I posted earlier.  It hadn't even crossed my mind earlier that OpenVPN might be involved, as this seemed to be a strictly internal issue (and since I hadn't made any changes recently.)
    It makes no difference whether anyone is connected via OpenVPN or not; if the tunnel is enabled, "arp who-has Palmetto" gets the answer "OpenVPN has it" - and presumably, the packet then gets sent through OpenVPN and into a black hole (especially if no-one is connected at the time!)  If I disable the tunnel, packets intended for Palmetto are routed through the WAN interface as they should be, and the ping succeeds.  I could delete the tunnel and re-create it, but I would rather not (unless I was sure that it would fix the problem) because I dread the thought of re-creating and re-distributing certificates...

    In any case, I think I've looked in all the usual places.  I was hoping that someone would have some ideas/experience of UNusual places to look?



  • I am confused by the client arping for an address not on its subnet and getting a reply.  This may be some openvpn oddity, though.  Have you looked at the openvpn config (/conf/config.xml) to see if that address shows up?


  • Rebel Alliance Developer Netgate

    Yes, the contents of the OpenVPN client (on windows) config and the server on pfsense (/var/etc/openvpn_*.conf) might also help shed some light on the situation.



  • @jimp:

    Yes, the contents of the OpenVPN client (on windows) config and the server on pfsense (/var/etc/openvpn_*.conf) might also help shed some light on the situation.

    Windows config file (tz.ovpn):

    
    client
    dev tap
    dev-node OpenVPN
    proto udp
    remote OurPublicIP 1194
    resolv-retry infinite
    nobind
    persist-key
    persist-tun
    mute-replay-warnings
    
    ca tz-ca.crt
    cert tz-marc-hp.crt
    key tz-marc-hp.key
    
    ns-cert-type server
    cipher BF-CBC
    comp-lzo
    verb 3
    
    

    cat openvpn_server0.conf

    
    writepid /var/run/openvpn_server0.pid
    #user nobody
    #group nobody
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    dev tun
    proto udp
    cipher BF-CBC
    up /etc/rc.filter_configure
    down /etc/rc.filter_configure
    tls-server
    ifconfig 192.168.35.1 192.168.35.2
    push "route 192.168.254.0 255.255.255.0"
    lport 1194
    push "dhcp-option DNS 192.168.254.254"
    push "dhcp-option NBT 4"
    ca /var/etc/openvpn_server0.ca
    cert /var/etc/openvpn_server0.cert
    key /var/etc/openvpn_server0.key
    dh /var/etc/openvpn_server0.dh
    comp-lzo
    dev tap0
    server-bridge 192.168.254.254 255.255.255.0 192.168.254.150 192.168.254.160
    
    


  • @danswartz:

    I am confused by the client arping for an address not on its subnet and getting a reply.  This may be some openvpn oddity, though.

    I'm not sure what you mean by "the client" in this context.  I was pinging from the pfSense shell… I do wonder where the reply came from.  Is there any switch I can use with tcpdump (or another tool) to find out?

    @danswartz:

    Have you looked at the openvpn config (/conf/config.xml) to see if that address shows up?

    Yes I have, and no it doesn't.  In fact, even the first octet doesn't appear anywhere in the file.  I can post it if you'd like, but it would obviously require some obfuscation.

    I think the answer lies in the fact that somebody somewhere on the network is answering that ARP who-has, and if I can find out who it is (and stop it) the problem will be solved.  Perhaps I'm in for a day of good old-fashioned unplugging and re-plugging… I'm hoping for a more modern answer, though.



  • sorry misread who was doing the arp.  you could try 'tcpdump -lnvvv' instead.  what i think must be going on is that the local IP (192.168.x.x) would not be arping for the palmetto address normally, much less getting a reply, so this must be coming to/from the tunnel.  What does 'clog /var/log/openvpn.log' show?


  • Rebel Alliance Developer Netgate

    Why are you using "dev tap" on the client? That is probably your problem. And why the server-bridge line on the server side?

    iirc, tap devices are more for bridging than routing. You probably want that to be tun.

    If you are trying to bridge, you want that to be tap on both sides, but I'm not familiar with a bridged setup.



  • Good point about tap vs tun.



  • @jimp:

    Why are you using "dev tap" on the client? That is probably your problem. And why the server-bridge line on the server side?

    iirc, tap devices are more for bridging than routing. You probably want that to be tun.

    If you are trying to bridge, you want that to be tap on both sides, but I'm not familiar with a bridged setup.

    I'm using tap intentionally because I do use bridging.  (It's not necessary at this office, but I have been using a standard configuration across all of my installations - much less hassle that way!)  One of my offices has a permanent (or, at least, we WISH it were permanent) IPSec tunnel to a hosting company.  The hosting company's road-warrior IPSec VPN (from a company that rhymes with Crisco) is much less reliable than the site-to-site, so I decided to have my road warriors connect to the office network and access the hosted network via bridging.  It's been very stable and very easy to add/remove clients, as opposed to dealing with the hosting people…  Once I got that set up and working, I standardized my OpenVPN setup; this particular office doesn't currently need bridging, but it's been working smoothly.

    I can change the OpenVPN setup to use tun instead; again, I'd prefer not to unless I'm sure it will fix this problem, because it means modifying all the client machines (most of which are at the doctors'/employees' houses.)  I have this same setup in nearly a dozen offices, running for over two years now, and this is the first time anything like this has come up.  Perhaps this is a hidden danger of using bridging, in which case I should expect to get hit with it at my other sites sooner or later; meanwhile, I'd like to get to the bottom of why it happened here.

    1. It's persistent across reboots.
    2. It doesn't matter whether any OpenVPN clients are connected or not.
    3. As far as I can tell, it's only this one IP address that's affected. (Google.com and forum.pfsense.org, for instance, work just fine.)

    These things make me think that pfSense has stored this (bad) routing information somewhere.  At this point it's more of an intellectual curiosity to me to find out where.  Should I perhaps ask on a FreeBSD forum?



  • @danswartz:

    sorry misread who was doing the arp.  you could try 'tcpdump -lnvvv' instead.  what i think must be going on is that the local IP (192.168.x.x) would not be arping for the palmetto address normally, much less getting a reply, so this must be coming to/from the tunnel.  What does 'clog /var/log/openvpn.log' show?

    
    # clog /var/log/openvpn.log
    Jan  1 22:55:25 pfsense openvpn[345]: OpenVPN 2.0.6 i386-portbld-freebsd7.2 [SSL] [LZO] built on Dec  4 2009
    Jan  1 22:55:25 pfsense openvpn[345]: WARNING: file '/var/etc/openvpn_server0.key' is group or others accessible
    Jan  1 22:55:25 pfsense openvpn[345]: WARNING: Since you are using --dev tap, the second argument to --ifconfig must be a netmask, for example something like 255.255.255.0\. (silence this warning with --ifconfig-nowarn)
    Jan  1 22:55:25 pfsense openvpn[345]: TUN/TAP device /dev/tap0 opened
    Jan  1 22:55:25 pfsense openvpn[345]: /sbin/ifconfig tap0 192.168.35.1 netmask 192.168.35.2 mtu 1500 up
    Jan  1 22:55:25 pfsense openvpn[345]: /etc/rc.filter_configure tap0 1500 1574 192.168.35.1 192.168.35.2 init
    Jan  1 22:55:25 pfsense openvpn[352]: UDPv4 link local (bound): [undef]:1194
    Jan  1 22:55:25 pfsense openvpn[352]: UDPv4 link remote: [undef]
    Jan  1 22:55:25 pfsense openvpn[352]: Initialization Sequence Completed
    
    


  • Just thought I'd post the eventual solution, in case anyone else ever has the same problem.  I added a static route:

    Interface  Network  Gateway  Description

    WAN 216.251.231.64/32 (our gateway) Palmetto

    • in other words, I added an explicit rule to reinforce what should be happening anyway.  And now it works.  What caused the original problem, I don't know…

Locked