ESX/Pfsense bridge mode/Arp response takes too long



  • I have set up pfSense as one of my VMs in ESX 3.5.
    This VM has a connection to my real network and another connection to a virtual network where I want to place all my machines that need to be protected. I have set up pfSense in bridge mode for packet filtering. I have a windows machine (VM) attached to the virtual network and I have configured pfSense to allow all outbound traffic and certain inbound traffic (RDP). I noticed that I cannot access the machine from the outside using MS RDP unless I log in to the machine using VMWare Infrastructure client and initiate an outbound connection first. Then I noticed that it takes too long before the outbound connection gets established, one to a few minutes sometimes due to the arp request (for the gateway address) being delayed so long. I have another VM that is connected to the real network (it is not behind PFSense) and works absolutely fine.
    I hope someone can help out with this strange issue ?
    Below is the captured traffic from PFSense when the machine tries to establish an outside connection and it takes 4 mins for the reply to come back.

    15:15:43.247838 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:43.248443 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:48.317403 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:48.317890 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:53.818591 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:53.818902 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:59.317226 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:15:59.317509 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:01.757649 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 243: (tos 0x0, ttl 128, id 24377, offset 0, flags [none], proto UDP (17), length 229) 10.11.1.96.138 > 10.11.255.255.138: [udp sum ok]

    NBT UDP PACKET(138) Res=0x1102 ID=0x98E0 IP=10 (0xa).11 (0xb).1 (0x1).96 (0x60) Port=138 (0x8a) Length=187 (0xbb) Res2=0x0
    SourceName=XPVM2005        NameType=0x20 (Server)
    DestName=WORKGROUP      NameType=0x1E (Browser Server)

    SMB PACKET: SMBtrans (REQUEST)
    SMB Command  =  0x25
    Error class  =  0x0
    Error code    =  0 (0x0)
    Flags1        =  0x0
    Flags2        =  0x0
    Tree ID      =  0 (0x0)
    Proc ID      =  0 (0x0)
    UID          =  0 (0x0)
    MID          =  0 (0x0)
    Word Count    =  17 (0x11)
    TotParamCnt=0 (0x0)
    TotDataCnt=33 (0x21)
    MaxParmCnt=0 (0x0)
    MaxDataCnt=0 (0x0)
    MaxSCnt=0 (0x0)
    TransFlags=0x0
    Res1=0x3E8
    Res2=0x0
    Res3=0x0
    ParamCnt=0 (0x0)
    ParamOff=0 (0x0)
    DataCnt=33 (0x21)
    DataOff=86 (0x56)
    SUCnt=3 (0x3)
    Data: (6 bytes)
    [000] 01 00 00 00 02 00                                \001\000\000\000\002\000
    smb_bcc=50
    Name=\MAILSLOT\BROWSE
    BROWSE PACKET
    BROWSE PACKET:
    Type=0xF (LocalMasterAnnouncement)
    UpdateCount=0x8000
    Res1=0xFC
    AnnounceInterval=10 (0xa)
    Name=XPVM2005        NameType=0x00 (Workstation)
    MajorVersion=0x5
    MinorVersion=0x1
    ServerType=0x51007
    ElectionVersion=0x10F
    BrowserConstant=0xAA55
    Data: (1 bytes)
    [000] 00                                                \000

    15:16:04.817813 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:04.818339 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:10.318662 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:10.318991 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:15.817373 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:15.817719 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:21.317792 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:21.318163 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:26.817474 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:26.817802 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:32.317599 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:32.317963 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:37.818408 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:37.819056 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:43.317404 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:43.317923 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:48.817780 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:48.818279 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:54.317317 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:54.317667 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:59.817106 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:16:59.817543 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:05.318277 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:05.318682 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:10.817347 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:10.817780 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:16.319128 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:16.319743 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:21.817611 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:21.818189 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:27.317442 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:27.318120 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:32.818605 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:32.818984 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:38.319033 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:38.319341 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:43.818909 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:43.819309 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:49.317599 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:49.318154 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:54.747264 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:17:54.747583 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:00.247443 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:00.247715 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:05.747495 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:05.747877 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:11.247677 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:11.248087 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:16.747323 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:16.747609 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:22.247294 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:22.247764 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:27.747339 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:27.747628 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:33.248269 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:33.248662 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:38.747458 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:38.747766 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:44.247206 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:44.247493 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:49.749670 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:49.750045 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:55.247289 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:18:55.247604 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:00.748857 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:00.749147 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:06.247580 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:06.247894 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:11.747568 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:11.747855 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:17.247309 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:17.247760 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:22.747465 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:22.747923 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:28.248576 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:28.248833 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:33.747488 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:33.747906 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:39.248718 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:39.249122 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:44.747872 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:44.748215 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:50.248585 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:50.248924 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:55.747158 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:19:55.747749 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:20:01.248141 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:20:01.248613 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:20:06.748045 00:0c:29:34:b9:85 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.11.1.1 tell 10.11.1.96
    15:20:06.748749 00:09:6b:63:20:8b > 00:0c:29:34:b9:85, ethertype ARP (0x0806), length 60: arp reply 10.11.1.1 is-at 00:00:5e:00:01:6f
    15:20:06.750060 00:0c:29:34:b9:85 > 00:00:5e:00:01:6f, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 24422, offset 0, flags [none], proto ICMP (1), length 60) 10.11.1.96 > 4.2.2.1: ICMP echo request, id 512, seq 54272, length 40



  • Dunno if it helps but I had to set the virtual switches to allow promiscuous mode on my ESXi box before pfsense could correctly forward traffic (although i'm running it as a filtering bridge)



  • Thanks for the reply, I have already set to promiscous mode in ESX, otherwise it would not work at all. The current setup does work in general except the weird arp problem and yes I am using it as a filtering bridge.


Locked