Xen DMZ config dropping packets between LAN and DMZ



  • Ok… I'll admit I'm at the limit of my understanding on this. Here's what I have:

    pfsense box is a VM under a host with a virtualized (emulated) LAN interface and a physical WAN interface (ie, host cannot see WAN hardware, it's physically passed to the VM). I've recently added another bridge to the equation, which I've called my "DMZ". pfsense has this DMZ defined as a separate subnet defined as 10.4.14.2/28.

    Connected to this DMZ is another VM sharing the above bridge that I call "jailer". This VM is running FreeBSD and runs a series of jails. The host does not have a DMZ address, but the jails do - and each jail is configured as a /32 subnet. Oddly enough, this appears to work (I would be inclined to think communication would be impossible with a /32 subnet, so I'm scratching my head as far as why any traffic gets through). Here's the rub, though... the firewall cuts all such traffic off (specifically, SSH was attempted) within 1 minute of the initial connection with a TCP:PA block in the logs.

    Ok, now for more specific information. I'll work this from the top down, starting with the physical host config.

    Physical host network config:```
    gorgon ~ # ifconfig
    bridge0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet 10.4.12.19  netmask 255.255.255.0  broadcast 10.4.12.255
            inet6 fe80::5888:b9ff:fef7:457  prefixlen 64  scopeid 0x20
            inet6 mypublicnet::2  prefixlen 64  scopeid 0x0 <global>ether d0:50:99:3b:c4:6d  txqueuelen 1000  (Ethernet)
            RX packets 742458407  bytes 3253065238235 (2.9 TiB)
            RX errors 0  dropped 92  overruns 0  frame 0
            TX packets 560651070  bytes 1574077522620 (1.4 TiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    bridge1: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::5c0a:a5ff:fe82:d9a0  prefixlen 64  scopeid 0x20
            ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
            RX packets 14  bytes 392 (392.0 B)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 0  bytes 0 (0.0 B)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    lo: flags=73<up,loopback,running>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10 <host>loop  txqueuelen 1  (Local Loopback)
            RX packets 7694015  bytes 128085900456 (119.2 GiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 7694015  bytes 128085900456 (119.2 GiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    network: flags=4163<up,broadcast,running,multicast>  mtu 1500
            ether d0:50:99:3b:c4:6d  txqueuelen 1000  (Ethernet)
            RX packets 2882631830  bytes 3330552643602 (3.0 TiB)
            RX errors 0  dropped 0  overruns 108047  frame 0
            TX packets 3453753129  bytes 4909360242646 (4.4 TiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
            device memory 0xf0400000-f047ffff</up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></global></up,broadcast,running,multicast>

    
    Next, my pfsense xen config:```
    gorgon ~ # cat /etc/xen/pfsense.cfg 
    memory = 1024
    vcpus = 2
    acpi = 1
    apic = 1
    name = "pfsense"
    
    uuid = "d590e33f-cc36-4aac-a509-e7b0d7afd4b3"
    
    # PVHVM stuff
    builder = "hvm"
    firmware_override = "hvmloader"
    boot = "c"
    
    vif = [ 'mac=00:16:3e:ae:bd:cc,bridge=bridge0,script=/etc/xen/scripts/vif-bridge.noTXoffload', 'mac=00:16:3e:f8:de:ad,bridge=bridge1,script=/etc/xen/scripts/vif-bridge.noTXoffload' ]
    disk = [ '/dev/mapper/vg0-efreet,raw,hda,w' ]
    device_model_version = 'qemu-xen-traditional'
    
    # Necessary for getting the serial console in `xm console`
    serial = "pty"
    on_poweroff = 'destroy'
    on_reboot = 'restart'
    on_crash = 'destroy'
    pci = [ '00:19.0' ]
    ```Little special here other than the scripts which disable TX offload for the virtual interfaces (resolves bad checksum errors detected by pfsense). Next, the **jailer** vm:```
    gorgon ~ # cat /etc/xen/jailer.cfg 
    memory = 4096
    vcpus = 4
    acpi = 1
    apic = 1
    name = "jailer"
    
    uuid = "b0634902-82d1-421a-8f73-ec702d28bd1c"
    
    # PVHVM stuff
    builder = "hvm"
    firmware_override = "hvmloader"
    boot = "c"
    
    vif = [ 'mac=00:16:3e:fe:ce:af,bridge=bridge0', 'mac=00:16:3e:8f:df:cf,bridge=bridge1' ]
    disk = [ 'format=raw, vdev=xvda, access=rw, target=/dev/mapper/vg0-jailer' ]
    
    device_model_version = 'qemu-xen-traditional'
    
    # Necessary for getting the serial console in `xm console`
    serial = "pty"
    on_poweroff = 'destroy'
    on_reboot = 'restart'
    on_crash = 'destroy'
    ```Really, nothing special going on here… the pfsense config:```
     WAN (wan)       -> em0        -> v4: my.public.ipv4.address/29
     LAN (lan)       -> xn0        -> v4: 10.4.12.10/24
                                      v6: my:public:ipv6:address::1/64
     HENETV6 (opt1)  -> gif0       -> v6: my:ipv6:tunnel:endpoint::2/128
     DMZ (opt2)      -> xn1        -> v4: 10.4.14.2/28
    ```My jailer config:```
    root@jailer: # ifconfig
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
            inet 127.0.0.1 netmask 0xff000000 
            nd6 options=21 <performnud,auto_linklocal>xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:fe:ce:af
            inet 10.4.12.21 netmask 0xffffff00 broadcast 10.4.12.255 
            inet6 fe80::216:3eff:fefe:ceaf%xn0 prefixlen 64 scopeid 0x2 
            inet6 2001:470:5:745::5 prefixlen 64 
            inet 10.4.12.22 netmask 0xffffffff broadcast 10.4.12.22 
            inet6 2001:470:5:745::7 prefixlen 128 
            inet 10.4.12.24 netmask 0xffffffff broadcast 10.4.12.24 
            inet6 2001:470:5:745::9 prefixlen 128 
            inet 10.4.12.23 netmask 0xffffffff broadcast 10.4.12.23 
            inet6 2001:470:5:745::8 prefixlen 128 
            inet 10.4.12.20 netmask 0xffffffff broadcast 10.4.12.20 
            inet6 2001:470:5:745::a prefixlen 128 
            inet 10.4.12.26 netmask 0xffffffff broadcast 10.4.12.26 
            inet6 2001:470:5:745::c prefixlen 128 
            inet6 2001:470:5:745::6 prefixlen 128 
            inet6 2001:470:5:745::b prefixlen 128 
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
            status: active
    xn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:8f:df:cf
            inet 10.4.14.8 netmask 0xfffffff0 broadcast 10.4.14.15 
            inet 10.4.14.3 netmask 0xffffffff broadcast 10.4.14.3 
            inet 10.4.14.4 netmask 0xffffffff broadcast 10.4.14.4 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet manual
            status: active
    lo1: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.2 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo2: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.3 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo3: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.4 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo4: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.5 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo5: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.6 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo6: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.7 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo7: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.8 netmask 0xffffffff 
            nd6 options=29 <performnud,ifdisabled,auto_linklocal>root@jailer: # jls
       JID  IP Address      Hostname                      Path
         1  10.4.12.22      ns1.mydomain.com            /usr/jails/ns1.mydomain.com
         3  10.4.12.24      sql.mydomain.com            /usr/jails/sql.mydomain.com
         5  10.4.12.23      ldap1.mydomain.com          /usr/jails/ldap1.mydomain.com
         6  10.4.12.20      kerberos1.mydomain.com      /usr/jails/kerberos1.mydomain.com
         7  10.4.12.26      imap.mydomain.com           /usr/jails/imap.mydomain.com
        12  10.4.14.3       web.mydomain.com            /usr/jails/web.mydomain.com
        13  10.4.14.4       mail1.mydomain.com          /usr/jails/mail1.mydomain.com</performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast>
    ```As you can see, I've added a /28 address to the jailer host (10.4.14.8/28) so I could at least conceive of a communication path between the pfsense vm and the jailer. This seems to have had no effect.
    
    Here's what works:
    
    1\. Communication from LAN to DMZ initiates (but gets interrupted)
    2\. Communication from DMZ to LAN appears to work (only ping has been tested)
    3\. Communication from pfsense to DMZ appears to work (ping test only)
    
    So… what's causing pfsense to start dropping packets?


  • Hrm… thinking about this setup, is it possible that the jailer host, which knows about the 10.4.12.0/24 network, could be cutting the pfsense router out of the loop after initial contact? ie, communication from host 10.4.12.6/24 to 10.4.14.4/28 must go through the router, as 10.4.12.6 has no interface on the 10.4.14.0/28 network… but the response doesn't need to go through the router, as the "jailer" knows about both networks and may route directly from 10.4.14.4 to 10.4.12.6 via 10.4.12.21 (the jailer's 10.4.12.0/24 address)... I'm not entirely sure how to resolve this kind of problem... Would I be required to create another VM that is only aware of the 10.4.14.0/28 subnet for the DMZ - essentially a second "jailer" running jails that are only aware of that subnet and the router? Is there another approach?


Log in to reply