Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Xen DMZ config dropping packets between LAN and DMZ

    Scheduled Pinned Locked Moved Firewalling
    2 Posts 1 Posters 710 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K Offline
      kshots
      last edited by

      Ok… I'll admit I'm at the limit of my understanding on this. Here's what I have:

      pfsense box is a VM under a host with a virtualized (emulated) LAN interface and a physical WAN interface (ie, host cannot see WAN hardware, it's physically passed to the VM). I've recently added another bridge to the equation, which I've called my "DMZ". pfsense has this DMZ defined as a separate subnet defined as 10.4.14.2/28.

      Connected to this DMZ is another VM sharing the above bridge that I call "jailer". This VM is running FreeBSD and runs a series of jails. The host does not have a DMZ address, but the jails do - and each jail is configured as a /32 subnet. Oddly enough, this appears to work (I would be inclined to think communication would be impossible with a /32 subnet, so I'm scratching my head as far as why any traffic gets through). Here's the rub, though... the firewall cuts all such traffic off (specifically, SSH was attempted) within 1 minute of the initial connection with a TCP:PA block in the logs.

      Ok, now for more specific information. I'll work this from the top down, starting with the physical host config.

      Physical host network config:```
      gorgon ~ # ifconfig
      bridge0: flags=4163<up,broadcast,running,multicast>  mtu 1500
              inet 10.4.12.19  netmask 255.255.255.0  broadcast 10.4.12.255
              inet6 fe80::5888:b9ff:fef7:457  prefixlen 64  scopeid 0x20
              inet6 mypublicnet::2  prefixlen 64  scopeid 0x0 <global>ether d0:50:99:3b:c4:6d  txqueuelen 1000  (Ethernet)
              RX packets 742458407  bytes 3253065238235 (2.9 TiB)
              RX errors 0  dropped 92  overruns 0  frame 0
              TX packets 560651070  bytes 1574077522620 (1.4 TiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

      bridge1: flags=4163<up,broadcast,running,multicast>  mtu 1500
              inet6 fe80::5c0a:a5ff:fe82:d9a0  prefixlen 64  scopeid 0x20
              ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
              RX packets 14  bytes 392 (392.0 B)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 0  bytes 0 (0.0 B)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

      lo: flags=73<up,loopback,running>  mtu 65536
              inet 127.0.0.1  netmask 255.0.0.0
              inet6 ::1  prefixlen 128  scopeid 0x10 <host>loop  txqueuelen 1  (Local Loopback)
              RX packets 7694015  bytes 128085900456 (119.2 GiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 7694015  bytes 128085900456 (119.2 GiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

      network: flags=4163<up,broadcast,running,multicast>  mtu 1500
              ether d0:50:99:3b:c4:6d  txqueuelen 1000  (Ethernet)
              RX packets 2882631830  bytes 3330552643602 (3.0 TiB)
              RX errors 0  dropped 0  overruns 108047  frame 0
              TX packets 3453753129  bytes 4909360242646 (4.4 TiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
              device memory 0xf0400000-f047ffff</up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></global></up,broadcast,running,multicast>

      
Next, my pfsense xen config:```
gorgon ~ # cat /etc/xen/pfsense.cfg 
memory = 1024
vcpus = 2
acpi = 1
apic = 1
name = "pfsense"

uuid = "d590e33f-cc36-4aac-a509-e7b0d7afd4b3"

# PVHVM stuff
builder = "hvm"
firmware_override = "hvmloader"
boot = "c"

vif = [ 'mac=00:16:3e:ae:bd:cc,bridge=bridge0,script=/etc/xen/scripts/vif-bridge.noTXoffload', 'mac=00:16:3e:f8:de:ad,bridge=bridge1,script=/etc/xen/scripts/vif-bridge.noTXoffload' ]
disk = [ '/dev/mapper/vg0-efreet,raw,hda,w' ]
device_model_version = 'qemu-xen-traditional'

# Necessary for getting the serial console in `xm console`
serial = "pty"
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'
pci = [ '00:19.0' ]
```Little special here other than the scripts which disable TX offload for the virtual interfaces (resolves bad checksum errors detected by pfsense). Next, the **jailer** vm:```
gorgon ~ # cat /etc/xen/jailer.cfg 
memory = 4096
vcpus = 4
acpi = 1
apic = 1
name = "jailer"

uuid = "b0634902-82d1-421a-8f73-ec702d28bd1c"

# PVHVM stuff
builder = "hvm"
firmware_override = "hvmloader"
boot = "c"

vif = [ 'mac=00:16:3e:fe:ce:af,bridge=bridge0', 'mac=00:16:3e:8f:df:cf,bridge=bridge1' ]
disk = [ 'format=raw, vdev=xvda, access=rw, target=/dev/mapper/vg0-jailer' ]

device_model_version = 'qemu-xen-traditional'

# Necessary for getting the serial console in `xm console`
serial = "pty"
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'
```Really, nothing special going on here… the pfsense config:```
 WAN (wan)       -> em0        -> v4: my.public.ipv4.address/29
 LAN (lan)       -> xn0        -> v4: 10.4.12.10/24
                                  v6: my:public:ipv6:address::1/64
 HENETV6 (opt1)  -> gif0       -> v6: my:ipv6:tunnel:endpoint::2/128
 DMZ (opt2)      -> xn1        -> v4: 10.4.14.2/28
```My jailer config:```
root@jailer: # ifconfig
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21 <performnud,auto_linklocal>xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:fe:ce:af
        inet 10.4.12.21 netmask 0xffffff00 broadcast 10.4.12.255 
        inet6 fe80::216:3eff:fefe:ceaf%xn0 prefixlen 64 scopeid 0x2 
        inet6 2001:470:5:745::5 prefixlen 64 
        inet 10.4.12.22 netmask 0xffffffff broadcast 10.4.12.22 
        inet6 2001:470:5:745::7 prefixlen 128 
        inet 10.4.12.24 netmask 0xffffffff broadcast 10.4.12.24 
        inet6 2001:470:5:745::9 prefixlen 128 
        inet 10.4.12.23 netmask 0xffffffff broadcast 10.4.12.23 
        inet6 2001:470:5:745::8 prefixlen 128 
        inet 10.4.12.20 netmask 0xffffffff broadcast 10.4.12.20 
        inet6 2001:470:5:745::a prefixlen 128 
        inet 10.4.12.26 netmask 0xffffffff broadcast 10.4.12.26 
        inet6 2001:470:5:745::c prefixlen 128 
        inet6 2001:470:5:745::6 prefixlen 128 
        inet6 2001:470:5:745::b prefixlen 128 
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
        status: active
xn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:8f:df:cf
        inet 10.4.14.8 netmask 0xfffffff0 broadcast 10.4.14.15 
        inet 10.4.14.3 netmask 0xffffffff broadcast 10.4.14.3 
        inet 10.4.14.4 netmask 0xffffffff broadcast 10.4.14.4 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet manual
        status: active
lo1: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.2 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo2: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.3 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo3: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.4 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo4: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.5 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo5: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.6 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo6: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.7 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo7: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.8 netmask 0xffffffff 
        nd6 options=29 <performnud,ifdisabled,auto_linklocal>root@jailer: # jls
   JID  IP Address      Hostname                      Path
     1  10.4.12.22      ns1.mydomain.com            /usr/jails/ns1.mydomain.com
     3  10.4.12.24      sql.mydomain.com            /usr/jails/sql.mydomain.com
     5  10.4.12.23      ldap1.mydomain.com          /usr/jails/ldap1.mydomain.com
     6  10.4.12.20      kerberos1.mydomain.com      /usr/jails/kerberos1.mydomain.com
     7  10.4.12.26      imap.mydomain.com           /usr/jails/imap.mydomain.com
    12  10.4.14.3       web.mydomain.com            /usr/jails/web.mydomain.com
    13  10.4.14.4       mail1.mydomain.com          /usr/jails/mail1.mydomain.com</performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast>
```As you can see, I've added a /28 address to the jailer host (10.4.14.8/28) so I could at least conceive of a communication path between the pfsense vm and the jailer. This seems to have had no effect.

Here's what works:

1\. Communication from LAN to DMZ initiates (but gets interrupted)
2\. Communication from DMZ to LAN appears to work (only ping has been tested)
3\. Communication from pfsense to DMZ appears to work (ping test only)

So… what's causing pfsense to start dropping packets?
      1 Reply Last reply Reply Quote 0
      • K Offline
        kshots
        last edited by

        Hrm… thinking about this setup, is it possible that the jailer host, which knows about the 10.4.12.0/24 network, could be cutting the pfsense router out of the loop after initial contact? ie, communication from host 10.4.12.6/24 to 10.4.14.4/28 must go through the router, as 10.4.12.6 has no interface on the 10.4.14.0/28 network… but the response doesn't need to go through the router, as the "jailer" knows about both networks and may route directly from 10.4.14.4 to 10.4.12.6 via 10.4.12.21 (the jailer's 10.4.12.0/24 address)... I'm not entirely sure how to resolve this kind of problem... Would I be required to create another VM that is only aware of the 10.4.14.0/28 subnet for the DMZ - essentially a second "jailer" running jails that are only aware of that subnet and the router? Is there another approach?

        1 Reply Last reply Reply Quote 0
        • First post
          Last post
        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.