Xen DMZ config dropping packets between LAN and DMZ
-
Ok… I'll admit I'm at the limit of my understanding on this. Here's what I have:
pfsense box is a VM under a host with a virtualized (emulated) LAN interface and a physical WAN interface (ie, host cannot see WAN hardware, it's physically passed to the VM). I've recently added another bridge to the equation, which I've called my "DMZ". pfsense has this DMZ defined as a separate subnet defined as 10.4.14.2/28.
Connected to this DMZ is another VM sharing the above bridge that I call "jailer". This VM is running FreeBSD and runs a series of jails. The host does not have a DMZ address, but the jails do - and each jail is configured as a /32 subnet. Oddly enough, this appears to work (I would be inclined to think communication would be impossible with a /32 subnet, so I'm scratching my head as far as why any traffic gets through). Here's the rub, though... the firewall cuts all such traffic off (specifically, SSH was attempted) within 1 minute of the initial connection with a TCP:PA block in the logs.
Ok, now for more specific information. I'll work this from the top down, starting with the physical host config.
Physical host network config:```
gorgon ~ # ifconfig
bridge0: flags=4163<up,broadcast,running,multicast> mtu 1500
inet 10.4.12.19 netmask 255.255.255.0 broadcast 10.4.12.255
inet6 fe80::5888:b9ff:fef7:457 prefixlen 64 scopeid 0x20
inet6 mypublicnet::2 prefixlen 64 scopeid 0x0 <global>ether d0:50:99:3b:c4:6d txqueuelen 1000 (Ethernet)
RX packets 742458407 bytes 3253065238235 (2.9 TiB)
RX errors 0 dropped 92 overruns 0 frame 0
TX packets 560651070 bytes 1574077522620 (1.4 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0bridge1: flags=4163<up,broadcast,running,multicast> mtu 1500
inet6 fe80::5c0a:a5ff:fe82:d9a0 prefixlen 64 scopeid 0x20
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 14 bytes 392 (392.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<up,loopback,running> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10 <host>loop txqueuelen 1 (Local Loopback)
RX packets 7694015 bytes 128085900456 (119.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7694015 bytes 128085900456 (119.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0network: flags=4163<up,broadcast,running,multicast> mtu 1500
ether d0:50:99:3b:c4:6d txqueuelen 1000 (Ethernet)
RX packets 2882631830 bytes 3330552643602 (3.0 TiB)
RX errors 0 dropped 0 overruns 108047 frame 0
TX packets 3453753129 bytes 4909360242646 (4.4 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xf0400000-f047ffff</up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></global></up,broadcast,running,multicast>Next, my pfsense xen config:``` gorgon ~ # cat /etc/xen/pfsense.cfg memory = 1024 vcpus = 2 acpi = 1 apic = 1 name = "pfsense" uuid = "d590e33f-cc36-4aac-a509-e7b0d7afd4b3" # PVHVM stuff builder = "hvm" firmware_override = "hvmloader" boot = "c" vif = [ 'mac=00:16:3e:ae:bd:cc,bridge=bridge0,script=/etc/xen/scripts/vif-bridge.noTXoffload', 'mac=00:16:3e:f8:de:ad,bridge=bridge1,script=/etc/xen/scripts/vif-bridge.noTXoffload' ] disk = [ '/dev/mapper/vg0-efreet,raw,hda,w' ] device_model_version = 'qemu-xen-traditional' # Necessary for getting the serial console in `xm console` serial = "pty" on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'destroy' pci = [ '00:19.0' ] ```Little special here other than the scripts which disable TX offload for the virtual interfaces (resolves bad checksum errors detected by pfsense). Next, the **jailer** vm:``` gorgon ~ # cat /etc/xen/jailer.cfg memory = 4096 vcpus = 4 acpi = 1 apic = 1 name = "jailer" uuid = "b0634902-82d1-421a-8f73-ec702d28bd1c" # PVHVM stuff builder = "hvm" firmware_override = "hvmloader" boot = "c" vif = [ 'mac=00:16:3e:fe:ce:af,bridge=bridge0', 'mac=00:16:3e:8f:df:cf,bridge=bridge1' ] disk = [ 'format=raw, vdev=xvda, access=rw, target=/dev/mapper/vg0-jailer' ] device_model_version = 'qemu-xen-traditional' # Necessary for getting the serial console in `xm console` serial = "pty" on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'destroy' ```Really, nothing special going on here… the pfsense config:``` WAN (wan) -> em0 -> v4: my.public.ipv4.address/29 LAN (lan) -> xn0 -> v4: 10.4.12.10/24 v6: my:public:ipv6:address::1/64 HENETV6 (opt1) -> gif0 -> v6: my:ipv6:tunnel:endpoint::2/128 DMZ (opt2) -> xn1 -> v4: 10.4.14.2/28 ```My jailer config:``` root@jailer: # ifconfig lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 <performnud,auto_linklocal>xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:fe:ce:af inet 10.4.12.21 netmask 0xffffff00 broadcast 10.4.12.255 inet6 fe80::216:3eff:fefe:ceaf%xn0 prefixlen 64 scopeid 0x2 inet6 2001:470:5:745::5 prefixlen 64 inet 10.4.12.22 netmask 0xffffffff broadcast 10.4.12.22 inet6 2001:470:5:745::7 prefixlen 128 inet 10.4.12.24 netmask 0xffffffff broadcast 10.4.12.24 inet6 2001:470:5:745::9 prefixlen 128 inet 10.4.12.23 netmask 0xffffffff broadcast 10.4.12.23 inet6 2001:470:5:745::8 prefixlen 128 inet 10.4.12.20 netmask 0xffffffff broadcast 10.4.12.20 inet6 2001:470:5:745::a prefixlen 128 inet 10.4.12.26 netmask 0xffffffff broadcast 10.4.12.26 inet6 2001:470:5:745::c prefixlen 128 inet6 2001:470:5:745::6 prefixlen 128 inet6 2001:470:5:745::b prefixlen 128 nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual status: active xn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:8f:df:cf inet 10.4.14.8 netmask 0xfffffff0 broadcast 10.4.14.15 inet 10.4.14.3 netmask 0xffffffff broadcast 10.4.14.3 inet 10.4.14.4 netmask 0xffffffff broadcast 10.4.14.4 nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet manual status: active lo1: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.2 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo2: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.3 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo3: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.4 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo4: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.5 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo5: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.6 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo6: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.7 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>lo7: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.8 netmask 0xffffffff nd6 options=29 <performnud,ifdisabled,auto_linklocal>root@jailer: # jls JID IP Address Hostname Path 1 10.4.12.22 ns1.mydomain.com /usr/jails/ns1.mydomain.com 3 10.4.12.24 sql.mydomain.com /usr/jails/sql.mydomain.com 5 10.4.12.23 ldap1.mydomain.com /usr/jails/ldap1.mydomain.com 6 10.4.12.20 kerberos1.mydomain.com /usr/jails/kerberos1.mydomain.com 7 10.4.12.26 imap.mydomain.com /usr/jails/imap.mydomain.com 12 10.4.14.3 web.mydomain.com /usr/jails/web.mydomain.com 13 10.4.14.4 mail1.mydomain.com /usr/jails/mail1.mydomain.com</performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,ifdisabled,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,tso4,lro></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast> ```As you can see, I've added a /28 address to the jailer host (10.4.14.8/28) so I could at least conceive of a communication path between the pfsense vm and the jailer. This seems to have had no effect. Here's what works: 1\. Communication from LAN to DMZ initiates (but gets interrupted) 2\. Communication from DMZ to LAN appears to work (only ping has been tested) 3\. Communication from pfsense to DMZ appears to work (ping test only) So… what's causing pfsense to start dropping packets?
-
Hrm… thinking about this setup, is it possible that the jailer host, which knows about the 10.4.12.0/24 network, could be cutting the pfsense router out of the loop after initial contact? ie, communication from host 10.4.12.6/24 to 10.4.14.4/28 must go through the router, as 10.4.12.6 has no interface on the 10.4.14.0/28 network… but the response doesn't need to go through the router, as the "jailer" knows about both networks and may route directly from 10.4.14.4 to 10.4.12.6 via 10.4.12.21 (the jailer's 10.4.12.0/24 address)... I'm not entirely sure how to resolve this kind of problem... Would I be required to create another VM that is only aware of the 10.4.14.0/28 subnet for the DMZ - essentially a second "jailer" running jails that are only aware of that subnet and the router? Is there another approach?