LAN clients on KVM bridge can't connect through firewall
-
Hi,
I have pfSense installed as a guest under KVM running on Debian 9.2.
There is one other active guest, running FreeBSD 11.1.
The KVM server has a a software bridge at 192.168.1.26/24. It's associated with a NIC on that subnet:
auto eno1 iface eno1 inet manual auto br0 iface br0 inet static address 192.168.1.26 netmask 255.255.255.0 gateway 192.168.1.254 bridge_ports eno1 bridge_stp off bridge_maxwait 0 bridge_fd 0The pfSense LAN is on a virtual NIC connected to the bridge on 192.168.1.0/24.
This is what things look like on the KVM server:
[orac.1145] $ ifconfig eno1 eno1: flags=4163<up,broadcast,running,multicast>mtu 1500 ether 50:46:5d:76:25:9b txqueuelen 1000 (Ethernet) RX packets 3131064 bytes 666408021 (635.5 MiB) RX errors 0 dropped 40512 overruns 0 frame 0 TX packets 2085772 bytes 2071053850 (1.9 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 20 memory 0xf7c00000-f7c20000 [orac.1146] $ ifconfig br0 br0: flags=4163<up,broadcast,running,multicast>mtu 1500 inet 192.168.1.26 netmask 255.255.255.0 broadcast 192.168.1.255 ether 50:46:5d:76:25:9b txqueuelen 1000 (Ethernet) RX packets 2932839 bytes 567227483 (540.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 874493 bytes 1931119811 (1.7 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [orac.1147] $ ifconfig vnet3 vnet3: flags=4163<up,broadcast,running,multicast>mtu 1500 ether fe:54:00:de:4a:fe txqueuelen 1000 (Ethernet) RX packets 26190 bytes 17428082 (16.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 35805 bytes 4633540 (4.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</up,broadcast,running,multicast></up,broadcast,running,multicast></up,broadcast,running,multicast>On the pfSense KVM guest, the network looks like this (vtnet0 is unused – it was the "dummy WAN" initially, before ppp was configured).
[2.4.0-RELEASE][admin@pfSense.my.domain]/root: ifconfig -a vtnet0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=6c07bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6>ether 52:54:00:85:d0:2f hwaddr 52:54:00:85:d0:2f nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active vtnet1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=6c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6>ether 52:54:00:de:4a:fe hwaddr 52:54:00:de:4a:fe inet6 fe80::5054:ff:fede:4afe%vtnet1 prefixlen 64 scopeid 0x2 inet 192.168.1.37 netmask 0xffffff00 broadcast 192.168.1.255 inet 192.168.1.254 netmask 0xffffffff broadcast 192.168.1.254 nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 <performnud,auto_linklocal>groups: lo enc0: flags=0<> metric 0 mtu 1536 nd6 options=21 <performnud,auto_linklocal>groups: enc pflog0: flags=100 <promisc>metric 0 mtu 33160 groups: pflog pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync syncpeer: 224.0.0.240 maxupd: 128 defer: on syncok: 1 ppp0: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492 inet 121.44.11.88 --> 10.64.64.0 netmask 0xffffffff inet6 fe80::c88c:8100:109:bd37%ppp0 prefixlen 64 scopeid 0x7 nd6 options=21 <performnud,auto_linklocal></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></promisc></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6></broadcast,simplex,multicast>Noth that the default route for the 192.168.1.0/24 network is at 192.168.1.254 and an IP alias on vtnet1.
The WAN is connected to a ppp link on a 3G USB modem. No known problems.
The other active KVM guest is connected to the same bridge as the LAN at 192.168.1.36.
The packet filter rules look like this ("pfctl -s all"):
TRANSLATION RULES: no nat proto carp all nat-anchor "natearly/*" all nat-anchor "natrules/*" all nat on ppp0 inet from 127.0.0.0/8 to any port = isakmp -> 121.44.11.88 static-port nat on ppp0 inet from 192.168.1.0/24 to any port = isakmp -> 121.44.11.88 static-port nat on ppp0 inet from 127.0.0.0/8 to any -> 121.44.11.88 port 1024:65535 nat on ppp0 inet from 192.168.1.0/24 to any -> 121.44.11.88 port 1024:65535 no rdr proto carp all rdr-anchor "relayd/*" all rdr-anchor "tftp-proxy/*" all rdr on ppp0 inet proto tcp from any to 121.44.11.88 port = 32022 -> 192.168.1.24 port 22 rdr-anchor "miniupnpd" all FILTER RULES: scrub on ppp0 all fragment reassemble scrub on vtnet1 all fragment reassemble anchor "relayd/*" all anchor "openvpn/*" all anchor "ipsec/*" all block drop in log quick inet from 169.254.0.0/16 to any label "Block IPv4 link-local" block drop in log quick inet from any to 169.254.0.0/16 label "Block IPv4 link-local" block drop in log inet all label "Default deny rule IPv4" block drop out log inet all label "Default deny rule IPv4" block drop in log inet6 all label "Default deny rule IPv6" block drop out log inet6 all label "Default deny rule IPv6" pass quick inet6 proto ipv6-icmp all icmp6-type unreach keep state pass quick inet6 proto ipv6-icmp all icmp6-type toobig keep state pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echorep keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echorep keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbrsol keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state block drop log quick inet proto tcp from any port = 0 to any label "Block traffic from port 0" block drop log quick inet proto udp from any port = 0 to any label "Block traffic from port 0" block drop log quick inet proto tcp from any to any port = 0 label "Block traffic to port 0" block drop log quick inet proto udp from any to any port = 0 label "Block traffic to port 0" block drop log quick inet6 proto tcp from any port = 0 to any label "Block traffic from port 0" block drop log quick inet6 proto udp from any port = 0 to any label "Block traffic from port 0" block drop log quick inet6 proto tcp from any to any port = 0 label "Block traffic to port 0" block drop log quick inet6 proto udp from any to any port = 0 label "Block traffic to port 0" block drop log quick from <snort2c>to any label "Block snort2c hosts" block drop log quick from any to <snort2c>label "Block snort2c hosts" block drop in log quick proto carp from (self) to any pass quick proto carp all no state block drop in log quick proto tcp from <sshlockout>to (self) port = ssh label "sshlockout" block drop in log quick proto tcp from <webconfiguratorlockout>to (self) port = https label "webConfiguratorlockout" block drop in log quick from <virusprot>to any label "virusprot overload table" block drop in log quick on ppp0 from <bogons>to any label "block bogon IPv4 networks from WAN" block drop in log quick on ppp0 from <bogonsv6>to any label "block bogon IPv6 networks from WAN" block drop in log on ! ppp0 inet from 121.44.11.88 to any block drop in log inet from 121.44.11.88 to any block drop in log on ppp0 inet6 from fe80::c88c:8100:109:bd37 to any block drop in log quick on ppp0 inet from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8" block drop in log quick on ppp0 inet from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8" block drop in log quick on ppp0 inet from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12" block drop in log quick on ppp0 inet from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16" block drop in log quick on ppp0 inet6 from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7" block drop in log on ! vtnet1 inet from 192.168.1.0/24 to any block drop in log inet from 192.168.1.37 to any block drop in log inet from 192.168.1.254 to any block drop in log on vtnet1 inet6 from fe80::5054:ff:fede:4afe to any pass in quick on vtnet1 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server" pass in quick on vtnet1 inet proto udp from any port = bootpc to 192.168.1.37 port = bootps keep state label "allow access to DHCP server" pass out quick on vtnet1 inet proto udp from 192.168.1.37 port = bootps to any port = bootpc keep state label "allow access to DHCP server" pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass in on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback" pass out on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback" pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself" pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself" pass out route-to (ppp0 10.64.64.0) inet from 121.44.11.88 to ! 121.44.11.88 flags S/SA keep state allow-opts label "let out anything from firewall host itself" pass in quick on vtnet1 proto tcp from any to (vtnet1) port = https flags S/SA keep state label "anti-lockout rule" pass in quick on vtnet1 proto tcp from any to (vtnet1) port = http flags S/SA keep state label "anti-lockout rule" pass in quick on vtnet1 proto tcp from any to (vtnet1) port = ssh flags S/SA keep state label "anti-lockout rule" anchor "userrules/*" all pass in quick on vtnet1 inet from 192.168.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule" pass in quick on ppp0 reply-to (ppp0 10.64.64.0) inet proto tcp from any to 192.168.1.24 port = ssh flags S/SA keep state label "USER_RULE: NAT External ssh access to ding" anchor "tftp-proxy/*" all No queue in use STATES: vtnet1 udp 255.255.255.255:7423 <- 192.168.1.2:1035 NO_TRAFFIC:SINGLE vtnet1 udp 255.255.255.255:7423 <- 192.168.1.3:46996 NO_TRAFFIC:SINGLE ppp0 icmp 121.44.11.88:16960 -> 192.231.203.3:16960 0:0 vtnet1 udp 224.0.0.251:5353 <- 192.168.1.35:5353 NO_TRAFFIC:SINGLE vtnet1 tcp 192.168.1.37:22 <- 192.168.1.145:22038 ESTABLISHED:ESTABLISHED vtnet1 tcp 150.101.195.202:80 <- 192.168.1.10:49690 CLOSING:ESTABLISHED ppp0 tcp 121.44.11.88:16478 (192.168.1.10:49690) -> 150.101.195.202:80 ESTABLISHED:CLOSING ppp0 udp 121.44.11.88:123 -> 203.135.184.123:123 MULTIPLE:SINGLE ppp0 udp 121.44.11.88:123 -> 13.55.50.68:123 MULTIPLE:SINGLE ppp0 udp 121.44.11.88:123 -> 27.124.125.250:123 MULTIPLE:SINGLE vtnet1 tcp 111.221.29.104:443 <- 192.168.1.7:49882 ESTABLISHED:ESTABLISHED ppp0 tcp 121.44.11.88:1113 (192.168.1.7:49882) -> 111.221.29.104:443 ESTABLISHED:ESTABLISHED vtnet1 tcp 104.98.26.36:443 <- 192.168.1.10:49689 CLOSING:ESTABLISHED ppp0 tcp 121.44.11.88:37824 (192.168.1.10:49689) -> 104.98.26.36:443 ESTABLISHED:CLOSING vtnet1 tcp 23.40.74.230:443 <- 192.168.1.10:49692 CLOSING:ESTABLISHED ppp0 tcp 121.44.11.88:33607 (192.168.1.10:49692) -> 23.40.74.230:443 ESTABLISHED:CLOSING vtnet1 tcp 17.252.252.41:443 <- 192.168.1.10:49693 ESTABLISHED:ESTABLISHED ppp0 tcp 121.44.11.88:8945 (192.168.1.10:49693) -> 17.252.252.41:443 ESTABLISHED:ESTABLISHED vtnet1 tcp 23.40.74.230:443 <- 192.168.1.10:49694 CLOSING:ESTABLISHED ppp0 tcp 121.44.11.88:13807 (192.168.1.10:49694) -> 23.40.74.230:443 ESTABLISHED:CLOSING vtnet1 udp 202.127.210.37:123 <- 192.168.1.147:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:25400 (192.168.1.147:123) -> 202.127.210.37:123 SINGLE:NO_TRAFFIC vtnet1 udp 103.242.68.69:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:21197 (192.168.1.36:123) -> 103.242.68.69:123 SINGLE:NO_TRAFFIC vtnet1 udp 192.231.203.132:123 <- 192.168.1.26:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:57787 (192.168.1.26:123) -> 192.231.203.132:123 SINGLE:NO_TRAFFIC vtnet1 udp 103.214.220.220:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:19718 (192.168.1.36:123) -> 103.214.220.220:123 SINGLE:NO_TRAFFIC vtnet1 udp 202.127.210.37:123 <- 192.168.1.27:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:23807 (192.168.1.27:123) -> 202.127.210.37:123 SINGLE:NO_TRAFFIC vtnet1 udp 103.239.8.22:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:41032 (192.168.1.36:123) -> 103.239.8.22:123 SINGLE:NO_TRAFFIC vtnet1 udp 255.255.255.255:67 <- 0.0.0.0:68 NO_TRAFFIC:SINGLE vtnet1 tcp 172.217.17.46:443 <- 192.168.1.145:17495 ESTABLISHED:ESTABLISHED ppp0 tcp 121.44.11.88:54736 (192.168.1.145:17495) -> 172.217.17.46:443 ESTABLISHED:ESTABLISHED vtnet1 tcp 172.217.17.46:443 <- 192.168.1.145:17496 ESTABLISHED:ESTABLISHED ppp0 tcp 121.44.11.88:8140 (192.168.1.145:17496) -> 172.217.17.46:443 ESTABLISHED:ESTABLISHED vtnet1 udp 150.203.22.28:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:43931 (192.168.1.36:123) -> 150.203.22.28:123 SINGLE:NO_TRAFFIC vtnet1 udp 122.252.184.186:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:51527 (192.168.1.36:123) -> 122.252.184.186:123 SINGLE:NO_TRAFFIC vtnet1 udp 150.203.1.10:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:23369 (192.168.1.36:123) -> 150.203.1.10:123 SINGLE:NO_TRAFFIC vtnet1 udp 203.0.178.191:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:20164 (192.168.1.36:123) -> 203.0.178.191:123 SINGLE:NO_TRAFFIC ppp0 udp 121.44.11.88:123 -> 203.122.222.149:123 MULTIPLE:SINGLE ppp0 udp 121.44.11.88:123 -> 121.0.0.42:123 MULTIPLE:SINGLE ppp0 udp 121.44.11.88:123 -> 203.217.19.190:123 MULTIPLE:SINGLE ppp0 udp 121.44.11.88:123 -> 103.214.220.220:123 MULTIPLE:SINGLE vtnet1 udp 192.168.1.255:137 <- 192.168.1.7:137 NO_TRAFFIC:SINGLE vtnet1 udp 224.0.0.252:5355 <- 192.168.1.7:61576 NO_TRAFFIC:SINGLE vtnet1 udp 202.127.210.37:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:48946 (192.168.1.36:123) -> 202.127.210.37:123 SINGLE:NO_TRAFFIC vtnet1 udp 224.0.0.252:5355 <- 192.168.1.7:58273 NO_TRAFFIC:SINGLE vtnet1 udp 203.12.160.2:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:55055 (192.168.1.36:123) -> 203.12.160.2:123 SINGLE:NO_TRAFFIC vtnet1 udp 203.0.178.191:123 <- 192.168.1.147:123 SINGLE:MULTIPLE ppp0 udp 121.44.11.88:4996 (192.168.1.147:123) -> 203.0.178.191:123 MULTIPLE:SINGLE vtnet1 udp 150.203.22.28:123 <- 192.168.1.27:123 SINGLE:MULTIPLE ppp0 udp 121.44.11.88:30245 (192.168.1.27:123) -> 150.203.22.28:123 MULTIPLE:SINGLE vtnet1 udp 203.0.178.191:123 <- 192.168.1.27:123 SINGLE:MULTIPLE ppp0 udp 121.44.11.88:51593 (192.168.1.27:123) -> 203.0.178.191:123 MULTIPLE:SINGLE vtnet1 udp 203.12.160.2:123 <- 192.168.1.147:123 SINGLE:MULTIPLE ppp0 udp 121.44.11.88:8374 (192.168.1.147:123) -> 203.12.160.2:123 MULTIPLE:SINGLE vtnet1 udp 239.255.255.250:1900 <- 192.168.1.3:39102 NO_TRAFFIC:SINGLE vtnet1 udp 203.26.24.6:123 <- 192.168.1.36:123 NO_TRAFFIC:SINGLE ppp0 udp 121.44.11.88:27473 (192.168.1.36:123) -> 203.26.24.6:123 SINGLE:NO_TRAFFIC INFO: Status: Enabled for 0 days 01:18:06 Debug: Urgent Interface Stats for vtnet1 IPv4 IPv6 Bytes In 2810962 59578 Bytes Out 8089433 0 Packets In Passed 22366 94 Blocked 12 533 Packets Out Passed 16747 0 Blocked 0 0 State Table Total Rate current entries 67 searches 98715 21.1/s inserts 10645 2.3/s removals 10578 2.3/s Counters match 11431 2.4/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s map-failed 0 0.0/s LABEL COUNTERS: Block IPv4 link-local 11420 0 0 0 0 0 0 0 Block IPv4 link-local 3861 0 0 0 0 0 0 0 Default deny rule IPv4 3861 251 13429 251 13429 0 0 0 Default deny rule IPv4 10825 0 0 0 0 0 0 0 Default deny rule IPv6 11421 533 54050 533 54050 0 0 0 Default deny rule IPv6 7560 0 0 0 0 0 0 0 Block traffic from port 0 11358 0 0 0 0 0 0 0 Block traffic from port 0 9646 0 0 0 0 0 0 0 Block traffic to port 0 10827 0 0 0 0 0 0 0 Block traffic to port 0 9646 0 0 0 0 0 0 0 Block traffic from port 0 11359 0 0 0 0 0 0 0 Block traffic from port 0 533 0 0 0 0 0 0 0 Block traffic to port 0 533 0 0 0 0 0 0 0 Block traffic to port 0 533 0 0 0 0 0 0 0 Block snort2c hosts 11358 0 0 0 0 0 0 0 Block snort2c hosts 11358 0 0 0 0 0 0 0 sshlockout 11358 0 0 0 0 0 0 0 webConfiguratorlockout 696 0 0 0 0 0 0 0 virusprot overload table 4398 0 0 0 0 0 0 0 block bogon IPv4 networks from WAN 4398 0 0 0 0 0 0 0 block bogon IPv6 networks from WAN 243 0 0 0 0 0 0 0 Block private networks from WAN block 10/8 241 0 0 0 0 0 0 0 Block private networks from WAN block 127/8 241 0 0 0 0 0 0 0 Block private networks from WAN block 172.16/12 241 0 0 0 0 0 0 0 Block private networks from WAN block 192.168/16 241 0 0 0 0 0 0 0 Block ULA networks from WAN block fc00::/7 241 0 0 0 0 0 0 0 allow access to DHCP server 4132 45 14760 45 14760 0 0 22 allow access to DHCP server 0 0 0 0 0 0 0 0 allow access to DHCP server 10043 0 0 0 0 0 0 0 pass IPv4 loopback 10804 46 5025 23 1553 23 3472 23 pass IPv4 loopback 46 0 0 0 0 0 0 0 pass IPv6 loopback 579 0 0 0 0 0 0 0 pass IPv6 loopback 23 0 0 0 0 0 0 0 let out anything IPv4 from firewall host itself 11336 46 5025 23 3472 23 1553 23 let out anything IPv6 from firewall host itself 6964 0 0 0 0 0 0 0 let out anything from firewall host itself 6964 38388 10795851 19201 8898594 19187 1897257 6943 anti-lockout rule 11345 1759 633803 832 152875 927 480928 6 anti-lockout rule 10 0 0 0 0 0 0 0 anti-lockout rule 10 837 110758 320 35324 517 75434 2 USER_RULE: Default allow LAN to any rule 11336 33368 9515823 18065 1982752 15303 7533071 3557 USER_RULE: NAT External ssh access to ding 7244 0 0 0 0 0 0 0 TIMEOUTS: tcp.first 120s tcp.opening 30s tcp.established 86400s tcp.closing 900s tcp.finwait 45s tcp.closed 90s tcp.tsdiff 30s udp.first 60s udp.single 30s udp.multiple 60s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start 58200 states adaptive.end 116400 states src.track 0s LIMITS: states hard limit 97000 src-nodes hard limit 97000 frags hard limit 5000 table-entries hard limit 200000 TABLES: bogons bogonsv6 snort2c sshlockout virusprot webConfiguratorlockout OS FINGERPRINTS: 758 fingerprints loaded</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c>The WAN and LAN firewall rules are "standard" (there is one extra rule for port-forwarding inbound ssh connections, with is untested).
The KVM server (192.168.1.26) and other guest (192.168.1.36) can't traverse the firewall.
Everything else seems to work fine.
All clues gratefully accepted…
-
Hi,
I have looked at the packet traces from multiple perspectives.
The KVM bridge connects the KVM host and all its guests (including the pfSense firewall) to the LAN (192.168.1.0/24).
The default gateway on the LAN is 192.168.1.254, and that's an IP alias on the pfSense LAN interface (192.168.1.37).
The pfSense firewall works perfectly for all hosts not associated with the KVM bridge.
Hosts associated with the KVM bridge (i.e. the KVM server itself, and all its guests except the pfSense firewall itself) can't establish connections through the pfSense firewall. Tracing a TCP connection from a KVM guest to an Internet host shows the initial SYN entering the pfSense firewall via the default route, and being re-transmitted out the WAN interface. The SYN is then re-transmitted several times before the connection times out. The expected SYN/ACK response never happens.
This feels very much like NAT (or masquerading) is not turned on for those hosts on the KVM bridge. The NAT table looks like this:
[2.4.2-RELEASE][admin@pfSense.oakes.consulting]/root: pfctl -s nat no nat proto carp all nat-anchor "natearly/*" all nat-anchor "natrules/*" all nat on ppp0 inet from 127.0.0.0/8 to any port = isakmp -> 121.44.4.84 static-port nat on ppp0 inet from 192.168.1.0/24 to any port = isakmp -> 121.44.4.84 static-port nat on ppp0 inet from 127.0.0.0/8 to any -> 121.44.4.84 port 1024:65535 nat on ppp0 inet from 192.168.1.0/24 to any -> 121.44.4.84 port 1024:65535 no rdr proto carp all rdr-anchor "relayd/*" all rdr-anchor "tftp-proxy/*" all rdr on ppp0 inet proto tcp from any to 121.44.4.84 port = 32022 -> 192.168.1.24 port 22 rdr-anchor "miniupnpd" allMy setup mst be very similar to that used by ProxMox users.
Any ideas appreciated.
Cheers,
-
Hi,
Does anyone have pfSense working under KVM?
I need to figure out if I have a configuration issue which can be resolved or something more difficult.
Cheers,
-
Hi,
For those who follow, I just upgraded to 2.4.2-RELEASE-p1 (amd64) and the problem instantly went away.
Cheers,
-
Oops, spoke too soon.
Ping now works, where it did not before, but wget does not (ritz is a FreeBSD 11.1 KVM client, so very similar to the psSense firewall):
[ritz.132] ping -c1 google.com PING google.com (172.217.25.142): 56 data bytes 64 bytes from 172.217.25.142: icmp_seq=0 ttl=54 time=99.929 ms --- google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 99.929/99.929/99.929/0.000 ms [ritz.133] wget google.com --2017-12-27 16:43:50-- http://google.com/ Resolving google.com (google.com)... 172.217.25.142, 2404:6800:4006:804::200e Connecting to google.com (google.com)|172.217.25.142|:80... failed: Operation timed out. Connecting to google.com (google.com)|2404:6800:4006:804::200e|:80... failed: No route to host.So, no trouble getting ICMP forwarded through the firewall, with outbound SNAT working, but not so with TCP.
As above, only those LAN clients associated with the KVM bridge (other than the pfSense firewall) have this problem (i.e. the KVM host and the KVM client named ritz).
Very odd that SNAT outbound from the firewall is working selectively.
Cheers,
-
Hello @gpw928 did you ever resolve this problem?
I can observe exactly the same behaviour. I can see the SYN packet entering the pfSense and I can "see" it leaving the psSense too. According to tcpdump it has been masqueraded too.
I observe the same odd problem with inter vlan traffic too. I CAN send icmp echo request between host and other machines or between clients on differen vlans, but as soon as I am trying to telnet into a different machine it does not work.
I tried to connect an USB network card directly to the pfSense as WAN interface, so it even does not loop back through the networking stack of KVM, but still no success. Actually there must be something malformed about the package that arrives on the virtualized pfSense box though. Trying to trace this down further I found some different guys obdviously seeming to observe the same strange behaviour but noone has a solution yet. I have the pfSense currently running on an esxi host and I tried to transfer the config 1:1, no luck.
I can connect from from physical lan clients to the internet, and from one physical device to another one on a different vlan. I can NEVER connect to VM running on the KVM host itself or FROM a VM running on the host. I neither can configure the host to use the pfSense box as a gateway without loosing internet connectivity.
well, actually it seems I happen to fix it RIGHT NOW after nearly two days of troubleshooting. It seems using e1000 network interfaces instead of virtio does the trick.
still not 100% sure but at least I was able to get some tcp connections running now.
-
if anyone runs into the same problem- using fully virtualized network cards helps (e1000, rtl8139) but has am major performance impact, really high cpu utilization in comparison to my esxi setup I had before, although this kvm box has a better cpu. i would strongly advise against it, unless you have so much horsepower, it does not matter.
but I came across this: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox.html
in the last paragraph it states:
"Because the hardware checksum offload is not yet disabled, accessing pfSense webGUI might be sluggish. This is NORMAL and is fixed in the following step."
well, setting this option seems to help for me for any traffic passing the pfSense VM. I can now run virtio paravirtualized network interfaces and have a good performance over all, at relatively low cpu consumption.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.