PfSense won't forward traffic form LAN server to internet



  • ★ My setup:

    Host (ArchLinux, nanoBox):
    Physical interfaces: with eth0 (no ip) and wlan0 (hostapd).
    Virtual interfaces: br0 (static IP 192.168.7.2 assigned with netctl profile)

    Guest (pfSense inside KVM):
    Guest interfaces:
    vtnet0 - bridged to eth0
    vtnet1 - bridged to br0 (192.168.7.1)

    My Host is also a web sever. I do not know if this is good practice, but br0 is the interface which which host services connect to internet.

    I assign static IP to br0 for two reasons:
    1 - when I set it to DHCP it does not receive any IP. I have no idea why - it worked for a month before, but it just doesn’t anymore.
    2 - it allows me to access the host even when pfSense VM is down

    ★ What works:

    • accessing internet from Host
    • accessing internet from WiFi clients connecting to the internet using hostapd
    • port forwarding from the internet to WiFi clients connecting to the internet using hostapd

    ★★★ What does not work: ★★★

    • port forwarding from the internet to the Host! This means that any service running on Host (http, ssh, VNC etc.) is not accessible from the Internet.

    I have no idea why, because:
    ➤ All the NAT and Firewall rules on pfSense are set up and identical (config restore) to when they worked on VirtualBox setup. NAT rules: http://i.imgur.com/jqzqYYu.png  Firewall rules: http://i.imgur.com/pfuuwAz.png
    (nanoBox alias is assigned to 192.168.7.2 ip. Replacing alias with the ip itself makes no difference)
    ➤ The static IP of br0 is the same as the IP of the destination host in every IP rule.
    ➤ Host can access internet.
    ➤ Host can be pinged on its IP (192.168.7.2) from pfSense
    ➤ NAT and Firewall rules worked for any other internal IP (for example, a VNC server on one of WiFi clients)

    ★ Configs

    Host's ifconfig

    # ifconfig 
    br0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet 192.168.7.2  netmask 255.255.255.0  broadcast 192.168.7.255
            inet6 fe80::4ccb:a9ff:feb7:5617  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 0  (Ethernet)
            RX packets 2825  bytes 330247 (322.5 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 3339  bytes 802554 (783.7 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    enp2s0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::5ea1:75a3:7d46:befd  prefixlen 64  scopeid 0x20
            ether 00:90:27:77:fb:02  txqueuelen 1000  (Ethernet)
            RX packets 223027  bytes 20719723 (19.7 MiB)
            RX errors 0  dropped 178  overruns 0  frame 0
            TX packets 6747  bytes 2101069 (2.0 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<up,loopback,running>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10 <host>loop  txqueuelen 0  (Local Loopback)
            RX packets 12388  bytes 1341938 (1.2 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 12388  bytes 1341938 (1.2 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    macvtap0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::26f4:1e55:97a0:c0cb  prefixlen 64  scopeid 0x20
            ether 00:90:27:77:fb:02  txqueuelen 500  (Ethernet)
            RX packets 217268  bytes 20328935 (19.3 MiB)
            RX errors 8919  dropped 8919  overruns 0  frame 0
            TX packets 6620  bytes 2073711 (1.9 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    vnet0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::5d6b:398c:6b44:d602  prefixlen 64  scopeid 0x20
            ether fe:54:00:6f:2e:15  txqueuelen 500  (Ethernet)
            RX packets 4558  bytes 4062075 (3.8 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 4583  bytes 624983 (610.3 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    wlp1s0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::6e57:fe92:1321:1521  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 1000  (Ethernet)
            RX packets 6040  bytes 811010 (792.0 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 7038  bytes 4986969 (4.7 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</up,broadcast,running,multicast></up,broadcast,running,multicast></up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></up,broadcast,running,multicast>
    

    /etc/netctl/br

    Interface=br0
    Connection=bridge
    BindsToInterfaces=()
    #IP=dhcp
    IP=static
    Address=('192.168.7.2/24')
    Routes=('192.168.0.0/24 via 192.168.7.1')
    Gateway=('192.168.7.1')
    ## Ignore (R)STP and immediately activate the bridge
    #SkipForwardingDelay=yes
    

    ★ Notes that may or may not be relevant:

    • When KVM starts, it creates an apparently useless vnet0 interface on host. I say it is useless because when I was using VirtualBox in identical setup, it did not create anything beyond the interfaces listed at the beginning of this post, and things worked fine. vnet0 remains IP-less. However, if I destroy br0, vnet0 immediately receives a DHCP IP from pfSense guest.

    • when I look at DHCP logs on pfSense, I can see repetitive entries:

    Apr 4 08:12:03 	dhcpd: DHCPDISCOVER from 4e:cb:a9:b7:56:17 via vtnet1
    Apr 4 08:12:03 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:12:04 	dhcpd: DHCPOFFER on 192.168.7.102 to 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:12:52 	dhcpd: DHCPDISCOVER from fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:12:52 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:12:53 	dhcpd: DHCPOFFER on 192.168.7.100 to fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:13:06 	dhcpd: DHCPDISCOVER from 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:13:07 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:13:07 	dhcpd: DHCPOFFER on 192.168.7.102 to 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:13:56 	dhcpd: DHCPDISCOVER from fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:13:56 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:13:57 	dhcpd: DHCPOFFER on 192.168.7.100 to fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:14:10 	dhcpd: DHCPDISCOVER from 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:14:11 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:14:12 	dhcpd: DHCPOFFER on 192.168.7.102 to 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:15:00 	dhcpd: DHCPDISCOVER from fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:15:01 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:15:01 	dhcpd: DHCPOFFER on 192.168.7.100 to fe:54:00:6f:2e:15 (nanoBox) via vtnet1
    Apr 4 08:15:15 	dhcpd: DHCPDISCOVER from 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    Apr 4 08:15:16 	dhcpd: unexpected ICMP Echo Reply from 85.28.128.1
    Apr 4 08:15:16 	dhcpd: DHCPOFFER on 192.168.7.102 to 4e:cb:a9:b7:56:17 (nanoBox) via vtnet1
    

    ★ Things I tried:

    • disabling DHCP server on pfSense
    • changing host's static IP
    • in Firewall/NAT ruels, replacing alias nanoBox with the ip itself
    • many other things I can't remember


  • Could you try to take a screenshot?

    You are saying you can't port forward from the internet to the host, but it should be the other way arround so you are port forwarding your server to the wan interface :)



  • @notaduck:

    Could you try to take a screenshot?

    Yes, but of what?

    @notaduck:

    You are saying you can't port forward from the internet to the host, but it should be the other way arround so you are port forwarding your server to the wan interface :)

    Well, I need my server on the LAN to be accessible from WAN. Sorry for the confusion.


  • Banned

    When you look at the 80 NAT rule, you do NOT have a firewall rule associated with that. WHY?



  • Check your firewall logs.



  • @doktornotor,
    Good point. I set it up,
    http://i.imgur.com/0LrJCzv.png
    but the problem persists.

    @EMWEE,
    In the firewall logs, I can see nothing relevent, ie related to port :80, or the external IP I am trying to access the webserver from.


  • Banned

    Move it above the block all rule and then its fine.



  • @Supermule:

    Move it above the block all rule and then its fine.

    I don't think I have a block all rule. And if you mean "RFC 1918 networks" and "Reserved/not assigned by IANA", then it is not possible.


  • Banned

    Do you have an outbound nat rule in place that seperate subnet?



  • @Supermule:

    Do you have an outbound nat rule in place that seperate subnet?

    http://i.imgur.com/L5FrSeb.png
    Is that it?



  • @lockheed:

    @Supermule:

    Move it above the block all rule and then its fine.

    I don't think I have a block all rule. And if you mean "RFC 1918 networks" and "Reserved/not assigned by IANA", then it is not possible.

    You can try unchecking the box “Block private networks” on the screen Interfaces WAN (at the bottom) and see if that solves your problem.



  • @gjaltemba:

    You can try unchecking the box “Block private networks” on the screen Interfaces WAN (at the bottom) and see if that solves your problem.

    That's one of those things I tried and forgot to mention  :(


  • Banned

    Perhaps you could finally look at the firewall logs?!?!?


  • Banned

    Nope…

    It looks like this...

    Menu -> Firewall -> NAT -> Outbound




  • @doktornotor:

    Perhaps you could finally look at the firewall logs?!?!?

    What am I looking for? I already said I could see nothing related to the IP I am accessing the web server from.

    @Supermule:

    Nope…

    It looks like this...

    Menu -> Firewall -> NAT -> Outbound

    That's what I have:
    http://i.imgur.com/fWWY3XA.png



  • @lockheed:

    @gjaltemba:

    You can try unchecking the box “Block private networks” on the screen Interfaces WAN (at the bottom) and see if that solves your problem.

    That's one of those things I tried and forgot to mention  :(

    But the screen shot of the firewall rules shows Block private networks is in effect. I am confused.



  • @gjaltemba:

    But the screen shot of the firewall rules shows Block private networks is in effect. I am confused.

    Because I re-enabled it after I found out it did not change anything. Anyway, now I have it disabled again.



  • @lockheed:

    ★ My setup:

    Host (ArchLinux, nanoBox):
    Physical interfaces: with eth0 (no ip) and wlan0 (hostapd).
    Virtual interfaces: br0 (static IP 192.168.7.2 assigned with netctl profile)

    Guest (pfSense inside KVM):
    Guest interfaces:
    vtnet0 - bridged to eth0
    vtnet1 - bridged to br0 (192.168.7.1)

    My Host is also a web sever. I do not know if this is good practice, but br0 is the interface which which host services connect to internet.

    Does this mean pfsense WAN interface is assigned to vtnet1, has a static ip of 192.168.7.1 and pfsense LAN interface is assigned to vtnet0?



  • @gjaltemba:

    Does this mean pfsense WAN interface is assigned to vtnet1, has a static ip of 192.168.7.1 and pfsense LAN interface is assigned to vtnet0?

    No, the other way around:
    eth0-WAN-85.x.x.x-vtnet0
    br0-LAN-192.168.7.1-vtnet1



  • @lockheed:

    @gjaltemba:

    Does this mean pfsense WAN interface is assigned to vtnet1, has a static ip of 192.168.7.1 and pfsense LAN interface is assigned to vtnet0?

    No, the other way around:
    eth0-WAN-85.x.x.x-vtnet0
    br0-LAN-192.168.7.1-vtnet1

    Does 192.168.7.2 have 192.168.7.1 as gateway? I am guessing not because it has internet with pfsense in shutdown.



  • @gjaltemba:

    Does 192.168.7.2 have 192.168.7.1 as gateway?
    Yes.
    @gjaltemba:

    I am guessing not because it has internet with pfsense in shutdown.

    No, it doesn't. It only has LAN if I set br0 to static IP. I can then connect to it with my laptop (also with static IP) which connects to hostapd (bridged with br0).



  • @lockheed:


    Well, I need my server on the LAN to be accessible from WAN.

    Test approach: simplify your config, exclude your reliance on aliases & name(s).

    You need probably:
    [Firewall: NAT: Port Forward] with a rule like:
    WAN TCP * * WAN address 80 192.168.x.y 80



  • @hda:

    Test approach: simplify your config, exclude your reliance on aliases & name(s).

    Thanks, but
    @lockheed:

    (nanoBox alias is assigned to 192.168.7.2 ip. Replacing alias with the ip itself makes no difference)

    I have a new find that might shed some light on the source of the problem.

    When I am on the host, pinging google.com works BUT going to google.com in a browser or with wget does not. In fact, no URL address works at all in any other capacity than PING.


  • Banned

    Change your outbound nat for the subnet to include UDP as well.



  • @Supermule:

    Change your outbound nat for the subnet to include UDP as well.

    Like so? http://i.imgur.com/4jgDqJj.png
    It didn't help.

    Also, please remember that those issues are experienced only on the Host of the pfSense VM. Every other pfSense manager LAN client works just fine.


  • Banned

    @lockheed:

    @Supermule:

    Change your outbound nat for the subnet to include UDP as well.

    Like so? http://i.imgur.com/4jgDqJj.png
    It didn't help.

    There is nothing useful visible there at all regarding protocol. (And please, learn to use the IMG tag.)



  • @doktornotor:

    There is nothing useful visible there at all regarding protocol. (And please, learn to use the IMG tag.)

    I selected ALL protocols. As for IMG, I am giving links to images because I did not want to clutter the thread with auto-displaying images.



  • I just created an identical pfSense on VirtualBox and cloned the config on it. Everything works fine.

    Here's the ifconfig of KVM setup:

    ifconfig 
    br0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet 192.168.7.2  netmask 255.255.255.0  broadcast 192.168.7.255
            inet6 fe80::4ccb:a9ff:feb7:5617  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 0  (Ethernet)
            RX packets 2825  bytes 330247 (322.5 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 3339  bytes 802554 (783.7 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    enp2s0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::5ea1:75a3:7d46:befd  prefixlen 64  scopeid 0x20
            ether 00:90:27:77:fb:02  txqueuelen 1000  (Ethernet)
            RX packets 223027  bytes 20719723 (19.7 MiB)
            RX errors 0  dropped 178  overruns 0  frame 0
            TX packets 6747  bytes 2101069 (2.0 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<up,loopback,running>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10 <host>loop  txqueuelen 0  (Local Loopback)
            RX packets 12388  bytes 1341938 (1.2 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 12388  bytes 1341938 (1.2 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    macvtap0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::26f4:1e55:97a0:c0cb  prefixlen 64  scopeid 0x20
            ether 00:90:27:77:fb:02  txqueuelen 500  (Ethernet)
            RX packets 217268  bytes 20328935 (19.3 MiB)
            RX errors 8919  dropped 8919  overruns 0  frame 0
            TX packets 6620  bytes 2073711 (1.9 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    vnet0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::5d6b:398c:6b44:d602  prefixlen 64  scopeid 0x20
            ether fe:54:00:6f:2e:15  txqueuelen 500  (Ethernet)
            RX packets 4558  bytes 4062075 (3.8 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 4583  bytes 624983 (610.3 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    wlp1s0: flags=4163<up,broadcast,running,multicast>  mtu 1500
            inet6 fe80::6e57:fe92:1321:1521  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 1000  (Ethernet)
            RX packets 6040  bytes 811010 (792.0 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 7038  bytes 4986969 (4.7 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</up,broadcast,running,multicast></up,broadcast,running,multicast></up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></up,broadcast,running,multicast>
    

    and of a much cleaner, and - more importantly - working VirtualBox setup:

    # ifconfig 
    br0: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet 192.168.7.2  netmask 255.255.255.0  broadcast 192.168.7.255
            inet6 fe80::a288:69ff:fe0d:5c41  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 0  (Ethernet)
            RX packets 4999  bytes 1686341 (1.6 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 9269  bytes 2203282 (2.1 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    enp2s0: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet 192.168.11.13  netmask 255.255.255.0  broadcast 192.168.11.255
            inet6 fe80::201:2eff:fe4e:4b99  prefixlen 64  scopeid 0x20
            ether 00:01:2e:4e:4b:99  txqueuelen 1000  (Ethernet)
            RX packets 175668  bytes 58689989 (55.9 MiB)
            RX errors 0  dropped 35  overruns 0  frame 0
            TX packets 33594  bytes 2862399 (2.7 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<up,loopback,running>mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10 <host>loop  txqueuelen 0  (Local Loopback)
            RX packets 44600  bytes 11957420 (11.4 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 44600  bytes 11957420 (11.4 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    wlp1s0: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet6 fe80::a288:69ff:fe0d:5c41  prefixlen 64  scopeid 0x20
            ether a0:88:69:0d:5c:41  txqueuelen 1000  (Ethernet)
            RX packets 4400  bytes 1698452 (1.6 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 8264  bytes 2315002 (2.2 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</up,broadcast,running,multicast></host></up,loopback,running></up,broadcast,running,multicast></up,broadcast,running,multicast> 
    

    So it looks like KVM is not suitable for hosting pfSense VM if host machine is required have access to the internet. It is a shame as I was hoping for KVM to be not just working, but a superior solution.

    Can someone move this thread to Virtualization?


Log in to reply