Routing some LAN address to internet doesn't work, others do.



  • I upgraded not too long ago and noticed yesterday that no servers behind pfSense were receiving any updates.  It no traffic is going out, only incoming traffic is working.

    pfSense runs in a KVM virtual machine under Proxmox and all was well using pfSense 2.2.2.

    Both ports are bridged in the VM host platform configured as follows:

    # cat /etc/network/interfaces 
    # network interface settings
    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet manual
    
    auto eth1
    iface eth1 inet manual
    
    auto eth3
    iface eth3 inet manual
    
    iface eth2 inet manual
    
    auto bond0
    iface bond0 inet manual
    	slaves eth0 eth1
    	bond_miimon 100
    	bond_mode 802.3ad
    	bond_xmit_hash_policy layer2
    
    auto vmbr0
    iface vmbr0 inet static
    	address  192.168.121.33
    	netmask  255.255.255.0
    	gateway  192.168.121.1
    	bridge_ports bond0
    	bridge_stp off
    	bridge_fd 0
    #	gateway  192.168.121.1
    
    auto vmbr1
    iface vmbr1 inet manual
    	bridge_ports eth3
    	bridge_stp off
    	bridge_fd 0
    
    

    Therefore the KVM host cannot reach anything on the WAN port, since there is no IP address allocated.

    The pfSense basic config is:
    1 WAN port (vtnet1)
    1 LAN port (vtnet0)
    7 Virtual IP defined
    I connect to the LAN via IPsec/L2tp and can access all the servers/services in this way.

    Problem is that no traffic is passed beyond the WAN port.  I can ping xxx.71.69.226 (the WAN port address), but not 225, which is the gateway.  However, from xxx.71.69.226 I can ping the 225 gateway and anything else outside.

    Here's the routing table:

    $ netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags      Netif Expire
    default            41.71.69.225       UGS      vtnet1
    xxx.71.69.224/28    link#2             U        vtnet1
    xxx.71.69.226       link#2             UHS         lo0
    xxx.71.69.228       link#2             UHS         lo0
    xxx.71.69.228/32    link#2             U        vtnet1
    xxx.71.69.229       link#2             UHS         lo0
    xxx.71.69.229/32    link#2             U        vtnet1
    xxx.71.69.230       link#2             UHS         lo0
    xxx.71.69.230/32    link#2             U        vtnet1
    xxx.71.69.231       link#2             UHS         lo0
    xxx.71.69.231/32    link#2             U        vtnet1
    xxx.71.69.232       link#2             UHS         lo0
    xxx.71.69.232/32    link#2             U        vtnet1
    xxx.71.69.233       link#2             UHS         lo0
    xxx.71.69.233/32    link#2             U        vtnet1
    xxx.71.69.234       link#2             UHS         lo0
    xxx.71.69.234/32    link#2             U        vtnet1
    127.0.0.1          link#5             UH          lo0
    192.168.120.240    link#7             UH        l2tp0
    192.168.120.241    link#8             UH        l2tp1
    192.168.120.248    link#7             UHS         lo0
    192.168.121.0/24   link#1             U        vtnet0
    192.168.121.1      link#1             UHS         lo0
    
    

    The default gateway should receive all the traffic, but it's not happening.  Is there a freeBSD method to disable/enable routing (like Linux has), or is routing always allowed by default?

    I don't know how to troubleshoot this any further, so I need some help please.



  • I also see this:

    cat /etc/defaults/rc.conf | grep gateway
    defaultrouter="NO"		# Set to default gateway (or NO).
    gateway_enable="NO"		# Set to YES if this host will be a gateway.
    ipxgateway_enable="NO"		# Set to YES to enable IPX routing.
    forward_sourceroute="NO"	# do source routing (only if gateway_enable is set to "YES")
    ipv6_defaultrouter="NO"		# Set to IPv6 default gateway (or NO).
    ipv6_gateway_enable="NO"	# Set to YES if this host will be a gateway.
    				# ipv6_gateway_enable="NO".
    
    

    Is that the way it's supposed to be with pfSense?  It's seems like freeBSD's routing is turned off, or what?


  • Banned

    /etc/defaults/rc.conf is NOT the place to mess with.



  • @doktornotor:

    /etc/defaults/rc.conf is NOT the place to mess with.

    Not, I won't mess with anything, especially in freeBSD!  I was just looking around to see what I can discover to give me an indication of what may be causing the problem.  I'm pretty sure I didn't change anything since I upgraded, since everything "just worked" once it was set up.

    So what do I do next?



  • Idea : remove temporary the Promox thing. From what I understand, this 'tool' does have huge effect on routing because it assigns port/address to internal VM's etc.

    pfSense that stops routing to the outside : of course, we've seen that before on this forum. Like people mix up LAN & WAN, using the same IP mask on LAN and WAN, these kinds of issues  (you get the picture ;))
    The thing is, you have a system-in-a-system and you declare that the 'system' as a whole doesn't work.
    One of the solutions is : break it up in pieces to analyze them all.

    Btw: I've no experience with "put a firewall in a proxmox" - I tend to assign a dedicated box to that kind of tasks.
    It seems more a proxmox issue to me.



  • (I have renamed the question to more correctly describe the symptoms)

    If have done a lot of digging and testing re this issue and the news is somewhat better than I at first reported:

    The routing works from some of the virtual machines, but not from others.

    To illustrate the point:

    machine A: www

    www:~$ ip addr show
    1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
    2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether ca:04:b6:f7:84:c4 brd ff:ff:ff:ff:ff:ff
        inet 192.168.121.38/24 brd 192.168.121.255 scope global eth0
          valid_lft forever preferred_lft forever
        inet6 fe80::c804:b6ff:fef7:84c4/64 scope link
          valid_lft forever preferred_lft forever

    @www:~$ ip route show
    default via 192.168.121.1 dev eth0
    192.168.121.0/24 dev eth0  proto kernel  scope link  src 192.168.121.38

    machine B: redmine

    redmine:~$ ip addr show
    1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
    2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 3a:50:f6:52:b9:24 brd ff:ff:ff:ff:ff:ff
        inet 192.168.121.50/24 brd 192.168.121.255 scope global eth0
          valid_lft forever preferred_lft forever
        inet6 fe80::3850:f6ff:fe52:b924/64 scope link
          valid_lft forever preferred_lft forever

    redmine:~$ ip route show
    default via 192.168.121.1 dev eth0
    192.168.121.0/24 dev eth0  proto kernel  scope link  src 192.168.121.50

    Both machines are configured with the same virtual hardware, yet machine A can access the internet (anything beyond the pfSense WAN port) while machine B can only access up to the WAN port, but not beyond that.

    I have shut down machine A and then change machine B's ip address to 12.168.121.38 and when I restart it, it can access the internet perfectly.

    So this all points to some obscure setting in pfSense that allows 192.168.121.38 out, but 192.168.121.50 not.

    Were could that be?</broadcast,multicast,up,lower_up></loopback,up,lower_up></broadcast,multicast,up,lower_up></loopback,up,lower_up>



  • Since finding out some address route to the internet and others not, I checked which actually do route.  Turns out that only those with 1:1 NAT entries in pfSense can access the internet.

    In Firewall: NAT: Outbound I have selected "Automatic outbound NAT rule generation (IPsec passthrough included) ".  It seems that this is not sufficient.

    What should I do to make this work as expected?  Should I remove the 1:1 NAT and only put in inbound NAT rules?



  • I was stuck with this issue, no replies, so I went through all the settings again.  Lo and behold!  There was no upstream gateway set on the WAN port (although I'm sure it was there at some stage before).

    After setting it, all is well.