Routing some LAN address to internet doesn't work, others do.
-
I upgraded not too long ago and noticed yesterday that no servers behind pfSense were receiving any updates. It no traffic is going out, only incoming traffic is working.
pfSense runs in a KVM virtual machine under Proxmox and all was well using pfSense 2.2.2.
Both ports are bridged in the VM host platform configured as follows:
# cat /etc/network/interfaces # network interface settings auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto eth1 iface eth1 inet manual auto eth3 iface eth3 inet manual iface eth2 inet manual auto bond0 iface bond0 inet manual slaves eth0 eth1 bond_miimon 100 bond_mode 802.3ad bond_xmit_hash_policy layer2 auto vmbr0 iface vmbr0 inet static address 192.168.121.33 netmask 255.255.255.0 gateway 192.168.121.1 bridge_ports bond0 bridge_stp off bridge_fd 0 # gateway 192.168.121.1 auto vmbr1 iface vmbr1 inet manual bridge_ports eth3 bridge_stp off bridge_fd 0Therefore the KVM host cannot reach anything on the WAN port, since there is no IP address allocated.
The pfSense basic config is:
1 WAN port (vtnet1)
1 LAN port (vtnet0)
7 Virtual IP defined
I connect to the LAN via IPsec/L2tp and can access all the servers/services in this way.Problem is that no traffic is passed beyond the WAN port. I can ping xxx.71.69.226 (the WAN port address), but not 225, which is the gateway. However, from xxx.71.69.226 I can ping the 225 gateway and anything else outside.
Here's the routing table:
$ netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 41.71.69.225 UGS vtnet1 xxx.71.69.224/28 link#2 U vtnet1 xxx.71.69.226 link#2 UHS lo0 xxx.71.69.228 link#2 UHS lo0 xxx.71.69.228/32 link#2 U vtnet1 xxx.71.69.229 link#2 UHS lo0 xxx.71.69.229/32 link#2 U vtnet1 xxx.71.69.230 link#2 UHS lo0 xxx.71.69.230/32 link#2 U vtnet1 xxx.71.69.231 link#2 UHS lo0 xxx.71.69.231/32 link#2 U vtnet1 xxx.71.69.232 link#2 UHS lo0 xxx.71.69.232/32 link#2 U vtnet1 xxx.71.69.233 link#2 UHS lo0 xxx.71.69.233/32 link#2 U vtnet1 xxx.71.69.234 link#2 UHS lo0 xxx.71.69.234/32 link#2 U vtnet1 127.0.0.1 link#5 UH lo0 192.168.120.240 link#7 UH l2tp0 192.168.120.241 link#8 UH l2tp1 192.168.120.248 link#7 UHS lo0 192.168.121.0/24 link#1 U vtnet0 192.168.121.1 link#1 UHS lo0The default gateway should receive all the traffic, but it's not happening. Is there a freeBSD method to disable/enable routing (like Linux has), or is routing always allowed by default?
I don't know how to troubleshoot this any further, so I need some help please.
-
I also see this:
cat /etc/defaults/rc.conf | grep gateway defaultrouter="NO" # Set to default gateway (or NO). gateway_enable="NO" # Set to YES if this host will be a gateway. ipxgateway_enable="NO" # Set to YES to enable IPX routing. forward_sourceroute="NO" # do source routing (only if gateway_enable is set to "YES") ipv6_defaultrouter="NO" # Set to IPv6 default gateway (or NO). ipv6_gateway_enable="NO" # Set to YES if this host will be a gateway. # ipv6_gateway_enable="NO".Is that the way it's supposed to be with pfSense? It's seems like freeBSD's routing is turned off, or what?
-
/etc/defaults/rc.conf is NOT the place to mess with.
-
/etc/defaults/rc.conf is NOT the place to mess with.
Not, I won't mess with anything, especially in freeBSD! I was just looking around to see what I can discover to give me an indication of what may be causing the problem. I'm pretty sure I didn't change anything since I upgraded, since everything "just worked" once it was set up.
So what do I do next?
-
Idea : remove temporary the Promox thing. From what I understand, this 'tool' does have huge effect on routing because it assigns port/address to internal VM's etc.
pfSense that stops routing to the outside : of course, we've seen that before on this forum. Like people mix up LAN & WAN, using the same IP mask on LAN and WAN, these kinds of issues (you get the picture ;))
The thing is, you have a system-in-a-system and you declare that the 'system' as a whole doesn't work.
One of the solutions is : break it up in pieces to analyze them all.Btw: I've no experience with "put a firewall in a proxmox" - I tend to assign a dedicated box to that kind of tasks.
It seems more a proxmox issue to me. -
(I have renamed the question to more correctly describe the symptoms)
If have done a lot of digging and testing re this issue and the news is somewhat better than I at first reported:
The routing works from some of the virtual machines, but not from others.
To illustrate the point:
machine A: www
www:~$ ip addr show
1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether ca:04:b6:f7:84:c4 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.38/24 brd 192.168.121.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::c804:b6ff:fef7:84c4/64 scope link
valid_lft forever preferred_lft forever@www:~$ ip route show
default via 192.168.121.1 dev eth0
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.38machine B: redmine
redmine:~$ ip addr show
1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 3a:50:f6:52:b9:24 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.50/24 brd 192.168.121.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::3850:f6ff:fe52:b924/64 scope link
valid_lft forever preferred_lft foreverredmine:~$ ip route show
default via 192.168.121.1 dev eth0
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.50Both machines are configured with the same virtual hardware, yet machine A can access the internet (anything beyond the pfSense WAN port) while machine B can only access up to the WAN port, but not beyond that.
I have shut down machine A and then change machine B's ip address to 12.168.121.38 and when I restart it, it can access the internet perfectly.
So this all points to some obscure setting in pfSense that allows 192.168.121.38 out, but 192.168.121.50 not.
Were could that be?</broadcast,multicast,up,lower_up></loopback,up,lower_up></broadcast,multicast,up,lower_up></loopback,up,lower_up>
-
Since finding out some address route to the internet and others not, I checked which actually do route. Turns out that only those with 1:1 NAT entries in pfSense can access the internet.
In Firewall: NAT: Outbound I have selected "Automatic outbound NAT rule generation (IPsec passthrough included) ". It seems that this is not sufficient.
What should I do to make this work as expected? Should I remove the 1:1 NAT and only put in inbound NAT rules?
-
I was stuck with this issue, no replies, so I went through all the settings again. Lo and behold! There was no upstream gateway set on the WAN port (although I'm sure it was there at some stage before).
After setting it, all is well.
