Routing some LAN address to internet doesn't work, others do.

lifeboy

I upgraded not too long ago and noticed yesterday that no servers behind pfSense were receiving any updates. It no traffic is going out, only incoming traffic is working.

pfSense runs in a KVM virtual machine under Proxmox and all was well using pfSense 2.2.2.

Both ports are bridged in the VM host platform configured as follows:

# cat /etc/network/interfaces 
# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth3
iface eth3 inet manual

iface eth2 inet manual

auto bond0
iface bond0 inet manual
	slaves eth0 eth1
	bond_miimon 100
	bond_mode 802.3ad
	bond_xmit_hash_policy layer2

auto vmbr0
iface vmbr0 inet static
	address  192.168.121.33
	netmask  255.255.255.0
	gateway  192.168.121.1
	bridge_ports bond0
	bridge_stp off
	bridge_fd 0
#	gateway  192.168.121.1

auto vmbr1
iface vmbr1 inet manual
	bridge_ports eth3
	bridge_stp off
	bridge_fd 0

Therefore the KVM host cannot reach anything on the WAN port, since there is no IP address allocated.

The pfSense basic config is:
1 WAN port (vtnet1)
1 LAN port (vtnet0)
7 Virtual IP defined
I connect to the LAN via IPsec/L2tp and can access all the servers/services in this way.

Problem is that no traffic is passed beyond the WAN port. I can ping xxx.71.69.226 (the WAN port address), but not 225, which is the gateway. However, from xxx.71.69.226 I can ping the 225 gateway and anything else outside.

Here's the routing table:

$ netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            41.71.69.225       UGS      vtnet1
xxx.71.69.224/28    link#2             U        vtnet1
xxx.71.69.226       link#2             UHS         lo0
xxx.71.69.228       link#2             UHS         lo0
xxx.71.69.228/32    link#2             U        vtnet1
xxx.71.69.229       link#2             UHS         lo0
xxx.71.69.229/32    link#2             U        vtnet1
xxx.71.69.230       link#2             UHS         lo0
xxx.71.69.230/32    link#2             U        vtnet1
xxx.71.69.231       link#2             UHS         lo0
xxx.71.69.231/32    link#2             U        vtnet1
xxx.71.69.232       link#2             UHS         lo0
xxx.71.69.232/32    link#2             U        vtnet1
xxx.71.69.233       link#2             UHS         lo0
xxx.71.69.233/32    link#2             U        vtnet1
xxx.71.69.234       link#2             UHS         lo0
xxx.71.69.234/32    link#2             U        vtnet1
127.0.0.1          link#5             UH          lo0
192.168.120.240    link#7             UH        l2tp0
192.168.120.241    link#8             UH        l2tp1
192.168.120.248    link#7             UHS         lo0
192.168.121.0/24   link#1             U        vtnet0
192.168.121.1      link#1             UHS         lo0

The default gateway should receive all the traffic, but it's not happening. Is there a freeBSD method to disable/enable routing (like Linux has), or is routing always allowed by default?

I don't know how to troubleshoot this any further, so I need some help please.

lifeboy

I also see this:

cat /etc/defaults/rc.conf | grep gateway
defaultrouter="NO"		# Set to default gateway (or NO).
gateway_enable="NO"		# Set to YES if this host will be a gateway.
ipxgateway_enable="NO"		# Set to YES to enable IPX routing.
forward_sourceroute="NO"	# do source routing (only if gateway_enable is set to "YES")
ipv6_defaultrouter="NO"		# Set to IPv6 default gateway (or NO).
ipv6_gateway_enable="NO"	# Set to YES if this host will be a gateway.
				# ipv6_gateway_enable="NO".

Is that the way it's supposed to be with pfSense? It's seems like freeBSD's routing is turned off, or what?

doktornotor

/etc/defaults/rc.conf is NOT the place to mess with.

lifeboy

@doktornotor:

/etc/defaults/rc.conf is NOT the place to mess with.

Not, I won't mess with anything, especially in freeBSD! I was just looking around to see what I can discover to give me an indication of what may be causing the problem. I'm pretty sure I didn't change anything since I upgraded, since everything "just worked" once it was set up.

So what do I do next?

Gertjan

Idea : remove temporary the Promox thing. From what I understand, this 'tool' does have huge effect on routing because it assigns port/address to internal VM's etc.

pfSense that stops routing to the outside : of course, we've seen that before on this forum. Like people mix up LAN & WAN, using the same IP mask on LAN and WAN, these kinds of issues (you get the picture ;))
The thing is, you have a system-in-a-system and you declare that the 'system' as a whole doesn't work.
One of the solutions is : break it up in pieces to analyze them all.

Btw: I've no experience with "put a firewall in a proxmox" - I tend to assign a dedicated box to that kind of tasks.
It seems more a proxmox issue to me.

lifeboy

(I have renamed the question to more correctly describe the symptoms)

If have done a lot of digging and testing re this issue and the news is somewhat better than I at first reported:

The routing works from some of the virtual machines, but not from others.

To illustrate the point:

machine A: www

www:~$ ip addr show
1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether ca:04:b6:f7:84:c4 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.38/24 brd 192.168.121.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::c804:b6ff:fef7:84c4/64 scope link
valid_lft forever preferred_lft forever

@www:~$ ip route show
default via 192.168.121.1 dev eth0
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.38

machine B: redmine

redmine:~$ ip addr show
1: lo: <loopback,up,lower_up>mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 3a:50:f6:52:b9:24 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.50/24 brd 192.168.121.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::3850:f6ff:fe52:b924/64 scope link
valid_lft forever preferred_lft forever

redmine:~$ ip route show
default via 192.168.121.1 dev eth0
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.50

Both machines are configured with the same virtual hardware, yet machine A can access the internet (anything beyond the pfSense WAN port) while machine B can only access up to the WAN port, but not beyond that.

I have shut down machine A and then change machine B's ip address to 12.168.121.38 and when I restart it, it can access the internet perfectly.

So this all points to some obscure setting in pfSense that allows 192.168.121.38 out, but 192.168.121.50 not.

Were could that be?</broadcast,multicast,up,lower_up></loopback,up,lower_up></broadcast,multicast,up,lower_up></loopback,up,lower_up>

lifeboy

Since finding out some address route to the internet and others not, I checked which actually do route. Turns out that only those with 1:1 NAT entries in pfSense can access the internet.

In Firewall: NAT: Outbound I have selected "Automatic outbound NAT rule generation (IPsec passthrough included) ". It seems that this is not sufficient.

What should I do to make this work as expected? Should I remove the 1:1 NAT and only put in inbound NAT rules?

lifeboy

I was stuck with this issue, no replies, so I went through all the settings again. Lo and behold! There was no upstream gateway set on the WAN port (although I'm sure it was there at some stage before).

After setting it, all is well.