OSPF: pfSense and 2 Instances of VyOS in Separate Networks
-
I am experimenting with OSPF in my virtual home lab. I have pfSense VM and two instances of VyOS VMs connected to my router running Debian Server.
I have assigned a network subnet of 172.24.9.0/24 where 172.24.9.1 is the gateway to the Internet.
My setup is as follows:
Internet -> Debian Server (router running in LXC) -> pfSense (172.24.9.2)
Internet -> Debian Server (router) -> VyOS 1 (172.24.9.3)
Internet -> Debian Server (router) -> VyOS 2 (172.24.9.4)pfSense, VyOS 1, and VyOS 2 have the same OSPF area of 0.0.0.0. As for router-id:
pfSense: 172.24.255.1
VyOS 1: 172.24.255.2
VyOS 2: 172.24.255.3Private IP subnets for each instance:
pfSense: 10.249.0.0/24
VyOS 1: 10.249.1.0/24
VyOS 2: 10.249.2.0/24I am going to share my configuration, starting with pfSense, VyOS 1, and VyOS 2.
pfSense (using FRR OSPF):
OSPF settings:
Router ID: 172.24.255.1
Area: 0.0.0.0Interfaces settings:
WAN: 0.0.0.0 (area)
LAN: 0.0.0.0 (area), Non-broadcastFirewall:
WAN: Allow source IP 172.24.9.0/24
Outbound NAT:
Rule 1:
Interface: WAN
Source: any
Destination: any
NAT Address: WAN AddressRule 2:
Interface: WAN
Source: any
Destination: any
NAT Address: 10.249.0.0/16VyOS 1:
vyos@vyos-vlan2401r# show interfaces ethernet eth0 { address dhcp firewall { in { } local { } } hw-id 52:54:00:74:ba:4e } ethernet eth1 { address 10.249.1.1/24 firewall { in { } } hw-id 52:54:00:1e:8b:b0 } loopback lo { address 172.24.255.2/32 } [edit] vyos@vyos-vlan2401r# show protocols ospf area 0.0.0.0 { network 172.24.9.0/24 network 10.249.1.0/24 } log-adjacency-changes { } parameters { abr-type cisco router-id 172.24.255.2 } [edit] vyos@vyos-vlan2401r# show nat source rule 1 { outbound-interface eth0 translation { address masquerade } }VyOS 2:
vyos@vyos-vlan2402r# show interfaces ethernet eth0 { address dhcp hw-id 52:54:00:80:0d:2f } ethernet eth1 { address 10.249.2.1/24 hw-id 52:54:00:8a:c8:94 } loopback lo { address 172.24.255.3/32 } [edit] vyos@vyos-vlan2402r# show protocols ospf area 0.0.0.0 { network 172.24.9.0/24 network 10.249.2.0/24 } log-adjacency-changes { } parameters { abr-type cisco router-id 172.24.255.3 } [edit] vyos@vyos-vlan2402r# show nat source rule 1 { outbound-interface eth0 translation { address masquerade } } [edit]I decided to leave the firewall configurations for VyOS 1 and VyOS 2 out because this is what I consider my setup a "virtual test lab." My main router running in a Linux container (LXC) already has a firewall running NFtables.
There is repetition regarding area and router ID, however I wanted to provide an overview near the top of my post and provide deeper insight into my router setup.
So now I'm going to show some pings. I am going to first start out with VyOS 1 and VyOS 2, as the two instances have no trouble pinging each other.
vlan2401-host1 (LXC) connected to VyOS 1:
root@vmserver:/mnt/vm/lxc# lxc-attach vlan2401-host1 [root@vlan2401-host1 /]# ip a show eth0 2: eth0@if97: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:00:a6:d3:96 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.249.1.100/24 brd 10.249.1.255 scope global dynamic noprefixroute eth0 valid_lft 68611sec preferred_lft 68611sec inet6 fe80::ff:fea6:d396/64 scope link valid_lft forever preferred_lft forever [root@vlan2401-host1 /]# ping 10.249.0.101 PING 10.249.0.101 (10.249.0.101) 56(84) bytes of data. 64 bytes from 10.249.0.101: icmp_seq=1 ttl=62 time=1.44 ms 64 bytes from 10.249.0.101: icmp_seq=2 ttl=62 time=0.699 ms 64 bytes from 10.249.0.101: icmp_seq=3 ttl=62 time=0.752 ms ^C --- 10.249.0.101 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 23ms rtt min/avg/max/mdev = 0.699/0.962/1.437/0.338 ms [root@vlan2401-host1 /]# ping 10.249.2.100 PING 10.249.2.100 (10.249.2.100) 56(84) bytes of data. 64 bytes from 10.249.2.100: icmp_seq=1 ttl=62 time=0.789 ms 64 bytes from 10.249.2.100: icmp_seq=2 ttl=62 time=0.720 ms 64 bytes from 10.249.2.100: icmp_seq=3 ttl=62 time=0.742 ms ^C --- 10.249.2.100 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 32ms rtt min/avg/max/mdev = 0.720/0.750/0.789/0.036 msvlan2402-host1 (LXC) connected to VyOS 2:
root@vmserver:/mnt/vm/lxc# lxc-attach vlan2402-host1 [root@vlan2402-host1 /]# ip a show eth0 2: eth0@if95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:00:88:09:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.249.2.100/24 brd 10.249.2.255 scope global dynamic noprefixroute eth0 valid_lft 67799sec preferred_lft 67799sec inet6 fe80::ff:fe88:92b/64 scope link valid_lft forever preferred_lft forever [root@vlan2402-host1 /]# ping 10.249.0.101 PING 10.249.0.101 (10.249.0.101) 56(84) bytes of data. 64 bytes from 10.249.0.101: icmp_seq=1 ttl=62 time=1.19 ms 64 bytes from 10.249.0.101: icmp_seq=2 ttl=62 time=0.663 ms 64 bytes from 10.249.0.101: icmp_seq=3 ttl=62 time=0.840 ms ^C --- 10.249.0.101 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 17ms rtt min/avg/max/mdev = 0.663/0.899/1.194/0.220 ms [root@vlan2402-host1 /]# ping 10.249.1.100 PING 10.249.1.100 (10.249.1.100) 56(84) bytes of data. 64 bytes from 10.249.1.100: icmp_seq=1 ttl=62 time=0.845 ms 64 bytes from 10.249.1.100: icmp_seq=2 ttl=62 time=0.845 ms 64 bytes from 10.249.1.100: icmp_seq=3 ttl=62 time=0.795 ms ^C --- 10.249.1.100 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 31ms rtt min/avg/max/mdev = 0.795/0.828/0.845/0.033 msNote:
Both of the hosts listed above can ping 1.1.1.1. I want to try and keep my post short by showing the same pings to 1.1.1.1, but at the same time, I want to be thorough with all the details.
vlan2400-host1 (LXC) connected to pfSense:
Both of the hosts within the VyOS 1's and VyOS 2's network can ping each other and can ping pfSense. So here is where I run into trouble. pfSense cannot ping the hosts within VyOS 1's and VyOS 2's network. Allow me to repeat the NAT configuration for pfSense here:
Outbound NAT:
Rule 1:
Interface: WAN
Source: any
Destination: any
NAT Address: WAN AddressRule 2:
Interface: WAN
Source: any
Destination: any
NAT Address: 10.249.0.0/16Now let's show the pings:
root@vmserver:/mnt/vm/lxc# lxc-attach vlan2400-host1 [root@vlan2400-host1 /]# ip a show eth0 2: eth0@if98: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:00:0a:fd:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.249.0.101/24 brd 10.249.0.255 scope global dynamic noprefixroute eth0 valid_lft 6742sec preferred_lft 6742sec inet6 fe80::ff:fe0a:fdee/64 scope link valid_lft forever preferred_lft forever [root@vlan2400-host1 /]# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=32.7 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=31.4 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 2 received, 33.3333% packet loss, time 5ms rtt min/avg/max/mdev = 31.381/32.058/32.735/0.677 ms [root@vlan2400-host1 /]# ping 10.249.1.100 PING 10.249.1.100 (10.249.1.100) 56(84) bytes of data. ^C --- 10.249.1.100 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 55msWhat if I flip the rules around?
Outbound NAT:
Rule 1:
Interface: WAN
Source: any
Destination: any
NAT Address: 10.249.0.0/16Rule 2:
Interface: WAN
Source: any
Destination: any
NAT Address: WAN AddressNow I can ping the hosts in VyOS 1's and VyOS 2's network, but rather than showing the pings, all I can say is I can't ping an IP address over the Internet until I reverse the outbound rules.
Final thoughts and questions:
Initially, when I had trouble pinging the hosts behind other routers in my virtual test lab, I was troubleshooting the firewall until I did a search in the Internet and learned that it is due to a NAT. I tried hybrid outbound (here's another thread about outbound rule generation and FRR), but that did not work for me.
NAT seems to be working fine in both VyOS 1 and 2, so Is there anything I can do to resolve this NAT issue?
Also, is all of my information seem overly abundant? Or should I include a diagram? Let me know if I need to clean it up so that all information makes sense to anyone reading my post.
PS: I am A+ (January 2018), Network+ (June 2018), and Security+ certified (December, 2019) and have no on-the-job IT experience. Network+ did not get my hands dirty on networking except that I have multiple networks connected to my Debian server.
-
Hey everyone. I found a solution to my problem! Here's my new configuration for outbound NAT:
Rule 1:
Interface: WAN
Source: any
Destination: 10.249.0.0/16
NAT Address: 10.249.0.0/16Rule 2:
Interface: WAN
Source: any
Destination: any
NAT Address: WAN AddressOkay, so let me explain how I did it. First, the rules above have to be in specific order for OSPF and Internet traffic to work.
Let's say I want to talk to 10.249.2.100. pfSense will forward the traffic out to 172.24.9.x subnet to 172.24.9.4. Perhaps a traceroute might help in understanding what pfSense is doing:
[root@vlan2400-host1 /]# traceroute 10.249.2.100 traceroute to 10.249.2.100 (10.249.2.100), 30 hops max, 60 byte packets 1 pfSense-vlan2400r.lab.graysonpeddie.lan (10.249.0.1) 0.228 ms 0.210 ms 0.196 ms 2 172.24.9.4 (172.24.9.4) 0.785 ms 0.923 ms 0.933 ms 3 10.249.2.100 (10.249.2.100) 0.917 ms 0.901 ms 0.886 ms [root@vlan2400-host1 /]# traceroute 10.249.1.100 traceroute to 10.249.1.100 (10.249.1.100), 30 hops max, 60 byte packets 1 pfSense-vlan2400r.lab.graysonpeddie.lan (10.249.0.1) 0.258 ms 0.241 ms 0.225 ms 2 172.24.9.3 (172.24.9.3) 0.781 ms 0.765 ms 0.929 ms 3 10.249.1.100 (10.249.1.100) 1.113 ms 1.098 ms 1.203 msAs shown from the terminal above, I am able to ping both hosts. I could show the ping command used, however, just know that pinging hosts work. Both the destination address and "Other Subnet" field must have the same subnet for it to work. That's 10.249.0.0/16.
Okay, so here's my question. What if I want to reach out to the Internet by pinging 1.1.1.1?
Refer to the two rules above. Does 1.1.1.1 fall into the destination address's subnet for rule 1? No, so pfSense goes down to rule 2. Because the destination address is set to "any" and translation address is set to "Interface address." That rule matches, so pfSense can return a response from 1.1.1.1 back to my originating host.
If the destination address for rule 1 is set to "any," then pfSense will try to match the first rule and ping response from 1.1.1.1 will not return back to the originating host.
Everything works very well and I seem to have a good understanding of how NAT works in addition to know how OSPF works. If anyone tries out my solution, let me know how it goes.
Really, I should have figured that out in the first place, but because I asked for help and didn't get any response from anyone, I will leave my solution here in my new post so that anyone can gain knowledge of how to get pfSense to talk to other routers with OSPF and NAT.
Oh, and there's no such thing as "masquerade" in pfSense. There's either an "interface address" or "other subnet" so it's not as simple as setting translation address to "masquerade" as compared to VyOS or NFTables/iptables in Debian server.
-
Note: I'm going to make a new post instead of editing my existing one. I did not see an error message when I submit my changes, so I'm going to make a new post documenting my experience with NAT.
Okay, so I'm going to document my experience with NAT in pfSense.
Let's change the destination to 10.249.1.0/24.
Rule 1:
Interface: WAN
Source: any
Destination: 10.249.1.0/24
Translation Address: 10.249.0.0/16Pinging 10.249.1.100 works and pinging 10.249.2.100 does not work.
So, changing the destination to 10.249.2.0/24 makes pinging 2.100 working, but 1.100 does not.
I'm going to reset the destination to 10.249.0.0/24 and modify the translation address to 10.249.0.0/24. I'm going to give it a try:
Rule 1:
Interface: WAN
Source: any
Destination: 10.249.0.0/16
Translation Address: 10.249.0.0/24Setting translation address to 10.249.0.0/24 works fine when pinging .1.100 and .2.100.
What happens if I set the translation address to just interface address?
Setting the first rule to WAN address won't work because pfSense does not seem to reach back to .0.101 when I try to ping .1.100 and .2.100.
So it makes sense to think that pfSense will translate the packets back to the originating host's IP address and not the interface address (172.24.9.2). I don't know how that works, but hey, it works. Maybe somebody could explain why using the local /24 LAN subnet works.
