Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    OSPF: pfSense and 2 Instances of VyOS in Separate Networks

    NAT
    1
    3
    482
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GraysonPeddieG
      GraysonPeddie
      last edited by

      I am experimenting with OSPF in my virtual home lab. I have pfSense VM and two instances of VyOS VMs connected to my router running Debian Server.

      I have assigned a network subnet of 172.24.9.0/24 where 172.24.9.1 is the gateway to the Internet.

      My setup is as follows:

      Internet -> Debian Server (router running in LXC) -> pfSense (172.24.9.2)
      Internet -> Debian Server (router) -> VyOS 1 (172.24.9.3)
      Internet -> Debian Server (router) -> VyOS 2 (172.24.9.4)

      pfSense, VyOS 1, and VyOS 2 have the same OSPF area of 0.0.0.0. As for router-id:

      pfSense: 172.24.255.1
      VyOS 1: 172.24.255.2
      VyOS 2: 172.24.255.3

      Private IP subnets for each instance:

      pfSense: 10.249.0.0/24
      VyOS 1: 10.249.1.0/24
      VyOS 2: 10.249.2.0/24

      I am going to share my configuration, starting with pfSense, VyOS 1, and VyOS 2.

      pfSense (using FRR OSPF):

      OSPF settings:

      Router ID: 172.24.255.1
      Area: 0.0.0.0

      Interfaces settings:

      WAN: 0.0.0.0 (area)
      LAN: 0.0.0.0 (area), Non-broadcast

      Firewall:

      WAN: Allow source IP 172.24.9.0/24

      Outbound NAT:

      Rule 1:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: WAN Address

      Rule 2:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: 10.249.0.0/16

      VyOS 1:

      vyos@vyos-vlan2401r# show interfaces
 ethernet eth0 {
     address dhcp
     firewall {
         in {
         }
         local {
         }
     }
     hw-id 52:54:00:74:ba:4e
 }
 ethernet eth1 {
     address 10.249.1.1/24
     firewall {
         in {
         }
     }
     hw-id 52:54:00:1e:8b:b0
 }
 loopback lo {
     address 172.24.255.2/32
 }
[edit]
vyos@vyos-vlan2401r# show protocols ospf
 area 0.0.0.0 {
     network 172.24.9.0/24
     network 10.249.1.0/24
 }
 log-adjacency-changes {
 }
 parameters {
     abr-type cisco
     router-id 172.24.255.2
 }
[edit]
vyos@vyos-vlan2401r# show nat source 
 rule 1 {
     outbound-interface eth0
     translation {
         address masquerade
     }
 }

      VyOS 2:

      vyos@vyos-vlan2402r# show interfaces
 ethernet eth0 {
     address dhcp
     hw-id 52:54:00:80:0d:2f
 }
 ethernet eth1 {
     address 10.249.2.1/24
     hw-id 52:54:00:8a:c8:94
 }
 loopback lo {
     address 172.24.255.3/32
 }
[edit]
vyos@vyos-vlan2402r# show protocols ospf
 area 0.0.0.0 {
     network 172.24.9.0/24
     network 10.249.2.0/24
 }
 log-adjacency-changes {
 }
 parameters {
     abr-type cisco
     router-id 172.24.255.3
 }
[edit]
vyos@vyos-vlan2402r# show nat source 
 rule 1 {
     outbound-interface eth0
     translation {
         address masquerade
     }
 }
[edit]

      I decided to leave the firewall configurations for VyOS 1 and VyOS 2 out because this is what I consider my setup a "virtual test lab." My main router running in a Linux container (LXC) already has a firewall running NFtables.

      There is repetition regarding area and router ID, however I wanted to provide an overview near the top of my post and provide deeper insight into my router setup.

      So now I'm going to show some pings. I am going to first start out with VyOS 1 and VyOS 2, as the two instances have no trouble pinging each other.

      vlan2401-host1 (LXC) connected to VyOS 1:

      root@vmserver:/mnt/vm/lxc# lxc-attach vlan2401-host1
[root@vlan2401-host1 /]# ip a show eth0
2: eth0@if97: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:a6:d3:96 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.249.1.100/24 brd 10.249.1.255 scope global dynamic noprefixroute eth0
       valid_lft 68611sec preferred_lft 68611sec
    inet6 fe80::ff:fea6:d396/64 scope link 
       valid_lft forever preferred_lft forever
[root@vlan2401-host1 /]# ping 10.249.0.101
PING 10.249.0.101 (10.249.0.101) 56(84) bytes of data.
64 bytes from 10.249.0.101: icmp_seq=1 ttl=62 time=1.44 ms
64 bytes from 10.249.0.101: icmp_seq=2 ttl=62 time=0.699 ms
64 bytes from 10.249.0.101: icmp_seq=3 ttl=62 time=0.752 ms
^C
--- 10.249.0.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 23ms
rtt min/avg/max/mdev = 0.699/0.962/1.437/0.338 ms
[root@vlan2401-host1 /]# ping 10.249.2.100
PING 10.249.2.100 (10.249.2.100) 56(84) bytes of data.
64 bytes from 10.249.2.100: icmp_seq=1 ttl=62 time=0.789 ms
64 bytes from 10.249.2.100: icmp_seq=2 ttl=62 time=0.720 ms
64 bytes from 10.249.2.100: icmp_seq=3 ttl=62 time=0.742 ms
^C
--- 10.249.2.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 32ms
rtt min/avg/max/mdev = 0.720/0.750/0.789/0.036 ms

      vlan2402-host1 (LXC) connected to VyOS 2:

      root@vmserver:/mnt/vm/lxc# lxc-attach vlan2402-host1
[root@vlan2402-host1 /]# ip a show eth0
2: eth0@if95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:88:09:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.249.2.100/24 brd 10.249.2.255 scope global dynamic noprefixroute eth0
       valid_lft 67799sec preferred_lft 67799sec
    inet6 fe80::ff:fe88:92b/64 scope link 
       valid_lft forever preferred_lft forever
[root@vlan2402-host1 /]# ping 10.249.0.101
PING 10.249.0.101 (10.249.0.101) 56(84) bytes of data.
64 bytes from 10.249.0.101: icmp_seq=1 ttl=62 time=1.19 ms
64 bytes from 10.249.0.101: icmp_seq=2 ttl=62 time=0.663 ms
64 bytes from 10.249.0.101: icmp_seq=3 ttl=62 time=0.840 ms
^C
--- 10.249.0.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 17ms
rtt min/avg/max/mdev = 0.663/0.899/1.194/0.220 ms
[root@vlan2402-host1 /]# ping 10.249.1.100
PING 10.249.1.100 (10.249.1.100) 56(84) bytes of data.
64 bytes from 10.249.1.100: icmp_seq=1 ttl=62 time=0.845 ms
64 bytes from 10.249.1.100: icmp_seq=2 ttl=62 time=0.845 ms
64 bytes from 10.249.1.100: icmp_seq=3 ttl=62 time=0.795 ms
^C
--- 10.249.1.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 31ms
rtt min/avg/max/mdev = 0.795/0.828/0.845/0.033 ms

      Note:

      Both of the hosts listed above can ping 1.1.1.1. I want to try and keep my post short by showing the same pings to 1.1.1.1, but at the same time, I want to be thorough with all the details.

      vlan2400-host1 (LXC) connected to pfSense:

      Both of the hosts within the VyOS 1's and VyOS 2's network can ping each other and can ping pfSense. So here is where I run into trouble. pfSense cannot ping the hosts within VyOS 1's and VyOS 2's network. Allow me to repeat the NAT configuration for pfSense here:

      Outbound NAT:

      Rule 1:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: WAN Address

      Rule 2:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: 10.249.0.0/16

      Now let's show the pings:

      root@vmserver:/mnt/vm/lxc# lxc-attach vlan2400-host1
[root@vlan2400-host1 /]# ip a show eth0
2: eth0@if98: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:0a:fd:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.249.0.101/24 brd 10.249.0.255 scope global dynamic noprefixroute eth0
       valid_lft 6742sec preferred_lft 6742sec
    inet6 fe80::ff:fe0a:fdee/64 scope link 
       valid_lft forever preferred_lft forever
[root@vlan2400-host1 /]# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=32.7 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=31.4 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 2 received, 33.3333% packet loss, time 5ms
rtt min/avg/max/mdev = 31.381/32.058/32.735/0.677 ms
[root@vlan2400-host1 /]# ping 10.249.1.100
PING 10.249.1.100 (10.249.1.100) 56(84) bytes of data.
^C
--- 10.249.1.100 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 55ms

      What if I flip the rules around?

      Outbound NAT:

      Rule 1:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: 10.249.0.0/16

      Rule 2:
      Interface: WAN
      Source: any
      Destination: any
      NAT Address: WAN Address

      Now I can ping the hosts in VyOS 1's and VyOS 2's network, but rather than showing the pings, all I can say is I can't ping an IP address over the Internet until I reverse the outbound rules.

      Final thoughts and questions:

      Initially, when I had trouble pinging the hosts behind other routers in my virtual test lab, I was troubleshooting the firewall until I did a search in the Internet and learned that it is due to a NAT. I tried hybrid outbound (here's another thread about outbound rule generation and FRR), but that did not work for me.

      NAT seems to be working fine in both VyOS 1 and 2, so Is there anything I can do to resolve this NAT issue?

      Also, is all of my information seem overly abundant? Or should I include a diagram? Let me know if I need to clean it up so that all information makes sense to anyone reading my post.

      PS: I am A+ (January 2018), Network+ (June 2018), and Security+ certified (December, 2019) and have no on-the-job IT experience. Network+ did not get my hands dirty on networking except that I have multiple networks connected to my Debian server.

      1 Reply Last reply Reply Quote 0
      • GraysonPeddieG
        GraysonPeddie
        last edited by GraysonPeddie

        Hey everyone. I found a solution to my problem! Here's my new configuration for outbound NAT:

        Rule 1:
        Interface: WAN
        Source: any
        Destination: 10.249.0.0/16
        NAT Address: 10.249.0.0/16

        Rule 2:
        Interface: WAN
        Source: any
        Destination: any
        NAT Address: WAN Address

        Okay, so let me explain how I did it. First, the rules above have to be in specific order for OSPF and Internet traffic to work.

        Let's say I want to talk to 10.249.2.100. pfSense will forward the traffic out to 172.24.9.x subnet to 172.24.9.4. Perhaps a traceroute might help in understanding what pfSense is doing:

        [root@vlan2400-host1 /]# traceroute 10.249.2.100
traceroute to 10.249.2.100 (10.249.2.100), 30 hops max, 60 byte packets
 1  pfSense-vlan2400r.lab.graysonpeddie.lan (10.249.0.1)  0.228 ms  0.210 ms  0.196 ms
 2  172.24.9.4 (172.24.9.4)  0.785 ms  0.923 ms  0.933 ms
 3  10.249.2.100 (10.249.2.100)  0.917 ms  0.901 ms  0.886 ms
[root@vlan2400-host1 /]# traceroute 10.249.1.100
traceroute to 10.249.1.100 (10.249.1.100), 30 hops max, 60 byte packets
 1  pfSense-vlan2400r.lab.graysonpeddie.lan (10.249.0.1)  0.258 ms  0.241 ms  0.225 ms
 2  172.24.9.3 (172.24.9.3)  0.781 ms  0.765 ms  0.929 ms
 3  10.249.1.100 (10.249.1.100)  1.113 ms  1.098 ms  1.203 ms

        As shown from the terminal above, I am able to ping both hosts. I could show the ping command used, however, just know that pinging hosts work. Both the destination address and "Other Subnet" field must have the same subnet for it to work. That's 10.249.0.0/16.

        Okay, so here's my question. What if I want to reach out to the Internet by pinging 1.1.1.1?

        Refer to the two rules above. Does 1.1.1.1 fall into the destination address's subnet for rule 1? No, so pfSense goes down to rule 2. Because the destination address is set to "any" and translation address is set to "Interface address." That rule matches, so pfSense can return a response from 1.1.1.1 back to my originating host.

        If the destination address for rule 1 is set to "any," then pfSense will try to match the first rule and ping response from 1.1.1.1 will not return back to the originating host.

        Everything works very well and I seem to have a good understanding of how NAT works in addition to know how OSPF works. If anyone tries out my solution, let me know how it goes.

        Really, I should have figured that out in the first place, but because I asked for help and didn't get any response from anyone, I will leave my solution here in my new post so that anyone can gain knowledge of how to get pfSense to talk to other routers with OSPF and NAT.

        Oh, and there's no such thing as "masquerade" in pfSense. There's either an "interface address" or "other subnet" so it's not as simple as setting translation address to "masquerade" as compared to VyOS or NFTables/iptables in Debian server.

        1 Reply Last reply Reply Quote 0
        • GraysonPeddieG
          GraysonPeddie
          last edited by

          Note: I'm going to make a new post instead of editing my existing one. I did not see an error message when I submit my changes, so I'm going to make a new post documenting my experience with NAT.

          Okay, so I'm going to document my experience with NAT in pfSense.

          Let's change the destination to 10.249.1.0/24.

          Rule 1:
          Interface: WAN
          Source: any
          Destination: 10.249.1.0/24
          Translation Address: 10.249.0.0/16

          Pinging 10.249.1.100 works and pinging 10.249.2.100 does not work.

          So, changing the destination to 10.249.2.0/24 makes pinging 2.100 working, but 1.100 does not.

          I'm going to reset the destination to 10.249.0.0/24 and modify the translation address to 10.249.0.0/24. I'm going to give it a try:

          Rule 1:
          Interface: WAN
          Source: any
          Destination: 10.249.0.0/16
          Translation Address: 10.249.0.0/24

          Setting translation address to 10.249.0.0/24 works fine when pinging .1.100 and .2.100.

          What happens if I set the translation address to just interface address?

          Setting the first rule to WAN address won't work because pfSense does not seem to reach back to .0.101 when I try to ping .1.100 and .2.100.

          So it makes sense to think that pfSense will translate the packets back to the originating host's IP address and not the interface address (172.24.9.2). I don't know how that works, but hey, it works. Maybe somebody could explain why using the local /24 LAN subnet works.

          1 Reply Last reply Reply Quote 0
          • First post
            Last post