Slightly strange setup :: help/pointers appreciated
-
Dear List members
I am trying to figure out a way to set up a rather unusual setup. I have googled as much as I can, have been through all the pf docs available (specially http://www.pfsense.org/mirror.php?section=tutorials/mobile_ipsec/, didn't help much). Any pointers / suggestion is appreciated.
In simpler terms, I have the following setup
real IP network x.y.z.64/27 network
private network 192.168.3.0/24 network
private network 10.0.0.0/24 networkthe connection diagram would look like this
x.y.z.69
|
|a1
–----------------
| pfSense box 1 |
------------------
|a2 | 192.168.3.1
|
private network |
192.168.3.0/24 __|
|
|
| b1 | 192.168.3.2
–----------------
| pfSense box 2 |
------------------
|b2 | 10.0.0.1
private network |
10.0.0.0/24 ___|The objective is to establish a IPSec tunnel between a2 and b1, so that all traffic from 10.0.0.0/24 network ends up directly at the realIP land, ensuring prevention of evesdropping from the 192.168.3.0/24 network.
I have made several different attempts, which I'll try to explain below. but no matter what I do the setup does not seem to work. Therefore, requesting pointers/suggestions.
in all cases:
a. the gateway of box 1 is x.y.z.65 (default gateway for the realIP world)
b. in general the 192.168.3.0/24 network has its own default gateway, which is a different box
c. the gateway of box 2 is 192.168.3.1 – since all traffic is expected to pass via box 1
d. IPSec interface has all allowed in all directions
e. all the setups below have been tried in setups where
1. box 1 initiating a tunnel to box 2, box 2 initiating a tunnel to box 1
2. box 1 accepting connection from a mobile device, box 2 acting as a
mobile device initiating connection to box 1
f. In all cases, the tunnel went up alright and I could ping the immediate devices a1, a2, b1, b2 from both the boxes.case 1 :: 192 is LAN, rest is WAN
I have configured both the boxes as 192 network to be the LAN network, and both x.y.z and 10.0.0 to be WAN network -- since I do not wish the 10 network to interfere in general with 192 network (and vice-versa). Configured IPSec between a2 and b1. The link was up, and I could ping both a1 and b2 from the pfSense boxes, but no traffic went between the networkscase 2 :: 192 is WAN, rest is LAN
similar to above, but reversed view from the box. Again I could ping the immediate interfaces, but not the other hosts in any of the networkcase 3 and 4 :: same as the two above, but this time I configured another real IP in box 1 to do NAT, so that private network does not expect a direct routing
case 5 :: a1 WAN, a2 LAN, b1 WAN, b2 LAN
In this case, as long as the IPSec tunnel is not up, things are working fine and I can even browse the internet. However, as soon as the IPSec tunnel is active, it stops workingcase 6 :: as case 5, but with additional NAT on the realIP
I can ping the immediate interfaces as described above, but no traffic.I tried both with PFSense 2.0 BETA and stable 1.2.3 -- not mixed environment, both the boxes had exactly the same version installed in all the above tests. While using 1.2.3, the Virtual IP (in box 1) was Proxy ARP, while using 2.0 the virtual IP was IP Alias. I have lost track on how many times I have installed pfsense so far. Guess my brain is looking forward to a little break after nearly 10 days of fighting over this.
to make matters slighly worse, I have just been informed that there are two hosts in the 10.0.0 network which will need to be accessed from the 192 network (any host, or maybe certain hosts) for file sharing and RDP/VNC. I am guessing that would be simple via 1:1 mapping (i.e., one of the 192 address mapped to one of the 10 address) but have never done so. my experience with pfSense only goes to that far where I use it as a firewall/gateway for the network -- and am extremely pleased with the performance and functionality (DNS, DHCP, bandwidthd, nmap, etc.).
Again, any suggestion/pointer is highly appreciated -- as I believe I have exhausted all possible options that I could find in the mailing lists and all the links I could find via google.
Thanks to all
Shamim -
I think you're probably trying to do this the hard way - is there any reason you can't add another interface to one of the pfSense boxes?
Even if you can't, what's the actual risk (as opposed to the theoretical risk) of somebody performing an ARP poisoning attack (or similar) against your switch between the 2 pfSense boxes?
Alternatively, how about switching the networks around so that the traffic for the 10.0.0.0/24 network doesn't go through the 192.168.3.0/24 network, but the other way around? That will make your firewall rules much simpler and make it much easier to allow access to certain hosts on the 10.0.0.0/24 network from the 192.168.3.0/24 network.
-
To answer your questions, as much as I can
I think you're probably trying to do this the hard way - is there any reason you can't add another interface to one of the pfSense boxes?
No reason. I can stick in another card anytime I wish (the card is sitting there doing nothing as we speak). But in that event what kind of configuration do you suggest?
Even if you can't, what's the actual risk (as opposed to the theoretical risk) of somebody performing an ARP poisoning attack (or similar) against your switch between the 2 pfSense boxes?
frankly speaking, I do not know. as far as I can tell, the two pfSense boxes are connecting to two different switches – both at secured location (under lock and key). The two switches are communicating over fibre (insecure if you consider people might actually try to fiddle with that -- even I don't know which route they take).
Alternatively, how about switching the networks around so that the traffic for the 10.0.0.0/24 network doesn't go through the 192.168.3.0/24 network, but the other way around? That will make your firewall rules much simpler and make it much easier to allow access to certain hosts on the 10.0.0.0/24 network from the 192.168.3.0/24 network.
If I understand you correctly, you are suggesting a third interface on the second box and allocating certain IPs and then match them to internal IPs (1:1 mapping). Well, as I stated earlier, it can be done with no problem. But the main issue is to make sure that nothing/nobody can fiddle with the traffic between the boxes and the people from 10 network cannot access the 192 network at all, people at 192 network has access to ONLY designated boxes, and the people from 10 network is behind a NAT at the realIP world – that is the main objective.
I am not sure if I am getting this wrong -- since none of my google search indicated anything. Maybe I am not using the correct wording/terminology for the search -- but as stated, could not find anything in-favour/against what I am trying to do. One thread in this list suggested OpenVPN as a choice -- maybe I should give that a try. I have nothing against OpenVPN, just thought IPSec would be much more effective in this scenerio since the tunnel is created over the same subnet.
If I wish to take the OpenVPN path, can anyone suggest any how-to/tutorial that explains the setup for terminating the other end of the tunnel at a realIP world and has NAT enabled? Since most of the setups deal with private to private network, the NAT-end is expected to be quite rare, as far as I understand.
Any other suggestion/pointers?
Thanks to all for at least viewing my issue -- maybe it has given some food for thought ;D
Happy new year to all -
To answer your questions, as much as I can
I think you're probably trying to do this the hard way - is there any reason you can't add another interface to one of the pfSense boxes?
No reason. I can stick in another card anytime I wish (the card is sitting there doing nothing as we speak). But in that event what kind of configuration do you suggest?
If you put another interface into box 1 then you can simply directly connect the 10.0.0.0/24 network to it. Most of the problems you raised largely go away at that point, no issues of network traffic being sniffed and no complex routing issues.
At that point your diagram looks like
x.y.z.69
|
|a1
–----------------
| pfSense box 1 |
------------------
192.168.3.1 |a2 a3| 10.0.0.1
| |
private network | | private network
192.168.3.0/24 | | 10.0.0.0/24Now the traffic from either network to the Internet goes direct and you can easily create rules to allow access between the networks to suit yourself.
-
I think I might not have made the picture clearer earlier – I'll blame it all to the many sleepless nights and brain damage at my end ;D. The two boxes in question are physically located at two ends -- and the only path from one to the other is via the fibre link (192 network). If I wish to take your path, I will need to do a new layout of cabling from one end of my campus to the other -- which is NOT an option. I cannot use even vlans -- if you understand what I mean. Providing support for a small not-so-important group is not meant to be simple -- but that is what makes it more challenging and hence interesting.
As it happens, these two boxes and a reasonable amount of IPs (in all networks, including a few from the realIP) is available to be put to use as I feel absolutely necessary, and I have to find my way around it.
The place is closed at the moment, and the next time I go back is on the 4th, the solution ideally should be in place by the 5th latest. I am trying to set up a machine now where I might be able to run a few virtual instances of pfSense and see if I can achieve anything.
Thanks to all once again.
-
You could make number 2 a bridge, that should work.
However, if the fibre goes from one pfSense to the other, and not via switches, there isn't any way for an average person to sniff the traffic. Otherwise, any VPN between the LAN interface of number 1 and the WAN interface of number 2 should work (not from WAN to WAN).
-
Thank you for the suggestion, but could you please be a bit more specific?
what I am trying to understand is, when you say box one, I am gathering a1 is WAN and a2 is LAN, but when it comes to box2, which one are you referring to as LAN and WAN?
if I set a2 and b1 as LAN (both facing the other way as "outside"), do you suppose I could do a LAN to LAN tunnel? I know I can have a tunnel like that (it establishes itself without any error) – the only problem is it does not take data from one side to the other.
I am almost done setting up 4 pfSense virtual machines into my PC, now I can try to mimic a similar setup and see if I can ping from one end of the network to the other, or at least traceroute. Would appreciate if you could please let me know.
Thanks again
-
WAN is always the "outside" interface, LAN the "inside". Generally when setting up tunnels you set them up between the interfaces that connect the systems (in my experience). If you're having trouble passing data it's probably a routing table problem.
I think you'd probably be better off, if you can, replacing box 2 with a fibre connected switch and hooking the other end into a third interface on box 1. As I said though, if the fibre goes between the 2 pfSense boxes there isn't any way for others on the 192.168.3.0/24 network to sniff the traffic.
-
Thank you for your response. yes, I do understand WAN is outside and LAN is inside – but then again I am not very bright and for me anything and everything could be either inside or outside -- a matter of perspective. Hence I requested a little more detail (based on my original diagram) on which interface would be which for your suggestion to be tested.
As for direct fibre link, that is not possible. The only fibre link is between two switches, as I stated earlier. The switch near box 1 is connected to another switch near box 2 via fibre. switch near box 2 is solely used by the 192 series IPs (no chance of a vlan either, as stated earlier). Hence I am having to think of a secure tunnel to get the traffic -- while keeping both the network free from each others eyes.
If I cannot do it the ipsec way -- because of my stupidity and lack of understanding of how the tunnel should work -- maybe I should abandon the attempts or at least stop bugging people on forums.
thanks to everyone for at least viewing my little problem.
-
I've always found bugging people on forums an effective way of learning (even if sometimes just new swear words ;) ).
I've never tried to set up an IPsec, or any other, tunnel between 2 pfSense boxes, most of my site to site experience is with dedicated VPN devices or Cisco kit. That said, there are 2 stickies in the OpenVPN forum that cover using OpenVPN for site to site and there is a tutorial on IPsec between 2 pfSense hosts.
-
yap, seen them, followed them all even before posting the problem here, nada. As I have said before, in all events the links are established fine. I can even ping the nearest interface, but the packets simply does not route to the other side and vice versa. For example, as per my diagram, I can ping from a1 to b1, or b2 to a2. I tried with openvpn and i can even ping the other end of the tunnel (I used 169.254.13.0/24 as tunnel network), but somehow it is not going to the other side at all. I have checked the routing table and all seemed to be alright – no noticeable sign of error. But somehow the packets are not being routed. To be on the safe side, I even added firewall rules to allow traffic from any to any on all interfaces (including openvpn/IPSec/OPT1) -- and still nada.
I am having a hunch that the setup I am looking for is doable. It is just me unable to find the right trigger.
Thanks again.
-
In that case it sounds like a routing problem, or probably that you're abusing 169.254.13.0/24. Try using one of the RFC-1918 addresses instead and see if your problem goes away.
If it doesn't check your routing tables to ensure that clients know how to route to through the VPN. In particular you'll want to ensure that the default gateway for the "inner" pfSense host is down the IPsec tunnel.
-
trying to figure out what is wrong with the routing – as even after changing the network as per your suggestion (the tunnel is now 172.16.10.0/27) the packets don't seem to cooperate.
I have set up four virtual machines for testing, the diagram now goes like this
--------- 172.16.0.1 172.16.0.2 --------- 192.168.13.20
-----------| box a |---------------------------------| box b |---------|OPT1 10.2
--------- a1 b1 -------- b2 |
| OpenVPN tunnel
| using 172.16.10.0/27
--------- d1 c2 --------- c1 |
-----------| box d |---------------------------------| box c |---------| OPT1 10.1
--------- 172.16.13.1 172.16.13.3 -------- 192.168.13.19The tunnel is up, and the routing tables from the 4 boxes are like below
box a:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire 127.0.0.1 link#5 UH 0 5448 lo0 172.16.0.0/24 link#2 U 0 6594 le1 172.16.0.1 link#2 UHS 0 0 lo0 172.16.13.0/24 172.16.0.2 UGS 0 37 le1 192.168.56.0/24 link#1 U 0 1 le0 192.168.56.101 link#1 UHS 0 0 lo0box b:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 172.16.0.1 UGS 0 3051 le0 127.0.0.1 link#5 UH 0 97 lo0 172.16.0.0/24 link#1 U 0 1 le0 172.16.0.1 08:00:27:1f:a6:cb UHS 0 6718 le0 172.16.0.2 link#1 UHS 0 0 lo0 172.16.10.1 link#7 UH 0 7 ovpnc1 172.16.10.2 link#7 UHS 0 0 lo0 172.16.13.0/24 172.16.10.1 UGS 0 30 ovpnc1 192.168.13.0/27 link#2 U 0 3349 le1 192.168.13.20 link#2 UHS 0 0 lo0box c:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 172.16.13.1 UGS 0 0 le1 127.0.0.1 link#5 UH 0 4452 lo0 172.16.0.0/24 172.16.10.2 UGS 0 11 ovpns1 172.16.10.1 link#7 UHS 0 0 lo0 172.16.10.2 link#7 UH 0 7 ovpns1 172.16.13.0/24 link#2 U 0 2804 le1 172.16.13.3 link#2 UHS 0 0 lo0 192.168.13.0/27 link#1 U 0 2590 le0 192.168.13.19 link#1 UHS 0 0 lo0box d:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire 0.0.0.0/8 link#2 U 0 0 le1 127.0.0.1 link#5 UH 0 5048 lo0 172.16.0.0/24 172.16.13.3 UGS 0 7 le0 172.16.13.0/24 link#1 U 0 3152 le0 172.16.13.1 link#1 UHS 0 0 lo0Now time for some traceroute
from box a, tracing route to box d[2.0-BETA5][admin@pfSense.localdomain]/root(3): traceroute 172.16.13.1 traceroute to 172.16.13.1 (172.16.13.1), 64 hops max, 40 byte packets 1 172.16.0.2 (172.16.0.2) 4.019 ms 9.434 ms 4.227 ms 2 172.16.10.1 (172.16.10.1) 20.677 ms 12.546 ms 26.545 ms 3 * * * 4 * * * ^C [2.0-BETA5][admin@pfSense.localdomain]/root(4): traceroute 172.16.13.3 traceroute to 172.16.13.3 (172.16.13.3), 64 hops max, 40 byte packets 1 172.16.0.2 (172.16.0.2) 18.063 ms 15.014 ms 3.099 ms 2 * * * 3 * * * ^Cping from box a
[2.0-BETA5][admin@pfSense.localdomain]/root(5): ping 172.16.13.3 PING 172.16.13.3 (172.16.13.3): 56 data bytes 64 bytes from 172.16.13.3: icmp_seq=0 ttl=63 time=12.455 ms ^C --- 172.16.13.3 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 12.455/12.455/12.455/0.000 ms [2.0-BETA5][admin@pfSense.localdomain]/root(6): ping 172.16.13.1 PING 172.16.13.1 (172.16.13.1): 56 data bytes ^C --- 172.16.13.1 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossand a traceroute from box d to a
[2.0-BETA5][admin@pfSense.localdomain]/root(6): traceroute 172.16.0.1 traceroute to 172.16.0.1 (172.16.0.1), 64 hops max, 40 byte packets 1 172.16.13.3 (172.16.13.3) 5.914 ms 4.803 ms 4.672 ms 2 172.16.10.2 (172.16.10.2) 14.846 ms 12.389 ms 11.773 ms 3 * * * 4 * * * ^C [2.0-BETA5][admin@pfSense.localdomain]/root(7): traceroute 172.16.0.2 traceroute to 172.16.0.2 (172.16.0.2), 64 hops max, 40 byte packets 1 172.16.13.3 (172.16.13.3) 5.617 ms 7.013 ms 5.117 ms 2 * * * 3 * * * ^Cping from box d
[2.0-BETA5][admin@pfSense.localdomain]/root(8): ping 172.16.0.2 PING 172.16.0.2 (172.16.0.2): 56 data bytes 64 bytes from 172.16.0.2: icmp_seq=0 ttl=63 time=12.179 ms 64 bytes from 172.16.0.2: icmp_seq=1 ttl=63 time=12.879 ms ^C --- 172.16.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 12.179/12.529/12.879/0.350 ms [2.0-BETA5][admin@pfSense.localdomain]/root(9): ping 172.16.0.1 PING 172.16.0.1 (172.16.0.1): 56 data bytes ^C --- 172.16.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossI am not sure how much (if any) can be deduced from this tests. I am willing to run more tests if directed properly. But still cannot figure out what is going wrong in any of these boxes.
however, I can ping box a from box c, or d from b, and it pings quite happily.
FYI, box a has a1 LAN
box b has b1 wan, b2 lan
box c has c1 lan, c2 wan
box d has d1 land1 default route to c2
a1 default route to b1Thanks all.
-
I assume that 'b' and 'c' represent '1' and '2' in the original diagram? You'd probably be best off duplicating the exact same addressing etc as in the real one for your testing.
box a - no default gateway, but knows how to route to 172.16.13.0/24 via the correct gateway
box b - Route to 172.16.13.0/24 looks good (though obviously nothing beyond box d)
box c - Route to 172.16.0.0/24 looks good (though obviously nothing beyond box a)
box d - No default gateway, but knows to route to 172.16.0.0/24 via the correct gateway
What about NAT and firewall rules? With NAT fully enabled these boxes won't be doing just routing, which may cause most of your problems.
Do the firewall rules allow the traffic (ICMP) through?
-
I have added defaultroute in box a to make sure it pushes all packets via box b
As for NAT, I don't think any of the boxes are doing any NAT at this point – apart from the autogenerated rules by openvpn (I have Automatic outbound NAT rule generation selected), more so since I can ping the interfaces properly.
from box b
[2.0-BETA5][admin@box-b.local]/root(10): ping 172.16.13.1
PING 172.16.13.1 (172.16.13.1): 56 data bytes
64 bytes from 172.16.13.1: icmp_seq=0 ttl=63 time=13.907 ms
64 bytes from 172.16.13.1: icmp_seq=1 ttl=63 time=10.257 ms
64 bytes from 172.16.13.1: icmp_seq=2 ttl=63 time=12.189 ms
^C
–- 172.16.13.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.257/12.118/13.907/1.491 msfrom box c
[2.0-BETA5][admin@box-c.local]/root(14): ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: icmp_seq=0 ttl=63 time=11.507 ms
64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=9.639 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=63 time=19.691 ms
^C
–- 172.16.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.639/13.612/19.691/4.365 mshowever, when I'm trying from box a, I can ping all the way to the end of box c on both the external interface and the end of vpn tunnel
[2.0-BETA5][admin@pfSense.localdomain]/root(14): ping 172.16.13.3
PING 172.16.13.3 (172.16.13.3): 56 data bytes
64 bytes from 172.16.13.3: icmp_seq=0 ttl=63 time=20.001 ms
64 bytes from 172.16.13.3: icmp_seq=1 ttl=63 time=14.493 ms
^C
–- 172.16.13.3 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 14.493/17.247/20.001/2.754 ms
[2.0-BETA5][admin@pfSense.localdomain]/root(15): ping 172.16.10.1
PING 172.16.10.1 (172.16.10.1): 56 data bytes
64 bytes from 172.16.10.1: icmp_seq=0 ttl=63 time=14.902 ms
64 bytes from 172.16.10.1: icmp_seq=1 ttl=63 time=12.105 ms
64 bytes from 172.16.10.1: icmp_seq=2 ttl=63 time=14.444 ms
^C
–- 172.16.10.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.105/13.817/14.902/1.225 msor from box d, can ping all the way to the outer interface of box b or the vpn tunnel
[2.0-BETA5][admin@box-D]/root(7): ping 172.16.0.2
PING 172.16.0.2 (172.16.0.2): 56 data bytes
64 bytes from 172.16.0.2: icmp_seq=0 ttl=63 time=30.871 ms
64 bytes from 172.16.0.2: icmp_seq=1 ttl=63 time=16.862 ms
^C
–- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.862/23.866/30.871/7.004 ms
[2.0-BETA5][admin@box-D]/root(8): ping 172.16.10.2
PING 172.16.10.2 (172.16.10.2): 56 data bytes
64 bytes from 172.16.10.2: icmp_seq=0 ttl=63 time=9.543 ms
64 bytes from 172.16.10.2: icmp_seq=1 ttl=63 time=12.682 ms
^C
–- 172.16.10.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.543/11.113/12.682/1.569 mshowever, packets from a to d or d to a is lost at the immediate last interface (at the nearest ends of the vpn tunnel to the destination).
any clues?
-
Frankly at this point I'd be popping your favourite packet sniffer on the various links and seeing what's going on at the network layer. That'll tell you exactly how far the packets are getting, and possibly why they're not getting back.
