Slightly strange setup :: help/pointers appreciated
-
To answer your questions, as much as I can
I think you're probably trying to do this the hard way - is there any reason you can't add another interface to one of the pfSense boxes?
No reason. I can stick in another card anytime I wish (the card is sitting there doing nothing as we speak). But in that event what kind of configuration do you suggest?
If you put another interface into box 1 then you can simply directly connect the 10.0.0.0/24 network to it. Most of the problems you raised largely go away at that point, no issues of network traffic being sniffed and no complex routing issues.
At that point your diagram looks like
x.y.z.69
|
|a1
–----------------
| pfSense box 1 |
------------------
192.168.3.1 |a2 a3| 10.0.0.1
| |
private network | | private network
192.168.3.0/24 | | 10.0.0.0/24Now the traffic from either network to the Internet goes direct and you can easily create rules to allow access between the networks to suit yourself.
-
I think I might not have made the picture clearer earlier – I'll blame it all to the many sleepless nights and brain damage at my end ;D. The two boxes in question are physically located at two ends -- and the only path from one to the other is via the fibre link (192 network). If I wish to take your path, I will need to do a new layout of cabling from one end of my campus to the other -- which is NOT an option. I cannot use even vlans -- if you understand what I mean. Providing support for a small not-so-important group is not meant to be simple -- but that is what makes it more challenging and hence interesting.
As it happens, these two boxes and a reasonable amount of IPs (in all networks, including a few from the realIP) is available to be put to use as I feel absolutely necessary, and I have to find my way around it.
The place is closed at the moment, and the next time I go back is on the 4th, the solution ideally should be in place by the 5th latest. I am trying to set up a machine now where I might be able to run a few virtual instances of pfSense and see if I can achieve anything.
Thanks to all once again.
-
You could make number 2 a bridge, that should work.
However, if the fibre goes from one pfSense to the other, and not via switches, there isn't any way for an average person to sniff the traffic. Otherwise, any VPN between the LAN interface of number 1 and the WAN interface of number 2 should work (not from WAN to WAN).
-
Thank you for the suggestion, but could you please be a bit more specific?
what I am trying to understand is, when you say box one, I am gathering a1 is WAN and a2 is LAN, but when it comes to box2, which one are you referring to as LAN and WAN?
if I set a2 and b1 as LAN (both facing the other way as "outside"), do you suppose I could do a LAN to LAN tunnel? I know I can have a tunnel like that (it establishes itself without any error) – the only problem is it does not take data from one side to the other.
I am almost done setting up 4 pfSense virtual machines into my PC, now I can try to mimic a similar setup and see if I can ping from one end of the network to the other, or at least traceroute. Would appreciate if you could please let me know.
Thanks again
-
WAN is always the "outside" interface, LAN the "inside". Generally when setting up tunnels you set them up between the interfaces that connect the systems (in my experience). If you're having trouble passing data it's probably a routing table problem.
I think you'd probably be better off, if you can, replacing box 2 with a fibre connected switch and hooking the other end into a third interface on box 1. As I said though, if the fibre goes between the 2 pfSense boxes there isn't any way for others on the 192.168.3.0/24 network to sniff the traffic.
-
Thank you for your response. yes, I do understand WAN is outside and LAN is inside – but then again I am not very bright and for me anything and everything could be either inside or outside -- a matter of perspective. Hence I requested a little more detail (based on my original diagram) on which interface would be which for your suggestion to be tested.
As for direct fibre link, that is not possible. The only fibre link is between two switches, as I stated earlier. The switch near box 1 is connected to another switch near box 2 via fibre. switch near box 2 is solely used by the 192 series IPs (no chance of a vlan either, as stated earlier). Hence I am having to think of a secure tunnel to get the traffic -- while keeping both the network free from each others eyes.
If I cannot do it the ipsec way -- because of my stupidity and lack of understanding of how the tunnel should work -- maybe I should abandon the attempts or at least stop bugging people on forums.
thanks to everyone for at least viewing my little problem.
-
I've always found bugging people on forums an effective way of learning (even if sometimes just new swear words ;) ).
I've never tried to set up an IPsec, or any other, tunnel between 2 pfSense boxes, most of my site to site experience is with dedicated VPN devices or Cisco kit. That said, there are 2 stickies in the OpenVPN forum that cover using OpenVPN for site to site and there is a tutorial on IPsec between 2 pfSense hosts.
-
yap, seen them, followed them all even before posting the problem here, nada. As I have said before, in all events the links are established fine. I can even ping the nearest interface, but the packets simply does not route to the other side and vice versa. For example, as per my diagram, I can ping from a1 to b1, or b2 to a2. I tried with openvpn and i can even ping the other end of the tunnel (I used 169.254.13.0/24 as tunnel network), but somehow it is not going to the other side at all. I have checked the routing table and all seemed to be alright – no noticeable sign of error. But somehow the packets are not being routed. To be on the safe side, I even added firewall rules to allow traffic from any to any on all interfaces (including openvpn/IPSec/OPT1) -- and still nada.
I am having a hunch that the setup I am looking for is doable. It is just me unable to find the right trigger.
Thanks again.
-
In that case it sounds like a routing problem, or probably that you're abusing 169.254.13.0/24. Try using one of the RFC-1918 addresses instead and see if your problem goes away.
If it doesn't check your routing tables to ensure that clients know how to route to through the VPN. In particular you'll want to ensure that the default gateway for the "inner" pfSense host is down the IPsec tunnel.
-
trying to figure out what is wrong with the routing – as even after changing the network as per your suggestion (the tunnel is now 172.16.10.0/27) the packets don't seem to cooperate.
I have set up four virtual machines for testing, the diagram now goes like this
--------- 172.16.0.1 172.16.0.2 --------- 192.168.13.20
-----------| box a |---------------------------------| box b |---------|OPT1 10.2
--------- a1 b1 -------- b2 |
| OpenVPN tunnel
| using 172.16.10.0/27
--------- d1 c2 --------- c1 |
-----------| box d |---------------------------------| box c |---------| OPT1 10.1
--------- 172.16.13.1 172.16.13.3 -------- 192.168.13.19The tunnel is up, and the routing tables from the 4 boxes are like below
box a:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire 127.0.0.1 link#5 UH 0 5448 lo0 172.16.0.0/24 link#2 U 0 6594 le1 172.16.0.1 link#2 UHS 0 0 lo0 172.16.13.0/24 172.16.0.2 UGS 0 37 le1 192.168.56.0/24 link#1 U 0 1 le0 192.168.56.101 link#1 UHS 0 0 lo0box b:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 172.16.0.1 UGS 0 3051 le0 127.0.0.1 link#5 UH 0 97 lo0 172.16.0.0/24 link#1 U 0 1 le0 172.16.0.1 08:00:27:1f:a6:cb UHS 0 6718 le0 172.16.0.2 link#1 UHS 0 0 lo0 172.16.10.1 link#7 UH 0 7 ovpnc1 172.16.10.2 link#7 UHS 0 0 lo0 172.16.13.0/24 172.16.10.1 UGS 0 30 ovpnc1 192.168.13.0/27 link#2 U 0 3349 le1 192.168.13.20 link#2 UHS 0 0 lo0box c:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 172.16.13.1 UGS 0 0 le1 127.0.0.1 link#5 UH 0 4452 lo0 172.16.0.0/24 172.16.10.2 UGS 0 11 ovpns1 172.16.10.1 link#7 UHS 0 0 lo0 172.16.10.2 link#7 UH 0 7 ovpns1 172.16.13.0/24 link#2 U 0 2804 le1 172.16.13.3 link#2 UHS 0 0 lo0 192.168.13.0/27 link#1 U 0 2590 le0 192.168.13.19 link#1 UHS 0 0 lo0box d:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire 0.0.0.0/8 link#2 U 0 0 le1 127.0.0.1 link#5 UH 0 5048 lo0 172.16.0.0/24 172.16.13.3 UGS 0 7 le0 172.16.13.0/24 link#1 U 0 3152 le0 172.16.13.1 link#1 UHS 0 0 lo0Now time for some traceroute
from box a, tracing route to box d[2.0-BETA5][admin@pfSense.localdomain]/root(3): traceroute 172.16.13.1 traceroute to 172.16.13.1 (172.16.13.1), 64 hops max, 40 byte packets 1 172.16.0.2 (172.16.0.2) 4.019 ms 9.434 ms 4.227 ms 2 172.16.10.1 (172.16.10.1) 20.677 ms 12.546 ms 26.545 ms 3 * * * 4 * * * ^C [2.0-BETA5][admin@pfSense.localdomain]/root(4): traceroute 172.16.13.3 traceroute to 172.16.13.3 (172.16.13.3), 64 hops max, 40 byte packets 1 172.16.0.2 (172.16.0.2) 18.063 ms 15.014 ms 3.099 ms 2 * * * 3 * * * ^Cping from box a
[2.0-BETA5][admin@pfSense.localdomain]/root(5): ping 172.16.13.3 PING 172.16.13.3 (172.16.13.3): 56 data bytes 64 bytes from 172.16.13.3: icmp_seq=0 ttl=63 time=12.455 ms ^C --- 172.16.13.3 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 12.455/12.455/12.455/0.000 ms [2.0-BETA5][admin@pfSense.localdomain]/root(6): ping 172.16.13.1 PING 172.16.13.1 (172.16.13.1): 56 data bytes ^C --- 172.16.13.1 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossand a traceroute from box d to a
[2.0-BETA5][admin@pfSense.localdomain]/root(6): traceroute 172.16.0.1 traceroute to 172.16.0.1 (172.16.0.1), 64 hops max, 40 byte packets 1 172.16.13.3 (172.16.13.3) 5.914 ms 4.803 ms 4.672 ms 2 172.16.10.2 (172.16.10.2) 14.846 ms 12.389 ms 11.773 ms 3 * * * 4 * * * ^C [2.0-BETA5][admin@pfSense.localdomain]/root(7): traceroute 172.16.0.2 traceroute to 172.16.0.2 (172.16.0.2), 64 hops max, 40 byte packets 1 172.16.13.3 (172.16.13.3) 5.617 ms 7.013 ms 5.117 ms 2 * * * 3 * * * ^Cping from box d
[2.0-BETA5][admin@pfSense.localdomain]/root(8): ping 172.16.0.2 PING 172.16.0.2 (172.16.0.2): 56 data bytes 64 bytes from 172.16.0.2: icmp_seq=0 ttl=63 time=12.179 ms 64 bytes from 172.16.0.2: icmp_seq=1 ttl=63 time=12.879 ms ^C --- 172.16.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 12.179/12.529/12.879/0.350 ms [2.0-BETA5][admin@pfSense.localdomain]/root(9): ping 172.16.0.1 PING 172.16.0.1 (172.16.0.1): 56 data bytes ^C --- 172.16.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossI am not sure how much (if any) can be deduced from this tests. I am willing to run more tests if directed properly. But still cannot figure out what is going wrong in any of these boxes.
however, I can ping box a from box c, or d from b, and it pings quite happily.
FYI, box a has a1 LAN
box b has b1 wan, b2 lan
box c has c1 lan, c2 wan
box d has d1 land1 default route to c2
a1 default route to b1Thanks all.
-
I assume that 'b' and 'c' represent '1' and '2' in the original diagram? You'd probably be best off duplicating the exact same addressing etc as in the real one for your testing.
box a - no default gateway, but knows how to route to 172.16.13.0/24 via the correct gateway
box b - Route to 172.16.13.0/24 looks good (though obviously nothing beyond box d)
box c - Route to 172.16.0.0/24 looks good (though obviously nothing beyond box a)
box d - No default gateway, but knows to route to 172.16.0.0/24 via the correct gateway
What about NAT and firewall rules? With NAT fully enabled these boxes won't be doing just routing, which may cause most of your problems.
Do the firewall rules allow the traffic (ICMP) through?
-
I have added defaultroute in box a to make sure it pushes all packets via box b
As for NAT, I don't think any of the boxes are doing any NAT at this point – apart from the autogenerated rules by openvpn (I have Automatic outbound NAT rule generation selected), more so since I can ping the interfaces properly.
from box b
[2.0-BETA5][admin@box-b.local]/root(10): ping 172.16.13.1
PING 172.16.13.1 (172.16.13.1): 56 data bytes
64 bytes from 172.16.13.1: icmp_seq=0 ttl=63 time=13.907 ms
64 bytes from 172.16.13.1: icmp_seq=1 ttl=63 time=10.257 ms
64 bytes from 172.16.13.1: icmp_seq=2 ttl=63 time=12.189 ms
^C
–- 172.16.13.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.257/12.118/13.907/1.491 msfrom box c
[2.0-BETA5][admin@box-c.local]/root(14): ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: icmp_seq=0 ttl=63 time=11.507 ms
64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=9.639 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=63 time=19.691 ms
^C
–- 172.16.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.639/13.612/19.691/4.365 mshowever, when I'm trying from box a, I can ping all the way to the end of box c on both the external interface and the end of vpn tunnel
[2.0-BETA5][admin@pfSense.localdomain]/root(14): ping 172.16.13.3
PING 172.16.13.3 (172.16.13.3): 56 data bytes
64 bytes from 172.16.13.3: icmp_seq=0 ttl=63 time=20.001 ms
64 bytes from 172.16.13.3: icmp_seq=1 ttl=63 time=14.493 ms
^C
–- 172.16.13.3 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 14.493/17.247/20.001/2.754 ms
[2.0-BETA5][admin@pfSense.localdomain]/root(15): ping 172.16.10.1
PING 172.16.10.1 (172.16.10.1): 56 data bytes
64 bytes from 172.16.10.1: icmp_seq=0 ttl=63 time=14.902 ms
64 bytes from 172.16.10.1: icmp_seq=1 ttl=63 time=12.105 ms
64 bytes from 172.16.10.1: icmp_seq=2 ttl=63 time=14.444 ms
^C
–- 172.16.10.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.105/13.817/14.902/1.225 msor from box d, can ping all the way to the outer interface of box b or the vpn tunnel
[2.0-BETA5][admin@box-D]/root(7): ping 172.16.0.2
PING 172.16.0.2 (172.16.0.2): 56 data bytes
64 bytes from 172.16.0.2: icmp_seq=0 ttl=63 time=30.871 ms
64 bytes from 172.16.0.2: icmp_seq=1 ttl=63 time=16.862 ms
^C
–- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.862/23.866/30.871/7.004 ms
[2.0-BETA5][admin@box-D]/root(8): ping 172.16.10.2
PING 172.16.10.2 (172.16.10.2): 56 data bytes
64 bytes from 172.16.10.2: icmp_seq=0 ttl=63 time=9.543 ms
64 bytes from 172.16.10.2: icmp_seq=1 ttl=63 time=12.682 ms
^C
–- 172.16.10.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.543/11.113/12.682/1.569 mshowever, packets from a to d or d to a is lost at the immediate last interface (at the nearest ends of the vpn tunnel to the destination).
any clues?
-
Frankly at this point I'd be popping your favourite packet sniffer on the various links and seeing what's going on at the network layer. That'll tell you exactly how far the packets are getting, and possibly why they're not getting back.
