Lan 1 to Lan 2 Connection Fail
-
Hi,
Thanks for the replies and hope you won’t mind if I answer both in the same post..
Phil:
This really should be simple, and just work by default as long as there are firewall rules that allow the traffic.
Can we start again and get exactly what is where:a) Each interface name and its IP address on pfSense
There are 3 hardware interfaces.
wan: 10.0.x.x External, internet Default gateway -> upstream
homenet; 172.16.x.x Home, internal No gateway defined
labnet: 192.9.x.x Lab, internal. No gateway definedb) What rules are on each interface
wan: None, other than one to block bogon networks
homenet: One, to allow home net to anything
labnet: One, as per homenet.There’s also the admin anti lockout rule on labnet, on a non standard https port. Admin account name is different as well, but doubt if that should affect anything.
This is all working fine for outgoing access via either interface to the web, but if I try to telnet a homenet node to the labnet node, I get “no route to host” which is expected since there’s no rule or port forward defined to allow it.
c) What the clients have for their default gateway (hopefully the respective pfSense interface IP addresses)
Correct for both
d) Details of any other router/gateway device in the network
1 upstream from wan interface, none on homenet or labnet
As I understand it, pfsense blocks everything by default, so you need rules even for outgoing access. For that reason, it seems logical that if I want to access a host on labnet from a homenet host, I need a port forward or some sort of rule to allow it. Port forward on another pfsense box works fine incoming from the wan to a lan port, but it doesn’t seem to work from one internal port, to another on this box, so perhaps all ports are not created equal / have the same capabilities ?.
Ok, so I define a port forward rule for telnet as follows:
Src Src Dest Dest Nat Nat
Addr Ports Addr Ports Addr Ports172.16.x.x * housenet net 23 192.9.x.x 23
Sorry no screen shot, but haven’t got round to that yet..
Interface is housenet, protocol = tcp/udp, nat reflection = default and nat creates associated rule. Except for the interfaces, this is the same setup as that for the other pfsense box on the webserver, which works fine.
Using this rule, the telnet request is seen on homenet with wireshark, can be seen outgoing on the pfsense local console and the server on labnet replies, but the reply is lost on it’s way back into pfsense labnet interface. All the tcpdump trace info is in a previous post above, fyi.
Do you need anything else ?.
John,
Dude: … what part about NOT needing nat do you not understand ??
Well, all of it, unless you can tell me how it can be done without a port forward / nat, or rules of some sort :-). (Note the smiley :-)
All the debug info is in the previous posts, other than the rules, but if there’s anything I’ve missed this time, please let me know…
Regards,
Chris
-
Hi,
A bit more info:
Have also tried various variations on the above rule and also various switches in system -> advanced -> firewall-NAT, but none of it seems to work.
Is there any way to force wysiwig from post editor -> preview -> post. Formatting lost :-)…
Regards,
Chris
-
This is all working fine for outgoing access via either interface to the web, but if I try to telnet a homenet node to the labnet node, I get “no route to host” which is expected since there’s no rule or port forward defined to allow it.
Actually, it is expected to have a route and thus deliver your packet/s. pfSense (and every router I have ever seen) will route between local subnets by default.
homenet: One, to allow home net to anything
That rule should allow home to labnet, as well as homenet to google, homenet to facebook, homenet to anything.
There REALLY is no need to use NAT for this. There MUST be some other tricky thing that you have accidentally set up that is causing this not to work, or the target system in labnet does not respond to telnet from another subnet or…
Look in the Firewall log and make sure packets are not being blocked there. Then do some packet capture on homenet to verify the telnet initiation packet/s arrive, then on labnet to verify they leave labnet, then look for the response packet from labnet client on labnet and then homenet. Wherever the packet/s stop being seen is where to look next. -
Phil,
Thanks for the reply. The install is plain vanilla from the iso, with no special tweaks. I’ve been using pfsense for years now, with ipcop, freesco and packet filtering in the past, along with doing electronics / sw eng for work for decades, so hopefully not a complete newbie to this. Strange thing is that I’m pretty sure this worked on 2.03, but may have been ipcop, as it’s some time since I had this requirement set up.
If you read the op, you can see that I have been packet monitoring at 3 points: homenet via wireshark, pfsense and labnet via tcpdump. There’s a packet trace that proves that the reply from the remote server is being dropped at the pfsense labnet interface, on the way back in, as it is seen on tcpdump labnet, but not on tcpdump pfsense console.
While you and John both seem to think that packets between local interfaces are routed by default, in fact they appear not to be. As I said, pfsense blocks everything by default and you need outgoing rules just to access the wan from local.
Regards,
Chris
-
Dude post a screen shot of your rules for gosh sake, and your routing table.
I am going to say this ONE LAST time – there is NO NEED to NAT between local segments.. PERIOD! I have shown you this - it is FACT, you do NOT have to NAT between local networks segments no matter what address space your using.
as to this
"the remote server is being dropped at the pfsense labnet interface,"No dropped is the WRONG word.. Not seen is the right word from what you have shown.. Even if you had a block rule there the tcpdump would still show the packets if they hit the interface.
So if your saying the packets are not being seen there then you have another issue.. Validate that the packets that leave your client actually have the correct MAC for one. And what is between??
Had a very strange thing with a cisco switch awhile back where packets were not being forwarded in a vlan unless there was a SVI on the vlan..
https://tools.cisco.com/bugsearch/bug/CSCth74527
IF your not seeing the return packets as pfsense - then this issue has NOTHING to do with pfense!
-
John,
4 images to post, but call me clueless, how do you inline images on this board ?. There's the button, but paste file contents doesn't seem to do anything. Intuitive, or what ?. Help file != useful either.
Just to add, I know that the hardware is ok because I can telnet or ftp into the lab server from the pfsense console, as well from any other machine on labnet. Also tried a direct connection from the housenet machine (ip changed) to the server and that works fine. There can be some funnies with telnet between some machines, but not here. I use ftp from pfsense all the time to send the backup dump files to the server. All the lab machines and pfsense are on the same 3Com 29xx series switch.
The cisco link is behind a login, but I played with a pix515 some time ago and thought the user interface (windows client) unintuitive and primitive compared to pfsense, or even ipcop. Ymmv, of course :-)…
Regards,
Chris
-
click the preview button on the bottom so you get the full editor
if your wanting to link to a img else where - then use the tags and put in the url to your image.
-
John,
Thanks - Didn't have the wysiwyg editor selected, but still clueless. Click on add image, which then asks for the location, which I fill in for the first as:
i:\ImageFile\Screenshots\HouseRules.png
Which insertes a box into the reply, but preview sees nowt. Anyway, 4 images included as attachments…
Sometimes just easier to ask etc :-).
Regards,
Chris
-
…2 replies as the image > 300K total -
-
And how would pfsense forums have access to i:\ImageFile\Screenshots\HouseRules.png ?? If you want to do inline with img tags then you need to have some url that pfsense and for that matter the viewer of the fourms could access ie http://images.something.tld/image.png for example.
And what is that rule doing above your tenet rule? with the advanced tag on it?
Dude – for testing lets call it.. make your housenet rule like your labnet rule. Where source is the network and dest IP and port are any any. Where are your autobound nats - are they automatic or manual. And what are you doing in the advanced section -- that a tag on the rule. Remove all port forwards! And what our your outbound nats - post them.
But lets forget any rules you have on pfsense for now.
If your saying when you tcpdump on lapnet pfsense interface, you see the telnet packet go out to your 192.9.x.x address you don't see a response then pfsense has nothing to do with the issue.
Pfsense sends out packets to 192.9.x.x:23 from 172.16.x.x:random, and you don't see packets come back.. Then 172.16 did not answer, or sent it to the wrong place or something between 172.16 and pfsense interface is not forwarding along the packet.
tcpdump will show you all packets before they hit any nat rule or firewall rule. So if the packets are not there - there is NOTHING that pfsense could do.
btw not sure what your doing with the advanced options - but that rule makes the rule below it about telnet pointless and would never be used unless something in the advanced options would rule out telnet?
Also telnet is TCP, not UDP so having both tcp and udp is again pointless.
-
192.9.0.0/16 is allocated to Sun Microsystems and is not private address space.
Just fyi.
-
^ yeah we know, been brought up already.. Another ? he never answered after he stated he had answered everything, etc. ;)
"have given chapter and verse on the network topology and loads of debug info"
His wan is clearly private.. So at a loss to why you would be using public IP space behind a NAT.. Unless there was some other path to get to these boxes? Maybe he thinks its OK to just pull IP space out of the AIR and use it - maybe he works for Sun?
-
John,
This is getting hard work :-)
There might have been a time when I would have thought it very cool and a privilege to work for Sun, since they made some of the earliest and most competent unix systems around, Full gui when most pc's were still running dos. I’m in the uk, am probably too old now and do hardware and embedded systems development, not unix. Besides, they never asked me :-), they no longer make workstations anyway and they are owned by oracle now, which is a disaster.
If you actually read through the earlier posts, you would see that I said that the 192.9 lab network is historical. My very first unix box was a sun 3 system found in a junkyard. All the sun docs of the time used the 192.9 block, so that’s what I used. I have a load of respect for early unix development and like historical context, so never changed the 192 block for the lab, nor deleted old host names. Anyway, this is all nitpicking since pfsense really shouldn’t care what address block you use for any of the interfaces and there is no “correct” way to do it, other than the way you want it. The rules are what matters.
As for tcpdump traces, I have copied the traces twice to previous posts here, the second time with a description of what’s going on. Perhaps you did't understand it, so will try again in verbose mode. Note that tcpdump is tracing output from the pfsense labnet interface and the server replies. Server host name is firelight.
$ tcpdump -q host 172.16.100.205
tcpdump: verbose output suppressed,
listening on bge019:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
The above line shows the initial telnet request from the housenet node
19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
This is the first server repl
19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
This is the first retry from the housenet host
19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:31.290075 IP firelight.telnet > 172.16.100.205.53942: tcp 0This time, we get two replies for redundancy, telnet is assuming the packet was dropped, or therwise corrupted on return to the client.
19:00:36.922216 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
The third retry from housenet
19:00:36.922286 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:38.060093 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:42.930063 IP firelight.telnet > 172.16.100.205.57243: tcp 0Here we have three replies at once, again for redundancy. Telnet thinks this must be a very sh***y line.
So, the server is doing it’s best responding to the telnet request, but no one is at home on pfsense labnet receive and the replies are being dropped. This is confirmed by tcpdump on the pfsense console, which shows the outgoing request to the server, but not the reply.
You asked for the rules etc, so what's the solution, if there is one ?. The setup is about the most basic you could imagine, yet it doesn’t work as expected…
Regards,
Chris -
I would therefore conclude that traffic arriving on labnet interface with destination in homenet is somehow being routed elsewhere, not back to homenet. (Because there should be a state established by the 1st packet in your example, and the responses in the reverse direction will match that state and be allowed in/through regardless of any firewall rules on labnet interface).
What routing related stuff have you got defined - static routes?
What does the routing table look like - diagnostics->routes?
Do tcpdump on other interfaces (e.g. WAN) and see if the response packets are exiting there. (I guess you have checked that they are not exiting on homenet). -
Phil,
Thanks. I posted a screenshot of the routing table above - there's no routes defined other than the default, wan interface, which points upstream to the next level router.
The port forwarding rule is still in place. Without it, I just get a "no route to host" at the client.
Did another trace at the pfsense console:
tcpdump -i de0 -q host 172.16.100.205
de0 is the labnet interface. Anything in or out related to 172.16.100.205 on de0 should be traced. The strange thing is that while the telnet request shows up on the trace, the reply from the server (which really is on the wire), doesn't. What I get is the initial request, client -> server and several retries, but nothing else. Typically:
14:52:42.713118 IP 172.16.100.205 > 192.9.xxx.xxx.telnet :tcp 0
homenet client is the 172.16 source, server is the 192.9 destination.
The only way I can explain the discrepancy between the tcpdump traces, server vs pfsense, is that pfsense is tracing at a higher layer in the stack than where the reply is being dropped. That doesn't really make sense though, since tcpdump is generally understood to trace at hardware level or close. What do you think ?.
As for the state table, screenshot attached. This is immediately after the telnet request and shows a pair of relevant states, but no established connection ?. Line 1 looks like the keep alive ping upstream, 2-7 are management login related and 8-9 are telnet. Finally, the blanked out addresses in the telnet entries are the server address
I'm pretty rusty on low level network ops and will be digging out the Tcp/Ip Illustrated books later for more inspiration :-)…
Regards,
Chris
-
Can you sniff the wire itself somewhere with another system to see if the reply packet is actually there?
pfSense tcpdump should really see that response packet as long as it actually gets through the NIC hardware and firmware to be delivered to the FreeBSD driver. Any switch on labnet that connects the server and pfSense should learn MAC addresses from the first packet that is sent, and thus know which switch port the pfSense is in… Maybe a cable is half fallen out, and it can transmit through to the server, but the pair for the reverse direction are not connecting properly (but I think you said you can ping from pfSense to labnet server, so the physical layer must be OK).
I am struggling to think where the problem can be. At this point you have to make really sure that the response packet is actually leaving labnet server and getting onto the labnet wiring. -
Phil,
I can ping from pfsense, as well as ftp and just checked that again. There’s a debian machine that I can patch in but will need to hook up a hub, since all the boxes are on a switch. Ok, I can port miror the switch, but a hub is most likely quicker in this case and it don’t lie:-)
The other thing I’m going to do today is delete all the current rules and port forward, reboot, re-input the rules and reboot before checking again. None of this makes sense at present and it seems to me that maybe some part of the config has got out of sync. I was trying various combinations in the port forward definition, trying to make it work and the config files may be confused, depending on how the gui entered stuff translates down to the xml (?) files.
Failing that, reinstall from iso but try that first. Thanks for the help / inspiration and will report back later…
Regards,
Chris
-
John,
You asked about the stuff in the advanced section. Just a few minor changes to improve security:
-
None standard https management port
-
Randomise id field in ip packets
-
Insert stronger id into ip header for packets passing though the filter.
If you run zenmap against the wan port, it more or less susses out which os it is, but doesn't find much else. Going to try some of the Kali Linux stuff in the next few weeks, which may find some holes, though I doubt it. I wonder if the nsa have found a way in yet ?…
Regards,
Chris
-
-
What I wasn't clear in your other posts of your dumps is what actual interface you did the trace on.. Be it they were on pfsense, server or client.
Statements like this
"Pfsense console tcpdump also shows the outgoing request to the server on the interface. However, it never sees the server response."From a common sense point of why would you be sniffing on the interface on the other side of pfsense?? Until you had validated that it had showed up on the input interface to pfsense.
This trace for example.
You state this is the trace at the server..
"Also, I did show the output from tcpdump at the server, firelight:
19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0"So one statement from you says pfsense does not see the response, then you show that server sent the response.
As to you starting over - already suggested multiple times to do that.. YOU DO NOT NEED NAT!!! And from your port forward your trying to send anything going to port 23 on the "housenet net" which is the 172 space – so when would that even be hit? to the server on labnet.
And you have both tcp/udp ?? So yes I would START OVER.. And just do any any rules between your local segments.. Once that works, then you can block or allow what you want between your segments with firewall rules - not nats and or port forwards.
As to the advanced options - that is not what I am talking about.. I am talking you have set something in the advanced features of the rule.. See attached. What have you done there? Your labnet rule shows that an advanced feature was set on the rule - that is what I was asking about.
BTW - I can understanding wanting to be secure, etc. and sure if you want to use random in the IP ID, ok -- but now I might be mistaken - but pretty sure that would be changed again at your other nat since your pfsense wan is 10.x.x.x you clearly have to be doing NAT again to talk to the internet or your clients behind pfsense would have to be using a proxy to get out, etc. So your wanting to hide your OS from this 10.x.x.x network?
-
I tried reinstalling with the same rules as per the screenshots and it's still not working, so have to assume for now that this configuration is not supported at present. Will just have to find a different solution for now, but if you need any more info to confirm the findings, please let me know
Thanks for the help and suggestions in any case…
Regards,
Chris
