Lan 1 to Lan 2 Connection Fail
-
192.9.0.0/16 is allocated to Sun Microsystems and is not private address space.
Just fyi.
-
^ yeah we know, been brought up already.. Another ? he never answered after he stated he had answered everything, etc. ;)
"have given chapter and verse on the network topology and loads of debug info"
His wan is clearly private.. So at a loss to why you would be using public IP space behind a NAT.. Unless there was some other path to get to these boxes? Maybe he thinks its OK to just pull IP space out of the AIR and use it - maybe he works for Sun?
-
John,
This is getting hard work :-)
There might have been a time when I would have thought it very cool and a privilege to work for Sun, since they made some of the earliest and most competent unix systems around, Full gui when most pc's were still running dos. I’m in the uk, am probably too old now and do hardware and embedded systems development, not unix. Besides, they never asked me :-), they no longer make workstations anyway and they are owned by oracle now, which is a disaster.
If you actually read through the earlier posts, you would see that I said that the 192.9 lab network is historical. My very first unix box was a sun 3 system found in a junkyard. All the sun docs of the time used the 192.9 block, so that’s what I used. I have a load of respect for early unix development and like historical context, so never changed the 192 block for the lab, nor deleted old host names. Anyway, this is all nitpicking since pfsense really shouldn’t care what address block you use for any of the interfaces and there is no “correct” way to do it, other than the way you want it. The rules are what matters.
As for tcpdump traces, I have copied the traces twice to previous posts here, the second time with a description of what’s going on. Perhaps you did't understand it, so will try again in verbose mode. Note that tcpdump is tracing output from the pfsense labnet interface and the server replies. Server host name is firelight.
$ tcpdump -q host 172.16.100.205
tcpdump: verbose output suppressed,
listening on bge019:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
The above line shows the initial telnet request from the housenet node
19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
This is the first server repl
19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
This is the first retry from the housenet host
19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:31.290075 IP firelight.telnet > 172.16.100.205.53942: tcp 0This time, we get two replies for redundancy, telnet is assuming the packet was dropped, or therwise corrupted on return to the client.
19:00:36.922216 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
The third retry from housenet
19:00:36.922286 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:38.060093 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:42.930063 IP firelight.telnet > 172.16.100.205.57243: tcp 0Here we have three replies at once, again for redundancy. Telnet thinks this must be a very sh***y line.
So, the server is doing it’s best responding to the telnet request, but no one is at home on pfsense labnet receive and the replies are being dropped. This is confirmed by tcpdump on the pfsense console, which shows the outgoing request to the server, but not the reply.
You asked for the rules etc, so what's the solution, if there is one ?. The setup is about the most basic you could imagine, yet it doesn’t work as expected…
Regards,
Chris -
I would therefore conclude that traffic arriving on labnet interface with destination in homenet is somehow being routed elsewhere, not back to homenet. (Because there should be a state established by the 1st packet in your example, and the responses in the reverse direction will match that state and be allowed in/through regardless of any firewall rules on labnet interface).
What routing related stuff have you got defined - static routes?
What does the routing table look like - diagnostics->routes?
Do tcpdump on other interfaces (e.g. WAN) and see if the response packets are exiting there. (I guess you have checked that they are not exiting on homenet). -
Phil,
Thanks. I posted a screenshot of the routing table above - there's no routes defined other than the default, wan interface, which points upstream to the next level router.
The port forwarding rule is still in place. Without it, I just get a "no route to host" at the client.
Did another trace at the pfsense console:
tcpdump -i de0 -q host 172.16.100.205
de0 is the labnet interface. Anything in or out related to 172.16.100.205 on de0 should be traced. The strange thing is that while the telnet request shows up on the trace, the reply from the server (which really is on the wire), doesn't. What I get is the initial request, client -> server and several retries, but nothing else. Typically:
14:52:42.713118 IP 172.16.100.205 > 192.9.xxx.xxx.telnet :tcp 0
homenet client is the 172.16 source, server is the 192.9 destination.
The only way I can explain the discrepancy between the tcpdump traces, server vs pfsense, is that pfsense is tracing at a higher layer in the stack than where the reply is being dropped. That doesn't really make sense though, since tcpdump is generally understood to trace at hardware level or close. What do you think ?.
As for the state table, screenshot attached. This is immediately after the telnet request and shows a pair of relevant states, but no established connection ?. Line 1 looks like the keep alive ping upstream, 2-7 are management login related and 8-9 are telnet. Finally, the blanked out addresses in the telnet entries are the server address
I'm pretty rusty on low level network ops and will be digging out the Tcp/Ip Illustrated books later for more inspiration :-)…
Regards,
Chris
-
Can you sniff the wire itself somewhere with another system to see if the reply packet is actually there?
pfSense tcpdump should really see that response packet as long as it actually gets through the NIC hardware and firmware to be delivered to the FreeBSD driver. Any switch on labnet that connects the server and pfSense should learn MAC addresses from the first packet that is sent, and thus know which switch port the pfSense is in… Maybe a cable is half fallen out, and it can transmit through to the server, but the pair for the reverse direction are not connecting properly (but I think you said you can ping from pfSense to labnet server, so the physical layer must be OK).
I am struggling to think where the problem can be. At this point you have to make really sure that the response packet is actually leaving labnet server and getting onto the labnet wiring. -
Phil,
I can ping from pfsense, as well as ftp and just checked that again. There’s a debian machine that I can patch in but will need to hook up a hub, since all the boxes are on a switch. Ok, I can port miror the switch, but a hub is most likely quicker in this case and it don’t lie:-)
The other thing I’m going to do today is delete all the current rules and port forward, reboot, re-input the rules and reboot before checking again. None of this makes sense at present and it seems to me that maybe some part of the config has got out of sync. I was trying various combinations in the port forward definition, trying to make it work and the config files may be confused, depending on how the gui entered stuff translates down to the xml (?) files.
Failing that, reinstall from iso but try that first. Thanks for the help / inspiration and will report back later…
Regards,
Chris
-
John,
You asked about the stuff in the advanced section. Just a few minor changes to improve security:
-
None standard https management port
-
Randomise id field in ip packets
-
Insert stronger id into ip header for packets passing though the filter.
If you run zenmap against the wan port, it more or less susses out which os it is, but doesn't find much else. Going to try some of the Kali Linux stuff in the next few weeks, which may find some holes, though I doubt it. I wonder if the nsa have found a way in yet ?…
Regards,
Chris
-
-
What I wasn't clear in your other posts of your dumps is what actual interface you did the trace on.. Be it they were on pfsense, server or client.
Statements like this
"Pfsense console tcpdump also shows the outgoing request to the server on the interface. However, it never sees the server response."From a common sense point of why would you be sniffing on the interface on the other side of pfsense?? Until you had validated that it had showed up on the input interface to pfsense.
This trace for example.
You state this is the trace at the server..
"Also, I did show the output from tcpdump at the server, firelight:
19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0"So one statement from you says pfsense does not see the response, then you show that server sent the response.
As to you starting over - already suggested multiple times to do that.. YOU DO NOT NEED NAT!!! And from your port forward your trying to send anything going to port 23 on the "housenet net" which is the 172 space – so when would that even be hit? to the server on labnet.
And you have both tcp/udp ?? So yes I would START OVER.. And just do any any rules between your local segments.. Once that works, then you can block or allow what you want between your segments with firewall rules - not nats and or port forwards.
As to the advanced options - that is not what I am talking about.. I am talking you have set something in the advanced features of the rule.. See attached. What have you done there? Your labnet rule shows that an advanced feature was set on the rule - that is what I was asking about.
BTW - I can understanding wanting to be secure, etc. and sure if you want to use random in the IP ID, ok -- but now I might be mistaken - but pretty sure that would be changed again at your other nat since your pfsense wan is 10.x.x.x you clearly have to be doing NAT again to talk to the internet or your clients behind pfsense would have to be using a proxy to get out, etc. So your wanting to hide your OS from this 10.x.x.x network?
-
I tried reinstalling with the same rules as per the screenshots and it's still not working, so have to assume for now that this configuration is not supported at present. Will just have to find a different solution for now, but if you need any more info to confirm the findings, please let me know
Thanks for the help and suggestions in any case…
Regards,
Chris
-
John,
Just to clear up one misunderstanding, "pfsense console tcpdump" means that tcpdump was running on the pfsense console shell. It's a built in command and part of pfsense.
The packet is definately on the wire at the lab interface, but it's not seen by the pfsense shell tcpdump, so where can it be ?…
Regards,
Chris
-
I know what tcpdump is ;) and how to run it..
And again your saying its not there.. So lets clear up where your running it..
you have
client – pfsense -- server
where you telneting from client to server..
So which interface are you running tcpdump on? the client side or the server side?
You need to run it on the server side.. If your running on server side and you see packets go out to server. And wireshark/tcpdump running on server shows it sending replies to that traffic but you don't see those reply packets on pfsense then 110% sure pfsense has NOTHING to do with your issue.
All I can say is this takes all of 3 minutes to setup.. There is NO nat between local networks.. And its simple firewall rules to allow whatever traffic you want. As I showed you when I changed my dmz segment to 172.15
I assure you that pfsense supports this setup so your doing something wrong in the config. Or you have something else wrong on your network.
I would be happy to teamviewer in and take a look if you want.
Back to the tcpdump -- if you run it on the server side and see the return traffic. But don't see it on the client side of pfsense then yeah something is wrong with pfsense. So which exact interface are you running the tcpdump on pfsense client or server labnet or homenet.. Where if can keep your networks straight homenet is the client side and your telnet server is on the labnet side.