Lan 1 to Lan 2 Connection Fail



  • A question for the group:

    We have a pfsense machine with 3 interfaces, let’s call them wan, homenet, labnet. Snort is running on the wan interface and all unsolicited wan incoming traffic  is denied, no rules. Both  homenet and labnet have a single rule each to allow outgoing access to wan / anything for normal internet connectivity. Labnet has a sun sparc server running nfs, mysql vnc etc, which I would like to be able to access from a couple of housenet machines.

    Started by writing a nat rule to do this, but after trying just about every combination possible from the setup screens, just can’t seem to make it work. However, repatching the housenet machine (DL140, Suse) directly to the server and I can telnet in.to the server with no problems. Also use a second pfsense box to access a webserver, wan > webnet and there the nat works perfectly and never even blinks. Both pfsense’s have an identical hardware config, using old sff desktop machines.

    Wireshark is running on the housenet machine and tcpdump on the server. tcpdump is also running on the pfsense text console. If I try to telnet to the server, the packet can be seen on it’s way out on wireshark, as can the retries, but nothing comes back. At the server end, the packet  sequence is as follows:

    $ tcpdump -q host 172.16.100.205
    tcpdump: verbose output suppressed,
    listening on bge0

    19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:31.290075 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:36.922216 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:36.922286 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:38.060093 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:42.930063 IP firelight.telnet > 172.16.100.205.57243: tcp 0

    This shows the telnet request and server response, together with several retries. Pfsense console tcpdump also shows the outgoing request to the server on the interface. However, it never sees the server response, even though it’s clearly on the wire on that interface. Looks like it’s being blocked on the way back in, but why ?.

    I get the same response even deleting the nat rule altogether, which might (?) be due to the single outgoing  “access anything” rule on housenet and labnet, but no amount of tweaking seems to get the response back to the requesting machine. I spent a nearly a whole day on this yesterday and i’m probably missing something obvious. Any help or suggestions would be appreciated to put out of misery  :-)….

    Regards,

    Chris


  • LAYER 8 Global Moderator

    "Started by writing a nat rule to do this,"

    Why would you nat from lan to lan that are both rfc1918?

    Lets just be clear here – what are you to lan networks?  For example I have 3 lan segments

    192.168.1.0/24 lan
    192.168.2.0/24 wlan
    192.168.3.0/24 dmz

    Now I allow lan to talk to ANYHING it wants this is simple, wlan can talk to my ntp server on my lan, and then anything else that is ! my lan net (192.168.1.0/24)

    DMZ can talk to anything it wants that are not my locals - this is an alias that has lan, wlan and my openvpn network segments.

    There should be really no reason to nat between local segments.  There should be NO gateways on the interfaces on local segments..  The only thing that you need to setup is the correct filrewall rules on each segment to work.

    So why don't you let us know what your lan network segments are  - like I did above.. and post your rules like I did below.

    If you do not see the response from the server your talking too, and you see the packets leave pfsense -- then tells me the server has firewall and is not answering.. Or maybe has wrong gatewaysetup /mask and thinks to talk to your first segment it needs to send traffic to some other IP(gateway).








  • Hi John,

    Thanks for the reply. Like I said, there are 3 different subnets:

    Wan: 10.x.x.x
    Housent: 172.x.x.x
    Labnet: 192.x.x.x

    Sorry if I did’t make that clear.

    It seems reasonable that  we need a nat rule to talk from housenet to labnet. If not, perhaps you could tell me what I’m doing wrong ?.

    The 192.x.x.x network is historical and dates back to sun 3 days, while the 172.x.x.x network is not assigned and not routable, fwir. Besides, it makes the overall system more secure by having separate numerical subnets for each. It wouldn’t be easy to change either of these and in any case, pfsense nat should be able to handle it. I have no trouble with nat between  wan -> webnet on another pfsense box, so why between 2 x internal lan ports ?.

    Oh yes, system reboot today and without the nat rule, telnet returns “no route to host”, rather than just timing out as it does with the nat rule included, so it is doing something. It’s just not returning packets from the target, as is shown by tcpdump. I wonder why ?…

    Regards,

    Chris



  • John,

    Also, I did  show the output from tcpdump at the server, firelight:

    19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:31.290075 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:36.922216 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:36.922286 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:38.060093 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:42.930063 IP firelight.telnet > 172.16.100.205.57243: tcp 0

    Which shows an incoming packet from the 172 telnet request, line 1, the server response, line 2 then several similar retry attempts from the telnet request host.

    The telnet request packet can also be seen running tcpdump on the pfsense hardware console,, but the reply not there at all, even though it’s definitely on the wire, so it’s 1)  being ignored or 2) blocked at the input to the  labnet pfsense interface.

    If you have any idea of what the normal procedure is to nat across dissimilar local networks, would be interested to know…

    Regards,

    Chris


  • LAYER 8 Global Moderator

    "It seems reasonable that  we need a nat rule to talk from housenet to labnet. If not, perhaps you could tell me what I’m doing wrong ?."

    Why do you think you would need to nat??  NO you do not need a Nat to talk local network to local network.. If you have created some your going to have ISSUES most likely.

    Why are you hiding the lasts parts of your network - they are RFC1918, ie private - they are NOT routable on the public net.. Everyone has the same networks on their local networks.  If you tell them - it tells me NOTHING about your location, not can I talk to your pfsense box using a rfc1918 address.  172.16-31.x.x, 10.x.x.x, 192.168.x.x

    Once you give pfsense an IP address in that range it will know to use that interface to talk - route to that network.

    So I suggest you put your nat back in Auto..  And give out some into so we can help you.  When adding a network network, your labnetwork - all that you need to go is setup your firewall rules, since unlike the default lan pfsense will not create any firewall rules on opt interfaces.  And there should be NO gateway on LAN interfaces – or pfsense will think it is a WAN interface..

    Here is my routing table.. notice the 3 network segments and the route out the interfaces in those segments 192.168.x.253 are all my interfaces in my 3 lan segments.  Those 10.x networks are my openvpn segments.

    if you are trying to NAT there is no reason.. You just need to have simple firewall rules.  And boxes in each segment need to alk the pfsense address in that network segment.

    So for exmaple my pfsense IPs are

    192.168.1.253 lan
    192.168.2.253 wlan
    192.168.3.253 dmz

    So machines in wlan have 192.168.2.253 as their gateway off their network, and dmz devices point to 192.168.3.253 as their gateway.

    You only need to NAT when you go from private address space rfc1918 to public addresses, see the 24.13.x.x address in my route table that is my internet connection.  If your pfsense has a Wan address of 10.x.x.x then your behind a DOUBLE nat already -- you have another router in front of pfsense if your wan is 10.something..  There is normally NO reason to do this unless you can put your ISP device in bridge mode.

    edit:
    "If you have any idea of what the normal procedure is to nat across dissimilar local networks,"

    Again YOU DO NOT NEED to NAT between your own local networks!!!  Put your nat in auto, and delete any rules you have might of put in there..  This is what your outbound nat rules should look like - see attached.

    And there is not need for any port forwards between your networks..  I could not tell much from your tcpdump with that  firelight.telnet vs the address of firelight..

    If you show your box answering, but you don't see it on pfsense..

    19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0

    Then your box is setup wrong for its gateway or its network mask and thinks that IP address is local to its network..  Again its hard to tell where you talking with firelight vs an IP address.






  • NO you do not need a Nat to talk local network to local network

    What?!?

    If you have a client device behind the router it will send anything outside its subnet (assume 192.168.10.0/24) towards the router.

    How will the router know what to do with it without NAT?  (network address translation)

    If your using autonat then the router is doing it for you automatically.



  • LAYER 8 Global Moderator

    You clearly don't understand NAT do you..

    How will it know what to do with traffic to say 192.168.10.42 for example – is it will look in its routing table.. See the one I attached!  And since it has an interface directly attached to that 192.168.10.0/24 network -- its send the packets out that interface..

    Why do you think it would need to NAT that in anyway??  Maybe you need to look up what NAT and or NAPT is and when its needed.  You do not need to NAT anything if you can route between the networks.

    The reason you NAT to the public internet, is the public internet has no clue how to route rfc1918 address space - since its PRIVATE and designed to be used in your location.  Now if you want to have a device talk to say google which is on the public internet, and has a IP address that is routable on the internet - then you need to have an address google can talk to.  So your router that has a IP in public space changes your private space to its public ONE so google can talk back to you.

    It does that really with NAPT, so that more than one private address can share the 1 public IP you have.  This does not have to be done when your router knows how to talk to both networks, since it has an interface in both networks.

    With your wan address there on pfsense of 10.. Your doing NAPT twice when you talk to google..  Since google can not talk to a 10.x address

    So you have this

    pc 192.168.1.a --- 192.168.1.b pfsense 10.x.x.A -- 10.x.x.B router publicIP - internet -- google

    So your Natting twice in your setup with that private address on pfsense WAN..  What you normally have is this

    192.168.1.14 -- 192.168.1.1 pfsense publicIP - internet - google

    but for pfsense to route traffic between 2 network it has interfaces in say

    192.168.1.14 -- 192.168.1.1 pfsense 172.16.1.1 -- 172.16.1.3

    there is NO reason to NAT or NAPT that.. And it will cause you all kinds of problems doing so that there is NO reason to do.

    edit pfsense AUTO nat only nats to the WAN from its lan interfaces - so here is my NAT table.. You can see this by just changing to manual

    Notice there are NAT entries for my local networks to my WAN address ONLY, you don't see any NAT entries between the local networks..




  • Obviously we had different' teachers…

    from the "Firewall: Nat: Outbound:" page...

    If automatic outbound NAT selected, a mapping is automatically created for each interface's subnet



  • Hi John,

    This is in answer to your oringinel reply, 2 posts back

    And there should be NO gateway on LAN interfaces – or pfsense will think it is a WAN interface..

    No gateways are defined for either housenet or labnet, but one is defined for the wan 10.x.x.x network as you would expect.

    _if you are trying to NAT there is no reason.. You just need to have simple firewall rules.  And boxes in each segment need to alk the pfsense address in that network segment.

    You only need to NAT when you go from private address space rfc1918 to public addresses_

    Ok, but the housenet interface at 172.16.x.x is in private address space, while the server labnet is public, so to follow your argument, I do need nat to connect the two ?.

    Also, what you really seem to be saying here is that the wan interface is a special case in nat terms, where I would have expected nat to be applicable to any interface. Is this in fact the case ?.

    I will try a few simple rules to see if that can be made to work. I did a full restore from backup today to get the baseline back, but will try some firewall rules later to see if that can work.

    There is normally NO reason to do this unless you can put your ISP device in bridge mode.

    I have been using double layer hardware firewalls for years, with pfsense or other as the inner layer. I do have good tech reasons as well, but it’s probably not relevant to this discussion…

    Regards,

    Chris


  • LAYER 8 Global Moderator

    "Whatever…  Obviously we had different' teachers..."

    Your TEACH told you that you need to NAT to talk from say 192.168.1.0/24 to 192.168.2.0/24 ???  No I don't think so -- if they did, they sure and the F should not be teaching anything to do with networking.

    If automatic outbound NAT selected, a mapping is automatically created for each interface's subnet

    Yeah I agree with you - from the local private lan segments to your WAN segment, be that wan public or private does not matter pfsense will auto nat from lan to wan..  But it DOES not auto nat from lan to lan segments.. Se my attached rules from my automatic NAT.  All that nat rules nat from the local or openvpn segments to the WAN IPs.. there are no rules that nat between the local or openvpn segments because there is NO reason too, since pfsense has interfaces in all of those network and can talk to them..  While stuff on the WAN side only know how to talk to pfsense wan IP, not some rfc1918 space on one of its lan segments.


  • LAYER 8 Global Moderator

    "while the server labnet is public"

    Why would you be using PUBLIC address space on your local network, behind a NAT router??

    "I have been using double layer hardware firewalls for years"

    There is HUGE difference between firewalls between public network and having say another firewall between your DMZ and your local network.. But you sure and the hell do not need to double NAT to accomplish more than 1 firewall.. And that is behind the scope of this discussion I would agree yes.  But NORMALLY there is NO point to double natting..  But that is outside the current issue - what IP space that is on your wan currently other than the fact that is 10, so private is besides the point.

    I don't know why you would be running public space behind a NAT which you are if pfsense wan is 10..  I have to assume your just natting that outbound to pfsense 10 address - so what is the point of the public space?

    That being said - you don't have to NAT it to talk to another segment directly connected to pfsense.



  • Hi John,

    The 192.x.x.x network is historical, dating back to the earliest networked systems here in the lab. The server hosts file still has most of them listed as well, even if many of the machines were retired years ago. For me, ongoing context is important and that's the way it is, that's why :-).

    Is there a solution using the network as it is, or do I have to dig out an old smc barricade mini ethernet router to port forward  the subnets and get the job done while I try to work out what's wrong ?…

    Regards,

    Chris


  • LAYER 8 Global Moderator

    What??

    So your saying you have public address space behind nat 10.x – for what possible freaking use??

    But again its besides the POINT.. you do NOT need to nat from 2 networks directly connected to pfsense..

    Does not matter what I use for the network segments on pfsense they could be public, they could be private does not matter..  between lan I do NOT have to NAT..  I only need to nat to wan if devices connected to wan and beyond don't know how to route to the lan segments of pfsense..  Like the internet not knowing how to talk to 192.168.1.0/24

    If the devices connected beyond pfsense wan will know how to talk to 192.16.1.0/24 then I don't have to nat at pfsense.. I can nat when my networks talk to some network, where they wont know how to talk to my private networks -- ie the internet.

    Let say you have this - see attached.

    Does not matter what the lan networks are, be it they 192.168.x.x, 172.16-31.x.x or 42.15.0.0/23

    inetnum:        42.8.0.0 - 42.15.255.255
    netname:        SAMSUNGSDS-KR
    descr:          SamsungSDS Inc.

    Since I just pulled that network out of my A_S ;)

    Since pfsense has a interface in that network -- lets say 42.15.0.1/23 and has an IP in say another lan segment 192.168.1.1/24  there is NO need to NAT between 192.168.x.0/24 and 42.15.0.0/23

    because all the devices talking to pfsense as their gateway -- pfsense knows how to route to those 2 networks.

    Now when either of those 2 networks need to go out pfsense wan (that is connected to the internet) then Yes pfsense would need to NAT that.. since IPs beyond pfsense wan IP don't know how to get to 192.168.x.0 network connected to pfsense..  And if you want to use public IP space -- you wouldn't need to nat it -- if that public network is ROUTED to pfsense WAN address..  But with your 10.x.x.x on your wan I find that HIGHLY Unlikely!

    This is networking 101 - how is it your not understanding this??  When you state you always use double hardware firewall, ets. etc..




  • Hi John,

    Looking at your network, the reason why you don't need nat may be that all the subnets are on the same private block and pfsense may recognise that internally. It's difficult to say if this is the case without wading through the sources, since there appears to be no docs that describe pfsense internals, or perhaps there are ?.

    Anyway, all my subnets are not on the same private block and need to get this working.  A nat rule partially works, but doesn't return packets, so it looks like the smc box bridging / port forwarding the subnets while I try to figure out why nat is broken may be the best short term solution…

    Regards,

    Chris



  • pfsense knows how to route to those 2 networks.

    This is what Im (obviously incorrectly) heaping into the NAT arena…

    Seems to me I had to build NAT rules (since I was using manual NAT at the time) when I added my second LAN subnet a year ago.


  • LAYER 8 Global Moderator

    "Anyway, all my subnets are not on the same private block and need to get this working.  "

    Dude what part are you NOT understanding about 2 lan networks directly connect to pfsense NOT needing nat.

    If these 2 networks are directly connect to pfsense - then NO NAT IS NEEDED and is only going to cause you problems - because now if your trying to nat between LAN networks you going to have to do that manual, and your also going to have to create port forwards for the traffic you want to create to IPs behind the NAT, etc..

    Again - I am going to say this yet again.. THERE is NO NAT between LAN network segments directly connected to a pfsense..

    Do I really need to change my DMZ segment to some other network to show you that??  Really??


  • LAYER 8 Global Moderator

    ok LOOK – changed my DMZ network to 172.15.0.0/24 -- gave pfsense 172.15.0.1/24 address.. See attached.  Did not change my dmz rules.. It can talk to anything it wants other than my local networks..  But my local networks can create connections to it..

    That is a PUBLIC network space..  Not rfc1918.. But notice my pfsense has route to it..  And I can ping a host I brought up on 172.15.0.42/24 with gateway pointing to pfsense 172.15.0.1 address from my 192.168.1.0/24 network

    That took me all of what 2 minutes to setup??

    There is NO freaking NATS needed between 2 locally connected networks to pfsense..  I assure you there are NO nats between those networks!!

    What traffic I allow between 192.168 lan and 172.15 dmz would be my firewall rules ONLY - there are NO port forwards required for these 2 local network to talk to each other - no matter what IP space I use on them.










  • Well-  Thanks John!  I learned something new today.

    http://www.zytrax.com/tech/protocols/ip-classes.html#nat

    A well written NAT system also acts as a 'poor mans' firewall since it has the additional advantage that Internal IP addresses are not visible from outside the organisation

    Obviously this isn't something that LAN to LAN would want or need.

    Also- verified here as well and turned off all NAT and was still able to move around throughout the various LANS here. Including one of the VPNs to my office network.


  • Rebel Alliance

    :-[

    May i just one…....  "what if ?"

    What if of the "private" networks isn't using the pfSense as its GW ?

    Let's say that the 172.15.0.42 host is using as GW an IP other than your pfSense 172.15.0.1.... It would you be able to ping it ?

    :-[


  • LAYER 8 Global Moderator

    "Let's say that the 172.15.0.42 host is using as GW an IP other than your pfSense 172.15.0.1."

    When I have asked already multiple times in this thread..

    Or maybe has wrong gatewaysetup /mask and thinks to talk to your first segment it needs to send traffic to some other IP(gateway).

    If you show your box answering, but you don't see it on pfsense..

    19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0
    19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0

    Then your box is setup wrong for its gateway or its network mask and thinks that IP address is local to its network..

    But WHY would my 172.15.0.42 NOT use pfsense as its default gateway??  But yeah if that was the case, then sure you "could" nat so that my 172.5.0.42 saw the traffic as coming from its local network, and would not talk to a gateway to talk to it..

    If that is the case why has the OP not stated this - I have brought up that scenario a couple times already..  Real early in the thread even when he stated pfsense was not seeing a response.

    Its really simple when you need nat - you need nat when the dest would not know how to talk to the source IP, so you need to change the source IP to a IP that the dest can talk too..  Or you need to create routes so it does know..

    if you had like what your saying.

    192.168.1.0/24 pfsense 172.15.0.1 –- 172.15.0.42 -- 172.15.0.254 router otherIP -- other networks.

    Where .254 was say the default gateway for .42 you have 2 options..  You could nat traffic coming from 192.168.1.0/24 so it LOOKs like its really from 172.15.0.1 -- so .42 thinks its just local. Not my first choice since be it you use NAT or NAPT you made it more complicated than simple route and firewall rules (if even firewall between and not just router)

    OR!!  Simple option –> You put a route on .42 that says hey when you want to talk 192.168.1.0/24 use 172.15.0.1 as your gateway to that network and don't send it out your "default" gateway..

    This is like the pfsense routing table..  Since pfsense has routes to the networks involved, it doesn't send the traffic out its "default" gateway..  It sends the traffic out the interface connected to that network.



  • Hi John,

    I’ve tried to explain what I’m trying to do, a simple port forward between internal lan segments and have given chapter and verse on the network topology and loads of debug info. All  you seem to do is find fault and  criticize via various side issues, without once answering any of the questions directly.

    Don’t be offended, but it sounds like angry father syndrome, rather than understanding mentor. Perhaps clam down a bit and actually try to answer the questions ?…

    Regards,

    Chris



  • This really should be simple, and just work by default as long as there are firewall rules that allow the traffic.
    Can we start again and get exactly what is where:
    a) Each interface name and its IP address on pfSense
    b) What rules are on each interface
    c) What the clients have for their default gateway (hopefully the respective pfSense interface IP address)
    d) Details of any other router/gateway device in the network


  • LAYER 8 Global Moderator

    Dude I have gone over this and over this - what part about NOT needing nat do you not understand??

    You have not answered anything that has been asked..

    For starters does 172.x network use pfsense as its default gateway or point to something else?

    If your going to insist on NAT, your going to have to create it manually because pfsense does not NAT between LAN segments automatically.

    Do your pfsense LAN interfaces have gateways on them - see my example where there is NO gateway listed on LAN interfaces.

    Where are you rules - Post them!  And the networking setup from your devices.  Where is your pfsense routing table?

    As I showed you it takes literately only a couple of minutes to route traffic on pfsense between lan segment - there is NO need to NAT.

    As stated with your tcpdump if your saying your seeing pfsense send the packets, and seeing your box answer those packets but not being seen on pfsense.. Then you have something wrong with your client configuration.  Be it it thinks that source IP is local, be it has another route to that network - ie a different default gateway?  Or something between pfsense and this client not allowing the traffic (firewall?)

    Without some details NOBODY can help you.



  • Hi,

    Thanks for the replies and hope you won’t mind if I answer both in the same post..

    Phil:

    This really should be simple, and just work by default as long as there are firewall rules that allow the traffic.
    Can we start again and get exactly what is where:

    a) Each interface name and its IP address on pfSense

    There are 3 hardware interfaces.

    wan: 10.0.x.x         External, internet Default gateway ->  upstream
    homenet;         172.16.x.x Home, internal No gateway defined
    labnet: 192.9.x.x         Lab, internal.    No gateway defined

    b) What rules are on each interface

    wan: None, other than one to block bogon networks
    homenet:         One, to allow home net to anything
    labnet: One, as per homenet.

    There’s also the admin anti lockout rule on labnet, on a non standard https port. Admin account name is different as well, but doubt if that should affect anything.

    This is all working  fine for outgoing access via either interface to the web, but if I try to telnet a homenet node to the labnet node, I get “no route to host” which is expected since there’s no rule or port forward defined to allow it.

    c) What the clients have for their default gateway (hopefully the respective pfSense interface IP addresses)

    Correct for both

    d) Details of any other router/gateway device in the network

    1 upstream from wan interface,  none on homenet or labnet

    As I understand it, pfsense blocks everything by default, so you need rules even for outgoing access. For that reason, it seems logical that if I want to access a host on  labnet from a homenet host, I need a port forward or some sort of rule to allow it. Port forward on  another pfsense box works fine incoming from the wan to a lan port, but it  doesn’t seem to work from one internal port, to another on this box, so perhaps all ports are not created equal / have the same capabilities ?.

    Ok, so I define a port forward rule for telnet as follows:

    Src Src Dest Dest Nat Nat
    Addr Ports Addr Ports Addr Ports

    172.16.x.x * housenet net 23 192.9.x.x 23

    Sorry no screen shot, but haven’t got round to that yet..

    Interface is housenet, protocol = tcp/udp, nat reflection = default and nat creates associated rule. Except for the interfaces, this is the same setup as that for the other pfsense box on the webserver, which works fine.

    Using this rule, the telnet request is seen on homenet with wireshark, can be seen outgoing on the pfsense local console and the server on labnet replies, but the reply is lost on it’s way back into pfsense labnet interface. All the tcpdump trace info is in a previous post above, fyi.

    Do you need anything else ?.

    John,

    Dude: … what part about NOT needing nat do you not understand ??

    Well, all of it, unless you can tell me how it can be done without a port forward / nat, or rules of some sort :-). (Note the smiley :-)

    All the debug info is in the previous posts, other than the rules,  but if there’s anything I’ve missed  this time, please let me know…

    Regards,

    Chris



  • Hi,

    A bit more info:

    Have also tried various variations on the above rule and also various switches in system -> advanced -> firewall-NAT, but none of it seems to work.

    Is there any way to force wysiwig from post editor -> preview -> post.  Formatting lost :-)…

    Regards,

    Chris



  • This is all working  fine for outgoing access via either interface to the web, but if I try to telnet a homenet node to the labnet node, I get “no route to host” which is expected since there’s no rule or port forward defined to allow it.

    Actually, it is expected to have a route and thus deliver your packet/s. pfSense (and every router I have ever seen) will route between local subnets by default.

    homenet:          One, to allow home net to anything

    That rule should allow home to labnet, as well as homenet to google, homenet to facebook, homenet to anything.
    There REALLY is no need to use NAT for this. There MUST be some other tricky thing that you have accidentally set up that is causing this not to work, or the target system in labnet does not respond to telnet from another subnet or…
    Look in the Firewall log and make sure packets are not being blocked there. Then do some packet capture on homenet to verify the telnet initiation packet/s arrive, then on labnet to verify they leave labnet, then look for the response packet from labnet client on labnet and then homenet. Wherever the packet/s stop being seen is where to look next.



  • Phil,

    Thanks for the reply. The install is plain vanilla from the iso, with no special tweaks. I’ve been using pfsense for years now, with ipcop, freesco and packet filtering in the past, along with doing electronics / sw eng for work for decades,  so hopefully not a complete newbie to this. Strange thing is that I’m pretty sure this worked on 2.03, but may have been ipcop, as it’s some time since I had this requirement set up.

    If you read the op, you can see that I have been packet monitoring at 3 points: homenet via wireshark, pfsense and labnet via tcpdump. There’s a packet trace that proves that the reply from the remote server is being dropped at the pfsense labnet interface, on the way back in, as it is seen on tcpdump labnet, but not on tcpdump pfsense console.

    While you and John both seem to think that packets between local interfaces are routed by default, in fact they appear not to be. As I said, pfsense blocks everything by default and you need outgoing rules just to access the wan from local.

    Regards,

    Chris


  • LAYER 8 Global Moderator

    Dude post a screen shot of your rules for gosh sake, and your routing table.

    I am going to say this ONE LAST time – there is NO NEED to NAT between local segments..  PERIOD! I have shown you this - it is FACT, you do NOT have to NAT between local networks segments no matter what address space your using.

    as to this
    "the remote server is being dropped at the pfsense labnet interface,"

    No dropped is the WRONG word..  Not seen is the right word from what you have shown.. Even if you had a block rule there the tcpdump would still show the packets if they hit the interface.

    So if your saying the packets are not being seen there then you have another issue.. Validate that the packets that leave your client actually have the correct MAC for one.  And what is between??

    Had a very strange thing with a cisco switch awhile back where packets were not being forwarded in a vlan unless there was a SVI on the vlan..

    https://tools.cisco.com/bugsearch/bug/CSCth74527

    IF your not seeing the return packets as pfsense - then this issue has NOTHING to do with pfense!



  • John,

    4 images to post, but call me clueless, how do you inline images on this board ?. There's the button, but paste file contents doesn't seem to do anything. Intuitive, or what ?. Help file !=  useful either.

    Just to add, I know that the hardware is ok because I can telnet or ftp into the lab server from the pfsense console, as well from any other machine on labnet. Also tried a direct connection from the housenet machine (ip changed) to the server and that works fine. There can be some funnies with telnet between some machines, but not here. I use ftp from pfsense all the time to send the backup dump files to the server. All the lab machines and pfsense are on the same 3Com 29xx series switch.

    The cisco link is behind a login, but I played with a pix515 some time ago and thought the user interface (windows client) unintuitive and primitive compared to pfsense, or even ipcop. Ymmv, of course :-)…

    Regards,

    Chris


  • LAYER 8 Global Moderator

    click the preview button on the bottom so you get the full editor

    if your wanting to link to a img else where - then use the tags and put in the url to your image.




  • John,

    Thanks - Didn't have the wysiwyg editor selected, but still clueless. Click on add image, which then asks for the location, which I fill in for the first as:

    i:\ImageFile\Screenshots\HouseRules.png

    Which insertes a box into the reply, but preview sees nowt. Anyway, 4 images included as attachments…

    Sometimes just easier to ask etc :-).

    Regards,

    Chris






  • …2 replies as the image > 300K total -





  • LAYER 8 Global Moderator

    And how would pfsense forums have access to i:\ImageFile\Screenshots\HouseRules.png ??  If you want to do inline with img tags then you need to have some url that pfsense and for that matter the viewer of the fourms could access ie http://images.something.tld/image.png for example.

    And what is that rule doing above your tenet rule? with the advanced tag on it?

    Dude – for testing lets call it.. make your housenet rule like your labnet rule.  Where source is the network and dest IP and port are any any.  Where are your autobound nats - are they automatic or manual.  And what are you doing in the advanced section -- that a tag on the rule.  Remove all port forwards!  And what our your outbound nats - post them.

    But lets forget any rules you have on pfsense for now.

    If your saying when you tcpdump on lapnet pfsense interface, you see the telnet packet go out to your 192.9.x.x address you don't see a response then pfsense has nothing to do with the issue.

    Pfsense sends out packets to 192.9.x.x:23 from 172.16.x.x:random, and you don't see packets come back..  Then 172.16 did not answer, or sent it to the wrong place or something between 172.16 and pfsense interface is not forwarding along the packet.

    tcpdump will show you all packets before they hit any nat rule or firewall rule.  So if the packets are not there - there is NOTHING that pfsense could do.

    btw not sure what your doing with the advanced options - but that rule makes the rule below it about telnet pointless and would never be used unless something in the advanced options would rule out telnet?

    Also telnet is TCP, not UDP so having both tcp and udp is again pointless.



  • 192.9.0.0/16 is allocated to Sun Microsystems and is not private address space.

    Just fyi.


  • LAYER 8 Global Moderator

    ^ yeah we know, been brought up already.. Another ? he never answered after he stated he had answered everything, etc. ;)

    "have given chapter and verse on the network topology and loads of debug info"

    His wan is clearly private..  So at a loss to why you would be using public IP space behind a NAT..  Unless there was some other path to get to these boxes?  Maybe he thinks its OK to just pull IP space out of the AIR and use it - maybe he works for Sun?



  • John,

    This is getting hard work :-)

    There might have been a time when I would have thought it very cool and a privilege to work for Sun, since they made some of the earliest and most competent unix systems around, Full gui when most pc's were still running dos. I’m in the uk, am probably too old now and do hardware and embedded systems development, not unix. Besides, they never asked me :-), they no longer make workstations anyway and they are owned by oracle now, which is a disaster.

    If you  actually read through the earlier  posts, you would see that I said that the 192.9 lab network is historical. My very first unix box was a sun 3 system found in a junkyard. All the sun docs of the time used the 192.9 block, so that’s what I used. I have a load of respect for early unix development and like historical context, so never changed the 192 block for the lab, nor deleted old host names. Anyway, this is all nitpicking since pfsense really shouldn’t care what address block you use for any of the interfaces and there is no “correct” way to do it, other than the way you want it. The rules are what matters.

    As for tcpdump traces, I have copied the traces twice to previous posts here, the second time with a description of what’s going on. Perhaps you did't understand it, so will try again in verbose mode. Note that tcpdump is tracing output from the pfsense labnet interface and the server replies. Server host name is firelight.

    $ tcpdump -q host 172.16.100.205
    tcpdump: verbose output suppressed,
    listening on bge0

    19:00:27.909689 IP 172.16.100.205.53942 > firelight.telnet: tcp 0

    The above line shows the initial telnet request from the housenet node

    19:00:27.909804 IP firelight.telnet > 172.16.100.205.53942: tcp 0

    This is the first server repl

    19:00:30.914297 IP 172.16.100.205.53942 > firelight.telnet: tcp 0

    This is the first retry from the housenet host

    19:00:30.914362 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:31.290075 IP firelight.telnet > 172.16.100.205.53942: tcp 0

    This time, we get two replies for redundancy, telnet is assuming the packet was dropped, or therwise corrupted on return to the client.

    19:00:36.922216 IP 172.16.100.205.53942 > firelight.telnet: tcp 0

    The third retry from housenet

    19:00:36.922286 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:38.060093 IP firelight.telnet > 172.16.100.205.53942: tcp 0
    19:00:42.930063 IP firelight.telnet > 172.16.100.205.57243: tcp 0

    Here we have three replies at once, again for redundancy. Telnet thinks this must be a very sh***y line.

    So, the server is doing it’s best responding to the telnet request, but no one is at home on pfsense labnet receive and the replies are being dropped. This is confirmed by tcpdump on the pfsense console, which shows the outgoing request to the server, but not the reply.

    You asked for the rules etc, so what's the solution, if there is one ?. The setup is about the most basic you could imagine, yet it doesn’t work as expected…

    Regards,
    Chris



  • I would therefore conclude that traffic arriving on labnet interface with destination in homenet is somehow being routed elsewhere, not back to homenet. (Because there should be a state established by the 1st packet in your example, and the responses in the reverse direction will match that state and be allowed in/through regardless of any firewall rules on labnet interface).
    What routing related stuff have you got defined - static routes?
    What does the routing table look like - diagnostics->routes?
    Do tcpdump on other interfaces (e.g. WAN) and see if the response packets are exiting there. (I guess you have checked that they are not exiting on homenet).



  • Phil,

    Thanks. I posted a screenshot of the routing table above - there's no routes defined other than the default, wan interface, which points upstream to the next level router.

    The port forwarding rule is still in place. Without it, I just get a "no route to host" at the client.

    Did another trace at the pfsense console:

    tcpdump -i de0  -q  host 172.16.100.205

    de0 is the labnet interface. Anything in or out related to 172.16.100.205 on de0 should be traced. The strange thing is that while the telnet request shows  up on the trace, the reply from the server (which really is on the wire), doesn't. What I get is the initial request, client ->  server and several retries, but nothing else. Typically:

    14:52:42.713118 IP 172.16.100.205 > 192.9.xxx.xxx.telnet :tcp 0

    homenet client is the 172.16 source, server is the 192.9 destination.

    The only way I can explain the discrepancy between the tcpdump traces, server vs pfsense, is that pfsense is tracing at a higher layer in the stack  than where the reply  is being dropped. That doesn't really make sense  though, since tcpdump is generally understood to trace at hardware level or close. What do you think ?.

    As for the state table, screenshot attached. This is immediately after the telnet request and shows a pair of relevant states, but no established connection  ?. Line 1 looks like the keep alive ping upstream, 2-7 are management login related and 8-9 are telnet. Finally, the blanked out addresses in the telnet entries are the server address

    I'm pretty rusty on low level network ops and will be digging out the Tcp/Ip Illustrated books later for more inspiration :-)…

    Regards,

    Chris




  • Can you sniff the wire itself somewhere with another system to see if the reply packet is actually there?
    pfSense tcpdump should really see that response packet as long as it actually gets through the NIC hardware and firmware to be delivered to the FreeBSD driver. Any switch on labnet that connects the server and pfSense should learn MAC addresses from the first packet that is sent, and thus know which switch port the pfSense is in… Maybe a cable is half fallen out, and it can transmit through to the server, but the pair for the reverse direction are not connecting properly (but I think you said you can ping from pfSense to labnet server, so the physical layer must be OK).
    I am struggling to think where the problem can be. At this point you have to make really sure that the response packet is actually leaving labnet server and getting onto the labnet wiring.



  • Phil,

    I can ping from pfsense, as well as ftp and just checked that again. There’s a debian machine that I can patch in but will need to hook up a hub, since all the boxes are on a switch. Ok, I can port miror the switch, but a hub is most likely quicker in this case and it don’t lie:-)

    The other thing I’m going to do today is delete all the current rules and port forward, reboot, re-input the rules and reboot before checking again. None of this makes sense at present and it seems to me that maybe some part of the config has got out of sync. I was trying various combinations in the port forward definition, trying to make it work and the config files may be confused, depending on how the gui entered stuff translates down to the xml (?)  files.

    Failing that, reinstall from iso but try that first. Thanks for the help / inspiration and will report back later…

    Regards,

    Chris


Log in to reply