PfSense can ping all but one specific IP address in range



  • Hi,

    will try to give resume of my setup:
    1. I have a production network range 192.168.22.x
    2. We use a Veeam backup server for VMWare. The Veeam server IP is 192.168.22.20. This server can create a 'sandbox' to test virtual machines.
    3. I create virtual machine in a sandbox, and make at static IP address mapping from production network into the VM in the sandbox. The IP is 192.168.22.15 on production network (I have tryed other IP for the VM aswell, but all results are the same)
    4. I can ping and remote to the VM in the sandbox from every server/PC in the IP range 192.168.22.x.
    5. The only place I can NOT ping the VM in the sandbox from is the pfSense firewall.
    5.1. I use the correct interface to ping from
    5.2. the firewall can ping the Veeam server on 192.168.22.20 and other servers/PC's in the range, but not the VM in sandbox on 192.168.22.15.

    I have never seen this occur before, so any usefull input will be much appreciated.

    I need to get firewall to ping/connect to the 192.168.22.15 IP, because I need remote access the VM in the sandbox. The firewall is preventing me from achieving this.


  • Netgate Administrator

    I assume there is nothing in the firewall logs? Or the system logs?
    Possibly something to do with the MAC translation from the real to virtual NIC in VMware. Just speculation.

    If there's nothing in the logs it's probably a routing problem. Possible the VM doesn't have a route back to the pfSense box. Maybe the wrong subnet mask?

    Run a packet capture on the LAN interface to see if pings are leaving and returning at all.

    Steve



  • Give us a network map with IP and mask info.



  • @eeverglades:

    5. The only place I can NOT ping the VM in the sandbox from is the pfSense firewall.

    What does ping report when you attempt that? (The ping report is almost always more informative than "can not ping".)



  • Let me know my friend did you setup the vmbox network adapter to bridge connection to your ethernet card or wireless.by default the vmbox it will access Nat connection so you cant ping form the out side connection …

    check your both ipaddress range in vmbox and out side is match to ping both connection and post here network configuration here we cant assume with out clue dear friend



  • Wau, quite an active forum. Did not expect so many replies so soon.

    First a confession: the IP range is not 192.168.20.x/24, but 192.168.88.x/24

    So important IP address in that range are:
    192.168.88.15 = VM in sandbox
    192.168.88.20 = Veeam server (backup server)
    192.168.88.254 = pfSense firewall interface

    When I try packet capture from firewall with full detail setting for IP 192.168.88.15, and then ping the VM I get:
    07:41:47.733598 ARP, Request who-has 192.168.88.15 tell 192.168.88.254, length 28
    07:41:48.752283 ARP, Request who-has 192.168.88.15 tell 192.168.88.254, length 28
    07:41:49.772138 ARP, Request who-has 192.168.88.15 tell 192.168.88.254, length 28

    When I try the same capture for the Veeam backup server I get:
    07:47:44.423325 IP 192.168.88.254 > 192.168.88.20: ICMP echo request, id 29645, seq 0, length 64
    07:47:44.423642 IP 192.168.88.20 > 192.168.88.254: ICMP echo reply, id 29645, seq 0, length 64
    07:47:45.440352 IP 192.168.88.254 > 192.168.88.20: ICMP echo request, id 29645, seq 1, length 64
    07:47:45.440588 IP 192.168.88.20 > 192.168.88.254: ICMP echo reply, id 29645, seq 1, length 64
    07:47:46.460201 IP 192.168.88.254 > 192.168.88.20: ICMP echo request, id 29645, seq 2, length 64
    07:47:46.460534 IP 192.168.88.20 > 192.168.88.254: ICMP echo reply, id 29645, seq 2, length 64

    And as mentioned before, I can ping and RDP to the VM in the sandbox from any server/PC in the 192.168.88.x range.

    So I'm ??!??!??!????



  • Thank you for the info rakeshvijayan.

    The pfSense firewall is Virtual, so it has no physical Ethernetcard.

    All the other servers which can ping the VM in the sandbox are also virtual.

    I'm unable to ping from the 'inside' interface of the firewall.

    Hope I understood your post correctly, and my reply therefore is relevant.



  • how you configure pf ethernet card what ever may be there must be a  virtual interface in actual os of wmware to connect to virtual machine check there . if it whole are correct post a Image of you pf interface here we need to check the configuration that made there ….my static ip configuration is so with cidr 24



  • I have not mentioned that the 192.168.88.x/24 is also using Vlan 88.

    I tought that maybe this could be an issue, but after checking the Vswitch setup, I still can't find an error.

    I have also tryed with and without Vlan 88 set on the firewall interface on pfSense, but still no go.

    I have attached a Vsphere setup picture.

    The firewall is the one called 254.domain.lan.

    A server is the one called 102.domain.lan.

    102 can ping VM, but 254 can not.







  • Netgate Administrator

    How many interfaces do you have on the pfSense VM? Looks like just one.

    The fact that when you try to ping from it it is ARPing and getting no reply is not good.

    Steve



  • @eeverglades:

    When I try packet capture from firewall with full detail setting for IP 192.168.88.15, and then ping the VM I get:
    07:41:47.733598 ARP, Request who-has 192.168.88.15 tell 192.168.88.254, length 28

    This is pfSense trying to discover the MAC address of the system with IP address 192.168.88.15. That there is no reply suggests to me one or more of:
    1. The "plumbing" linking VMs doesn't include a system with IP address 192.168.88.15
    2. Such a system is configured to ignore ARPs.

    I suggest you do a packet capture in that VM to see if the ARP Requests are reaching it and it is responding.



  • Yes, it is highly likely that the ARP requests are not reaching the VM from the pfSense, but why?

    The VM accepts ping and RDP from all other devices in the IP range, so it is not an issues with the VM setup.

    It must be some sort of network issue…...

    The backupserver hosting the VM is physical, so all the traffic goes through a physical NIC. But why can the server access the VM, while the pfSense cant, when their setup seems identical?

    They are in the same IP range, and both are connected to Vlan 88. The backupserver is also connected to Vlan 88.


  • Netgate Administrator

    3. It is responding to ARPs but the pfSense box is not seeing the response.

    Perhaps the other VMs have already cached the MAC/IP of the server. Is the pfSense box the most recent VM?

    Can you ARP for that IP from any other machine?

    Steve



  • Hi Steve,

    Yes, all machines in the range can ping the VM in the sandbox.

    No, the pfSense has been in production for over 1 year.

    I'm not an network expert, since it's almost 10 years since i've studied ARP etc., and have forgotten all about it.

    I've tryed different IP's for static mapping to the VM in the sandbox, and all the servers can find the VM right away. But pfSense won't.



  • @eeverglades:

    I've tryed different IP's for static mapping to the VM in the sandbox, and all the servers can find the VM right away. But pfSense won't.

    I think if you want more specific help you will need to provide much more detail on your configuration. In particular, how pfSense is supposed to communicate with the "problem" VM. I don't know vSphere but I consider it suspicious that your previously posted vSphere configuration screenshot doesn't show the problem VM on the same VLAN as the pfSense x.x.x.254 interface.



  • Ok, i've tryed to make a drawing using paint (yes good old paint :) )

    Does this give you guys any possible soulutions or ideas for tools for problemsolving?



  • Netgate Administrator

    Hmm. I would try creating a different server in the sandbox and see if the results are any different.

    You haven't shown any VLANs on the diagram, I assume everything there is in the same VLAN?

    Check the MAC of the sandbox server against the real NIC and anything else in the chain. .20 .22 and .15 are presumably using the same physical NIC. There may be more than one device using the same MAC which is causing pfSense a problem. Do you have any other FreeBSD boxes to test with?

    Steve



  • FORM YOU PICTURE SHOW THAT YOU CONFIGURED IPS IN SAME RANGE NO NATING IS DOING THERE . MY SUGGESTION IS  TRY TO REMOVE  THE TICK FROM  Block private networks  Block bogon networks FROM THE INTERFACE . THIS MAY SOLVE YOU PROBLEM



  • Been on a long weekend vacation…...

    Rake, good suggestion, but unfortunately the boxes are unticked :(

    Stephen, the MAC's of the sandbox proxy and real NIC are different.

    I have checked the ARP table on the pfSense, and the IP of the sandbox proxy is in the table, although the IP of the VM in the sandbox is not.

    Note: I am NOT able to ping the ip of the sandbox proxy from the pfSense firewall either. All other servers in the range can ping the sandbox proxy IP.

    I am ever so close to jumping out the window (don't worry, only a 2 feet drop). This problem is just not logical.........


  • LAYER 8 Global Moderator

    So if pfsense is not on the same segment??

    The firewall is the one called 254.domain.lan.
    A server is the one called 102.domain.lan.
    102 can ping VM, but 254 can not.

    As mentioned above by wallabybob where is this 88.15 box connected to that vswitch?  If that vswitch is the 192.168.88.0/24 ??

    Show us this box your trying to ping on your vsphere setup.  And its ipconfig /all – I am guessing its a windows box?



  • Hi John,

    thank you for joining.

    If you check the drawing I made on top of page 2 in this tread, you might get the overview you need.

    The sandbox proxy is created by the backupserver, and is a linux as far as I know. I have no linux experience.

    Let me know if there is any other info you might need.


  • Netgate Administrator

    You didn't say whether you have any other FreeBSD machines on your network?
    One possibility is that FreeBSD, and hence pfSense, adhere strictly to the rules regarding IPs, routing, subnets etc. Other OSs not so much. Hence it's possible to have a setup that works from Windows and not FreeBSD.

    Steve



  • No, there are no other FreeBSD machines in that IP range. So maybe this could be relevant. You have any suggestions as to how I might check this? A complete novice with FreeBSD.

    I will just point out once more that pfSense can ping the backup server at .20 and all other machines in the range. Only the proxy/VM are 'unpingable'


  • LAYER 8 Global Moderator

    What do you mean by proxy for sandbox?  is that just another esxi host?  Or other vm, then show us its vwswitch setup, like you did with the esxi host containing the pfsense and 102 VMs


  • Netgate Administrator

    @eeverglades:

    You have any suggestions as to how I might check this?

    No not really. You could setup a FreeBSD (or some derivative of it) machine and see how that behaves but that's not a quick and easy test. To be honest I doubt that this is the cause but I thought I'd mention it since we seem to be running out of options. The only time I've heard of it was a Windows box that was using a gateway outside of its subnet something that pfSense refused to do in the same situation.

    Steve


  • LAYER 8 Global Moderator

    can we see the ipconfig /all from the 102 box that you say can ping this 88.15 box



  • To Johnpoz (the ipconfig /all of the 102, which can ping VM):

    Ethernet adapter DOMAIN.LAN:

    Connection-specific DNS Suffix  . :
      Description . . . . . . . . . . . : Broadcom BCM5716C NetXtreme II GigE (NDIS
    VBD Client) #36
      Physical Address. . . . . . . . . : 78-2B-CB-49-76-5F
      DHCP Enabled. . . . . . . . . . . : No
      Autoconfiguration Enabled . . . . : Yes
      Link-local IPv6 Address . . . . . : fe80::31dd:2adc:9c3a:9b1e%13(Preferred)
      IPv4 Address. . . . . . . . . . . : 192.168.88.10(Preferred)
      Subnet Mask . . . . . . . . . . . : 255.255.255.0
      Default Gateway . . . . . . . . . : 192.168.88.254
      DHCPv6 IAID . . . . . . . . . . . : 309865419
      DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-16-C8-11-52-78-2B-CB-49-76-60

    The default gateway is the pfSense, and all this works perfectly. The pfSense can also ping 102.

    The proxy for the sandbox is created by the backupserver, and is not part of the VMWare enviroment. Please view drawing on top of tread page 2.


  • Netgate Administrator

    I expected the above to show 192.168.88.102 not .10 is that just a typo?

    Steve



  • Hi Steve,

    you are absoluty right. Wrong server ipconfig. This is the config for 102:
    Ethernet adapter Domain.lan:

    Connection-specific DNS Suffix  . :
      Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #
    2
      Physical Address. . . . . . . . . : 00-50-56-96-42-07
      DHCP Enabled. . . . . . . . . . . : No
      Autoconfiguration Enabled . . . . : Yes
      Link-local IPv6 Address . . . . . : fe80::c52d:5bbd:1909:a310%14(Preferred)
      IPv4 Address. . . . . . . . . . . : 192.168.88.102(Preferred)
      Subnet Mask . . . . . . . . . . . : 255.255.255.0
      Default Gateway . . . . . . . . . : 192.168.88.254
      DHCPv6 IAID . . . . . . . . . . . : 318787670
      DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-18-36-15-5A-00-50-56-96-42-06


  • LAYER 8 Global Moderator

    I saw your paint drawing.. And it makes no sense to what you mean by Proxy??

    What proxy software?  Do you mean its doing NAT?  How does it proxy?

    If you saying 88.102 can talk to 88.15 how does it do it via the proxy?

    From your drawing there is no difference between the interface connected to the vswitch for your 88.102 box and pfsense at 88.102

    Ok just took a look real quick!!

    http://www.veeam.com/vmware-esx-backup.html

    And its doing NAT into your sandbox, clearly states the stuff in the sandbox are isolated and you connect to them via a masqueraded IP

    http://forums.veeam.com/viewtopic.php?f=24&t=9329&start=15#p40131
    Q: How is it possible to access temporary VMs in the isolated network from production network, if VMs in both networks have the same IP addresses?
    A: Each temporary VM is assigned so called "masquerade address" from selected masquerade network (part of virtual lab settings). Routing table on Veeam Backup server is automatically updated, and proxy appliance IP address in the production network is assigned as gateway for masquerade network. Acting as gateway, the proxy appliance performs address translation and substitutes masquerade IP address with real IP address in the isolated network. Although this sounds pretty complex, all happens transparently for you as a user.

    So what was the NAT setup you created when you installed this veem backup - it says you create that scheme when you setup.



  • Yes, correct. The masqrade IP range isolates the enviroment in the sandbox from production, and the masqrade IP range is only accessible from the Veeam server itself. In my case the masqrade range is 188.x.

    I have a server .88.101 in production. I then recreate this server in the sandbox, which means it gets the masqrade IP of 188.101 to the world outside of the sandbox.

    To be able to access the VM which is created in the sandbox from the production enviroment, you can map an unused IP adress in the production through to a VM in the Isolated enviroment.

    Ergo I have mapped IP 88.15 in production to 88.101 in sandbox, which means that when I try to ping 88.15, I am actually trying to reach the VM in the sandbox, which has an IP of 88.101.

    Hope you understanding my attempt at explaining.

    To sum up the pinging status:
    pfSense: .88.254
    production server 102: .88.102
    Veeam server: .88.20
    Proxy gateway 'inside' of veeam server: .88.22
    Masqrade IP for VM in sandbox: .188.101
    Production IP for VM in sandbox: .88.15

    source:     Dest:          Result:
    .88.20       .188.101      OK       (Veeam server is only server to be able to ping masqrade IP, since it is the only server that knows of the .188.x IP range)
    .88.102     .88.22          OK      (server 102 can ping proxy gw)
    .88.102     .88.15          OK      (server 102 can ping VM in sandbox)
    .88.254     .88.20          OK      (pfSense can ping Veeam server)
    .88.254     .88.22          NEG    (pfSense can NOT ping proxy)
    .88.254    .88.16          NEG    (pfSense can NOT ping VM in sandbox)


  • LAYER 8 Global Moderator

    So from a quick breeze over of the UG for the veem backup 6.5, I don't think your understanding how it works to be honest.

    So these labs or how you call them sandboxes?  Are created on VM HOST, so your saying in your drawing that this esx host is your backup server?

    Or is the LAB(sandbox) on your same ESXi host as your pfsense box?  Or s different host?

    While I agree your never going to get to this isolated VM in your lab/sandbox if you can not talk to the proxy - understanding your environment is a requirement in helping you find your problem.


    44 | Veeam Backup & Replication | VMware Environments | USER GUIDE | REV 5

    To enable communication between the outer world and VMs in the virtual lab, Veeam Backup & Replication uses a proxy appliance that is created and registered in the folder and resource pool of the virtual lab. The proxy appliance is a VM that acts as a gateway routing requests from the production network to the isolated network.
    To connect to isolated networks, Veeam Backup & Replication adds to the proxy appliance a vNIC adapter for each network. Each vNIC adapter gets an IP address from the network to which it is connected, which is typically the same as the IP address of a default gateway in the corresponding production network.

    So from this above statement - and your pic of your vswitch, where was this proxy/router VM connected to the vswitch that your pfsense 88.254 interface connected?  If you proxy VM and sandbox are on a different ESX host, lets see the vswitch setup there where you proxy and VM in your lab is connected.

    so if you have

    esxi host (production) <88.3> --- 192.168.88.0/24 --- <88.20> other esxi host (veem backup with labs and vms)

    Lets see the vm network from this other esxi host with the 88.20 address in your real world physical network.





  • Netgate Administrator

    Whilst this is clearly quite complex it doesn't really explain why a VM in the same host as the pfSense VM is able to reach the server VM in the lab/sandbox whilst pfSense is not.
    However there must be a load of settings that could be wrong or at least not what you thought they were. Why is it not responding to ARP requests from the pfSense box? I would seem that the proxy is either not responding or sending responses to wrong place.
    Perhaps because the pfSense box is at the end of the IP range, .254.  :-\

    Steve


  • LAYER 8 Global Moderator

    "Whilst this is clearly quite complex it doesn't really explain why a VM in the same host as the pfSense VM is able to reach the server VM in the lab/sandbox whilst pfSense is not."

    While I agree with the comment that its odd that pfsense can ping the host that it seems the proxy (I don't like that word in this setup - more of a router) is on.  But not the "proxy" and that the other host connected to the same vm switch as pfsense 88.254 address can

    .88.254    .88.20          OK      (pfSense can ping Veeam server)
    .88.254    .88.22          NEG    (pfSense can NOT ping proxy)

    But without understanding this setup more, its hard to know what the issue is.  Maybe there is a firewall?  Its clearly more complex that just a bunch of boxes connected to same switch in the same segment.

    Maybe this 88.102 box has another interface?  Would love to help the OP figure out his problem, but does not seem like we are getting a full picture of the setup.  And from how the OP is explaining it - its quite possible they don't fully understand it ;)

    Can we get a full picture of the VM networking setup from both his esxi host where pfsense sits and this veem exi host??  Or are these vms and labs on the 1 esxi Host??



  • Hi John,

    you are absoluty correct. Eggs all over my face :) I messed up the Veeam explaination. The isolated eviroment is created on an esxi, not on the Veeam backup server.

    I have migrated the relevant servers to the same esxi host, to give a better overview. I have tryed to take screenshots of the relevant setup.

    Virtal lab 2 is the proxy/router created for the isolated enviroment.

    Hope it helps….. And thank you for your patience.....

    From what I see in the attachments, there is absolutly no reason why 102(server) can ping the virtual Lab 2 at 88.22, and 254(pfsense) can not.

    ![Virtual Lab 2 - setup.PNG](/public/imported_attachments/1/Virtual Lab 2 - setup.PNG)
    ![Virtual Lab 2 - setup.PNG_thumb](/public/imported_attachments/1/Virtual Lab 2 - setup.PNG_thumb)
    ![Virtual Lab 2.PNG](/public/imported_attachments/1/Virtual Lab 2.PNG)
    ![Virtual Lab 2.PNG_thumb](/public/imported_attachments/1/Virtual Lab 2.PNG_thumb)
    ![Virtual Lab 2 - phys nic.PNG](/public/imported_attachments/1/Virtual Lab 2 - phys nic.PNG)
    ![Virtual Lab 2 - phys nic.PNG_thumb](/public/imported_attachments/1/Virtual Lab 2 - phys nic.PNG_thumb)


  • LAYER 8 Global Moderator

    Can we get a full picture please?  I can not make any sense of what your posting because its bits and pieces.

    In your first posting

    You did not show those virtual labs like you do in this time

    So where is the 88.20 IP??  Is that proxy/router vm or is that IP of your veem physical box?

    So why can pfsense not ping the interface of the proxy at 88.22 from 88.254 while interfaces are connected in the same switch?  Well looks like you have a vlan tag of 88 on that switch?  How is pfsense setup with vlans? etc..  Why are you using vlan tags?  I don't see the point of it in this setup??  What IP do you have for the vlan you created on pfsense of 88, what interface did you put that vlan? etc. etc.

    Can we get the network configuration of that VM proxy appliance that shows 2 interfaces one in the vswitch connected to pfsense interface and other in..

    Here is a question for you can pfsense at 88.254 ping 88.102?  Or 102 ping 254?

    So in your first posting you only show 2 virtual machines on that switch, it say 2 right there in your picture.  Now it says 4?  So were they just added??  Also lets say you wanted to use vlan 88, great – you sure would not put vlan 88 tag on both sides your vm proxy??

    From the switches yous how above both having vlan 88 tag, you have 1 leg of your proxy in a vswitch port group with vlan 88 tag, and the other leg where pfsense is and 88.102 box also having a vlan 88 tag??  That vm proxy is routing between networks from my quick breeze through of the UG..  So why would there be vlan 88 on both of its interfaces connected to 2 different network ip ranges?



  • 88.20 is the physical Veeam box.

    How is pfsense setup with vlans?
    I have tryed both with and without Vlan 88 configured on the .88.254 interface (em1) of pfSense. Makes no difference to pinging at all.

    Can we get the network configuration of that VM proxy appliance that shows 2 interfaces one in the vswitch connected to pfsense interface and other in..
    I have attached the config of the proxy. Don't know if it's enough for you. The interface in the proxy config with IP .88.254 is the inside of the isolated network. Maybe that could be the problem?

    Here is a question for you can pfsense at 88.254 ping 88.102?  Or 102 ping 254?
    Yes, i've noted several times that all the machines in the range can ping eachother. Only pfSense can not ping .22(proxy) or .15(VM)

    Now it says 4?  So were they just added??
    I posted just before: "I have migrated the relevant servers to the same esxi host, to give a better overview."  Therefore all the machines I find relevant are in the same picture now.

    So why would there be vlan 88 on both of its interfaces connected to 2 different network ip ranges?
    Hope I understand your long question correct. All these machines are in the .88.x/24 range.

    ![Proxy config.PNG](/public/imported_attachments/1/Proxy config.PNG)
    ![Proxy config.PNG_thumb](/public/imported_attachments/1/Proxy config.PNG_thumb)


  • LAYER 8 Global Moderator

    "config with IP .88.254 is the inside of the isolated network. Maybe that could be the problem?"

    Yah think ;)

    Why would the proxy answer pings to 88.254 when to him that is HIS address.  I am going to look at the UG again, I don't think your setup is correct having the same 88.0/24 on both sides of the proxy VM..  Makes no sense!  And can have all kinds of issues!!!

    So you arp for 88.22, which is basically hey who has 88.22 tell 88.254

    Well to that proxy VM, oh tell 88.254 – sure I will tell myself ;)



  • Very good. We seem to get closer to the target.

    It's a bit of a roundabout with the .88.254 address of the pfSense.
    It is the gateway for the production network, and therefore when you create the isolated enviroment and restore copies of the production network servers in that enviroment, all these isolated servers are still configured with the .88.254 GW. That is why Veeam normally recommends to use the same GW IP in the isolated enviroment, as in the production enviroment.

    But the problem I facing is that I want to access the isolated enviroment using the production GW, which is not possible since the production GW and isolated network GW have the same IP.

    I've seen Veeam mention somewhere that there is a solution for the isolated network to get Internet access, and I would guess that that solution would also solve the problem.
    Maybe by just giving the isolated GW a new available IP in the productionrange, and manually changing the GW IP of the servers in the isolated enviroment to that GW.
    And maybe adding a default route to the proxy pointing to the production GW.

    Going to do some reading. GREAT to finally be making some progresssssss.


  • Netgate Administrator

    Damn! I looked at your screen shot earlier and failed to spot that.  :-[

    Yes, that's certainly your problem. The sandboxed VM cannot possibly respond to any requests from the pfSense machine because it will just send any traffic to the proxy internal interface instead.

    I see why it is done like that, you can bring up a test VM with the production config.
    The answer here is to change the proxy internal interface IP. That will mean you have to manually configure the VM gateway as you say but it will resolve the problem.

    Steve


Log in to reply