2 devices in the same network, 1 can ping outside other cannot, same firewall rules..

huud

Hi,

This scenario is configured in VMware Workstation.

I have 1 Windows Server VM, 1 vCenter VM, 5 esxi VMs, pfSense VM.

I have 3 network interfaces on the pfSense

192.168.100.0/24 (vCenter, Windows Server)
192.168.101.0/24 (3 esxi hosts)
192.168.102.0/24 (2 esxi hosts)

I'm facing this issue where the vCenter is unable to ping any esxi host and vice versa. I'm unsure where the issue as once an hour or 2 has passed without using any of the VMs none of the device can then ping.

I have 3 gateways configured for each network, 192.168.100.1, 192.168.101.1, and 192.168.102.1

There are no special rules in the firewall, all LAN traffic is permitted between the 3 interfaces. Once the firewall is restarted again then the ping works. And some times when the same time vCenter cannot ping an esxi host the windows server can ping he same esxi host.

Note that the esxi hosts are not yet added to vCenter so I'm pinging then through the pfSense routing.

In the below images we can see the right CMD window the windows server (IP 192.168.100.2) can ping esxi host 192.168.101.10 but at the same time the vCenter server (IP 192.168.100.35) cannot ping the same esxi host. Both the windows server and the vCenter are in the same network.

For some reason I feel the pfSense is not maintaining the session, any other thoughts, any thoughts on this.

Thanks

bmeeks

Have you checked to be sure both hosts have the identical subnet mask and gateway defined?

I doubt there would be, but have you verified that a host firewall is not the culprit?

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

Have you checked to be sure both hosts have the identical subnet mask and gateway defined?

I doubt there would be, but have you verified that a host firewall is not the culprit?

Hi,

Yes I have confirmed both hosts are in the same subnet, and I'm not able to figure out where the issue as sometimes after a long pause the firewall can't ping its own directly connected interface. This was tested from the firewall console, this also makes it evident that the web GUI won't load at all.

Thanks

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

Have you checked to be sure both hosts have the identical subnet mask and gateway defined?

I doubt there would be, but have you verified that a host firewall is not the culprit?

Hi,

Yes I have confirmed both hosts are in the same subnet, and I'm not able to figure out where the issue as sometimes after a long pause the firewall can't ping its own directly connected interface. This was tested from the firewall console, this also makes it evident that the web GUI won't load at all.

Thanks

There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

huud

There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

Hi,

Yes all interfaces are on the firewall VM, there is no internet connection, all are workstation VMs. the esxi hosts are to be joined to the vCenter server and that is where the issue is. When I ping from the vCenter server to all the esxi hosts the ping works back and forth, when I add them after some time the server cannot find the host.

Here is the network diagram.

I get the following in the vCenter log when it stops pinging, this is when the firewall can't ping its own directly connected interface interface.

2020-05-06T14:28:49.967Z warning vpxd[04990 [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5d6100, h:40, <TCP '127.0.0.1 : 59236'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:49.968Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:49.968Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.028Z warning vpxd[04904] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78f03c2d00, h:42, <TCP '127.0.0.1 : 59244'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.029Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.029Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04832] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5c26c0, h:42, <TCP '127.0.0.1 : 59246'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.068Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04821] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78e831d150, h:42, <TCP '127.0.0.1 : 59248'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.102Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.]

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

Hi,

Yes all interfaces are on the firewall VM, there is no internet connection, all are workstation VMs. the esxi hosts are to be joined to the vCenter server and that is where the issue is. When I ping from the vCenter server to all the esxi hosts the ping works back and forth, when I add them after some time the server cannot find the host.

Here is the network diagram.

I get the following in the vCenter log when it stops pinging, this is when the firewall can't ping its own directly connected interface interface.
2020-05-06T14:28:49.967Z warning vpxd[04990 [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5d6100, h:40, <TCP '127.0.0.1 : 59236'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:49.968Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:49.968Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.028Z warning vpxd[04904] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78f03c2d00, h:42, <TCP '127.0.0.1 : 59244'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.029Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.029Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04832] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5c26c0, h:42, <TCP '127.0.0.1 : 59246'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.068Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04821] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78e831d150, h:42, <TCP '127.0.0.1 : 59248'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.102Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.]

Um---where is pfSense in this diagram? This is a pfSense forum, not Sophos. I assumed you were running pfSense when I replied.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

Hi,

Yes all interfaces are on the firewall VM, there is no internet connection, all are workstation VMs. the esxi hosts are to be joined to the vCenter server and that is where the issue is. When I ping from the vCenter server to all the esxi hosts the ping works back and forth, when I add them after some time the server cannot find the host.

Here is the network diagram.

I get the following in the vCenter log when it stops pinging, this is when the firewall can't ping its own directly connected interface interface.
2020-05-06T14:28:49.967Z warning vpxd[04990 [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5d6100, h:40, <TCP '127.0.0.1 : 59236'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:49.968Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:49.968Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.028Z warning vpxd[04904] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78f03c2d00, h:42, <TCP '127.0.0.1 : 59244'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.029Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.029Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04832] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5c26c0, h:42, <TCP '127.0.0.1 : 59246'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.068Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04821] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78e831d150, h:42, <TCP '127.0.0.1 : 59248'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.102Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.]
Um---where is pfSense in this diagram? This is a pfSense forum, not Sophos. I assumed you were running pfSense when I replied.

pfSense is being used a the moment, the diagram is from I was exploring the firewall to use. Was going through SophosXG, SonicWall and some others.

Thanks..

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

Hi,

Yes all interfaces are on the firewall VM, there is no internet connection, all are workstation VMs. the esxi hosts are to be joined to the vCenter server and that is where the issue is. When I ping from the vCenter server to all the esxi hosts the ping works back and forth, when I add them after some time the server cannot find the host.

Here is the network diagram.

I get the following in the vCenter log when it stops pinging, this is when the firewall can't ping its own directly connected interface interface.
2020-05-06T14:28:49.967Z warning vpxd[04990 [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5d6100, h:40, <TCP '127.0.0.1 : 59236'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:49.968Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:49.968Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.028Z warning vpxd[04904] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78f03c2d00, h:42, <TCP '127.0.0.1 : 59244'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.029Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.029Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04832] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5c26c0, h:42, <TCP '127.0.0.1 : 59246'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.068Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04821] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78e831d150, h:42, <TCP '127.0.0.1 : 59248'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.102Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.]
Um---where is pfSense in this diagram? This is a pfSense forum, not Sophos. I assumed you were running pfSense when I replied.
pfSense is being used a the moment, the diagram is from I was exploring the firewall to use. Was going through SophosXG, SonicWall and some others.

Thanks..

So the circle that says "Sophos XG" is actually a pfSense firewall? If so, then I would suspect the pf and pfctl issue I mentioned. To see if that is the case, check if you have "block bogons" enabled on any of your interfaces.

Does this interruption in traffic flow occur at a regular interval? The bug I mentioned happens every 15 minutes when the bogons tables are updated. The bug causes the firewall to appear to freeze for long periods.

One other note, the pfSense Dashboard page will be super sluggish and laggy when the firewall does not have Internet connectivity. This is because that page tries to contact the pfSense home servers to check for available updates. pfSense does not like having no Internet access. It can make things really slow.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:
There is a current problem with pfSense-2.4.5 related to a change made in FreeBSD-11.3-STABLE in the pf firewall engine. That problem manifests itself as lags and network stalls whenever large firewall tables are being manipulated. The IPv6 bogons table is one that will trigger this when it is enabled. pfBlockerNG and/or DNSBL can also trigger the issue when they have large tables of IP addresses. This issue is particularly evident in virtualized pfSense instances, but can be mitigated somewhat by cutting down the pfSense virtual machine to a single CPU core. Don't know if you are hitting that issue or not.

A diagram of how you have things configured would be helpful. For example, exactly where does the firewall sit in the scenario you outlined in your original post? Are all those listed networks interfaces off the firewall VM?

You virtual switch configuration is also important. Are you sure it is 100% correct?

Hi,

Yes all interfaces are on the firewall VM, there is no internet connection, all are workstation VMs. the esxi hosts are to be joined to the vCenter server and that is where the issue is. When I ping from the vCenter server to all the esxi hosts the ping works back and forth, when I add them after some time the server cannot find the host.

Here is the network diagram.

I get the following in the vCenter log when it stops pinging, this is when the firewall can't ping its own directly connected interface interface.
2020-05-06T14:28:49.967Z warning vpxd[04990 [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5d6100, h:40, <TCP '127.0.0.1 : 59236'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:49.968Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:49.968Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.028Z warning vpxd[04904] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78f03c2d00, h:42, <TCP '127.0.0.1 : 59244'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.029Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.029Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04832] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f791c5c26c0, h:42, <TCP '127.0.0.1 : 59246'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.068Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.068Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04821] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f78e831d150, h:42, <TCP '127.0.0.1 : 59248'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-05-06T14:28:50.102Z error vpxd[04851] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-05-06T14:28:50.102Z warning vpxd[04851] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.]
Um---where is pfSense in this diagram? This is a pfSense forum, not Sophos. I assumed you were running pfSense when I replied.
pfSense is being used a the moment, the diagram is from I was exploring the firewall to use. Was going through SophosXG, SonicWall and some others.

Thanks..
So the circle that says "Sophos XG" is actually a pfSense firewall? If so, then I would suspect the pf and pfctl issue I mentioned. To see if that is the case, check if you have "block bogons" enabled on any of your interfaces.

Does this interruption in traffic flow occur at a regular interval? The bug I mentioned happens every 15 minutes when the bogons tables are updated.

Yes the Sophos XG just the text, pfSense paused can be seen below..

There is no regular interval, it happens in 2 scenarios.

When he VMs are paused (as seen in the image above) for may a few hours, between 1-3.
When the VMs are running but not used for may be between 1-3.

I have no interface that connects to the internet, except initially when I configured the device the 1st time. In my case there is no connectivity between the device and its directly connected interfaces, the Web GUI, nor any device or its gateway interface on the firewall.

I will check the bogons table settings..

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

In my case there is no connectivity between the device and its directly connected interfaces, the Web GUI, nor any device or its gateway interface on the firewall.

This statement is confusing to me. If there is no connectivity between the devices how do you expect them to work? I'm obviously missing something from your description.

I think you also said you were running all of this on VMware Workstation. Running a hypervisor on top of a hypervisor (ESXi on top of VMware Workstation) is a little strange. That will set up with the virtual switches of ESXi having to cooperate with the virtual networks of VMware Workstation and Windows.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

In my case there is no connectivity between the device and its directly connected interfaces, the Web GUI, nor any device or its gateway interface on the firewall.

This statement is confusing to me. If there is no connectivity between the devices how do you expect them to work? I'm obviously missing something from your description.

I'm talking about when the problem occurs, then there is no connectivity between the firewall and its directly connected interface, otherwise it works fine..

The Bogons network is set for Monthly but I could not find anything relating to IPv6 in Bogons settings

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

In my case there is no connectivity between the device and its directly connected interfaces, the Web GUI, nor any device or its gateway interface on the firewall.

This statement is confusing to me. If there is no connectivity between the devices how do you expect them to work? I'm obviously missing something from your description.

I think you also said you were running all of this on VMware Workstation. Running a hypervisor on top of a hypervisor (ESXi on top of VMware Workstation) is a little strange. That will set up with the virtual switches of ESXi having to cooperate with the virtual networks of VMware Workstation and Windows.

I did that to reduce the amount of resource needed to run vCenter inside ESXi which is being run on Workstation.

I understand but it works fine when all VMs are in the same network and no firewall needed.

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

In my case there is no connectivity between the device and its directly connected interfaces, the Web GUI, nor any device or its gateway interface on the firewall.

This statement is confusing to me. If there is no connectivity between the devices how do you expect them to work? I'm obviously missing something from your description.

I'm talking about when the problem occurs, then there is no connectivity between the firewall and its directly connected interface, otherwise it works fine..

The Bogons network is set for Monthly but I could not find anything relating to IPv6 in Bogons settings

The blocking is configured on a per-interface basis. So open INTERFACES and the look at each configured firewall interface to see if "Block Bogons" is enabled.

If you have the bug, the interval will be predictable.

I will say that sometime back I frequently had problems with a Linux VM inside of VMware Workstation on a Windows 10 host. I routinely left the VM running and VMware Workstation running, but after some period of time (say a day or even a few hours), when I opened up the VMware window and accessed the GUI of the Linux VM I would find it (the Linux VM) had zero network connectivity. I was using "bridged mode". The only way I could "fix" the VM without shutting down and restarting VMware Workstation was to call up the Virtual Network Editor in VMware Workstation and essentially delete and then recreate the virtual network adaptors. Then everything was fine.

That was a while back on a slightly older VMware Workstation. Maybe you are hitting that kind of problem? You may have something going whacky with your virtual network adapters on the host.

huud

The blocking is configured on a per-interface basis. So open INTERFACES and the look at each configured firewall interface to see if "Block Bogons" is enabled.

If you have the bug, the interval will be predictable.

The Block Bogons checkbox is unchecked for all the 3 interfaces, except for WAN (the interface is disabled but the bock bogons checkbox is checked).

I will say that sometime back I frequently had problems with a Linux VM inside of VMware Workstation on a Windows 10 host. I routinely left the VM running and VMware Workstation running, but after some period of time (say a day or even a few hours), when I opened up the VMware window and accessed the GUI of the Linux VM I would find it (the Linux VM) had zero network connectivity. I was using "bridged mode". The only way I could "fix" the VM without shutting down and restarting VMware Workstation was to call up the Virtual Network Editor in VMware Workstation and essentially delete and then recreate the virtual network adaptors. Then everything was fine.

That was a while back on a slightly older VMware Workstation. Maybe you are hitting that kind of problem?

Could be, just that I never faced the issue as I never implemented a firewall in Workstation, I could observe that for a while to see if that is the issue.

bmeeks

See what impact unchecking that bogons block on the WAN has. I'm thinking, though, that if you are experiencing the problem on an interval different than 15 minutes that the bug is not what is causing the problem.

Look in the pfSense system log to see what is getting logged around the time you lose the connectivity. There may be some clues there.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

See what impact unchecking that bogons block on the WAN has. I'm thinking, though, that if you are experiencing the problem on an interval different than 15 minutes that the bug is not what is causing the problem.

Look in the pfSense system log to see what is getting logged around the time you lose the connectivity. There may be some clues there.

Thanks, I will observe that.

As for logs, are they accessible in /var/logs.. ?

bmeeks

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

See what impact unchecking that bogons block on the WAN has. I'm thinking, though, that if you are experiencing the problem on an interval different than 15 minutes that the bug is not what is causing the problem.

Look in the pfSense system log to see what is getting logged around the time you lose the connectivity. There may be some clues there.

Thanks, I will observe that.

As for logs, are they accessible in /var/logs.. ?

In the pfSense GUI under STATUS > SYSTEM MESSAGES.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@huud said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

See what impact unchecking that bogons block on the WAN has. I'm thinking, though, that if you are experiencing the problem on an interval different than 15 minutes that the bug is not what is causing the problem.

Look in the pfSense system log to see what is getting logged around the time you lose the connectivity. There may be some clues there.

Thanks, I will observe that.

As for logs, are they accessible in /var/logs.. ?

In the pfSense GUI under STATUS > SYSTEM MESSAGES.

Thanks..

bmeeks

pfSense uses clog, a binary circular logging engine. So while the files are in /var/log, they are in a binary format. You need to either view them within the pfSense GUI or use the command-line clog utility to dump the logs.

Also note that the circular nature of the logging means you may need to go into the log settings and increase the number of displayed entries. The default is usually a bit small. While on the logs tab in the GUI, look for the Settings tab. That's where you can configure the logging engine.

huud

@bmeeks said in 2 devices in the same network, 1 can ping outside other cannot, same firewall rules..:

pfSense uses clog, a binary circular logging engine. So while the files are in /var/log, they are in a binary format. You need to either view them within the pfSense GUI or use the command-line clog utility to dump the logs.

Also note that the circular nature of the logging means you may need to go into the log settings and increase the number of displayed entries. The default is usually a bit small. While on the logs tab in the GUI, look for the Settings tab. That's where you can configure the logging engine.

Appreciate clarifying that..