[closed] (unsolved) - Why does pfSense not reply on a ICMP echo request

CodeNinja · Jun 8, 2020, 8:58 AM

Hello,

I want to "link" 2 (internal) networks (old an new one) to each other and have some struggles with this. I already asked some questions one multiple forum's but i don't have it working yet. As i gained a lot of new knowledge about networking, i decided to "split" my issue into smaler issues and solve the one after the other. To understand my question a bit better it makes sense to take a look at the diagram bellow:

login-to-view

As you can see its possible to ping the zyxel firewall (192.168.222.1) from pfSense (192.168.222.2) successfully but not other way round. This is the first part i want to get working, maybe this automatically solves my other problems.

I traced the packets with the pfSense packet trace functionality while i try to ping pfSense from the Zyxel firewall.
The ping echo request (from Zyxel) is received by pfSense but it does not reply for some reason.

10:26:08.262126 IP 192.168.222.1 > 192.168.222.2: ICMP echo request, id 27486, seq 1, length 64
10:26:09.276425 IP 192.168.222.1 > 192.168.222.2: ICMP echo request, id 27486, seq 2, length 64
10:26:10.275579 IP 192.168.222.1 > 192.168.222.2: ICMP echo request, id 27486, seq 3, length 64
10:26:13.254841 ARP, Request who-has 192.168.222.2 tell 192.168.222.1, length 46
10:26:13.254863 ARP, Reply 192.168.222.2 is-at c1:9b:43:75:5a:65, length 28
10:26:16.775110 IP6 fe80::1:1 > ff02::1: ICMP6, router advertisement, length 56
10:26:16.932578 IP6 fe80::1:1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
10:26:19.472525 IP6 fe80::1:1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28

An image of the pfSense interface which is connected to the Zyxel firewall:
login-to-view

An image of the firewall rules for that interface (during ping request from Zyxel, you can see activity on the rule)
login-to-view

I suppose that the routes are configured correctly as pinging the zyxel from the "WAN2" port of pfSense works fine but incase someone wants to see the routing configuration:
login-to-view

login-to-view

Any idea how i can solve this issue or at least how i can find out why pfSense does not reply to the ping request?

stephenw10 · Jun 8, 2020, 2:07 PM

That sure looks like a problem with a missing firewall rule.

Make sure you don't have block private networks enabled on WAN2. I doesn't look like you do.

Make sure there are no floating block rules that might apply.

Other things that might block traffic could be; captive portal, Snort/Suricata, IPSec grabbing it.

Steve

CodeNinja · Jun 8, 2020, 2:07 PM

@stephenw10 Thanks for your reply, i appreciate it. I just checked the items/points you mentioned. Here my findings:

Make sure you don't have block private networks enabled on WAN2. I doesn't look like you do.

Private networks are not blocked on WAN2:
login-to-view

Make sure there are no floating block rules that might apply.

I have no floating rules configured at all:
login-to-view

Other things that might block traffic could be; captive portal, Snort/Suricata, IPSec grabbing it.

I have none of these things configured/used (at least not when its not configured by default)

This doesn't mean that i'm not happy with your answer as it confirms that it is on the pfSense side an probably has something to do with the firewall rules. Do you have any other suggestions i can check? or do you need more info? let me know what an i will post it!

Thanks in advance!

guardian · Jun 8, 2020, 5:36 PM

@CodeNinja By default ICMP pings are blocked on an interface. I can't be specific, but here are some ideas for troubleshooting: (I don't know your level of skill or what tools you have available, so I give a few ideas).

Can you ssh into pfsense form PC2?
(Original ideal about pfctl courtesy of @Derelict several years ago-much thanks @Derelict )
If so open a shell session and enter clog -f /var/log/filter.log | grep -i ',block,' and see if you get any output showing igb1 while you ping from the Zyzel. If you do, you will get a number see highlighted that will allow you to find the filter rule that is causing the block. (Partial sample-your numbers will be different):

Jun 8 12:55:42 pfsensenostname filterlog: 398,,,1489363031,interface,match,block,in,4,0x0,,64,5690,0,DF,1,icmp,...

Once you have got that number you can trace it back by opening another shell (or cancelling the log follow with Ctrl-C) and entering pfctl -vvsr | less then type /1489363031 (Your number-this is from the sample) and it will show you the rule. You can post your output if you need help. (You can also view blocked traffic from the GUI, but I find that the shell is easier and faster in many cases).

As an alternative, if this network is currently closed to the outside and you safely open up things without danger of getting hacked while you troubleshoot, just put an allow any rule at the top of each interface to see if you can ping. Most important is under Edit Firewall Rule Protocol should be ANY.

If the network isn't isolated, you could use
Protocol ICMP
and chose
ICMP Subtypes Echo Request and Echo Reply
to just allow ping traffic.

You could also do a promiscuous packet capture on igb1 (pfSense interface from Zyzel) to see if the pings are actually arriving at pfSense..

Hope that helps.

stephenw10 · Jun 8, 2020, 11:13 PM

Check the state table in pfSense whgen you are pinging it from the Zyxcel.
The firewall rules there look like states are being created but check they are on the correct interface and only that interface.
Replies going via a different interface would open states there.

Your gateway is not actually on the WAN2 interface so pfSense will not treat it as a WAN. Traffic coming in will not get reply-to tagged for example which could also allow it to leave via a different route if one more specific than /24 exists.

Steve

CodeNinja · Jun 9, 2020, 5:54 AM

@stephenw10 Nice tool... i did not know about this - state table - function.

I checked it before i start pinging from the Zyxel to pfSense and i noticed that there is already something "wierd" going, at least for me - the network noob - it looks unlogically....

login-to-view

As you can see, pfsense pings 192.168.2.1 from the WAN1 while i do no "manual" ping on any device...

When i start pinging pfSense from the Zyxel, a new entry spawns in the table. It only receives traffic when i understand it correctly..

login-to-view

CodeNinja · Jun 9, 2020, 6:25 AM

@guardian

(I don't know your level of skill or what tools you have available, so I give a few ideas)

I'm actually software engineer and had some networking during my study. I would say i'm far away from an expert but i'm no "newby" anymore. I understand - basic - things like routing protocols, routing, firewall rules and so on. I can install every tool i need to solve this but i'm probably not very familiar with all network tools. I understand the basics of wireshark but never used it for debugging a network. I'm familiar with Linux in general. Maybe i had to mention this in my question already, sorry about that.

Can you ssh into pfsense form PC2?

Yes, i can. I also configure(d) pfSense from this PC

If so open a shell session and enter clog -f /var/log/filter.log | grep -i ',block,' and see if you get any output showing igb1 while you ping from the Zyzel.

This looks like checking Status/System Logs/ Firewall in the interface? I checked this and have no entry for WAN2 or the 192.168.222.0 network here.

When i execute the command you suggest, i see a lot of entries but nothing about the 192.168.222.0 network, with/from/to igb1 or with icmp
This is the only entry since i started pinging from the Zyxel...
login-to-view

just put an allow any rule at the top of each interface to see if you can ping

I already did this before i post my question, just to give it a try. What i noticed during checking these settings (to answer your post correctly) is taht WAN2 (the interface which is connected with the Zyxel) has an "Anti lockout rule" which i can't delete. Maybe this has also something to do with my issue?

(WAN1 = internet/WAN -- WAN2 = connected with Zyxes -- AXN_INT = 10.x LAN -- others are not in use)
login-to-view

You could also do a promiscuous packet capture on igb1 (pfSense interface from Zyzel) to see if the pings are actually arriving at pfSense..

I already did this also.. i posted the result in my question but just to go for sure i did it once again. I receive the ping packets but no response is send. The "new" results of a capture on WAN2 (Zyxel to pfSense):
login-to-view
Capture on WAN1 (internet) to make sure the reply is not send over WAN1
login-to-view
No ICMP packets are send on WAN1, WAN2 or AXN_INT (the internal 10.x network)

Thanks for the extensive reply, unfortunately it did not solve my problem yet.. Any other suggestions or maybe info/details which would help you to understand my problem? Just let me know what you need an i will post it. Thanks in advance!

guardian · Jun 9, 2020, 8:47 AM

@CodeNinja said in Why does pfSense not reply on a ICMP echo request:

When i execute the command you suggest, i see a lot of entries but nothing about the 192.168.222.0 network, with/from/to igb1 or with icmp
This is the only entry since i started pinging from the Zyxel...
login-to-view

I'm assuming that you get one line for each ping?

pfctl -vvsr | less
Then at the : type /1000000103
see if that provides a hit as to what rule is doing the blocking.

just put an allow any rule at the top of each interface to see if you can ping

I already did this before i post my question, just to give it a try. What i noticed during checking these settings (to answer your post correctly) is taht WAN2 (the interface which is connected with the Zyxel) has an "Anti lockout rule" which i can't delete. Maybe this has also something to do with my issue?

(WAN1 = internet/WAN -- WAN2 = connected with Zyxes -- AXN_INT = 10.x LAN -- others are not in use)
login-to-view

Thanks for the extensive reply, unfortunately it did not solve my problem yet.. Any other suggestions or maybe info/details which would help you to understand my problem? Just let me know what you need an i will post it. Thanks in advance!

Just for a quick test, put the following at the top (or as high up as you can get on WAN2)

Action: Pass
Interface: WAN2
Address Family: IPv4
Protocol: ANY

Source:
Source Address: ANY

Destination
Destination Address: ANY

Log: Checked (Optional, but may make testing easier

Make sure to click the Apply Changes button after adding the rule.

CodeNinja · Jun 10, 2020, 6:24 PM

@guardian said in Why does pfSense not reply on a ICMP echo request:

Just for a quick test, put the following at the top (or as high up as you can get on WAN2)
Action: Pass
Interface: WAN2
Address Family: IPv4
Protocol: ANY
Source:
Source Address: ANY
Destination
Destination Address: ANY
Log: Checked (Optional, but may make testing easier
Make sure to click the Apply Changes button after adding the rule.

I think you missed some details in my answer.

I'm assuming that you get one line for each ping?

No, i do only get that line, it is not related to the network i have issues with... (192.168.2.1, instead of 192.168.222.0)

Just for a quick test, put the following at the top (or as high up as you can get on WAN2)

I already did that before posting my question, unfortunately without any result :(

guardian · Jun 10, 2020, 6:24 PM

@CodeNinja It has been a long time since I have had to troubleshoot a difficult problem, so what follows are some ideas that I have incomplete memories about.
When I was doing my initial setup these commands helped me solve many problems. You may need to fill in some details with additional questions or google searches.

As long as you have a backup of your configuration, the worst case situation would be reload the configuration from a backup and reoot. Depending on what systems you have running, you may also wish to disconnect the internet for part/all of your testing if you start messing around with pfctl too much.

Try the following command from the shell:

clog -f /var/log/filter.log

Run your pings and see if you can get some output that indicates which rule is causing the block.

Then get the rule number and search the output of the following commands.

Detailed operating firewall tables:
pfctl -vvsr | less

Read ONLY debug file generated by pfSense that shows the rules loaded by the GUI:
less /tmp/rules.debug

Once you find rule causing the block, if you can't figure it out yourself, you can ask a very specific question posting the relevant output.

If the system is quiet you may find that clearing the counters might also be helpful.

Check out the pfctl man page, there are a lot of diagnostic commands -- maybe something helpful.

As a last resort, if you can afford downtime, you may even be able to make a copy of rules.debug and add some rules for testing -- I would ask someone experienced to verify that this AFAIK either a firewall reboot or doing something on the GUI that causes the rules to reload should wipe out your debug code. I'm sure there is also a pfctl command to do it as well if you are doing a lot of tests it might be worth looking that up.

Hope that helps.

CodeNinja · Jun 12, 2020, 9:43 AM

@guardian Thanks for your time and support. We already have this problem for weeks no so my boss decided to make a "big bang" and just shut off the old network and go to the new one as we run out of time to make the switch. It will be a sh*tstorm but we have 4 days as yesterday was a free day here and today most employees are not in the office and off course we have the saturday and sunday.

Till now it looks not that bad and there is a lot of progress. I wil mark this question as closed.