Ping: sendto: Invalid argument && no arp with cisco 3750

zanon · Aug 5, 2013, 6:13 PM

Hi all,
During my experiments with pfsense 2.1 (2.1-RC1 (amd64) built on Thu Aug 1 19:39:40 EDT 2013) I have hit something that I rally could not understand.
start with the facts:
I am installing a new appliance with 4 NIC, having 3 BGP sessions (2 on em0, 1 on em1) , on em2 I have my public network ,
and em3 i am planning to use for carp and some internal IPMI.

I would like to concentrate on my public network. There I have few linuxes , 1 free testing bare esxi 5.1 with few linux playground machines .
In general everything works fine except the cisco stack. All Linux machines can see each other, have internet, can ping pfsense and can ping and connect to cisco stack.
cisco stack can ping and connect to any Linux machine but unable to communicate with pfsense.
pfsense can see and connect to any Linux machine but totally unaware about cisco stack.

when I ping pfsense –> cisco I do get error
PING 94.156.103.3 (94.156.103.3): 56 data bytes
ping: sendto: Invalid argument
ping: sendto: Invalid argument

tcpdump on em2 ..... silence

when i ping cisco --> pfsense i dont get anything on cisco

tcpdump on em2 looks like :
20:52:27.423526 IP (tos 0x0, ttl 255, id 50, offset 0, flags [none], proto ICMP (1), length 100)
94.156.103.3 > 94.156.103.1: ICMP echo request, id 10, seq 0, length 80
20:52:29.428863 IP (tos 0x0, ttl 255, id 51, offset 0, flags [none], proto ICMP (1), length 100)
94.156.103.3 > 94.156.103.1: ICMP echo request, id 10, seq 1, length 80

when i ping subnet from pfsens i do get response
64 bytes from 94.156.103.3: icmp_seq=0 ttl=255 time=3.276 ms
64 bytes from 94.156.103.3: icmp_seq=1 ttl=255 time=8.574 ms
and tcpdump looks like :
20:54:01.037300 IP (tos 0x0, ttl 255, id 25419, offset 0, flags [none], proto ICMP (1), length 84)
94.156.103.3 > 94.156.103.1: ICMP echo reply, id 25377, seq 3, length 64
20:54:18.552951 IP (tos 0xc0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 76)
94.156.103.3.123 > 94.156.103.1.123: [udp sum ok] NTPv3, length 48

I do installed arping package and have done few arpping 94.156.103.3
60 bytes from 7c:ad:74:4a:24:40 (94.156.103.3): index=0 time=3.200 msec
60 bytes from 7c:ad:74:4a:24:40 (94.156.103.3): index=1 time=3.996 msec
tcpdump looks like
20:55:58.440672 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 94.156.103.3 tell 94.156.103.1, length 28
20:55:58.442825 ARP, Ethernet (len 6), IPv4 (len 4), Reply 94.156.103.3 is-at 7c:ad:74:4a:24:40, length 46
20:55:59.440864 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 94.156.103.3 tell 94.156.103.1, length 28
20:55:59.443207 ARP, Ethernet (len 6), IPv4 (len 4), Reply 94.156.103.3 is-at 7c:ad:74:4a:24:40, length 46

, but there is no record in arp -a for cisco stack, only for other machines

i have checked everything . cisco stack is just installed no blocking or acl, no vlan , PF is taking care for access.
i tried with pfctl -d and re enable again …. i have moved cisco stack around IP ... nada

tbh I am starting to think that either i am crazy or i am doing something terrifically stupid.

wallabybob · Aug 5, 2013, 9:21 PM

@zanon:

when i ping cisco –> pfsense i dont get anything on cisco

tcpdump on em2 looks like :
20:52:27.423526 IP (tos 0x0, ttl 255, id 50, offset 0, flags [none], proto ICMP (1), length 100)
94.156.103.3 > 94.156.103.1: ICMP echo request, id 10, seq 0, length 80
20:52:29.428863 IP (tos 0x0, ttl 255, id 51, offset 0, flags [none], proto ICMP (1), length 100)
94.156.103.3 > 94.156.103.1: ICMP echo request, id 10, seq 1, length 80

PERHAPS you don't have a firewall rule on em2 to allow pings.You might see the ping attempts in the pfSense firewall logs (see Status -> System Logs, click on Firewall tab). The default pfSense configuration is to block incoming connections on all interfaces except LAN.

I have no experience with Cisco 3750. PERHAPS the Cisco is not configured to allow incoming "connections" on the interface you are accessing it on and consequently doesn't reply to ARP requests. The response you saw to arping MIGHT be because the previous ping attempt on the Cisco opened a temporary hole in the Cisco firewall (to allow ping responses) and that also allowed it to respond to ARPs.

zanon · Aug 5, 2013, 9:47 PM

Thanks for answer , but …
Well , as i said this is not working even if i stop pf at all .
as long as i can see the ping replays on multicasting are getting back (subnet scenario) ands when i ping from cisco scenario 2 i do get packets.
i just cant see outgoing packets from pfsense --> cisco
i cant see arp record in pfsense either. even after i do some arpings with result , after that there is no way to communicate from pfsense toward cisco stack. :(

thanks for the try tho :)

zanon · Aug 5, 2013, 10:47 PM

YAY :) :) :) :) found it !!!
tho is a bit misleading !!!
on em2 interface (dmz) i do have dhcp server .
turning on option

Enable Static ARP entries
is causing this behavior.
To Be Honest i do not understand why i am unable to communicate with my cisco stack even if i do have DHCP record with option set on* Create an ARP Table Static Entry for this MAC & IP Address pair.
but all my linux machines could do so …. maybe its a glitch or small bug .. nvm
I am happy :) :)

Have fun

mikeisfly · Aug 5, 2013, 11:36 PM

another thing to watch for when working with Cisco is to make sure that your trunk is using dot1q encapsulation. I believe on the 3750 it will default to ISL. use the command: switchport trunk encapsulation dot1q

Issue this command from the interface.