Pfsense with LAN adresse that is not set by GUI/Setup

johnpoz · Nov 11, 2012, 11:46 PM

I don't see how you could be pinging across a switch with different ports and 2 different cables and be getting .055 ms

What switch and network cards do you have??? Across a switch your going to see around .5 ms not .05 ms

If I ping myself on my linux box I can see under .100 ms, but anything else on the switch and your look at .400 to .600ms. I just don't buy it that your pinging across a switch and getting .055 ms RTT?

So why can you not show us ifconfig off your pfsense box? Hide any public IPs

–[2.1-BETA0][root@pfsense.local.lan]/root(4): ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:02
inet 192.168.1.253 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::250:56ff:fe00:2%em0 prefixlen 64 scopeid 0x1
inet6 2001:470:xx:xx::1 prefixlen 64
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:01
inet6 fe80::250:56ff:fe00:1%em1 prefixlen 64 scopeid 0x2
inet 24.13.xx.xx netmask 0xfffff800 broadcast 255.255.255.255
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vmx3f0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:0c:29:1e:18:90
media: Ethernet 10Gbase-T (autoselect)
status: no carrier
vmx3f1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:0c:29:1e:18:9a
media: Ethernet 10Gbase-T (autoselect)
status: no carrier
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
pflog0: flags=100 <promisc>metric 0 mtu 33200
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=0<> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
nd6 options=3 <performnud,accept_rtadv>ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
options=80000 <linkstate>inet6 fe80::250:56ff:fe00:2%ovpns1 prefixlen 64 scopeid 0xb
inet 10.0.200.1 –> 10.0.200.2 netmask 0xffffffff
nd6 options=3 <performnud,accept_rtadv>Opened by PID 7348
gif0: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1280
tunnel inet 24.13.xx.xx --> 216.66.77.230
inet6 fe80::250:56ff:fe00:2%gif0 prefixlen 64 scopeid 0xa
inet6 2001:470:xx:xx::2 prefixlen 64
nd6 options=3 <performnud,accept_rtadv>options=1 <accept_rev_ethip_ver>This would for FACT Show you want IPs your box could answer on and what MAC address.</accept_rev_ethip_ver></performnud,accept_rtadv></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

p0ker · Nov 12, 2012, 8:00 AM

As I said, it look strange that I wold get a sub 0.0x respond… I did the same test from another machine and got a more "normal" result.
So to put this machine out of the loop, I turned it off....

If it would be a simpel answer, that the ip and interface would be shown by ifconfig, I would not be asking question on this forum...
But here you go, my complete ifconfig.....

[2.0.1-RELEASE][root@pfsense.local]/root(1): ifconfig
bce0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu   ="" m,vlan_hwtso,linkstate="">ether 00:10:18:b8:db:b0
        inet6 fe80::210:18ff:feb8:dbb0%bce0 prefixlen 64 scopeid 0x1
        inet xxx.xxx.xxx.xxx netmask 0xffffff80 broadcast 255.255.255.255
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu   ="" m,vlan_hwtso,linkstate="">ether 00:10:18:b8:db:b2
        inet 172.16.0.1 netmask 0xffffffe0 broadcast 172.16.0.31
        inet6 fe80::210:18ff:feb8:dbb2%bce1 prefixlen 64 scopeid 0x2
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu   ="" m,vlan_hwtso,linkstate="">ether bc:30:5b:e5:7b:00
        inet 10.0.1.1 netmask 0xffffff80 broadcast 10.0.1.127
        inet6 fe80::be30:5bff:fee5:7b00%bce2 prefixlen 64 scopeid 0x3
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=c01bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu   ="" m,tso4,vlan_hwtso,linkstate="">ether bc:30:5b:e5:7b:01
        media: Ethernet autoselect (none)
        status: no carrier
pflog0: flags=100 <promisc>metric 0 mtu 33664
pfsync0: flags=0<> metric 0 mtu 1460
        syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
        nd6 options=3 <performnud,accept_rtadv>ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::210:18ff:feb8:dbb0%ovpns1 prefixlen 64 scopeid 0x9
        inet 10.10.10.1 --> 10.10.10.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 2136</performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu ></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu ></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu ></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsu ></up,broadcast,running,simplex,multicast>

p0ker · Nov 12, 2012, 9:06 AM

A picture is worth more then 1000 words?

If the text on some of the elements is to small, let me know and I'll tell you what it says…
I cleared my ARP table on the switch, and then did a mapping on everything it can see... (happy that there is not much traffic early in the morning..)
There is only one switch, so where you find more MACs behind one physical port is because they are on a wireless network. (The AP is on port GE19)

johnpoz · Nov 12, 2012, 2:12 PM

So here's the thing - you don't show that MAC, you don't show that IP. In doing a sniff you see the ping come in - but you don't see it go out.

If you DON'T see it go OUT, but you see an answer from the client that sent the ping. How do you think its pfsense answering?

As to .055 ms looking strange?? How about impossible?

Add -e to your tcpdump so we can see mac, you sure you don't have some sort of mirror/span port setup on your switch? As to why pfsense to would see those packets, but clearly it did not answer them. And even if it did - how would it be possible it did in .055 ms, when you ping its normal IP its take .500 ms??

So yeah that is really really odd - so that is a dual port nic, is it possible there is some sort of load balancing/teaming going on where it created a VIP and mac and freebsd just can not show this because of lack of software from broadcom? Is this .25 in your dhcp scope?

p0ker · Nov 12, 2012, 2:35 PM

@johnpoz:

As to .055 ms looking strange?? How about impossible?

Let's forget this one for now (the machine that I got the result from is turned off, and should be out of the loop).
We might take this behavior up on another tread if your up for it… :)

Guess I'm no hard core tcpdumper.. I see that I should have used the -e earlier... but here are the results..


[2.0.1-RELEASE][root@pfsense.local]/root(3): tcpdump -ibce2 -vv -n -e | grep ICMP
tcpdump: listening on bce2, link-type EN10MB (Ethernet), capture size 96 bytes
15:22:23.352700 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 47054, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 45884, seq 32, length 64
15:22:24.353708 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 15857, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 45884, seq 33, length 64
15:22:25.354754 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 12483, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 45884, seq 34, length 64
15:22:26.356435 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65454, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 45884, seq 35, length 64

Regarding span/mirror, I thought of setting up when I got the switch, but never got around to configure it. Checked right now STP was enable but not in use, disables it now, no differance.. othere places I've checked..

Port and VLAN Mirroring - None
Link Aggregation - None
STP Status & Global Settings - Disabled

@johnpoz:

So yeah that is really really odd - so that is a dual port nic, is it possible there is some sort of load balancing/teaming going on where it created a VIP and mac and freebsd just can not show this because of lack of software from broadcom? Is this .25 in your dhcp scope?

I have no idea… .25 is outside of my DHCP scope.. that's all I know.. and I did run some load balancing software (varnish/mod_security ++ other 3rd party extension) but they have all been removed... the only one I have left, is a export for OpenVPN clients..

Open for suggestions...

johnpoz · Nov 12, 2012, 4:37 PM

So can understand why you see the request.. if your switch shows that :02 mac on the port bce2 is connected too. But clearly its not sending out a reply.

But your saying 10.0.1.9 is seeing the response? And on .9 do a tcpdump, it shows the response coming from that :02 mac? Then why did you not see it on your pfsense dump??

Very very strange issue yes - but if pfsense is sending out the bce2 port – shouldn't you see it via the tcpdump??

p0ker · Nov 12, 2012, 5:23 PM

Never though of check the respond, just did… and yes I do get the respond from the :2 MAC...
How'ever I do get a bad chsum on all the request.....

Mac-mini:~ root# tcpdump -ien0 -vv -n -e | grep ICMP
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:17:43.480336 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 34074, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->df6d)!)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 3901, seq 6, length 64
18:17:43.480737 bc:30:5b:e5:7b:02 > c8:2a:14:36:3b:26, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 47612, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.25 > 10.0.1.9: ICMP echo reply, id 3901, seq 6, length 64
18:17:44.481574 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 55579, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->8b6c)!)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 3901, seq 7, length 64
18:17:44.481982 bc:30:5b:e5:7b:02 > c8:2a:14:36:3b:26, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 47613, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.25 > 10.0.1.9: ICMP echo reply, id 3901, seq 7, length 64
18:17:45.482753 c8:2a:14:36:3b:26 > bc:30:5b:e5:7b:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 21348, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->1124)!)
    10.0.1.9 > 10.0.1.25: ICMP echo request, id 3901, seq 8, length 64
18:17:45.483179 bc:30:5b:e5:7b:02 > c8:2a:14:36:3b:26, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 47614, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.25 > 10.0.1.9: ICMP echo reply, id 3901, seq 8, length 64

And yeah. I do agree, my thinking would also say that if the pfsense is answering it should show it on the tcpdump…

johnpoz · Nov 12, 2012, 7:01 PM

very very odd - only thing I can think of is card is putting it on the wire directly from a virtual mac that is outside the OS. Where it is getting the IP is strange as well - unless you had set it on the card at some point with some broadcom software or firmware you can access on the card.

What is the specific model number of the card - is there a way to flush is firmware settings?

wallabybob · Nov 12, 2012, 7:25 PM

@p0ker:

How'ever I do get a bad chsum on all the request…..

If the software is using the hardware to generate IP checksums on transmit then tcpdump won't necessarily see a correct IP checksum on the transmit frames.

johnpoz · Nov 12, 2012, 7:31 PM

talking this over with a couple of guys here, and they have never seen such a thing but agree it must be something on the card itself with load balance virtual mac. But have never seen the cards without OS interaction respond to ping, etc.

My only experience with these cards has been using the suite from broadcom in windows to setup the virtual mac for load balance or failover. But I would guess once this is set it would be in the nvram of the card and not require OS integration if the driver now being used does not have the full feature set, etc. Wouldn't think you could give it a IP though??

Can you access the cards firmware via bios on the card during post? If we can get the exact model number of the card with could lookup the documentation, etc.

p0ker · Nov 12, 2012, 7:40 PM

@johnpoz:

very very odd - only thing I can think of is card is putting it on the wire directly from a virtual mac that is outside the OS. Where it is getting the IP is strange as well - unless you had set it on the card at some point with some broadcom software or firmware you can access on the card.

What is the specific model number of the card - is there a way to flush is firmware settings?

U might be spot on! This is a Dell PowerEdge R200, and if I'm not mistaking it has some kind of "remote management" (not Drac, but BMC? )
As this is a remote location, I can not confirm this for a while, but I'm confident that this must be the case… There is NO other good solution, and this is really plausible...

Kinda feel a bit stupid right now....but a big thanks goes out to all the contributors. :)

chpalmer · Nov 12, 2012, 7:45 PM

Whats the IP of the wirless access point?

p0ker · Nov 12, 2012, 7:47 PM

10.0.1.5, but if you look at my last post I think this issue can be set to resolved… (for now at least..)

chpalmer · Nov 12, 2012, 7:57 PM

Yep- just found my glasses and re-read…

Good Luck!

johnpoz · Nov 12, 2012, 8:25 PM

" it has some kind of "remote management" (not Drac, but BMC? )"

Normally those would be their OWN port on the box though, not part of the normal nic. Remote management would be for outofband access normally and a different port than standard nic, even if built onboard and not a add on drac card, etc.

R200 - will look into what I see about that model.

edit: Yup looks like you can do a shared lan method. That has go to be it! Try telnet to the IP and see what prompt you get.