Strange arp problem
-
My ISP (Bredbandsbolaget) gives me 5 IPs with 10 Mbit/s upload per interface and a total of 100 Mbit/s downstream. In my pfsense machine i have a quad 100/10 card that gets 4 different IPs via dhcp from my ISP.
Sometimes i get strange connection problems and when i check in the logs I see these two messages repeating every second:
[repeat]
Nov 15 17:46:04 kernel: arp: 00:80:c8:ca:6e:09 is using my IP address 213.113.29.59!
Nov 15 17:46:04 kernel: arp: 213.113.29.33 is on dc1 but got reply from 00:00:0c:07:ac:01 on dc0
[/repeat]Here is some facts about my current configuration:
213.113.29.33 (00:00:0c:07:ac:01) is the gateway at my ISP.
dc0 (00:80:c8:ca:6e:09) is WAN1 213.113.29.43
dc1 (00:80:c8:ca:6e:0a) is WAN2 213.113.29.59
dc2 (00:80:c8:ca:6e:0b) is WAN3 213.113.29.51
dc3 (00:80:c8:ca:6e:0c) is WAN4 213.113.29.48When i'm writing this I'm using 1.2-RC3 built on Sat Oct 27 19:37:36 EDT 2007. I upgraded last time I had this problem but it didn't help.
When it's time for the interfaces to request some IPs. after dc0 gets it's IP it goes back to normal:
Nov 15 17:49:03 dhclient[49075]: DHCPREQUEST on dc0 to 255.255.255.255 port 67
Nov 15 17:49:03 dhclient[49075]: DHCPACK from 213.113.29.34
Nov 15 17:49:03 dhclient[49075]: bound to 213.113.29.43 – renewal in 1800 seconds.Then there is some DHCPREQUESTs on dc0, dc1 and dc2 to 172.21.249.78:
[repeat]
Nov 15 17:56:31 dhclient[459]: DHCPREQUEST on dc1 to 172.21.249.78 port 67
Nov 15 17:56:48 dhclient[538]: DHCPREQUEST on dc2 to 172.21.249.78 port 67
Nov 15 17:57:03 dhclient[617]: DHCPREQUEST on dc3 to 172.21.249.78 port 67
[/repeat]After a while dhclient tries to broadcast the DHCPREQUEST to 255.255.255.255 and it works:
Nov 15 17:59:23 dhclient[459]: DHCPREQUEST on dc1 to 255.255.255.255 port 67
Nov 15 17:59:23 dhclient[459]: DHCPACK from 213.113.29.34
Nov 15 17:59:23 dhclient[459]: bound to 213.113.29.59 – renewal in 1800 seconds.Nov 15 18:01:20 dhclient[617]: DHCPREQUEST on dc3 to 255.255.255.255 port 67
Nov 15 18:01:20 dhclient[617]: DHCPACK from 213.113.29.34
Nov 15 18:01:20 dhclient[617]: bound to 213.113.29.48 – renewal in 1800 seconds.Nov 15 18:02:50 dhclient[538]: DHCPREQUEST on dc2 to 255.255.255.255 port 67
Nov 15 18:02:50 dhclient[538]: DHCPACK from 213.113.29.34
Nov 15 18:02:51 dhclient[538]: bound to 213.113.29.51 – renewal in 1800 seconds.And after a while hell breaks loose again:
Nov 15 18:06:01 kernel: arp: 00:80:c8:ca:6e:09 is using my IP address 213.113.29.59!
Nov 15 18:06:01 kernel: arp: 213.113.29.33 is on dc1 but got reply from 00:00:0c:07:ac:01 on dc0A reboot usually solves the problem but after some weeks I get the same problem again. The problem can come after some weeks or after some hours of uptime.
Anybody that knows why it's doing this?
/Thomas Svedin
Update:
I just updated to RC3. I wonder if it will work because of the upgrade or the reboot. -
I'm starring at the logs and suddenly I see that the machine gets a new IP for dc0 (by sending the DHCPREQUEST to 255.255.255.255) but after that dc1 dc2 and dc3 starts to send their requests to 172.21.249.78 and does so the next 10 minutes. Then it gives up and tries 255.255.255.255 that works as usual.
No strange ARP messages this time but i think I'm on to something.
EDIT: Looks like I'm right… i think.
I opened the configuration file and what do i see?<webgui>172.21.249.78 <protocol>http</protocol> <certificate><private-key><port>81</port></private-key></certificate></webgui>
That's the same IP that the dhclient is sending it's requests to! I'm going to remove it and reboot the firewall now.
-
It still asks 172.21.249.78!
It looks like that arp message is a big part of the problem.
Another thing i noticed is that traffic to 172.21.249.78 is blocked by some rule that i can't find.
-
00:00:0c is a cisco device. It seems like your ISP is using an cisco router, and maybe 172 as an internal managment net. It may be some kind of proxy arp, who can make some strange things sometimes (in the cisco router) I will think that the bredbåndsbolaget has a missconfigurated router or something (but I may be wrong)
-
I dissabled WAN2, WAN3 and WAN4 and it has worked over night. It still fails when asking 172.21.249.78 for new IPs. After 10 minutes it broadcasts it's request and then it gets an answer.
Nov 16 07:46:46 dhclient[29408]: DHCPREQUEST on dc0 to 172.21.249.78 port 67 Nov 16 07:47:20 dhclient[29408]: DHCPREQUEST on dc0 to 172.21.249.78 port 67 Nov 16 07:49:08 last message repeated 4 times Nov 16 07:52:40 last message repeated 5 times Nov 16 07:54:03 dhclient[29408]: DHCPREQUEST on dc0 to 255.255.255.255 port 67 Nov 16 07:54:03 dhclient[29408]: DHCPACK from 213.113.29.34 Nov 16 07:54:03 dhclient[29408]: bound to 213.113.29.43 -- renewal in 1800 seconds.
I know my ISP is using proxy arp. Is is possible to force dhclient to ask 255.255.255.255 instead of something else?
-
I have seen this in my logs to. (Bredbandsbolaget)
Nov 16 23:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 22:58:35 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 22:58:34 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 22:58:34 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 22:55:53 last message repeated 9 times
Nov 16 22:51:02 last message repeated 4 times
Nov 16 22:48:50 last message repeated 3 times
Nov 16 22:48:25 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 22:18:25 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 22:18:25 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 22:18:25 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 22:10:49 last message repeated 9 times
Nov 16 22:04:29 last message repeated 7 times
Nov 16 22:02:31 last message repeated 4 times
Nov 16 22:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 22:02:03 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 21:32:04 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 21:32:03 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 21:32:03 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 21:30:59 last message repeated 20 times
Nov 16 21:23:42 last message repeated 7 times
Nov 16 21:21:53 last message repeated 3 times
Nov 16 21:21:18 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 21:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 21:00:20 dnsmasq[31200]: reading /var/dhcpd/var/db/dhcpd.leases
Nov 16 20:51:18 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 20:51:18 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 20:51:18 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 20:46:20 last message repeated 3 times
Nov 16 20:41:40 last message repeated 5 times
Nov 16 20:39:57 last message repeated 3 times
Nov 16 20:39:39 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 20:09:39 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 20:09:39 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 20:09:39 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 20:08:20 last message repeated 5 times
Nov 16 20:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 20:02:06 last message repeated 9 times
Nov 16 20:00:07 last message repeated 4 times
Nov 16 19:59:25 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 19:35:01 check_reload_status: check_reload_status is starting
Nov 16 19:29:25 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 19:29:25 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 19:29:25 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 19:26:49 last message repeated 2 times
Nov 16 19:21:11 last message repeated 3 times
Nov 16 19:19:16 last message repeated 3 times
Nov 16 19:18:48 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 19:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 19:01:47 dnsmasq[31200]: reading /var/dhcpd/var/db/dhcpd.leases
Nov 16 18:48:48 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 18:48:48 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 18:48:48 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 18:47:08 last message repeated 17 times
Nov 16 18:40:00 last message repeated 6 times
Nov 16 18:37:48 last message repeated 4 times
Nov 16 18:37:22 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 18:07:22 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 18:07:22 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 18:07:22 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 18:04:35 last message repeated 8 times
Nov 16 18:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 17:58:31 last message repeated 7 times
Nov 16 17:56:34 last message repeated 3 times
Nov 16 17:56:04 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 17:26:04 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 17:26:04 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 17:26:04 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 17:24:36 last message repeated 11 times
Nov 16 17:17:11 last message repeated 3 times
Nov 16 17:14:54 last message repeated 2 times
Nov 16 17:14:36 dhclient[31166]: DHCPREQUEST on ste0 to 172.21.248.97 port 67
Nov 16 17:02:15 dhcpd: uid lease 192.168.0.199 for client 00:02:e3:12:7a:10 is duplicate on 192.168.0/24
Nov 16 16:44:36 dhclient[31166]: bound to 85.228.247.245 – renewal in 1800 seconds.
Nov 16 16:44:36 dhclient[31166]: DHCPACK from 85.228.240.1
Nov 16 16:44:36 dhclient[31166]: DHCPREQUEST on ste0 to 255.255.255.255 port 67
Nov 16 16:43:48 last message repeated 8 times
Nov 16 16:36:02 last message repeated 3 times
Nov 16 16:34:30 last message repeated 2 times -
I'm having the same problem, using 5 dynamic ip:s within the same broadcast domain from bredbandsbolaget on five 3com nic's. Getting disconnects and the damn log is spammed with dhclient trying DHCPREQUEST on 172.21.249.X and kernel reporting x is using my IP address x…
Anyone have a clue on what to do?
-
Just a thought: do you get the same IPs every time? Or does your ISP give the IP based on your hardware's MAC adresses? If so, you could disable DHCP and just try static IP instead, giving each interface the IP it normally gets via DHCP. I did so on our cable provider 'cause he had some strange issues with a 192.168-ish management network between his DHCP servers and our gateway.
-
Yeah, i've tried that, but as soon as the lease time goes out, i loose the connection.