Can't route public/29 IP block to VMs on lan
-
@viragomann - thanks for the reply and the advice on obscurification (now largely redundant), but I'm not sure I understand:
xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.
My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)
We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.
Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.
Yes, that's correct.
My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)
Well.
We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.
Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.
So it's fine. It wasn't clear from the above obscuring.
Now, as mentioned, first of all you should get the communication within the OPT subnet to work.
Presumed the firewall rules allow it, you should be able to ping the pfSense interface. -
@viragomann - thanks. So from the pfsense box I can ping the DMZ interface (aaa.bbb.206.177)
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.177
PING aaa.bbb.206.177 (aaa.bbb.206.177): 56 data bytes
64 bytes from aaa.bbb.206.177: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from aaa.bbb.206.177: icmp_seq=1 ttl=64 time=0.202 ms
^C
--- aaa.bbb.206.177 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.063/0.133/0.202/0.069 msBut not an IP address in that range assigned to the RPi machine
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
PING aaa.bbb.206.178 (aaa.bbb.206.178): 56 data bytes
^C
--- aaa.bbb.206.178 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet lossI've put 'allow any any rules' in at the top of the firewall for all interfaces WAN/LAN/DMZ - just as a test, but still no connectivity.
From the RPi machine, it cannot ping aaa.bbb.206.177.
Just makes no sense to me. The pfsense instance is running on proxmox and I'm beginning to think there is some issue at the proxmox/virtualisation layer that is blocking the connectivity. I've stopped the firewall that runs within proxmox - but still no joy.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
But not an IP address in that range assigned to the RPi machine
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
Before the pfSense is sending a packet to that IP, it does an ARP lookup. If it gets a proper ARP response it add the IP to its Diagnostic > ARP table.
Check if you see the respective entry there.
If not run packet capture on the interface, the destination machine is connected to (DMZ?) and enter ARP at protocol.
Then start a ping from pfSense. You should see the ARP packets then, at least the ARP request from pfSense.If IP is already in the ARP table sniff the ICMP traffic, while you ping the machine.
-
@viragomann - thanks again. The arp table does not look as I would expect it to. I've reproduced it below
[2.7.2-RELEASE][admin@pfsense.lan]/root: arp -a -n | grep '81\.' ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]Now ping the IP and the ARP entry is atted to the table:
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] ? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 1196 seconds [ethernet]However,
2a:8a:1c:ec:d5:20is not the correct MAC address for the interface on the RPi. I can't find what device corresponds to2a:8a:1c:ec:d5:20.This is the output of tcpdump picking out the arp traffic:
16:52:49.386790 ARP, Request who-has aaa.bbb.206.178 tell aaa.bbb.206.177, length 28 16:52:49.394704 ARP, Reply aaa.bbb.206.178 is-at 2a:8a:1c:ec:d5:20, length 46(An example of proxyarp?)
What is also strange is that if I manually replace the MAC address for aaa.bbb.206.178 with the correct one, I still cannot ping the device. When looking at tcpdump, I see the ICMP echoo request but no reply.
The MAC address in the arp table from the RPi showed as 'incomplete' against the aaa.bbb.206.177 address. Again, if I try and delete that and set the MAC entry on the RPi to the correct MAC of the aaa.bbb.206.177 address, that is also still not ping'able.
Again, thanks for your time.
-
@MrHedgehog 2a:8a:1c is unknown.. normally a pi would have a mac that starts with b8:27:eb, atleast the 2 I have currently running do.
pi@pi-ntp:~ $ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.32 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::8849:df44:1c76:170a prefixlen 64 scopeid 0x20<link> ether b8:27:eb:31:70:ab txqueuelen 1000 (Ethernet)pi@pihole:~ $ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.10 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::3829:d17c:fb3d:1b4d prefixlen 64 scopeid 0x20<link> ether b8:27:eb:38:d8:4d txqueuelen 1000 (Ethernet) -
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi.
Possibly a dynamic MAC of Proxmox?
-
@viragomann how would poxmox have anything to do with a physical pi, if its just a vm - why would he call them pis
Maybe a diagram how this is all connected would help..
Pfsense is virtual right.. and it tied to a physical interface on the host.. which some other device, a raspberry pi that is what I read when he said RPi - but maybe its some other vm?
Been a while since I played with proxmox - but why would it use random macs.. Yeah a drawing would be helpful.. If these are just other vms on the lan side network of pfsense on a virtual switch.. Look at the vm what does it show for its mac? That is what you should be seeing when you arp for it..
For example in virtual machine manager.. I can see and change the mac of the vm..
Are these VMs on a natted network in proxmox? is the pfsense interface on this vm network, attached to the same virtual switch on the vm host?
-
@johnpoz said in Can't route public/29 IP block to VMs on lan:
@viragomann how would poxmox have anything to do with a physical pi,
Something else that isn’t the Pi has the IP, if ARP finds the wrong MAC. It’s a question of what.
@MrHedgehog Have you tried a different IP in your subnet?
-
@viragomann - I am thinking that might be the case and am going to swap out the NVMe card that has the proxmox/pfsense install with another one and do a bare metal install of pfsense on that to strip out the virtualisation layer (this makes it easy to 'rollback' - internet connection being an essential service for my wife!)
-
@SteveITS - I have tried the other IP addresses in the /29 range. But the odd thing is - all the IPs in that range ARP to the same MAC address, eg from
arp -apon the pfsense machine:? (aaa.bbb.206.180) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 115 seconds [ethernet] ? (aaa.bbb.206.182) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 130 seconds [ethernet] ? (aaa.bbb.206.183) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 124 seconds [ethernet] ? (aaa.bbb.206.176) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 127 seconds [ethernet] ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] <- DMZ interface on pfsense ? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 611 seconds [ethernet] ? (aaa.bbb.206.179) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 139 seconds [ethernet]I have not idea (at the moment) what 2a:8a:1c:ec:d5:20 corresponds to.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
strip out the virtualisation layer
great idea!
