Can't route public/29 IP block to VMs on lan
-
@viragomann said in Can't route public/29 IP block to VMs on lan:
Many thanks for your reply.
I have setup the OPT1 interface with the IP of 81.187.2xx.1y7 and assigned 81.187.2xx.1y8 and 81.187.2xx.1y9 to the interfaces of the VMs with their default gateway set to 81.187.2xx.1y7 (this has worked on a different model of router). the OPT1 interface uses the vtnet0 port which is also given over to the WAN interface
Why, since the devices in the OPT1 subnet are VMs anyway. So you could assign an additional virtual interface for this subnet.
So not all of the devices in the OPT1 subnet are VMs but they were the easiest test ones to create. The VMs are on a prommox instance that is separate to the pfsense router. There is a single dumb switch that hangs off the LAN interface with the proxmox server attached. I'm not familiar enough with pfsense to following what you mean by "you could assign an additional virtual interface for this subnet." Do you mean at the proxmox level or the pfsense level?
I'm surprised I can't find a guide on setting up a similar arrangment. How would you go about setting up this arrangement?
That setup does not seem to work, I cannot access an HTTPS webpage on 81.187.2xx.1y8 from outside of the network.
Does the machine even allow access from outside of it's subnet? By default this is blocked.
I believe I have added firewall rules to allow access, if that is what you mean?
More fundamentally, when logged into the console of a VM, I am unable to ping it's default route, ie the 81.187.2xx.1y7 address
Your firewall rule seems to allow only echo replies.
You have to allow echo requests.
I have changed that - thanks for spotting it.
Thanks again for the reply.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
So not all of the devices in the OPT1 subnet are VMs
There is a single dumb switch that hangs off the LAN interface with the proxmox server attached.
Do you mean at the proxmox level or the pfsense level?Yes. But if your Proxmox hardware has only two NICs and you don't have a VLAN-capable switch your options on this are limited.
Anyway, basically it should also work with the shared NIC with PPPoE.
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
Does the machine even allow access from outside of it's subnet? By default this is blocked.
I believe I have added firewall rules to allow access, if that is what you mean?
I mean, the firewall on the destination device.
However, at least pings between pfSense and the devices on OPT1 should work. -
@MrHedgehog Is the /29 being routed to your WAN IP? If so Netgate has a recipe for this:
https://docs.netgate.com/pfsense/en/latest/recipes/route-public-ip-addresses.html -
@SteveITS - Hi there,
Thanks for the reply and yes it is being routed (eg I can see in the firewall logs blocks for randon scanners on my IP range). I've followed that document but I just can't get it to work. I'm going to pick through it again this evening with a physical host to test with.
-
@MrHedgehog Check the OPT1 interface has a mask of /29 not the default /32.
Agree w/ above, start with Diagnostics/Ping on pfSense to ensure pfSense can ping the web server VM, and go up from there. If pinging from OPT1 to the VM fails it's not a firewall problem it's a network problem or the firewall on the VM.
-
@SteveITS Hi there and thanks for reply. I do appreciate the time taken to read and comment.
I have checked the netmask on the DMZ interface and confimed that it is a /29. I've reproduced the output of ifconfig because as you suggest there seems to be some fundamental routing/network issue that I am missing. I have plugged a Raspberry Pi into the switch on the LAN and IP'd it with 81.187.2x6.1y8 netmast 255.255.255.248. This cannot ping it's default route, set to 81.187.2x6.1y7 (ie the DMZ interface). I've checked and there is no firewall enabled on the Pi.
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: DMZ options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE> ether bc:24:11:54:4e:82 inet 81.187.2x6.1y7 netmask 0xfffffff8 broadcast 81.187.2x6.1y3 inet6 fe80::be24:11ff:fe54:4e82%vtnet0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: LAN options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE> ether bc:24:11:ab:31:4e inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::be24:11ff:feab:314e%vtnet1 prefixlen 64 scopeid 0x2 inet6 2001:8b0:4b4:24c::1 prefixlen 64 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> enc0: flags=0 metric 0 mtu 1536 options=0 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0x0 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=100<PROMISC> metric 0 mtu 33152 options=0 groups: pflog pfsync0: flags=0 metric 0 mtu 1500 options=0 maxupd: 128 defer: off version: 1400 syncok: 1 groups: pfsync pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492 description: WAN options=0 inet 217.169.1xx.1yy --> 81.187.8x.1y7 netmask 0xffffffff inet6 fe80::be24:11ff:fe54:4e82%pppoe0 prefixlen 64 scopeid 0x7 inet6 2001:8b0:1111:1111:0:ffff:d9a9:110b prefixlen 128 pltime 3600 vltime 7200 nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>The route table (IPv4 only) looks like this:
Internet: Destination Gateway Flags Netif Expire default 81.187.8x.1y7 UGS pppoe0 81.187.8x.1y7 link#7 UH pppoe0 81.187.2x6.1y6/29 link#1 U vtnet0 81.187.2x6.1y7 link#4 UHS lo0 127.0.0.1 link#4 UH lo0 192.168.1.0/24 link#2 U vtnet1 192.168.1.1 link#4 UHS lo0 217.169.1x.1y link#4 UHS lo0 217.169.20.20 81.187.8x.1y7 UGHS pppoe0 217.169.20.21 81.187.8x.1y7 UGHS pppoe0I'm either missing something simple or there is some deeper issue.
I also cannot understand the arp table (and this may well be my lack of knowledge), I've reproduced a section below:
? (81.187.2x6.1y0) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 344 seconds [ethernet] ? (81.187.2x6.1y1) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 351 seconds [ethernet] ? (81.187.2x6.1y2) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 313 seconds [ethernet] ? (81.187.2x6.1y3) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 302 seconds [ethernet] ? (81.187.2x6.1y6) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 418 seconds [ethernet] ? (81.187.2x6.1y7) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] ? (81.187.2x6.1y8) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 1186 seconds [ethernet] ? (81.187.2x6.1y9) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 364 seconds [ethernet]I think this is proxyarp (?) but this feels like an issue.
Thanks.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZ
options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
ether bc:24:11:54:4e:82
inet 81.187.2x6.1y7 netmask 0xfffffff8 broadcast 81.187.2x6.1y3When you have a /29, it would be better for understanding to obscure the first three octets, but not the last one.
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
description: WAN
options=0
inet 217.169.1xx.1yy --> 81.187.8x.1y7 netmask 0xffffffffSo 81.187.8x.1y7 is assigned to the pppoe0. Than you cannot use the same network on another interface as well!
We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.
-
@viragomann - thanks for the reply and the advice on obscurification (now largely redundant), but I'm not sure I understand:
xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.
My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)
We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.
Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.
Yes, that's correct.
My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)
Well.
We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.
Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.
So it's fine. It wasn't clear from the above obscuring.
Now, as mentioned, first of all you should get the communication within the OPT subnet to work.
Presumed the firewall rules allow it, you should be able to ping the pfSense interface. -
@viragomann - thanks. So from the pfsense box I can ping the DMZ interface (aaa.bbb.206.177)
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.177
PING aaa.bbb.206.177 (aaa.bbb.206.177): 56 data bytes
64 bytes from aaa.bbb.206.177: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from aaa.bbb.206.177: icmp_seq=1 ttl=64 time=0.202 ms
^C
--- aaa.bbb.206.177 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.063/0.133/0.202/0.069 msBut not an IP address in that range assigned to the RPi machine
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
PING aaa.bbb.206.178 (aaa.bbb.206.178): 56 data bytes
^C
--- aaa.bbb.206.178 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet lossI've put 'allow any any rules' in at the top of the firewall for all interfaces WAN/LAN/DMZ - just as a test, but still no connectivity.
From the RPi machine, it cannot ping aaa.bbb.206.177.
Just makes no sense to me. The pfsense instance is running on proxmox and I'm beginning to think there is some issue at the proxmox/virtualisation layer that is blocking the connectivity. I've stopped the firewall that runs within proxmox - but still no joy.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
But not an IP address in that range assigned to the RPi machine
[2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
Before the pfSense is sending a packet to that IP, it does an ARP lookup. If it gets a proper ARP response it add the IP to its Diagnostic > ARP table.
Check if you see the respective entry there.
If not run packet capture on the interface, the destination machine is connected to (DMZ?) and enter ARP at protocol.
Then start a ping from pfSense. You should see the ARP packets then, at least the ARP request from pfSense.If IP is already in the ARP table sniff the ICMP traffic, while you ping the machine.
-
@viragomann - thanks again. The arp table does not look as I would expect it to. I've reproduced it below
[2.7.2-RELEASE][admin@pfsense.lan]/root: arp -a -n | grep '81\.' ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]Now ping the IP and the ARP entry is atted to the table:
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] ? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 1196 seconds [ethernet]However,
2a:8a:1c:ec:d5:20is not the correct MAC address for the interface on the RPi. I can't find what device corresponds to2a:8a:1c:ec:d5:20.This is the output of tcpdump picking out the arp traffic:
16:52:49.386790 ARP, Request who-has aaa.bbb.206.178 tell aaa.bbb.206.177, length 28 16:52:49.394704 ARP, Reply aaa.bbb.206.178 is-at 2a:8a:1c:ec:d5:20, length 46(An example of proxyarp?)
What is also strange is that if I manually replace the MAC address for aaa.bbb.206.178 with the correct one, I still cannot ping the device. When looking at tcpdump, I see the ICMP echoo request but no reply.
The MAC address in the arp table from the RPi showed as 'incomplete' against the aaa.bbb.206.177 address. Again, if I try and delete that and set the MAC entry on the RPi to the correct MAC of the aaa.bbb.206.177 address, that is also still not ping'able.
Again, thanks for your time.
-
@MrHedgehog 2a:8a:1c is unknown.. normally a pi would have a mac that starts with b8:27:eb, atleast the 2 I have currently running do.
pi@pi-ntp:~ $ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.32 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::8849:df44:1c76:170a prefixlen 64 scopeid 0x20<link> ether b8:27:eb:31:70:ab txqueuelen 1000 (Ethernet)pi@pihole:~ $ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.10 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::3829:d17c:fb3d:1b4d prefixlen 64 scopeid 0x20<link> ether b8:27:eb:38:d8:4d txqueuelen 1000 (Ethernet) -
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi.
Possibly a dynamic MAC of Proxmox?
-
@viragomann how would poxmox have anything to do with a physical pi, if its just a vm - why would he call them pis
Maybe a diagram how this is all connected would help..
Pfsense is virtual right.. and it tied to a physical interface on the host.. which some other device, a raspberry pi that is what I read when he said RPi - but maybe its some other vm?
Been a while since I played with proxmox - but why would it use random macs.. Yeah a drawing would be helpful.. If these are just other vms on the lan side network of pfsense on a virtual switch.. Look at the vm what does it show for its mac? That is what you should be seeing when you arp for it..
For example in virtual machine manager.. I can see and change the mac of the vm..
Are these VMs on a natted network in proxmox? is the pfsense interface on this vm network, attached to the same virtual switch on the vm host?
-
@johnpoz said in Can't route public/29 IP block to VMs on lan:
@viragomann how would poxmox have anything to do with a physical pi,
Something else that isn’t the Pi has the IP, if ARP finds the wrong MAC. It’s a question of what.
@MrHedgehog Have you tried a different IP in your subnet?
-
@viragomann - I am thinking that might be the case and am going to swap out the NVMe card that has the proxmox/pfsense install with another one and do a bare metal install of pfsense on that to strip out the virtualisation layer (this makes it easy to 'rollback' - internet connection being an essential service for my wife!)
-
@SteveITS - I have tried the other IP addresses in the /29 range. But the odd thing is - all the IPs in that range ARP to the same MAC address, eg from
arp -apon the pfsense machine:? (aaa.bbb.206.180) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 115 seconds [ethernet] ? (aaa.bbb.206.182) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 130 seconds [ethernet] ? (aaa.bbb.206.183) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 124 seconds [ethernet] ? (aaa.bbb.206.176) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 127 seconds [ethernet] ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] <- DMZ interface on pfsense ? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 611 seconds [ethernet] ? (aaa.bbb.206.179) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 139 seconds [ethernet]I have not idea (at the moment) what 2a:8a:1c:ec:d5:20 corresponds to.
-
@MrHedgehog said in Can't route public/29 IP block to VMs on lan:
strip out the virtualisation layer
great idea!
