Solved: Unknown servers on VLAN
-
I have servers in a data center. Each customer has their own untagged VLAN and they even said it is completely secure, for my own traffic only, fully separated from anyone elses traffic.
I also know that VLAN is inherently insecure so I'm just trying to understand what I have since I can see servers that aren't mine on my own VLAN.
I have OPT1 connected to the VLAN and nothing on that network. When I use nmap, I can see servers I don't own in the 192.168.0.0 and 10.0.0.0 networks.
What does this mean since the DC is telling me this is perfectly normal.
-
What are you actually seeing?
If you run a pcap do you see traffic from other machines? I would not expect to if the VLAN is configured correctly.
Steve
-
I'm using tcpdump to monitor and don't see any traffic. Isn't it odd that I can see servers that aren't mine though?
nmap 192.168.0.0/16 Starting Nmap 7.91 ( https://nmap.org ) at 2022-03-14 14:18 MST sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.9, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.9:80 A ttl=55 id=17315 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.10, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.10:80 A ttl=43 id=8247 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.15, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.15:80 A ttl=43 id=11396 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.16, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.16:80 A ttl=56 id=58879 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.9, 16) => Permission denied Offending packet: TCP 1.1.1.1:38149 > 192.168.0.9:80 A ttl=57 id=65386 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.10, 16) => Permission denied Offending packet: TCP 1.1.1.1:38149 > 192.168.0.10:80 A ttl=42 id=8900 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.15, 16) => Permission denied Offending packet: TCP 1.1.1.1:38149 > 192.168.0.15:80 A ttl=43 id=31041 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.16, 16) => Permission denied Offending packet: TCP 1.1.1.1:38149 > 192.168.0.16:80 A ttl=57 id=51456 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.27, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.27:80 A ttl=46 id=38816 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(4, packet, 40, 0, 192.168.0.30, 16) => Permission denied Offending packet: TCP 1.1.1.1:38148 > 192.168.0.30:80 A ttl=40 id=691 iplen=40 seq=0 win=1024 Omitting future Sendto error messages now that 10 have been shown. Use -d2 if you really want to see them. Nmap scan report for xxx.loc (192.168.1.3) (this is the IP on the interface connected to the VLAN) Host is up (0.000060s latency). Not shown: 998 filtered ports PORT STATE SERVICE 53/tcp open domain 80/tcp open http Nmap scan report for 192.168.7.106 - NOT MY SERVER/HOST Host is up (0.0017s latency). All 1000 scanned ports on 192.168.7.106 are filtered Nmap scan report for 192.168.100.10 - NOT MY SERVER/HOST Host is up (0.0013s latency). Not shown: 998 filtered ports PORT STATE SERVICE 22/tcp closed ssh 179/tcp closed bgp Nmap scan report for 192.168.200.10 - NOT MY SERVER/HOST Host is up (0.0016s latency). Not shown: 998 filtered ports PORT STATE SERVICE 22/tcp closed ssh 179/tcp closed bgp Nmap done: 65536 IP addresses (4 hosts up) scanned in 2153.17 seconds -
@lewis said in Unknown servers on VLAN:
the DC is telling me this is perfectly normal.
No its not normal to run different customers on the same L2.. Customers should be completely isolated - and you shouldn't be able to see or talk to any other customers in the DC. Unless both customers got with the DC and asked for their networks to be able to talk to each other.
-
Scanning the 10 network is not done yet but I'm not really understanding what I'm seeing here. Some of the IPs show a 0 at the end. Can't say I've ever seen that before.
10.0.4.0:80?
Of course the 1.1.1.1 IP is just me replacing mine.
nmap 10.0.0.0/8 >10-network sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.4.0, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.4.0:80 A ttl=37 id=33365 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.5.0, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.5.0:80 A ttl=47 id=13896 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.10.0, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.10.0:80 A ttl=41 id=41671 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.11.0, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.11.0:80 A ttl=47 id=57958 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.14.0, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.14.0:80 A ttl=49 id=21374 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.1.1, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.1.1:80 A ttl=49 id=17469 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.2.1, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.2.1:80 A ttl=47 id=36688 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.5.1, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.5.1:80 A ttl=56 id=54437 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.6.1, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.6.1:80 A ttl=49 id=65370 iplen=40 seq=0 win=1024 sendto in send_ip_packet_sd: sendto(5, packet, 40, 0, 10.0.7.1, 16) => Permission denied Offending packet: TCP 1.1.1.1:63779 > 10.0.7.1:80 A ttl=44 id=3248 iplen=40 seq=0 win=1024 Omitting future Sendto error messages now that 10 have been shown. Use -d2 if you really want to see them. -
@lewis said in Unknown servers on VLAN:
nmap 10.0.0.0/8 >10-network
Keep in mind when you do a scan like that, and your not binding to a specific interface to send the traffic out on, it would just go out your normal route, ie your default gateway if your not attached to that network, or don't have a specific route to that network.
So the DC could have other devices that are able to be gotten too via the routing that are rfc1918 addresses..
PORT STATE SERVICE 53/tcp open domain 80/tcp open http53 for example is DNS, this could be DNS that is meant to be able to get to on the DC network by any customer, or some 80 (web server or maybe a gui to that box?)
Your showing closed on those others, where a firewall actually rejected that traffic vs just dropping it.
22/tcp closed ssh 179/tcp closed bgpIf running bgp, and ssh but blocking your IP - this could be router on the DC network.
This could be normal and ok.. What wouldn't be ok is running multiple customers on the same L2 network just with different IP space..
As example - with a typical ISP they most likely run rfc1918 space on the internal ISP network, that you may or may not be able to get too.. Say their internal dhcp server that hands out dhcp to their customers for example. Or their routers internal loopback address or something that could be rfc1918, etc.. While the ISP doesn't route those rfc1918 outside of their network, they could be using internally to their network which you are a part of as a customer - same thing goes in a DC.
Unless you were scanning the actual L2 your connected to via sending the traffic out a specific interface that doesn't actually have the IP range your scanning on it - and getting replies. What your seeing via internal routing on the DC network could be normal and ok.
Some of those IPs your seeing come back - do a traceroute too them, if your worried about other devices on on the same L2 you might have to use a specific interface with -e and even might have to spoof an address in the network range your scanning with -S
If you can get to some DC servers via routing, even if rfc1918 this could be fine - might be something that needs to be open to clients in the DC, or could be lack of proper security controls? What you should NOT be able to see is other "customers' devices/servers on the same L2 as you, or even via routing - unless of course the server is meant to be open to the public - if this is the case it might be a slight lack of controls if you can actually get to them via routing in the DC..
What I would be concerned with is being able to talk or other devices to be able to talk to you on the same L2 as your talking too using a different IP range then yours, or even the same IP range. Customers devices should be isolated at L2 for sure.. And even internally via L3, but depending if the service is suppose to be open to the public - it might be that their routing internally on the DC also allows rfc1918 to get there - not the best thing, but not the end of the world either.
I wouldn't be all that worried if your seeing traffic from other IP ranges on your wan interface - the default block rfc1918 rule would stop any other DC customers from talking to some services you have open via public IP(s) that might reside on the same interface via their internal DC rfc1918 address. But I would be a bit concerned if on your lan side interfaces if you were seeing other devices on the same L2 as you, just using different IP space and not actually isolated via vlan..
-
@johnpoz said in Unknown servers on VLAN:
Customers should be completely isolated
Actually, that depends on the sort of service they're paying for. There are plenty of customers on virtual machines, which means they're sharing everything below the VM. I have also worked with Q in Q over fibre, where customers share a fibre and also sharing a fibre by using different wavelengths and passing through a splitter.
That said, if they have their own VLAN, they shouldn't be seeing other VLANs, which might point to a misconfigured switch.
-
@jknott said in Unknown servers on VLAN:
customers on virtual machines, which means they're sharing everything below the VM
While sure this might be the case - but in the DCs my company runs. Customers are normally on their own VM host. I would have to get with that team if there is any instances at all of shared host.. Their could be some, but while they might share physical hardware - there would still be isolation practices put in place.
Their own vswitch, which is isolated to their specific vlan. While they might share the same physical host, and physical networking infrastructure etc... Customer A shouldn't be seeing traffic of customer B.. And they shouldn't be able to talk to their other devices no matter what they do - like just changing IP ranges for example ;)
Most customers in our DC are larger and have many different servers or VMs, etc. We don't cater single use VMs etc. Where you might get some customer with only 1 or couple VMs, etc. While this might be the case in hosts of vps where any user can just give a CC and get a VM on a month to month basis sort of thing.
Big issue with say meltdown and spectre was that very issue of customers on the same VM host being able to get info from other customers vms, etc. Segregation of customers to their own VM hosts makes for good practice against such sorts of exploits.
Even when customers VMs share the same host - they should still be isolated where A can not see B traffic. And vise versa.
-
Mmm, I'm not sure what you are actually testing there. I assume 1.1.1.1 is an IP you have substituted there? On which interface?
If you are seeing broadcasts from other unknown IPs on the DCLAN in a packet capture that's much more concerning IMO.
Steve
-
@stephenw10 said in Unknown servers on VLAN:
unknown IPs on the DCLAN in a packet capture that's much more concerning IMO.
exactly! Looks like to me he is just scanning a IP range, which would be attempted via routing if the IP range is not actually attached. Or you didn't force use of a specific interface, and use a specific source IP that was in the range you were scanning.
He is just scanning the DC network - so yeah its pretty much a given there is going to be some stuff on the DC network he can prob get to.. The ones with 22 and BGP screams a router to me.. Which since show "closed" say rejected vs dropped. Which could be common setup for blocking your own internal networks.. I reject stuff I block internally - so the client doesn't waste time with a retrans, etc.
I believe the default scan would be a syn scan, so showing closed means got back a RST.. If got no answer or got back an icmp unreachable then it would show as filtered.
-
The 1.1.1.1 is my edited WAN IP
To clarify, I have my own hardware in a rack at the DC, nothing shared, not a shared vm environment. I have my own WAN blend but also have a LAN connection to communicate with some other servers in the DC which aren't in my rack. However, that LAN is in fact a VLAN that is assigned to each customer and I was told that it is completely secure.
I'm scanning from the pfsense box which has three connected interfaces. One to WAN, one to a LAN switch then servers behind that and one I've called DCLAN which is my LAN/VLAN network.
When scanning and using the specific interface of DCLAN, all I really get is;
snip
setup_target: failed to determine route to 192.168.255.253
setup_target: failed to determine route to 192.168.255.254
setup_target: failed to determine route to 192.168.255.255
snip
But without entering any interface is when I see other servers.
I should not be seeing any private IP's through my WAN and I only have 10.0.0.1/24 IPs being used on my LAN so that's why I'm curious where these other servers are coming from since that would be from the DCLAN.I've asked the DC but of course, they have to explain what private IPs are like I know nothing what so ever which is frustrating. Meaning, any time you even bring up networking questions, they seem to get evasive or almost belligerent while as a customer, I'm just trying to understand what I'm seeing and why.
Update: Ah, today, someone says yes they didn't notice I had mentioned the internal LAN/VLAN so they are looking into it. I love this DC really but it's kinda funny how defensive they get when you ask about networking. In a DC, we're truly all together in this LOL.
-
Yes, that's expected then. Using the WAN as source you're not actually scanning anything in the DCLAN VLAN. You would need to set the DCLAN as source but when you do that it can only scan things in that subnet or for which is has a route using that interface.
So you could set the interface to have a huge subnet and it will ARP for everything.
Or run a pcap since almost everything will send a broadcast at some point and you will see it.Steve
-
@lewis said in Unknown servers on VLAN:
I should not be seeing any private IP's through my WAN
This is not actually true... While sure in a perfect world nobody would use rfc1918 in a DC.. A DC does have limited IP space - so internally they will almost always leverage rfc1918 space on devices in their network that is used to route other traffic.. example here is my trace to google from my isp..
Tracing route to www.google.com [172.217.1.100] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms sg4860.local.lan [192.168.9.253] 2 9 ms 10 ms 18 ms d47-69-1-60.col.wideopenwest.com [69.47.60.1] 3 11 ms 12 ms 8 ms 10.52.33.194 4 13 ms 16 ms 12 ms 76-73-164-154.knology.net [76.73.164.154] 5 18 ms 12 ms 13 ms static-75-76-101-196.knology.net [75.76.101.196]Look at hop 3 in my trace - this is clearly my isp network.. But it comes back as rfc1918.. Now that might also have a public IP, but the trace might have been responded with by the loopback or something. Or internally my isp might be using rfc1918 as their transit networks.. All perfectly fine and normal. Even if looks a bit odd via trace, etc.
rfc1918 are IPs that route, they are just not suppose to route over the public internet - but they for sure can route on any internal network, like an ISPs or a DCs etc.. Just like you can route them on your own local network.. They can route them on their internal network..
-
Thanks, lots of interesting information.
It makes sense that internally, they could be using both public and private IPs but on my VLAN which is supposed to be private, I would have thought I would see nothing but my own resources.
What I should do then is when I have everything fully set up and working is to test this in a more controlled way. As mentioned, using a large subnet or pcap.
I'll get back to this as I'd like to better understand the results of scanning networks.
-
@lewis you want really weird... Is when the DC uses public IP space internally, that is not theirs ;)
Lets just say I know a DC (multiple actually) where they leverage some of the DOD space internally in the DC.. So while tracing stuff that is internally to the DC you might see rfc1918, rfc1918, DOD space IP, rfc1918, where you traced too..
Your like WTF ;) I tried telling them hey that not really a good idea... But who am I some lowly network engineer from some backwater LBU.. Who am to question the almighty powers that be.. They don't need to get to those networks, none of their customers need to get to those networks (they hope).. And it frees up rfc1918 space they would of used that their customers can now use.. Now if those networks ever leaked out via bgp - they would be in some shit for sure..
I tried bringing it up multiple times - wasn't going to be a hill I died on, and just let it drop.. Ok you clearly no better than me ;) hehehe
edit: So I just scanned that 10 IP in my isp network, and I get back this is open
[22.01-RELEASE][admin@sg4860.local.lan]/var/db: nmap -Pn 10.52.33.194 Host discovery disabled (-Pn). All addresses will be marked 'up' and scan times will be slower. Starting Nmap 7.91 ( https://nmap.org ) at 2022-03-15 10:34 CDT Nmap scan report for 10.52.33.194 Host is up (0.013s latency). Not shown: 999 filtered ports PORT STATE SERVICE 646/tcp open ldp Nmap done: 1 IP address (1 host up) scanned in 127.12 secondsIf I had to guess since LDP is used for exchanging labels in a mpls network, that is one of their routers and why I see it in my trace.. Now is that an optimal setup - prob not ;) But prob not a big deal because onlything that could possible get to that would be stuff internal to the isp network, or clearly their customers ;) hehehe
-
@johnpoz LOL, sometimes we get into trouble just for being curious I guess. But on the other hand, if no one ever questioned anything, things would come apart. Where is the balance? In your pay level maybe? :)
-
DC tells me it's their own hardware and as someone mentioned, because I was scanning all interfaces.
Thank you for the responses. Always learning :).
-
@lewis said in Unknown servers on VLAN:
because I was scanning all interfaces.
From the command you showed it wouldn't be scanning all interfaces, other then the interfaces you had that were in the networks you scanned. And then it would just send networks that were not directly attached out your default gateway. Which would be routed through your DCs network via your default gateway.
-
There are three NICs connected in the pfsense box.
One is WAN and two are LANs. All are of course routed internally so I'm sure LAN/WAN meet here and there.When I ran the nmap, I didn't specify an interface but I did specify a network.
-
@lewis said in Unknown servers on VLAN:
When I ran the nmap, I didn't specify an interface but I did specify a network.
Exactly - so if you scanned 192.168/16 and you had say 2 lan networks of say 192.168.100/24 and 192.168.200/24
It would scan those 2 networks because they are directly attached sending the traffic out those interface. But when it say scanned 192.168.101/24 out of your 16 it would of sent that to the default gateway out the wan.
As to lan/wan meeting??
If you scanned 10/8 and you had no interfaces in a 10.x network and or no other routes to get to 10.x then all of that traffic would of been sent to the default gateway.