Office 365 URLs and IP address ranges
-
If its a plain txt file, then you can use the pfSense URL Table aliases to collect these txt files on a schedule… Or use the pfBlockerNG package to parse the URL to collect the domains.... pfBlockerNG doesn't need the feed to be in plain text, and can parse the html to collect the IPs... You can then use the "Alias" in your firewall rules/nat etc as required...
-
Even if it were RSS or HTML, things like pfBlockerNG could make sense of it. In the current form, it's just incredibly useless POS. Cannot believe then don't have anything in machine-readable format for enterprise use.
-
It is in XML format, that is how it is published, which is machine readable…..
"Use the https://go.microsoft.com/fwlink/?LinkId=533185 to view a single list with all of the endpoints or to automatically process changes." -
@kuberan thanks for the info, how would you go about implementing this?
-
I would like to automate this… but for now I edited the XML file Aliases and loaded that... I think pfBlockerNG can be used.. but I need to test it first.... I was hoping there is a simpler solution
-
You won't achieve full automation easily because XML is essentially free form and to write a parser for an arbitrary XML file you need to know a bit about the structure of the file (called the schema). There are XML parsers for PHP but the parts that know about a particular XML schema (like this what MS uses) are not going be included in base pfSense, they are a clear case for an addon package.
-
well it is working and updating from their files, so unless they change their format etc, Ill continue to use it.
-
@daschenbrener what advice do you have. How did you do it?
-
how i used it is through pfBlockerNG.
I added it to the rules
see picture attached
and it automatically parsed the file and is working
 -
Is that a lucky accident or is PFBlockerNG smart enough to parse the XML properly?
-
this is a first for me, but i believe that it is designed that way as pfsense can uses xml files already as well.
-
@kpa:
Is that a lucky accident or is PFBlockerNG smart enough to parse the XML properly?
It doesn't really parse it, it uses a bunch of crazy regexps to get IPs out of the XML/HTML/whatnot file.
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L4037
-
@daschenbrener: thank you… I will give this a try.... :-)
-
my pleasure ,now to find files like this for other companies , hmm, maybe there is something here i can start, a list of the various locations for ip address . Just like the one for microsoft.
-
Good idea… it just needs to be a trusted list....
-
@kuberan
I use the Powershell script from Microsoft. It generates the IPv4 addresses in a text file. This is uploaded daily to an FTP server via task scheduling. I have deactivated ipv6 and Urls in the Powershell-Script.
Maybe this will help other users.<# Get-O365WebServiceUpdates.ps1 From https://aka.ms/ipurlws v1.1 8/6/2019 DESCRIPTION This script calls the REST API of the Office 365 IP and URL Web Service (Worldwide instance) and checks to see if there has been a new update since the version stored in an existing $Env:TEMP\O365_endpoints_latestversion.txt file in your user directory's temp folder (usually C:\Users\<username>\AppData\Local\Temp). If the file doesn't exist, or the latest version is newer than the current version in the file, the script returns IPs and/or URLs that have been changed, added or removed in the latest update and writes the new version and data to the output file $Env:TEMP\O365_endpoints_data.txt. USAGE Run as a scheduled task every 60 minutes. PARAMETERS n/a PREREQUISITES PS script execution policy: Bypass PowerShell 3.0 or later Does not require elevation https://docs.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide #> #Requires -Version 3.0 # web service root URL $ws = "https://endpoints.office.com" # path where output files will be stored #$versionpath = $Env:TEMP + "\O365_endpoints_latestversion.txt" $versionpath = "C:\Powershell\O365_endpoints_latestversion.txt" #$datapath = $Env:TEMP + "\O365_endpoints_data.txt" $datapath = "C:\Powershell\O365_endpoints_data.txt" # fetch client ID and version if version file exists; otherwise create new file and client ID if (Test-Path $versionpath) { $content = Get-Content $versionpath $clientRequestId = $content[0] $lastVersion = $content[1] Write-Output ("Version file exists! Current version: " + $lastVersion) } else { Write-Output ("First run! Creating version file at " + $versionpath + ".") $clientRequestId = [GUID]::NewGuid().Guid $lastVersion = "0000000000" @($clientRequestId, $lastVersion) | Out-File $versionpath } # call version method to check the latest version, and pull new data if version number is different $version = Invoke-RestMethod -Uri ($ws + "/version/Worldwide?clientRequestId=" + $clientRequestId) if ($version.latest -gt $lastVersion) { Write-Host "New version of Office 365 worldwide commercial service instance endpoints detected" # write the new version number to the version file @($clientRequestId, $version.latest) | Out-File $versionpath # invoke endpoints method to get the new data $endpointSets = Invoke-RestMethod -Uri ($ws + "/endpoints/Worldwide?clientRequestId=" + $clientRequestId) # filter results for Allow and Optimize endpoints, and transform these into custom objects with port and category # URL results $flatUrls = $endpointSets | ForEach-Object { $endpointSet = $_ $urls = $(if ($endpointSet.urls.Count -gt 0) { $endpointSet.urls } else { @() }) $urlCustomObjects = @() if ($endpointSet.category -in ("Allow", "Optimize")) { $urlCustomObjects = $urls | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; url = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $urlCustomObjects } # IPv4 results $flatIp4s = $endpointSets | ForEach-Object { $endpointSet = $_ $ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() }) # IPv4 strings contain dots $ip4s = $ips | Where-Object { $_ -like '*.*' } $ip4CustomObjects = @() if ($endpointSet.category -in ("Allow", "Optimize")) { $ip4CustomObjects = $ip4s | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; ip = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $ip4CustomObjects } # IPv6 results $flatIp6s = $endpointSets | ForEach-Object { $endpointSet = $_ $ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() }) # IPv6 strings contain colons $ip6s = $ips | Where-Object { $_ -like '*:*' } $ip6CustomObjects = @() if ($endpointSet.category -in ("Optimize")) { $ip6CustomObjects = $ip6s | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; ip = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $ip6CustomObjects } # write output to screen Write-Output ("Client Request ID: " + $clientRequestId) Write-Output ("Last Version: " + $lastVersion) Write-Output ("New Version: " + $version.latest) Write-Output "" Write-Output "IPv4 Firewall IP Address Ranges" ($flatIp4s.ip | Sort-Object -Unique) -join "," | Out-String Write-Output "IPv6 Firewall IP Address Ranges" ($flatIp6s.ip | Sort-Object -Unique) -join "," | Out-String Write-Output "URLs for Proxy Server" ($flatUrls.url | Sort-Object -Unique) -join "," | Out-String Write-Output ("IP and URL data written to " + $datapath) # write output to data file # Write-Output "Office 365 IP and UL Web Service data" | Out-File $datapath # Write-Output "Worldwide instance" | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output ("Version: " + $version.latest) | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output "IPv4 Firewall IP Address Ranges" | Out-File $datapath -Append ($flatIp4s.ip | Sort-Object -Unique) -join "`r`n" | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output "IPv6 Firewall IP Address Ranges" | Out-File $datapath -Append # ($flatIp6s.ip | Sort-Object -Unique) -join "," | Out-File $datapath -Append Write-Output "" | Out-File $datapath -Append # Write-Output "URLs for Proxy Server" | Out-File $datapath -Append # ($flatUrls.url | Sort-Object -Unique) -join "," | Out-File $datapath -Append } else { Write-Host "Office 365 worldwide commercial service instance endpoints are up-to-date." } $file = "C:\Powershell\O365_endpoints_data.txt" # Do not use special characters # . / % & etc. for password!!! $ftpuri = "ftp://username:password@my-ftp-server.com/ms365.txt" $webclient = New-Object System.Net.WebClient $uri = New-Object System.Uri($ftpuri) $webclient.UploadFile($uri, $file)
