Office 365 URLs and IP address ranges
-
You won't achieve full automation easily because XML is essentially free form and to write a parser for an arbitrary XML file you need to know a bit about the structure of the file (called the schema). There are XML parsers for PHP but the parts that know about a particular XML schema (like this what MS uses) are not going be included in base pfSense, they are a clear case for an addon package.
-
well it is working and updating from their files, so unless they change their format etc, Ill continue to use it.
-
@daschenbrener what advice do you have. How did you do it?
-
how i used it is through pfBlockerNG.
I added it to the rules
see picture attached
and it automatically parsed the file and is working
 -
Is that a lucky accident or is PFBlockerNG smart enough to parse the XML properly?
-
this is a first for me, but i believe that it is designed that way as pfsense can uses xml files already as well.
-
@kpa:
Is that a lucky accident or is PFBlockerNG smart enough to parse the XML properly?
It doesn't really parse it, it uses a bunch of crazy regexps to get IPs out of the XML/HTML/whatnot file.
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L4037
-
@daschenbrener: thank you… I will give this a try.... :-)
-
my pleasure ,now to find files like this for other companies , hmm, maybe there is something here i can start, a list of the various locations for ip address . Just like the one for microsoft.
-
Good idea… it just needs to be a trusted list....
-
@kuberan
I use the Powershell script from Microsoft. It generates the IPv4 addresses in a text file. This is uploaded daily to an FTP server via task scheduling. I have deactivated ipv6 and Urls in the Powershell-Script.
Maybe this will help other users.<# Get-O365WebServiceUpdates.ps1 From https://aka.ms/ipurlws v1.1 8/6/2019 DESCRIPTION This script calls the REST API of the Office 365 IP and URL Web Service (Worldwide instance) and checks to see if there has been a new update since the version stored in an existing $Env:TEMP\O365_endpoints_latestversion.txt file in your user directory's temp folder (usually C:\Users\<username>\AppData\Local\Temp). If the file doesn't exist, or the latest version is newer than the current version in the file, the script returns IPs and/or URLs that have been changed, added or removed in the latest update and writes the new version and data to the output file $Env:TEMP\O365_endpoints_data.txt. USAGE Run as a scheduled task every 60 minutes. PARAMETERS n/a PREREQUISITES PS script execution policy: Bypass PowerShell 3.0 or later Does not require elevation https://docs.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide #> #Requires -Version 3.0 # web service root URL $ws = "https://endpoints.office.com" # path where output files will be stored #$versionpath = $Env:TEMP + "\O365_endpoints_latestversion.txt" $versionpath = "C:\Powershell\O365_endpoints_latestversion.txt" #$datapath = $Env:TEMP + "\O365_endpoints_data.txt" $datapath = "C:\Powershell\O365_endpoints_data.txt" # fetch client ID and version if version file exists; otherwise create new file and client ID if (Test-Path $versionpath) { $content = Get-Content $versionpath $clientRequestId = $content[0] $lastVersion = $content[1] Write-Output ("Version file exists! Current version: " + $lastVersion) } else { Write-Output ("First run! Creating version file at " + $versionpath + ".") $clientRequestId = [GUID]::NewGuid().Guid $lastVersion = "0000000000" @($clientRequestId, $lastVersion) | Out-File $versionpath } # call version method to check the latest version, and pull new data if version number is different $version = Invoke-RestMethod -Uri ($ws + "/version/Worldwide?clientRequestId=" + $clientRequestId) if ($version.latest -gt $lastVersion) { Write-Host "New version of Office 365 worldwide commercial service instance endpoints detected" # write the new version number to the version file @($clientRequestId, $version.latest) | Out-File $versionpath # invoke endpoints method to get the new data $endpointSets = Invoke-RestMethod -Uri ($ws + "/endpoints/Worldwide?clientRequestId=" + $clientRequestId) # filter results for Allow and Optimize endpoints, and transform these into custom objects with port and category # URL results $flatUrls = $endpointSets | ForEach-Object { $endpointSet = $_ $urls = $(if ($endpointSet.urls.Count -gt 0) { $endpointSet.urls } else { @() }) $urlCustomObjects = @() if ($endpointSet.category -in ("Allow", "Optimize")) { $urlCustomObjects = $urls | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; url = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $urlCustomObjects } # IPv4 results $flatIp4s = $endpointSets | ForEach-Object { $endpointSet = $_ $ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() }) # IPv4 strings contain dots $ip4s = $ips | Where-Object { $_ -like '*.*' } $ip4CustomObjects = @() if ($endpointSet.category -in ("Allow", "Optimize")) { $ip4CustomObjects = $ip4s | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; ip = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $ip4CustomObjects } # IPv6 results $flatIp6s = $endpointSets | ForEach-Object { $endpointSet = $_ $ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() }) # IPv6 strings contain colons $ip6s = $ips | Where-Object { $_ -like '*:*' } $ip6CustomObjects = @() if ($endpointSet.category -in ("Optimize")) { $ip6CustomObjects = $ip6s | ForEach-Object { [PSCustomObject]@{ category = $endpointSet.category; ip = $_; tcpPorts = $endpointSet.tcpPorts; udpPorts = $endpointSet.udpPorts; } } } $ip6CustomObjects } # write output to screen Write-Output ("Client Request ID: " + $clientRequestId) Write-Output ("Last Version: " + $lastVersion) Write-Output ("New Version: " + $version.latest) Write-Output "" Write-Output "IPv4 Firewall IP Address Ranges" ($flatIp4s.ip | Sort-Object -Unique) -join "," | Out-String Write-Output "IPv6 Firewall IP Address Ranges" ($flatIp6s.ip | Sort-Object -Unique) -join "," | Out-String Write-Output "URLs for Proxy Server" ($flatUrls.url | Sort-Object -Unique) -join "," | Out-String Write-Output ("IP and URL data written to " + $datapath) # write output to data file # Write-Output "Office 365 IP and UL Web Service data" | Out-File $datapath # Write-Output "Worldwide instance" | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output ("Version: " + $version.latest) | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output "IPv4 Firewall IP Address Ranges" | Out-File $datapath -Append ($flatIp4s.ip | Sort-Object -Unique) -join "`r`n" | Out-File $datapath -Append # Write-Output "" | Out-File $datapath -Append # Write-Output "IPv6 Firewall IP Address Ranges" | Out-File $datapath -Append # ($flatIp6s.ip | Sort-Object -Unique) -join "," | Out-File $datapath -Append Write-Output "" | Out-File $datapath -Append # Write-Output "URLs for Proxy Server" | Out-File $datapath -Append # ($flatUrls.url | Sort-Object -Unique) -join "," | Out-File $datapath -Append } else { Write-Host "Office 365 worldwide commercial service instance endpoints are up-to-date." } $file = "C:\Powershell\O365_endpoints_data.txt" # Do not use special characters # . / % & etc. for password!!! $ftpuri = "ftp://username:password@my-ftp-server.com/ms365.txt" $webclient = New-Object System.Net.WebClient $uri = New-Object System.Uri($ftpuri) $webclient.UploadFile($uri, $file)