Automated scripts for Private Internet Access port forwarding

Bagpuss

On pfSense, crontab will get rebuilt from config.xml - so manual edit changes will get overwritten.
Install the Cron package, and use that to add the crontab entry.

Hi Phil,

Thanks for the feedback. I wasn't aware of this.

Can you tell me in what circumstances pfSense will do this? I've had a manually entered crontab entry running for a number of days, and it's still okay.
I'm not totally sure that my manual entry will be overwritten, as it's in a completely different crontab file.

The crontab which is built from the config.xml appears to be /etc/crontab.
The crontab which is changed by crontab -e is /var/cron/tabs/root.

I'm thinking that the two can co-exist without interfering with each other.

Having said that, it does make a lot of sense to use the Cron package, and manage things from the GUI.

Andy.

VTOLfreak

Thank you for this. :) Any chance of packing this up into a handy plugin for pfsense with a GUI page?

I don't really like the idea of adding too much scripts and files to my pfsense setup outside of plugins. Rightnow I'm putting off trying the 2.1.1 snapshots because I dont know if upgrading will wipe out all the changes I made to my 2.1 setup. If this script was in a plugin, pfsense takes care of setting it all up for you.

phil.davis

Can you tell me in what circumstances pfSense will do this?

Definitely at boot time - crontab, passwd databases, all that sort of "system management" setting gets rebuilt from the settings specified in config.xml
For crontab and passwd I don't think there are run-time events that cause them to rebuild from scratch, so manual edits probably survive a long time.
I definitely recommend NOT manually editing any files. Do it all through the GUI and available packages. If the available GUI+packages does not have a button for what you need, then put a feature request in RedMine and (if you have some coding skills) make the GUI enhancements and submit a pull request on GitHub.
It is an Open Source project, and if everyone contributes features that they need then we all get the benefit - or the software bloat ;)

sacredgaming

Do think this would work with plex server for viewing outside my network?

dalesd

Hi Andy,
I'm also a PIA customer and I'm trying your script for port forwarding.

So far, it's not working. Where do I find the log file so I can troubleshoot this? Your post says to check the /var/log/messages file, but there isn't one.

dalesd

Turns out I had my cron entry set up wrong. I fixed that and now the log is showing what's going on.


Mar 1 09:12:19	php: /index.php: New alert found: pfSense is restoring the configuration /conf/backup/config-1393680553.xml
Mar 1 09:12:19	php: /index.php: pfSense is restoring the configuration /conf/backup/config-1393680553.xml
Mar 1 09:12:19	php: /index.php: New alert found: No config.xml found, attempting last known config restore.
Mar 1 09:12:19	php: /index.php: No config.xml found, attempting last known config restore.
Mar 1 09:02:00	root: pia-port: New port number () inserted into config file.
Mar 1 09:02:00	root: pia-port: Created new PIA Client ID.

When I run the script in a terminal, here's what I get:

[2.1-RELEASE][admin@pfsense.localdomain]/usr/local/bin(51): ./pia-port 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   149    0    57  100    92    245    395 --:--:-- --:--:-- --:--:--   519
expr: syntax error

[: -gt: unexpected operator
I/O warning : failed to load external entity "/rule[descr="Torrent"]/local-port"
[2.1-RELEASE][admin@pfsense.localdomain]/usr/local/bin(52):

I think curl isn't working or installed correctly, because this happens:


[2.1-RELEASE][admin@pfsense.localdomain]/var/log(46): pkg_add -r curl           
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/curl.tbz... Done.
pkg_add: package 'curl-7.24.0' or its older version already installed
[2.1-RELEASE][admin@pfsense.localdomain]/var/log(47): curl                      
curl: Command not found.

I don't know how to fix that.

jgorman

Hello,

Thanks a lot for this script.

I'm getting the following:

cron: login_getclass: unknown class 'pia-port # #'

I should not, I created the pia-port using the Diagnostics-Edit File from within pfsense and not by ssh-ing it ont the machine…

Don't know if that makes a difference.

Any help would be greatly appreciated.

THANKS!

ppierre

Nice script.

Juste an edge case that I've encounter. I've got this reply :

{"error":"port forwarding not available for this region"}

Ewrything was fine, but in my setup I haven't setup the VPN as default route (only one IP go through it).

So I had to add this to your script :

curl [b]--interface [/b]"$LOCAL_IP" ...

And everything went smooth. It seem IPA need to get the port request from his own network.
Maybe you should add it to your post. In case someone would try a similar setup.

zounder1

To clarify this. Change the "PORT" line in the the script with this:

PORT=curl –interface $LOCAL_IP -d "user=$USERNAME&pass=$PASSWORD&client_id=$(cat $PIACLIENTID)&local_ip=$LOCAL_IP" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment

This should be put into V1.02 of this script. For setups with multiple gateways you need this to ensure the request goes out via the proper PIA VPN connection. With this change the script will try and use the default gateway which may not be the correct gateway.

Hats off to Bagpuss the creator of this script! This is awesome.

Hats off to ppierre for adding that important last catch with the script.

@ppierre:

Nice script.

Juste an edge case that I've encounter. I've got this reply :
{"error":"port forwarding not available for this region"} 
Ewrything was fine, but in my setup I haven't setup the VPN as default route (only one IP go through it).

So I had to add this to your script :
curl [b]--interface [/b]"$LOCAL_IP" ...
And everything went smooth. It seem IPA need to get the port request from his own network.
Maybe you should add it to your post. In case someone would try a similar setup.

Bagpuss

@zounder1:

To clarify this. Change the "PORT" line in the the script with this:

PORT=curl –interface $LOCAL_IP -d "user=$USERNAME&pass=$PASSWORD&client_id=$(cat $PIACLIENTID)&local_ip=$LOCAL_IP" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment

This should be put into V1.02 of this script. For setups with multiple gateways you need this to ensure the request goes out via the proper PIA VPN connection. With this change the script will try and use the default gateway which may not be the correct gateway.

Hats off to Bagpuss the creator of this script! This is awesome.

Hats off to ppierre for adding that important last catch with the script.

Not checked this thread for a while, and I've just realised that I hadn't posted v1.02 where I did indeed catch this bug. My apologies.
My fix is slightly different, but it achieves the same thing.

Attached to this post is v1.03, where I've caught another issue.
I kept seeing entries in my syslog which said 'pfSense is restoring the configuration /conf/backup/config-1393680553.xml' or similar.
Further debugging revealed that the PIA address wasn't resolving for curl, and hence the script returned a NULL for the port number.
I wasn't catching this error, and so the script was copying an empty /tmp/config.pia over the config.xml.

The new script fixes this, and also provides the return code from curl, so you can try and work out what went wrong.

pia-port.v103.txt

Bagpuss

@dalesd:

Hi Andy,
I'm also a PIA customer and I'm trying your script for port forwarding.

So far, it's not working. Where do I find the log file so I can troubleshoot this? Your post says to check the /var/log/messages file, but there isn't one.

Sorry. That should have been /var/log/system.log

Bagpuss

As an aside, my original instructions contain details for managing cron from the command line.
Whilst this works, there is a cron GUI package available, and this works just as well.

I'm pretty sure that anyone who is okay with installing this stuff in the first place can use cron,
but the pfSense preferred way is to use the GUI package. I'm told that changes to the underlying
pfSense OS going forward might mean that command line cron entries are lost across reboots.
The GUI package stores the cron entries in your config.xml, thereby ensuring their survival.

binaryjay

Amazing post. I was in the middle of reinventing this wheel (moving away from openvpn client on specific hosts) but this saved me a bunch of time. Had to make some minor changes to account for nanoBSD which I have PM'd you about. May be worth checking if the filesystem is ro and remounting before continuing to make your script pfsense distribution agnostic.

Also, instead of force deleting config.cache (doesn't that seem a little messy?), would it not be better to just run:
/etc/rc.reload_all

? Scratch that, running that manually seems to kill everything and requires a reboot.

After I started running this script I've been seeing inetd complaining in the logs about connection aborts and I wonder if it has something to do with the unexpected blowing away of config.cache.

Imaginos

So these two commands are causing an interesting problem:

pkg_add -r curl
pkg_add -r xmlstarlet

They both reach out to ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/… and both generate the following error.

Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz' by URL

A brief investigation reveals that the error is correct and that file path does not exist. There is a packages-8-stable, 8.4-release, 9-stable and others, but not an 8.3 flavor.

You can see the directory here http://ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/.

I'm at a loss of how to proceed and my kung fu with this is 30 years old and not potent enough to figure it out.

Cino

@Imaginos:

So these two commands are causing an interesting problem:
pkg_add -r curl
pkg_add -r xmlstarlet
They both reach out to ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/… and both generate the following error.
Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz' by URL
A brief investigation reveals that the error is correct and that file path does not exist. There is a packages-8-stable, 8.4-release, 9-stable and others, but not an 8.3 flavor.

You can see the directory here http://ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/.

I'm at a loss of how to proceed and my kung fu with this is 30 years old and not potent enough to figure it out.

searching with help your kung fu

https://forum.pfsense.org/index.php?topic=78935.0

AMD64
setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/

I386
setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.3-release/

zounder1

@Cino:

@Imaginos:
So these two commands are causing an interesting problem:
pkg_add -r curl
pkg_add -r xmlstarlet
They both reach out to ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/… and both generate the following error.
Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz' by URL
A brief investigation reveals that the error is correct and that file path does not exist. There is a packages-8-stable, 8.4-release, 9-stable and others, but not an 8.3 flavor.

You can see the directory here http://ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/.

I'm at a loss of how to proceed and my kung fu with this is 30 years old and not potent enough to figure it out.
searching with help your kung fu

https://forum.pfsense.org/index.php?topic=78935.0

AMD64
setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/

I386
setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.3-release/

I used:
setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/Latest/

Then I could install curl.

But quite the output for xmlstarlet
pkg_add -r xmlstarlet
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/Latest/xmlstarlet.tbz… Done.
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/All/pkg-config-0.25_1.tbz... Done.
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/All/libxml2-2.7.8_2.tbz... Done.
pkg_add: warning: package 'libxml2-2.7.8_2' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/All/libgpg-error-1.10.tbz... Done.
pkg_add: warning: package 'libgpg-error-1.10' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
pkg_add: warning: package 'libgpg-error-1.10' requires 'gettext-0.18.1.1', but 'gettext-0.18.3.1' is installed
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/All/libgcrypt-1.5.0.tbz... Done.
pkg_add: warning: package 'libgcrypt-1.5.0' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
pkg_add: warning: package 'libgcrypt-1.5.0' requires 'gettext-0.18.1.1', but 'gettext-0.18.3.1' is installed
Fetching http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/All/libxslt-1.1.26_3.tbz... Done.
pkg_add: warning: package 'libxslt-1.1.26_3' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
pkg_add: warning: package 'libxslt-1.1.26_3' requires 'gettext-0.18.1.1', but 'gettext-0.18.3.1' is installed
pkg_add: warning: package 'xmlstarlet-1.0.5' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
pkg_add: warning: package 'xmlstarlet-1.0.5' requires 'gettext-0.18.1.1', but 'gettext-0.18.3.1' is installed

It seems to have installed xmlstarlet despite the warnings. But I'm new to this package manager so I'm not sure if this is asking for trouble or if it just installs and hopes that everything runs with different versions of the dependencies.

Soloam

@ppierre:

Nice script.

Juste an edge case that I've encounter. I've got this reply :
{"error":"port forwarding not available for this region"} 
Ewrything was fine, but in my setup I haven't setup the VPN as default route (only one IP go through it).

So I had to add this to your script :
curl [b]--interface [/b]"$LOCAL_IP" ...
And everything went smooth. It seem IPA need to get the port request from his own network.
Maybe you should add it to your post. In case someone would try a similar setup.

Hello, I'm having the same error

{"error":"port forwarding not available for this region"}

All runs well, the pia_port.txt is created but in the log I get that error.

Can any one help me? Using last version of the script.

Best Regards
Soloam

saytar

Not sure I can help as I don't have My VPN back up yet since my box crashed and I had to reflash and set it backup,,,But before the crash I did have PIA up and working on Pfsense 2.1.3.

I didn't have any trouble with it changing IP's when they switched around…my logs showed Pfsense detecting the change and just remaking the tunnel.....and I never noticed it............until I happened to check the logs.....

Their is a how to on line somewhere I went by to set it up. The setup was for ONLY the Client setup. I think maybe everyone is Making the Issue More complex than needed.

Check this link. http://www.bodenzord.com/archives/324
Maybe it will help

Chow

khyr0n

OMG This post is the Holy Grail I had to revive it! (And because I have a question)

Thanks a lot! I only changed
PORT=curl -d "user=$USERNAME&pass=$PASSWORD&client_id=$(cat $PIACLIENTID)&local_ip=$LOCAL_IP" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment

to

PORT=curl -k -d "user=$USERNAME&pass=$PASSWORD&client_id=$(cat $PIACLIENTID)&local_ip=$LOCAL_IP" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment

Because I was getting an error… Now it's fine!

BUT...

Would anyone have an idea on how to go about reading the pia_port.txt file from a remote windows machine running utorrent client???

Thanks again!

Soloam

Hello, yes I had to make that change to, something regarding certificates on the SSH.

Also I made a small change to Bagpuss pfsense script, to make it use a Alias with a port, instead of changing the rules. That way I can use the torrent port in other rules.

It assumes a port alias with the name "TorrentBoxPortPIA".


#!/bin/sh
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

# Private Internet Access Advanced Port Forward Script for pfSense
# v1.0 (21st January 2014)

# Pre-requisites for this script:
# pfSense v2.1 (Port forward NAT return destination broken in earlier versions)
# curl - pkg_add -r curl
# xmlstarlet - pkg_add -r xmlstarlet

# Add your PIA username and password
USERNAME=USER
PASSWORD=PASSWORD
PIACLIENTID=/cf/conf/pia_client_id
CONFFILE=/cf/conf/config.xml

# Check to see if we have a valid PIA Client ID file.
# If not, create one. Linux is included for illustration only.
if [ ! -e $PIACLIENTID ]; then

	# OSX/FreeBSD (pfSense)
	head -n 100 /dev/urandom | md5 > $PIACLIENTID

	# Linux
	#head -n 100 /dev/urandom | md5sum | tr -d " -" > $PIACLIENTID

	logger "pia-port: Created new PIA Client ID."
fi

# Find out the tunnelling device for your VPN and get your IP address.
# There are several options presented here. Personally, I prefer to use
# the interface which I know relates to my VPN tunnel for forwarding.

#DEVICE=`ifconfig | grep -o "tun[0-9]"`
#LOCAL_IP=`ifconfig $DEVICE | grep -Po "(?<=addr.)[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*"`
LOCAL_IP=`ifconfig ovpnc1 | grep "inet " | cut -d\  -f2`

# Get the port number for the forwarded port
PORT=`curl -d "user=$USERNAME&pass=$PASSWORD&client_id=$(cat $PIACLIENTID)&local_ip=$LOCAL_IP" -k https://www.privateinternetaccess.com/vpninfo/port_forward_assignment`

PORTNUM=`echo $PORT | grep -oE "[0-9]+"` 

# Some error detection. If PORTNUM is longer than 5 characters, we know that
# an error has been returned. We log it to syslog, and exit.
len=`expr $PORTNUM : '.*'`
echo $len

if [ $len -gt 5 ]; then
	logger "pia-port: $PORTNUM"
	exit 0
fi

logger "pia-port: Port number acquired: $PORTNUM"

# Get current NAT port number using xmlstarlet to parse the config file.
CURPORT=`xml sel -t -v '//alias[name="TorrentBoxPortPIA"]/address' $CONFFILE`

logger "pia-port: Current port forward: $CURPORT"

# The port mapping doesn't always change. 
# We don't want to force pfSense to re-read it's config if we don't need to.
if [ "$CURPORT" = "$PORTNUM" ]; then
	logger "pia-port: Port not changed. Exiting."
	exit 0
fi

# Port forward has changed, so we update the rules in the config file.
xml ed -u '//alias[name="TorrentBoxPortPIA"]/address' -v $PORTNUM $CONFFILE > /tmp/config.pia

# Put the config file in the correct location.
cp /tmp/config.pia $CONFFILE

# Create a file in the pfSense web server root that contains the current port.
# This can then be read by other hosts in order to update the open port in
# whatever torrent client is in use.
echo $PORTNUM > /usr/local/www/pia_port.txt

# Force pfSense to re-read it's config
rm /tmp/config.cache

logger "pia-port: New port number ($PORTNUM) inserted into config file."

Best Regards