[SOLVED] How to auto download backup file from pfsense to local folder via SCP?

[anonymous01]

I am running a virtual pfsense 2.4 on virtualbox. I have enabled ssh and able to log into pfsense using WinSCP. I also installed the cron service on my pfsense.

Since I already enabled the Auto Config Backup feature, I wanted to transfer the backup files from the pfsense to my local pc desktop every single minute. Everything runs on root for testing purposes.

Questions:
1. Any idea how to write the script for this and where do I upload my script to specificall to which path or file?
2. How to configure cron? and where to configure?

Here's my script so far
Source (192.168.13.102 is my pfsense wan ip)
Destination (C:\Users\aria\Desktop\pfSense is the folder i created on desktop) :

#!/bin/bash
scp -r admin@192.168.13.102:/cf/conf/backup C:\Users\aria\Desktop\pfSense

Not too sure how to write the scripts....

The Cron services i set up on pfsense has a schedule to run and execute command every minute every day.... but im not too sure what to write in the command section that is available on the gui...

Sorry I am a total beginner, might not understand how firewalls and machine works

[Gertjan]

Check this : https://github.com/KoenZomers/pfSenseBackup

A backup every minute ?

[anonymous01]

not really every minute, but if possible, every time there is a new configuration made a back up version will be made. I know this has been automatically done by the ACB feature provided by pfsense.... I did check the link you provided, but i am still clueless as to where to go from there.

My main objectives:
I now wanted to copy the back up files from the pfsense to my local machine.

The overall research i have found so far seems to tell me to do as follow:

1. Set up ssh, provide a ssh key to the user that performs this task, in this case the admin.
2. Create a bash script to allow file transfer from pfsense to local folder.
3. Enable Cron Job on pfsense for scheduling.
4. Configuration.xml file will now always be saved in your local folder and pfsense config file as well. Smtg like a duplication from the pfsense.

This copy and file transfer actions are all via WinSCP

[Gertjan]

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

I now wanted to copy the back up files from the pfsense to my local machine.

That what the tools does.
That's what I'm using for the last 2 years or so :
It's a Windows program, executes from your PC, using a Windows "cron task" (scheduled task) executes the program every day or so, whenever you want actually, and saves the file in a folder on pour PC.

Btw : clueless ? : your saw the instructions at the bottom of https://github.com/KoenZomers/pfSenseBackup ??

About :

up ssh, provide a ssh key to the user that performs this task, in this case the admin.

You always have to set up SSH access - or have a direct console access if your ppfSEnse is close to you. SSH is actually not some option. It's the most important access pfSense has.

Create a bash script to allow file transfer from pfsense to local folder.

That is what the program is /does ...

Enable Cron Job on pfsense for scheduling.

NOOP.
Schedule the program on your local device so it connect to pfSense as a client via SSH, and it will download the config.xml

Configuration.xml file will now always be saved in your local folder and pfsense config file as well.

Who is Configuration.xml ?
You'll be downloading a file called :

[stephenw10]

See: https://docs.netgate.com/pfsense/en/latest/backup/remote-config-backup.html

[anonymous01]

@Gertjan sorry for the late reply...yes it says that my ssl cert could not be established when running the exe file.

I tried to create one for my firewall, however, due to the administrator restriction, I could not add my firewall ssl cert to the windows certificate manager.

Could you advise on this if there is other solution to enable this?
ref for ssl cert: https://www.ceos3c.com/pfsense/pfsense-generate-ssl-certificate-https-pfsense/

Hence, I am wondering if I should just create a bash script of my own and upload it to scp. But it doesn't seem to be working even referencing other people's script and modify to my own. Maybe I did something wrong but I could not figure out where.

Thank you for assisting tho

[anonymous01]

@stephenw10
Yes I did read through this. I assume I only needed the "push it" method.

I am not too sure how to do this step:
Create a cron job on the pfSense firewall that would copy /cf/conf/config.xml to the remote system with scp.

For the curl and wget, I can't seem to get it working, better yet, and sorry for this stupid question, where do I run and type these curl or wget code on? Just need a confirmation.

[anonymous01]

Okay here's what I got so far
My script (test01.sh) :
#!/bin/sh
scp -r root@192.168.13.100:/cf/conf/backup C:\Users\aria\Desktop\pfSense

I saved this test01.sh script on the /bin/ directory.
I get it to run using ./ command on the pfsense shell interface, but it still prompt me to enter the password of my pfsense. Regardless, after i entered the password, it just freeze. No error message, nothing...until i ctrl c to escape process.

Assuming it works, on the pfsense gui, do i set up a schedule on the cron services? Then at the command insert /bin/test01.sh?

If i just run the
scp -r root@192.168.13.100:/cf/conf/backup C:\Users\aria\Desktop\pfSense
on my windows cmd, it works perfectly fine, file transfer is successful. However, i do not want to change my script to powershell and set up a task scheduler as all of this would be redundant.

I want to run it on pfsense by pushing the config to my local. Not pulling the config. Sorry i do not understand the logic behind these.

Btw, my pfsense system>advanced>secure shell doesnt seem to have an option to disable the password login for ssh

[Gertjan]

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

where do I run and type these curl or wget code on?

You saw this :

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

ref for ssl cert: https://www.ceos3c.com/pfsense/pfsense-generate-ssl-certificate-https-pfsense/

in that arctice, where the gys spoke about some "back door", hje was talking about the console or ssh access. It's not a back door, but actually an important admin access. You'll be having a command line access.

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

Create a cron job on the pfSense firewal

It already exists, but you can't "see" it in the GUI (although, type "cron -e" on the command line, as any other FreeBSD/Linux device on the planet) and you see it. You'll be needing the .... Cron pfSense package to see and edit it in the GUI.

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

doesnt seem to have an option to disable the password login for ssh

That would be a huge security hole.

[stephenw10]

You can use SSH key instead of password login which is much easier to script if you are pulling the files from somewhere else.

More info about pushing the files with SCP is here:
https://docs.netgate.com/pfsense/en/latest/book/backup/alternate-remote-backup-techniques.html#push-with-scp

Steve

[anonymous01]

@stephenw10 sorry yes i could ssh successfully now.

I created the above script, tried with the #!/bin/bash and #!/bin/sh header

But pfsense cant seem to read this script, i either get permission denied or failed to connect to pfsense.
(error code 3 and 127 from winscp)

Winscp is login as root, chmod 777 the script, tried moving the script to /root/, to /usr/bin/, to /bin/ but still cant seem to execute it.

Both admin and user privilege has got the scp related privilege.

[Gertjan]

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

the scrip

How and where did you create this script ?
What do you see when you type

file script

where script is the script name.

[anonymous01]

@Gertjan

I created the script on notepad ++ and saved it as unix script file type specifically . sh
Then i insert this script onto my remote site which is log in as pfsense root on winscp.
Double check that my backup config is saved at /cf/conf/backup directory

my script file name as testbckup.sh

192.168.13.100 is my pfsense ip
192.168.13.10 is my local machine ip
pfsense is the local folder i wanted my config to saved at

My script basically looks like this

#! /bin/sh
scp - r root@192.168.13.100:/cf/conf/backup C:/Users/aria/Desktop/pfsense

What i hv tried and changed:

1. scp - r root@192.168.13.100:/cf/conf/backup aria@myuserdomain:/Users/aria/Desktop/pfsense

2. #! /bin/bash

3. #! /usr/bin/sh (with & without sh)

4. scp - r root@192.168.13.100:/cf/conf/backup aria@192.168.13.10:/Users/aria/Desktop/pfsense

....
Is my first time writing bash and scp stuff, pls bear with me

[anonymous01]

@Gertjan

What do you see when you type

Sorry for being stupid but what do you mean by this?

And for the cron gui, yes i am aware i hv to download that package. It is available in my pfsense gui.

[stephenw10]

So you are running that script/command on another host and it's copying the full backup folder from pfSense to a third host?

Does it work if you run it directly from the command line on that client?

Steve

[Gertjan]

@anonymous01 said in How to auto download backup file from pfsense to local folder via SCP?:

Sorry for being stupid but what do you mean by this?

file script

is a command to be typed on the command line.
'script' is the name of your script.

Example :

I ghave this script file called "script" :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: ls -al script
-rw-r--r--  1 root  wheel  1121 Feb 25 15:55 script

As you can see, it's not executable - no problem, we can handle that :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: chmod 0744 script
[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: ls -al script
-rwxr--r--  1 root  wheel  1121 Feb 25 15:55 script

Note : with a decent ssh client, the script file name changed color.

When I try to execute the script file :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: ./script
./script: Command not found.

it won't work.

This is the script file :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: cat script
#!/bin/sh
# https://forum.pfsense.org/index.php?topic=134352.msg737158#msg737158
#make sure the directory for the python libraries is in the chroot
mkdir -p /var/unbound/usr/local/lib/python2.7
#link the actual python library directory to the chroot's directory
mount -t nullfs /usr/local/lib/python2.7 /var/unbound/usr/local/lib/python2.7
#copy the python script to the /var/unbound directory so
#unbound-checkconf can find it
rm -f /var/unbound/var/unbound/netflix-no-aaaa.py
cp /root/netflix-no-aaaa.py /var/unbound/netflix-no-aaaa.py
#make sure unbound can read it
chown unbound:unbound /var/unbound/netflix-no-aaaa.py
#create a /var/unbound directory in the /var/unbound directory so that
#unbound can find the script
mkdir -p /var/unbound/var/unbound
#copy the python module into the /var/unbound/var/unbound directory under the chroot #directory
rm -f /var/unbound/var/unbound/netflix-no-aaaa.py
ln -s /var/unbound/netflix-no-aaaa.py /var/unbound/var/unbound/netflix-no-aaaa.py
#make sure unbound can read it
# bla bla bla bla ..........

I check what type of file it is :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: file script
script: POSIX shell script, ASCII text executable, with CRLF line terminators

And now you know that something is wrong - at least with my example, my script file.
It's Windows (CRLF) encoded. Not Unix.
Welcome in our world, where every OS uses it's own ASCI/ANSI file encoding scheme.

A script file that works would show this as a result :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: file unbound-p.sh
unbound-p.sh: POSIX shell script, ASCII text executable

[anonymous01]

@stephenw10
since my script only consist of one command line which is the scp...

Remote site: pfSense
Local: Windows host

I just manually type into my local windows cmd and it works! The backup config is successfully transfered from pfSense to my window host.

However when I tried to upload this script on pfSense...it couldnt execute the script.

[stephenw10]

If you're running it from pfSense itself though the scp line will be different. Something like:

scp /cf/conf/backup aria@192.168.13.10:/Users/aria/Desktop/pfsense

As long as the target has the pfSense public key registered for the aria user that should work.

Steve

[anonymous01]

@Gertjan

Thank you for explaining!! However I checked but it seems like it is executable, I think either the issue is reside either on the ssh I set up or winSCP

Running this on putty, I have you know once again I wrote this on notepad++ and saved it as unix like file

Script name and overall properties:

Script content:

Script file type:

Just in case, it is my first time setting up ssh key so here is how i set up.

1. Pfsense System Advanced: enabled ssh, tried enabling and disabling agent forwarding, ssh set as key pw or public key.
2. User Manager Admin: uses SSH-2 public key
3. User Manager User: uses SSH-2 public key (same public key as admin)
4. Putty and WinSCP: Uses private key generated from the same session as public key.
5. No passphrase is used in this occasion for testing purposes

[anonymous01]

@stephenw10

scp /cf/conf/backup aria@192.168.13.10:/Users/aria/Desktop/pfsense

I tried this. so do I just type this command on the cron interface or at the pfsense command prompt ? Or still make it as a .sh script and upload it?

If it is the latter, again it is frozen after changing command.

winscp:

//either that or it keeps showing this error:
ssh: connect to host 192.168.13.10 port 22: Connection refused lost connection

for how i set up ssh pls refer to my second latest reply.
I am starting to think if it is my office firewall blocking the downloads. As for my own host, i hv disabled all firewall.