PfSense Hyper-V .vhdx growing like crazy

[MrChris]

So I created a Hyper-V based PfSense fw and its working really well. But im seeing that the VHDX file is growing rapidly faster than I would have expected. Even if I "Compact or Optomize" the VHD. The size never lowers. It started out as around 5 or so GB then about a week later its now 85GB'ish. I thought it might be logs or bandwidth reports or something but doesnt look like those take up nearly that much space. The VHD is dynamic with a max size of I think 120GB. Iv worked with VHDs for years and I have a decent understanding of how they work. but dang. Where is all the space going?

[2.5.2-RELEASE][root@pfsense.lan]/: df -h
Filesystem              Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default    120G    902M    119G     1%    /
devfs                   1.0K    1.0K      0B   100%    /dev
pfSense/tmp             119G    316K    119G     0%    /tmp
pfSense/var             119G    5.4M    119G     0%    /var
pfSense/home            119G    108K    119G     0%    /home
pfSense/var/log         119G    744K    119G     0%    /var/log
pfSense/cf              119G     96K    119G     0%    /cf
pfSense                 119G     96K    119G     0%    /pfSense
pfSense/var/empty       119G     96K    119G     0%    /var/empty
pfSense/var/db          119G    9.6M    119G     0%    /var/db
pfSense/var/cache       119G    120K    119G     0%    /var/cache
pfSense/cf/conf         119G    2.6M    119G     0%    /cf/conf
pfSense/var/tmp         119G    144K    119G     0%    /var/tmp
/dev/md0                3.4M    136K    3.0M     4%    /var/run
devfs                   1.0K    1.0K      0B   100%    /var/dhcpd/dev

Thanks in advance for any feedback!

~MC

[Bob.Dig]

@mrchris Mine is two month old and is 10 Gigs in size. But I delete the logs almost daily.

1% of 119GiB - ufs

Filesystem                                         Size    Used   Avail Capacity  Mounted on
/dev/gptid/...                                     119G    1.5G    108G     1%    /
devfs                                              1.0K    1.0K      0B   100%    /dev
/dev/md0                                           3.4M    200K    2.9M     6%    /var/run
/lib                                               119G    1.5G    108G     1%    /var/unbound/lib
devfs                                              1.0K    1.0K      0B   100%    /var/unbound/dev
/var/log/pfblockerng                               119G    1.5G    108G     1%    /var/unbound/var/log/pfblockerng
/usr/local/share/GeoIP                             119G    1.5G    108G     1%    /var/unbound/usr/local/share/GeoIP
/usr/local/bin                                     119G    1.5G    108G     1%    /var/unbound/usr/local/bin
/usr/local/lib                                     119G    1.5G    108G     1%    /var/unbound/usr/local/lib
devfs                                              1.0K    1.0K      0B   100%    /var/dhcpd/dev

Spoiler

[KOM]

@mrchris What are you looking at that makes you believe you're missing this space? Your df output shows only 1% of your disk is used.

[MrChris]

@kom
When I view the actual VHDX file (pfsense harddrive image) on my Windows Hyper-V Host. Its 85GB in size. About a week ago after I was all done setting it up and getting it running the way I wanted, the VHDX was about 5GB or so which was perfectly fine. After sitting running for about a week with little to no changes to the config the VHDX has grown way more than I would have expected.

~MC

[MrChris]

@kom

TBH I dont think its a file in the FreeBSD image as much as the VHDX went rouge or something. I mounted the VHD on another linux host and browsed around it and dont see any crazy large files.

[KOM]

@mrchris I'm a VMware guy so I don't have anything to say about your HypverV config. Maybe their changed-block tracking took a dump or got confused. You could try installing to a new VM and then restoring your config. What's important is that pfSense doesn't think it has a problem, so at worst you're out 120G of space on your SAN.

[MrChris]

@kom

As much as I wish this is a SAN, its not :). Its a pretty beefy home server with lots of memory and cpu. my home router took a dump and all my other arm based routers cant actually handle gig up/down and have herd lots of good stuff about pfsense. All my old boxes didnt support AES or were too old. So I decided to play with it and see if I could build a PfSense Home Router/Firewall in a VM while utilizing 2 of the many NICs I have on the server while handling GB up and GB down on WAN/LAN. After a week or so of fumbling around. I did it. yay me. I make regular webui config backups. but also like to get backups of the OS Image too. but with a VHDX going rouge that makes it difficult. I have lots of space to work with but not that much to waste.

The Host is a Windows host and I like to keep it that way at all odds. I may give VMware a play with and see whats up. Or maybe rebuild a new hyper-v image and restore config etc..

I mainly wanted to check if anyone else has seen this behavior with pfsense in a hyper-v setup. I have experience with linux but never really played with FreeBSD. Im more a debian person myself! :)

Other than that I am impressed with PfSense. I like it and kinda wanna keep it in my mix if possible.

Thanks for the feedback!

~MC

[serbus]

Hello!

I have seen this on *nix running in hyperv with dynamic vhdx files. Check the blocksize with get-vhd. You might see this wild growth with some guest filesystems if your vhdx is using the default blocksize of 32M. Optimize-vhd can have mixed results trying to compact these filesystems. Using a smaller blocksizebytes, like 1M, when creating a dynamic vhdx can slow, but not eliminate, the excessive growth. I have defaulted to just using fixed size vhdx files in most cases and not messing around with the maintenance or the inevitable wasted space.

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/best-practices-for-running-linux-on-hyper-v

John

[SteveITS]

@mrchris Unless you need the space you could just create a smaller disk. Netgate appliances have as little as 8 GB, and there's not much need for more without something like Squid caching or large log files. Can't get bigger than what it is. :)

[MrChris]

@serbus
good point! I didnt think of that.

@SteveITS
This makes sense as it doesnt look like its a file on the pfsense filesystem that is growing and eating space as much as the dynamic vhdx itsself. I will see about creating a new 8GB flat vhdx and seeing how it behaves. I don't need any crazy caching or anything that I know of. Thanks for the info folks. much appreciated.

~MC

[provels]

You can see my VM specs in my sig, on 2012R2. No issues.

[2.5.2-RELEASE][root@fw.workgroup]/: df -h
Filesystem              Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default    5.0G    823M    4.2G    16%    /
devfs                   1.0K    1.0K      0B   100%    /dev
pfSense/tmp             4.2G    552K    4.2G     0%    /tmp
pfSense/cf              4.2G     96K    4.2G     0%    /cf
pfSense/var             4.2G     17M    4.2G     0%    /var
pfSense                 4.2G     96K    4.2G     0%    /pfSense
pfSense/home            4.2G    128K    4.2G     0%    /home
pfSense/cf/conf         4.2G    8.9M    4.2G     0%    /cf/conf
pfSense/var/cache       4.4G    213M    4.2G     5%    /var/cache
pfSense/var/log         4.2G    1.2M    4.2G     0%    /var/log
pfSense/var/db          4.3G    129M    4.2G     3%    /var/db
pfSense/var/empty       4.2G     96K    4.2G     0%    /var/empty
pfSense/var/tmp         4.2G    120K    4.2G     0%    /var/tmp
/dev/md0                3.4M    132K    3.0M     4%    /var/run
devfs                   1.0K    1.0K      0B   100%    /var/dhcpd/dev

[LesserBloops]

I hit this when I tried using ZFS as the file system with expanding disks. It went runaway and eventually caused an outage.

I've been meaning to go back and try again with a fixed size disk instead of an expanding one, but haven't got around to it yet, and EXT seems to be fine...