Drive Space Incorrect Raid 5

chrisfox75

So i have been running pfSense for a little while now about a month and just noticed when i do a df -h for disk free i do not see all of my disk space that was formatted and partitioned. I tested this by reinstalling pfsense in another box and that box had the same issue and reported the SAME space as the current box i have running pfSense 2.4.5 patch 1.

I am running a Dell R720 with 4 - 3tb drives in a Raid 5 array which should format out to about 7-8 tb after the 1 drive overhead for raid 5 but the system is only reporting 1.8 TB free. The other system i built had 4 - 4tb drives in it and reported the exact same amount of space free

Any ideas why Freebsd 11.3 is reporting this way and is there a patch or a way to fix it??

Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/5eab42b8cd102c13    1.9T    1.8G    1.8T     0%    /
devfs                          1.0K    1.0K      0B   100%    /dev
/dev/md0                       3.4M    128K    3.0M     4%    /var/run
devfs                          1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                          1.0K    1.0K      0B   100%    /cf/named/dev

Interestingly enough everything works fine and i thought it might be a problem with the drives thats why i tried another system to see if it was different. Just a basic Dell R720 with a Perc 710 Mini raid controller card. The drives report fine in the Perc Controller, when i installed pfsense and partitioned the drives they reported correctly but the system is just not reporting the correct size for the raid array.

Any suggestions?
Thanks
Chris Fox

DaddyGo

@chrisfox75

Hi,

how did you set up RAID 5?

by H710 RAID setup? (Press CTRL-R during boot)
or with the pfSense installer? (FreeBSD)

What do you want to achieve with TBs on pfSense?

it is absolutely unnecessary, because:
https://docs.netgate.com/pfsense/en/latest/book/hardware/minimum-hardware-requirements.html

in most cases we use 16GB / 32GB SSDs (unfortunately they are no longer available), so they remained 64 /128 vagy 256 GB

RAID is not absolutely necessary, unless it is a high-availability environment...

by the way, if you are already running such a powerful machine for pfSense
why not configured with ZFS RAID? (via pfSense installer)

Note:

the hardware (H710) is supported under FreeBSD 11.3, this should not be a problem
pfSense is not a database server....

+++++edit:
suggestion, visit here and perform tests:

johnpoz

@DaddyGo said in Drive Space Incorrect Raid 5:

What do you want to achieve with TBs on pfSense?

That is a very good question - curious minds want to know ;)

When you install pfsense it will create the partition on your disk/array for its use.. This is what df would show.. That partition size.

stephenw10

The output of geom disk list might be more useful here.

chrisfox75

@stephenw10

This is interesting geom disk list reports the correct size even though df -h does not.

Geom name: mfid0
Providers:
1. Name: mfid0
   Mediasize: 9000103968768 (8.2T)
   Sectorsize: 512
   Mode: r2w2e7
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

DaddyGo

@chrisfox75

because everything is fine with the disks / array
take this into account @johnpoz "When you install pfsense it will create the partition on your disk/array for its use.. This is what df would show.. That partition size." !!!!

chrisfox75

The partition size on installation was the full size of the array when installed so why the discrepency.. As i wrote in my first post, this happened 2 different times 2 different machines and different drives. Partition size reflects array size but DF -H does not report the correct partition size.

DaddyGo

@chrisfox75

then again, back here:

"how did you set up RAID 5?

by H710 RAID setup? (Press CTRL-R during boot)
or with the pfSense installer? (FreeBSD)"

chrisfox75

@DaddyGo said in Drive Space Incorrect Raid 5:

H710 RAID setup?

H710 RAID setup

chrisfox75

Advanced Properties
Status	 
Name	Virtual Disk 0
Device Description	Virtual Disk 0 on Integrated RAID Controller 1
State	Online
Layout	RAID-5
Size	8382.00 GB
Span Count	1
Block Size	512 bytes
Bus Protocol	SAS
Media Type	HDD
Operational State	Not Applicable
Read Policy	Adaptive Read Ahead
Write Policy	Write Back
Stripe Size	64K
Disk Cache Policy	Default
Enhanced Cache	Not Applicable
Progress	Not Applicable
Bad Blocks Found	No
Secured	No
Remaining Redundancy	1
T10 PI Status	Disabled
Controller	PERC H710 Mini (Embedded)
View Physical Disks

DaddyGo

@chrisfox75

it depends on a hardware-based RAID and Dell card
configure ZFS arrays (RAID 5) with pfSense installer

+++++edit:

genuine Dell stuff is not supported under FreeBSD - by Dell
do you see FreeBSD OP system somewhere here?

chrisfox75

@DaddyGo

My Perc H710 does not have IT mode enabled its running as a dell card, i dont see a benefit in running ZFS software raid when the hardware raid runs just fine

I wonder if i went wrong was with the partition scheme during setup

I used MBR - Dos Partitions not sure how that would matter unless there is a size limit on the partition that i dont know about and really what option should be used? But after selecting MBR i did see all of the partitions and their correct sizes that they should have been, if the array is reporting correctly why isnt the OS reporting the same

Someone else has had to run into this before. Doubt it has anything to do with pfSense, its all FreeBSD

chrisfox75

@DaddyGo said in Drive Space Incorrect Raid 5:

@chrisfox75

it depends on a hardware-based RAID and Dell card
configure ZFS arrays (RAID 5) with pfSense installer

+++++edit:

genuine Dell stuff is not supported under FreeBSD - by Dell
do you see FreeBSD OP system somewhere here?

If FreeBSD is not compatible then tell me why for one it is running and why you just said

" It depends on a hardware-based RAID and Dell card
configure ZFS arrays (RAID 5) with pfSense installer "

So now your saying if i reconfigure the partitions as ZFS it will report correctly and run fine when your paste of the controller said the opposite?

DaddyGo

@chrisfox75

We run Dell R210II hardwares (3 pcs.) with PERC H200 in native mode...

the ZFS is a high-security file system, that’s a fact
therefore, we use RAID 5 mode on these pfSense devices

and yes we configured the RAID mode with the pfSense installer
works fantastic

BTW:
FreeBSD supports this card (H710)
otherwise, Dell will not address FreeBSD compatibility

I think no one in this world...
have not yet installed TBs (gigamegaterrepeta) of HDD / SSDs for pfSense
pointless and damn unsafe
large hard drives consume a lot of power

this is an NGFW / router!

chrisfox75

So basically if i dont conform and rebuild this box using ZFS this thing will never report the correct drive space to the OS. Obviously something is just strange here..

DaddyGo

@chrisfox75

Juppp,

okay, I don't think anyone can answer that for you
because no one makes such a crazy configuration
everyone here deals with the basic (advanced) operation of the firewall and gives advice

so experiment as described above

I’ve worked with Dell a lot in the past, so my opinion is that the issue here is this hardware RAID software RAID

if you want to be absolutely sure, throw up a FreeBSD 12 ..... on this "iron" and see what happens

I don't think so(!!!!!), but it could also be that pfSense itself doesn't see it anymore.....TBs
because it makes no sense, anyway

chrisfox75

@DaddyGo

I think no one in this world...
have not yet installed TBs (gigamegaterrepeta) of HDD / SSDs for pfSense
pointless and damn unsafe
large hard drives consume a lot of power

this is an NGFW / router!

I have built numerous Checkpoint Firewalls over the years running TB's of storage for customers, and for at the time the largest ISP in the world UUNET before MCI Worldcom bought them and everything ran great for Gigabytes and Terabytes of storage so i wouldnt say nobody in this world has done it, and im sure im not the only one running a Dell R720 with a few TB of storage on the unit, you work with the hardware you have and make the best out of it, im not planning on going out and buying a brand new server with a few gigs of storage for a firewall/router and most people understand that.

I appreciate your help and direction but could do without the direction of the best way you think it should be done when everyone has their own, and besides why im doing it this way was completely irrelevant to the initial question i posted as to why the OS is reporting the wrong size for the partitions

First by posting this that was not helpful at all

"genuine Dell stuff is not supported under FreeBSD - by Dell
do you see FreeBSD OP system somewhere here?"

and then stating

"BTW:
FreeBSD supports this card (H710)
otherwise, Dell will not address FreeBSD compatibility"

Which completely through me for a loop not understanding if it was or it wasnt. Just cause it works sometimes doesnt mean its completely compatible.

Again, the help is appreciated, after all the confusion i found a site that explained the whole process to me regarding partitions on FreeBSD then what i got outta being confused.

Being nice goes a long way with others..
Thanks for your assistance @DaddyGo
Have a blessed day.

DaddyGo

@chrisfox75 said in Drive Space Incorrect Raid 5:

Being nice goes a long way with others..

allow me one last word - my friend!

you probably misunderstand me, because of my shitty english (sorry, I speak three languages and English is not the primary)

I dissolve the confusion pictures in your head:

"BTW:
FreeBSD supports this card (H710) =
https://www.freebsd.org/cgi/man.cgi?query=mrsas&sektion=4

"genuine Dell stuff is not supported under FreeBSD - by Dell
do you see FreeBSD OP system somewhere here?" =

BTW:
I still don't understand what TBs storage capacity is needed on pfSense ???

Have a nice day!

stephenw10

Try geom part list.

It looks like mounted slice just isn't any bigger that 2TB.

Steve