Yes, that’s a clickbait title. But technically it’s possible if I dropped all drive redundancy…
I recently saw an advert for a server that was just too good to be true. It promised to bring a huge amount of storage and future options in a quite hard-to-believe 1U. The price it was on offer for was also unreal. I wasn’t looking for a new storage server but knew I would be able to make good use of this one. It would allow me to consolidate a few bits and pieces so it just had to be done….
Table of Contents
Intro
The server is a D51PH-1ULH from a vendor called Quanta Cloud Technology. They are a vendor that doesn’t seem to be very well known outside of the HyperScalers. It’s described as an Ultra dense storage server and with the capacity for 12×3.5″ plus 4×2.5″ drives it’s hard to argue with that. It also comes with IPMIv2.0 and 2x10Gb SFP+ connections. One really interesting thing is that at first glance the drives don’t look like they are hot-swap. However, the server comes with a giant hard drive tray that slides out the front and then individual drives can be swapped.
The seller also threw in a few 480GB Samsung Enterprise SSDs and 16x8TB Seagate Exos making it the bargain of the century. This allowed me to fully populate the chassis and still have 4 8TB drives spare. I typically don’t like using second hard storage but in this instance, I was willing to make an exception.
OS Setup
I have stuck with TrueNAS scale as the OS of choice for my dedicated storage systems. It’s something I’m familiar with and served my needs in the past well. However, I have decided to leave this as primarily a pure “Storage Server”. The available storage will be served out as a mixture of iSCSI, NFS, and SMB natively by TrueNAS and then I will run a MINIO container on top to offer S3-based object services.
Hardware
I swapped out the 480GB Samsung drives for some larger 2TB consumer-based drives. The current configuration is listed below.
The servers also included rails which was a much appreciated benefit. The only downside is I underestimated just how long these servers are. The good news is I can adjust my Startech rack to make it longer and accommodate this. The downside is I need to derack the rest of my gear to do it. So for now it’s sitting on top of one of my other Supermicro servers.
| CPU | 2x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz |
| Memory | 256 DDR4 |
| Boot Device | 80GB Intel SSD |
| HDD | 12x ST8000NM0075 8TB 12Gb SAS |
| SDD | 3x2TB Samsung Evo |
| NIC | 2x 10Gb |
Pool Configuration
For performance reasons, I have chosen to go with 6x Mirrored vDevs 2x wide. Conceptually this is similar to RAID 10. This uses all 12 of the 3.5-inch drives in a single pool. This gives a usable pool capacity of just under 43.5TB
Although I lose 50% of the usable capacity I wanted the write performance that this gives
This is then complemented by three 2TB SSDs being used as L2ARC (Read cache) and as I have so much read cache the drives basically just do writes.
Performance
I haven’t bothered benchmarking the setup from a performance perspective as I have actual workloads on it that would be annoying to move. However, from what I am seeing in the latency graphs of vCentre I am happy with the results.
As you can see below the ARC is holding all of the really hot data in RAM.
The warm data is being served from the 6TB of SSD in the server, however it has not finished warming up yet. The L2ARC data can also be compressed so the reality is that it’s probably closer to the equivalent of 12TB.
A benefit of TrueNAS Scale is that the L2ARC persists between reboots. Otherwise, it would take a very long time to finish warming up with my workload.
This means the performance graphs of the actual disks tend to look like this. Only two are shown but all of the main drives appear like this. Yes, there are some reads (Blue) but the majority of the time they are only doing writes ( Pink) for the system.
Future Upgrades
There are a couple of upgrades I would like to add however they seem difficult to source.
The first is a dual port 25GbE Nic. As this is a Mezz card a standard Pcie device will not do.
3GS5BMA0000 Quanta ConnectX-4 LX Dual Port 25GbE SFP28 OCP Mezzanine NIC.
The second is a SataDOM to use as the TrueNAS install drive. This would allow me to utilize an extra 2.5 inch drive for IO purposes
DOM: SD134C-128GM-PT-A
