In this video I try to test out the Icy Dock ToughArmor M.2 NVMe Mobile Rack for PCIe slots. It supports 2 removable NVMe drives placed in the included sleds, and (almost) fits an x8 or x16 lane PCIe slot that supports bifurcation to get 4 juicy lanes to each of the SSDs. I test it out with Samsung 990 Pro and Western Digital SN850X 4TB drives.

That being said, it turns out that not all bifurcation implementations are created equal, nor are all PCIe slots. Hence the frustrating aspect. Buy it with a generous return policy in place, if you buy it at all.

Just to be 100% clear: This is not sponsored in any way. If you couldn’t tell from the length of the video and the conclusion. But anyways, I paid for it out of pocket and bought it for my own purposes.

Oh yeah, at the very end there’s an update to my video on using the Dell EMC PowerEdge T640 as a backup “server” with 18x 20TB disks.

For home or low-demand virtualization, you can’t beat the Dell R905 for price. Consider that, as of the time I’m writing this post, you can get an R905 like I did with 128 GB of RAM for about $960. That’s only a little more than the cost of the RAM!

That machine has 4x AMD Opteron 8356 quad-core CPUs @ 2.3Ghz and two built-in 10gbps ethernet ports (plus 2 gigabit ports), a PERC6i controller, dual PSUs, and a full enterprise DRAC. It’s a little old, but you can’t get those specs in any other machine in a nearby price range. (OK, the R900 and the HP DL580 are very similar in spec and sometimes comparably priced).

I chose an R905 in part because it only has 4 big 120mm fans (in redundant pairs). Larger and fewer fans generally makes for a quieter machine, and sometimes this enterprise hardware can get awfully loud. I was really lucky when I got it racked and found that it was indeed very quiet. Small office quiet, even under moderate load and in a 79F degree room. (I think the fans were spinning somewhere around 2000 RPM).

It all fell apart though when I added a PERC6/E to support an HP MSA70 DAS to supplement the R905’s measly 8 2.5″ hot-swap drive bays. For some reason the BMC (which monitors and controls the fans, amongst other things) decided that the PERC6/E was the last straw and started running the fans at 5175 RPM constantly. And that is most certainly not quiet.

After assessing my options, I decided to modify the fans. I didn’t have a solid plan, but I figured that some 555 timers, a few capacitors and resistors could change the tachometer signal coming out of the fan to trick the BMC into thinking that the fan was spinning faster than it actually was. Of course, I didn’t want to make these modifications on the only four fans I had on hand — the four that were actually keeping the system cool — so I decided to buy another four from eBay. They were only $40 all-in!

One eBay listing in particular caught my eye. It said “Buyer may receive a fan produced by one of 6 different manufacturers (see picture below). But, no matter who the manufacturer is, the fan is guaranteed!!” The manufacturers included Delta, NMB, Nidec, and JMC/Datech. And interestingly enough, from the pictures it looked like the different brands had different blade configurations.

Images borrowed from the auction page of red_planet_trading on eBay

My system originally came with Nidec UltraFlo fans which have 5 steeply canted blades, an airflow rating of 250CFM and a max power draw of 6 Amps. The Delta PFC1212DE fans which I’d ordered had 7 blades with a more shallow cant, an airflow rating of 205CFM and a max power draw of 4.8 Amps:

The Delta fan I ordered. Picture is from the auction listing by steals_an_deals. They shipped quickly and the order went smoothly.

From the specifications (particularly the lower power draw), I surmised that the Delta fans might spin more slowly than the Nidecs to move the same amount of air. On the other hand, the lower CFM rating on the Deltas might mean that they would need to spin faster to keep up with the airflow of the Nidecs. I held off on ordering 555 timers until I could try out the new fans.

Happily, the Deltas are significantly quieter! When the server’s cover is taken off, the fans spool up to maximum RPM. As I hot-swapped each Nidec for a Delta, I could hear the sound decreasing. After putting the cover back on, the fans gradually throttled down and I monitored their speed through Dell OpenManage. When the Deltas hit ~5175 RPM (the speed at which the Nidecs had been running), they were significantly quieter. They eventually leveled off around 4500 RPM and a very tolerable volume (though not as low as prior to adding the PERC6/E).

The moral of the story is this: There are probably many OEM-approved fans out there for your server. Look into their specifications (many times the mfr. will publish sound levels at varying RPMs), and find the most “silent” ones. It’s not always worth it to risk messing up your cooling and burning out your system with custom mods!

Update: Another interesting note from all of this is that the fans made for Dell have 5 wires, whereas on the spec sheets for the generic version of the same fans they only have four wires. The four wires are for power, PWM control, tachometer, and ground. I don’t know why the Dells have the fifth. My guess is that it’s a sense lead so the BMC can identify the fan from amongst the different brands and models. If anyone knows the real answer, please leave a comment!

Background

I recently purchased a Dell BMX blade chassis on eBay that came loaded with ten 2x dual-core, 24GB PE1955 blades.   For a system with 40 cores, 240 GB of RAM and two 16-port gigabit switches in a 7U frame, $2500 was too good to pass up.  Especially since shipping cost me a mere 2 hour round-trip to Brooklyn.

I didn’t really need ten more servers, so I figured I’d sell off 5 or 6 of them (hopefully breaking even on the entire system — the 4GB FB-DIMMs in 6 of the servers is worth $1450 – $1900 alone).  The remaining servers would be all mine!

But I forgot to research the noise!  The chassis screams like a jet engine when powering up.  Even after it settles down to “idle”, with a couple of blades powered up the thing is still as loud as a decent-sized vacuum cleaner!

Research

I searched high and low for solutions to the fan noise.  Not a single search result discussed quieting this system for home use.   The only advice I could find was “don’t buy one”.  Heh.

I didn’t want to swap out fans with slower ones like this guy or this guy did on PowerEdge rack servers.   I don’t mind a little soldering, but there are two problems with that approach when dealing with the blade system:

1. There are 16 fans in the chassis (3 in each of the 4 PSUs, and 2 in each of the 2 fan modules).  That’s a lot of soldering, a lot of fans to buy, and a lot of things to potentially screw up.
2. More importantly, I’m concerned that quieter, slower RPM fans wouldn’t move enough air through the chassis during peak utilization and peak ambient temperatures.  I don’t want to find out the hard way when 5 of my production servers start shutting themselves down during the summer months.

My solution was to hack the DRAC/MC firmware.  The DRAC monitors fan speed and temperature, and I assumed it controlled the fans based upon various sensor readings.  I wanted to change the parameters for fan operation, for example by allowing the fans to run at minimal levels through a broader temperature range.

In fumphering around with a hex editor, looking for any reference to “fan” (hey, you gotta start somewhere), I found this:

So there was a way to control fan speed through the DRAC, without having to modify the firmware!  What I found seemed to be the text for a menu, but of course no such menu is exposed.  (I checked via serial, telnet, and web interface to no avail).

A little more searching, and I found this nugget:

Amongst the text for the “?” (or “help”) command in the DRAC/MC CLI, there was a command I’d never seen before in the help text itself, nor anywhere on Google:  fanctrl.[0, 1, 2 3]

Undocumented Commands

In addition to the fanctrl command, there are quite a few others that don’t appear in the normal help text.  I’ve listed the more intriguing ones below, along with my best guess as to their usage. There are a lot of detailed config commands that you can view in a slightly cleaned up version of the relevant raw data.

Don’t want to/can’t use the console for some reason?  Just go to http://[DRAC IP]/debug_command.htm after logging in and you can execute any command from there and see the output.

Conclusion

My goal was to reduce fan speed  in order to make the Dell blade system usable in my open basement.  Sadly, even with the most throttled of speed that fanctrl has to offer, it’s still too loud for my (and more importantly the wife’s) taste.  The entire system is going up for sale on eBay, minus the chassis.  It’s too expensive to ship it via regular carrier, and lets just say that I have a Visio layout for a basement data room lying around.

If you have a dedicated server room or closet, a Dell 1855/1955 blade system may be the right choice for you for home use so long as you tweak the fanctrl settings and don’t let the system heat up too much.

To the best of my ears’ reckoning, an “untweaked” chassis with 5 powered-on blades makes noise equivalent to a leaf blower with some fans running as low as ~5000 RPM and others at ~10,000 RPM.  On setting “0” (ultra-low, if I’m correct) it’s just a little bit louder than my 10 year-old dryer with some fans at ~2500 RPM and the fastest ones at ~7500 RPM.

If you are going to replace fans, the small ones at the front of the PSUs are generally the fastest, smallest, and loudest.  Unfortunately, there are still 8 of them.  And if any one of them falls below RPM threshold, all the other fans will speed up to compensate.