In this video I create a new backup server from a Dell EMC PowerEdge T640 with 18x 20TB Seagate Exos refurbished drives. Also there’s some more ranting about backups in general. But idk, there are chapters so you can skip to whatever. And it’s not technically a server, in that it doesn’t serve files. It’s really more of a client that takes files from other servers and holds onto them. But I call it a backup server because hardware-wise it’s a server. Anywho, that’s the description. It’s probably not optimal for the YouTube algorithm, but hey, I’m not an influencer even though I have a TikTok account that I don’t use.

For posterity, here’s a transcript of the video in case you’d rather read than listen. But beware! It’s as transcribed by YouTube and so may include some errors and lack of punctuation (and etc).

hi everybody I’m Scott and this is a poweredge t640 with an 18 drive capacity up front uh you can see it’s got space for six up here and then some blanks for two rows of six more below that this is going to be my new backup server this monitor shows that it’s been under test for this is bad to look at one second well it’s still dim but yeah it’s been under test for almost 24 hours and because this is an eBay special and by special I mean it’s just fine it’s not like I don’t mean that in negative way got off eBay because I’m you know it’s a little bit of an older machine I think the t640 has been around for a while but this particular model is from 2021 so as of recording this it’s only about 3 years old which actually makes it one of the newer systems I’ve had down here in a while the reason it’s new isn’t because it’s particularly great it’s only with silver CPUs I think 8 core 2.1 GHz the CPUs aren’t important because this thing is not going to be doing anything CPU intensive if you couldn’t guess from the front of it it’s going to be doing something dis intensive and that is backing up because I said it’s a backup server and what do I mean by that it’s basically just going to be a file server only it’s not going to be accessible from the outside it is only going to pull files into it from other servers for the purposes of backups and it’s also I’m not going to reside in my basement I have a closet upstairs on the second floor of my house that currently has two computers and a disc array yeah there it is uh kind of teetering on top of one of the computers and those are both obviously homebuilt machines they’re nothing fancy whatsoever spec-wise in fact they’re pretty awful but they just have a ton of discs not enough discs though and that is the reason for this guy because one of them is completely full on space and the other one’s getting close yeah as of right now I have some video F that aren’t even getting backed up which I don’t like this guy will be paired with 18 drives oh I should get the drives most important part of the project is the [ __ ] drives but I forgot them these are the aforementioned drives right here well it’s a box these drives were another eBay purchase and right here we have 400 terabytes of raw storage in the form of these see gate exos I’m assuming that’s how you pronounce that x22 drives 20 tab a piece and there are 20 of them as I said before this is an 18 Bay chassis so obviously it’s two extra drives but that’s good because these were sold as refurbished in other words used I don’t think they’re manufacturer refurbished Al they do look like they’re in really good condition and I got them obviously much cheaper than new drives of this capacity so while it’s a risk I visited a well visited that sounds weird I found a couple of posts on Reddit about people who bought these drives or similar drives from the same seller who sells them in bulk and they all had good experiences this was on groups like uh serve the home and some other Enthusiast groups let’s say who generally know what they’re talking about and probably wouldn’t spam that for no reason so yeah hopefully these are okay I’m going to test them out obviously before I put them in production this is for a home backup server that’s why I put production in quotes and I should note that most of the data it’s going to be back up to these drives is on This Server right here and this dis array right here and between the two of those there’s about 150 terabytes of storage that are that’s in use and even those are getting kind of full so I’m probably going to be upgrading them soon as well maybe with more drives from the seller if they work out so it’s a lot of data it’s a lot of video files and obviously with this much storage on the backup server it’ll be more than enough to back that up and plus I have some other servers dis images database server backups stuff like that that are all going to go on this machine as well as another off-site machine I didn’t mention the reason this is upstairs is in case the basement floods or there’s a fire or something at least presumably This Server would be unaffected being two stories up in a completely different part of the house it’s not a huge house but still it’s uh at least out of the basement and that’s why I also have off-site backup servers just in case the whole house burns down um I would lose some of my Roll video files and some other stuff that’s not Mission critical or hyper Mission critical but uh most my important data would be safe there and elsewhere as well and I’m trying to get more serious about my backup so uh not long from now I’ll have a video featuring these backup tapes because I have a lot of data that exists but doesn’t change a lot so I just want to back it up once store it off site on tape and this way uh a lot less concerned about hard drives failing so the first step before I can test all 20 of these drives or rather 18 at a time is to take them out of their baggies put them in sleds get the sleds caddies whatever you want to call them loaded into the server and I’m going to be running this this has I think an h330 a perk h330 card which is a RAID card I’m going to be using it in HBA mode so these will just be treated as a jbod just a bunch of discs and the OS will see each disc individually so I can use Smart modon Tools check the status of these drives see how old they are see how many hours they have on them Etc and the real burning test for these drives is simply going to be to create a raid array and I’m going be using software raid obviously Linux software raid and that’s raid six now thing I’m struggling with is it’s a really bad idea to put 18 20 terab drives in one big fat array

[Music] and that really comes down to drive failures during array rebuilds is with raid six you can withstand up to two Drive failures before data loss on the third drive failure so you say h it’s no big deal take the drive out the bad Drive put a new good drive in starts rebuilding that’s great the problem is that when you do that it puts all the discs in the system under heavier than normal load because this server is going to be under very low load normally if a disc is going to fail it’s probably going to fail under load especially at higher temperatures when the entire Drive cage gets a little hotter from all these drives working not working their asses off because you know they’ll be reading anyway the point is whole thing gets warmer drives work harder another Drive is more likely to fail during the rebuild than at any other time and besides this is just a certain probability that drive will fail at any time with a 20 tbte drive to rebuild assuming a 100 megabyte per second let’s say of uh data streaming into the drive so let’s say that’s 100 megabytes per second going into the drive that’s a gigabyte every 10 seconds a terabyte every 10,000 seconds and 20 terabytes every 200,000 seconds a day is 86,400 seconds so it’s going to take two and a half days call it under optimal conditions to rebuild one drive in the array is second Drive goes bad you replace that in the middle of the rebuild it’s probably going to slow things down overall so you’re probably I mean in theory it might not but it probably will so then you’re talking even longer to do the second Drive rebuild while the first drive is still rebuilding you’re putting the whole array under load for even longer another Drive failing is a definite possibility the question is do I split this up into two n dis arrays or just go for the Gusto and do one big fat 18 Drive raid six array it’s inadvisable but the difference is of course with two raid six arrays you have two parody drives uh two more parody drives than you would otherwise which is 40 terabytes of less usable storage to put another way with an 18 Drive array you have two parity drives and 16 data drives which is 320 terabytes of usable storage it’s less when formatted but let’s just say for round numbers about 300 terabytes of usable storage if I split it up into two n dis Aries that’s only 14 drives of uh data total which is 280 terabytes call it 260 formatted maybe 250 so yeah it it’s really do I want to lose that extra 40 terabytes of storage I’m thinking I’m going to take the risk and that’s only because because I wouldn’t advise you do this necessarily the only reason I think I’m going to take that risk is because I have an off-site backup server my main servers are all at least raid six I have raid Z3 I’m using ZFS on the server behind me so it’s really only in the event of catastrophic failure and plus I’m going to be doing tape backups and storing the tape securely off site so I’m not super concerned about having this fail contemporaneously with the catastrophic failure of the server down here anyway I’m still on the fence we’ll see at first I’m just going to put 18 drives in here we’re going to test the 18 drives I’m going to build an 18 disc raid six at first just to burn in the discs because that array rebuild will take quite a lot of time and work the discs pretty thoroughly and then run some tests on it man 400 terabytes is heavy though oh and with software raid an 18 Drive raid six or two nine Drive raid sixes but one thing you might notice is a problem is that with the 18 drives I’m not putting the OS on those discs those are just going to be data drives there are internal USB ports in here I think I actually don’t know well anyway it doesn’t matter if they’re internal USB ports I have booted servers like this off of USB sticks internally I like to have all my discs externally accessible both for ease of swapping out and just to see the little blinking lights it’s a thing I have so you might notice there’s a 5 and a qu Drive Bay up here unfortunately unlike Dell 740 XDS they and maybe even some regular 740s this does not have any Drive Bays on the back I don’t think you can even spec that with their Tower line so what I did get is an icy Dock Drive cage which holds six 2.5 in drives this comes with a couple of fans on the back and it fits into one 5 and a qu Drive Bay which non- coincidentally is of course what we have up here it’s meant for an optical drive when you order it from Dell that’s usually what they would throw into that slot but this will go in there instead and this is nothing fancy there’s no Active Electronics in here it’s just a pass through it has two set of power connectors in the back I think it’s just connected like one of those for triplet of drives on either side and as you can see it has six SATA ports so those are just passrs um it’s a there’s just a dumb back plane back there the only other thing you might notice is this switch labeled HLA which I believe is the fan mode switch high low or Auto I’m heavily considering just taking these fans off completely because as I’ll show you later inside the chassis of the t640 there’s a fan right behind the 500 Quin Drive Bay that should pull air through this at least relatively nicely and I believe I’m only going to use the middle to Bay for the 2os disc I don’t think I’m going to use any other Drive bays in here so those will have decent air flow and a good amount of air gap between them and any other surface in here and plus they’re only they’re they’re going to be used like almost never you know like some log files are going to get written to them and they’ll really only get red from when the thing is booting up which will be extremely rare so these ssds are not going to get a lot of use so I don’t really care about thermal properties too much and just in case you’re curious here is actually doesn’t have the model number but that that’s what I got anyway there’s no model number on here and I was just assuming that that is fan speed high low Auto yeah high low Auto um high is 100% low is 60% and auto is the fan will start at low setting at increased speed depending on the hard drive temperature well who’s putting hard drives in this now if the hard dis Drive exceeded over 40° Centigrade room temperature 25° the fan will increase its speed to maximum I don’t know where the temperature sensor in here would be but like I said I I think I’m just going to omit the fans entirely that’s because these do not look like particularly high quality fans so they’re likely to stop working anyway or at least make annoying noises eventually if not immediately and noise is another thing that highly recommends this chassis um I ran it through that memory test even under load super quiet quieter than any of the servers behind me in that rack and I’ve used this is my first t640 but I’ve had a t is that can you see it behind me no it’s on the top of this rack it’s not in use right now is a t620 I had t710 before that and even a t605 point is that the tower servers are generally much much quieter than the rack servers and that’s by Design because these are really for a more small to mediumsized business Market where they might not have a dedicated data room or even closet and this might be in the same room as employees so it’s highly desirable for this to be nearly silent I mean for those of you that are really particular that want like a zero DB gaming chassis this wouldn’t satisfy you it’s not like completely silent but it’s got you know these are bigger fans than you would find in even a two rack unit server and they’re just a bit chunkier as well so they can move more air at a lower RPM yeah a lot of fan blades and well this is what they look

like actually what manufacturer is this oh foxcon and even for Server Chassis uh fans do make a huge difference the big file server back the big file server back there it’s a super micr chest the fans it came with were super loud and annoying I think they’re even a little smaller than this they were like the type of the size of fans you’d find like a 2 Ru system because it’s basically two rack unit systems kind of stacked on top of each other in one shell more well more or less anyway point is fans were super loud and annoying I replaced them with this brand which I can’t remember but this is yeah and I didn’t modify anything the BIOS didn’t modify anything in the fans it was just a different brand different model of fan much quieter I mean like probably makes 20% the noise it did originally but air flow is still good uh temperatures didn’t go up and manufacturer I had that experience with a Dell server as well it shipped with some other brand of fan I replaced it with Delta fans that were actually speced by Dell they even came with the right uh pin out and the right connector on it for hot swapping this was in an r90 five I believe and again made the system much quieter it was just a different brand different manufacturer different blade design made all the difference in the world even though they were also official Dell Replacements so manufacturers tend to swap out fan brand so i’ I’ve seen a lot of arguments online like people are like oh that that that server is super loud it’s it’s very annoying I don’t know how you could say that’s a quiet server and someone else is like no no no I have one it’s it’s super quiet it’s great and then they chalk it up to a disagreement usually about like what each person thinks quiet is you know is a quiet machine 30 DB is it 40 DB is it 50 like what relatively speaking what do you consider quiet I have a suspicion that sometimes they’re buying the same exact model of server maybe even the same year of production but they came with different fans and they make a huge difference so if you got a noisy server before you go to any crazy modifications check eBay see if you can find some other brands of fan you can usually get them pretty damn cheap if they’re used you could test them out for yourself yeah obviously I don’t know offhand which fans the best and which are not but yeah yeah and probably even model to model from the same manufacturer might make a huge difference I’m not a fan aerodynamicist but I know from empirical experience it makes a huge difference rant over this chassis was nice and quiet quiet enough for my purposes we’ll see though we’ll see when it has 18 drives spinning up front generating heat and that heat being sucked through the chassis we’ll see how much more those fans spool up and how much louder it gets I have a fairly high tolerance for noise if you couldn’t tell let me turn off the noise reduction on this microphone that’s what it sounds like down here and I work down here and do all sorts of crap that’s why my voice might sound a little funny because I got to use a [ __ ] ton of noise reduction even though I’m using the right mic for the job that’s why I have this mic I don’t know 3 cm 2 cm from my face or my mouth hole it’s it’s on my face my mouth Hole uh who the [ __ ] calls it a mouth hole it’s like 2 cm from my mouth so obviously that will be the loudest noise it picks up background noise will be much quieter relative to my talking at this distance because most of the noise is coming from 10 12T back there the point is still too loud down here but not for me I don’t mind it as long as I’m not trying to record a video that’s not an h330 I think this might have an h740p in it actually all right so now uh I guess I’ll put the drives into caddies into time lapse and uh you guys call them caddies trays or sleds I usually call them caddies even though I don’t think that’s the correct term because I like cdrom Cades from back in the day um that’s my earliest like memory of refering to something like that so I I call them caddies but trays sleds whatever point is this did not the server did not come with them so I’m hoping I have 18 of them uh that fit the style chassis I’m pretty sure I do and weirdly the t640 uses the previous generation style of trays even though the r 740s i have use the uh newer style of tray don’t know why so I need 18 of these guys at least I’m 90% sure I need 18 of these guys yes 18 of these oh that’s one two 7

that’s actually just about perfect I think cuz I think all I got is E7 but I have one more trick up my sleeve I’m just whipping out my cell phone for Simplicity and oh boy does not like my fancy blue lighting um this that’s not my label this is how it came from an eBay seller

but not using this chassis right now and it’s got 12 trays perfect and this is a dr4100 in case you’re curious which I think is a r 730 XD pretty much woohoo and that’s 12 more okay so that’s actually 19 right there but this way I know I have some let’s say not actual Dell uh you know knockoffs from eBay floating around here and those I had a I had a bad run with those where the spring that retracts the Button had a uh clip that would snap off and then the button would never retract and would just stay down and this thing would just flap like that and you could kind of cool it back into place but that was a pain so if any of these are defective at least I have one extra and some of them are a little Dusty these were some of these were used in my home environment obviously not the ones from that server I just took them out of those are pretty clean so that’s

good oh yeah I need screws as well fortunately I have a relatively comprehensive collection of screws from over the years and I think yeah I bought a bunch of these in bulk for mounting 3.5 in drive to sleds these are kinds with the fluted heads um nope oh there’s our Visa

Mount standoffs I was hoping I had more I got more down there I could probably pick through it because I don’t think this is going to be enough I got 18 drives times four screws a piece is 72 screws and that does not look like 72 screws it says 424 uh times 3.5 in but I don’t think this bag is full I think I’ve already used some of these Anyway come on Scott let’s get this over with in the time lapse you’re you’re probably going to see me putting three screws in each tray CU look this isn’t an Enterprise environment I don’t give a crap as long as you have one screw and one screw on the end where the connector is by and large it’s going to be fine I’ll probably I’ll put three you know two on this end and then one either here or here and it’ll be more than enough to hold the drives and whatever oh and I almost forgot another thing you’re going to see me do is label the drives because I’m going to be using software raid the LED indicators on the front of the chass seat you’re not going to tell me which Drive is bad if a drive goes bad I’ll need to determine that through software and I’ll need to either look for the serial number of the bad drive or see which serial number is absent from the list of drives or you know whatever you can see here these are both software rate arrays actually so is this one and yeah those are the drive serial numbers really handy when replacing them I mean another way to determine which Drive is bad just to use the array and see which activity light is not flashing and that’s probably your bad drive but only probably so you’re better off knowing for sure interesting these say date of manufacturer 26th of September 2023 this video is being recorded in May of 2024 so less than a year old at least that’s probably their refurb date but still that’s a good sign I’ve actually had in my life very good luck with refurb drives from both Western Digital and Seagate in fact I found that Seagate refurbs tend to not fail but Seagate regular drives do and I know we can argue about which is better Western Digital or seate personally and back Blaze statistics back me up on this um generally speaking seates consumer low-end drives are much worse than Western digitals drives that being said seates Enterprise Drive Line which I think these are is usually in my experience anyway topnotch but all I can say from my personal experience and you know you could argue this all you want in the comments in my entire life I’ve had many more Seagate failures than Western Digital failures even though I prefer Western Digital and usually buy those so in other words I’ve owned more WD drives but have had more seates fail in other words I’ve done more Seagate r rmas or just you know throw them in the trash than Western diges even though I’ve had a lot more Western diges over the years and hdst also really good uh both before and after the Western Digital acquisition so the way I like to label them is on the first line I put 20 terabytes because eventually years and years from now 20 terab drives might be harder to get than like 22 or 24 terab drives so I might have a mix of capacities even if I’m only using 20 TB per Drive in the array and then the serial number is pretty short so I’m just going to put the whole thing on two lines and then the only irritating thing is this label maker does look it’s my favorite label maker or at least these are my favorite labels but it does waste a lot you can’t control the margins you can’t control how much extra it ex fuds unlike some other label makers which are a lot more uh generous with the amount of label you actually get to use so I got to cut off the extra margin make it small enough to fit on the drive tray and then they’re oriented like this in the chassis so that is going to go roughly there and obviously I don’t care about fitting it within the margins of the original Dell label like no I want a nice big legible marking on there even though it got a little [ __ ] up there but whatever I usually just do a quick double check uh zx20 yez zx20 yez because if you get it wrong in the label that can cause real confusion when you’re going to replace

it not the most ergonomic way to do this but and voila only 177 more to go populating a large array like this and having to put all these drives into

caddi is both something I dread doing and something that I sort of weirdly enjoy doing it’s like very calming very Zen relaxing in a way so yeah let’s uh skip to the time lapse and some music and uh be done with this in a jiffy from your perspective

d

[Music]

[Applause] Ed. note: There is no applause in this video whatsoever. WTF?

[Music]

[Music] and that’s the last one how long did that take I don’t even know it’s on the screen though because the video is keeping track well it’s not the best view in the world but let’s at least see the last one

Ah that’s satisfying okay so now we got 18 discs obviously I got two more drives down there in that box those will be spares I’ll test those in a separate enclosure like a USB you know what I mean anyway it’s already getting kind of late here I really just wanted to get these drives in the system so I can get them testing overnight like I said testing is just going to be building an array so I’m going to boot off love a live version of a buntu off this USB stick configure The Raid array like I said all 18 discs get it set up as raid six and get started building and just leave it overnight tomorrow I will put in the OS discs show you around the inside of the chass in case you’re curious and um [ __ ] one more thing oh we’ll test power consumption I I didn’t test it beforehand which is probably dumb but what I can do is just sort of like you know half pop out all the drives just pull out like that test the power consumption without the drives then test it with the drives so yeah let’s get this uh get this full booted up

oh you can’t see it from your angle but right I took the side panel [Music] off oh yeah what we got to do is change change this to HBA [Music] mode and I’ll have to set up the ID leader but like I said this is just to get the machine up and running enough to build a raid array and to make sure all discs are recognized actually [Music]

uh block probing did not discover any discs oh actually I remember reading online somebody somewhere said that you need to cold boot the system in order to uh get the RAID controller to switch to HBA mode I’m hoping that’s the problem um I’ve pulled power cables yeah there they are so let me try booting [Music]

[Applause] again uh oh terminal there we go g honestly I I don’t really run Linux servers with a guey like ever so this is

awkward

Yeehaw H oh good that is installed at least

okay uh s must be the USB stick right

so yeah good okay just want to make sure now it’s usually wise to partition the dries first but since this just for testing I haven’t decided what I’m going to do with this array ultimately um I’m just going to I guess it doesn’t matter if it’s level six in that case but whatever oh oops that’s supposed to be I’m an idiot equals

18 e okay well that’s cool I mean I guess this resolution is good for video so you guys can read this but man um okay 0% complete clean resyncing 16

devices by the way this is just in the difference between these two is whether you consider a kilobyte to be a bytes or 1024 bytes and so forth all the way up to terabyte that’s why those numbers are different but it’s it represents the same amount of bits ultimately well this is going to be useless at this resolution right um yeah because it’s cutting off the freaking

percentage well that’s not so bad then and then I will just leave this up all night uh just for the record it is oh good timing actually so I started that just about 300 a.m. um you can see the system time says 7 a.m. on May 19th it’s 3:00 a.m. New York time May 19th obviously I didn’t set up the time zone on this correctly anyway so yeah I’ll just uh Leave This Server running overnight and we’ll we’ll see how far that gets in the next 10 12 hours or so oh and I almost forgot Blinky lights

oo oo I’m I’m looking at a screen over there that’s why I’m

yeah that’s very nice so until tomorrow I’ve been Scott still be Scott tomorrow I don’t know why I said that bye sometime as is now past just a quick update here you can see the LEDs are still blinking it’s been I think 23 hours since I started the array build and it is 48% complete so we’re probably looking at about 2 Days full 48 hours is which is actually a little faster than I thought it would be which is

nice uh let’s go 10 oh my

God there we go so yeah it’s writing at about 1110 megabytes per second so yeah a little faster than 100 megabytes I gave a credit for originally so that’s cool I kind of just wanted to check the performance of this operation and by the way that’s 110 megabytes per dis which is about 2 gbt per second uh from through the HBA which is really good that’s like 16 gigabit uh what kind of load is this putting on the system 100% CPU so it’s CPU bound but it still seems like it’s probably maxing out the discs as well which is kind of improbable but well anyway just want to take a look at that as it’s going um I’ll check back in either tomorrow or probably the next day because it should finish about 26 hours from now maybe and it’s going to be super late at night then so once this is done we’ll take a look at the chass itself install the OS discs install an OS and get it actually syncing the uh files from the server oh the other thing I wanted to see was

uh

uh forgot oh

yeah yes that is one of the discs obviously uh what is it d a for everything yeah that’s what I wanted oh frig well that’s not

good all right well then if that random ass Reddit post I just looked at is correct then it’s actually zero yeah I knew it was something with the seate drives okay yeah okay okay okay cool now these are referb drives so who knows what their situation was be before I assume this was factory reset um I mean the 28 hours is probably just me I’m assuming these came with zero hours on the clock so uh yeah okay we’ll we’ll just stick with that I’m back it’s been a few days cuz I got busy with work and you know real life stuff but the array did finish building successfully uh the drive seemed to be fine still not seeing any Smart errors everything looks great so fa confident these drives are good drives but only time will really tell for that all right get the keyboard and stuff out of the way and then install the ssds in the 5 and a/4 inch Drive Bay in the IC do whatever it’s called oh and to go into this enclosure I just got two old ssds I haven’t been using for a while this one’s 250 gig and this one I think is 120 or8 gig it doesn’t matter if they’re different sizes I really don’t need more than like a couple of gig for the OS partition anyway so you know cuz I’m just going to run these in software raid one and I’ll just create a 120 gig uh boot drive you know the various partitions boot partition and the uh OS partition and I guess swap too why not have some swap even though it doesn’t matter

this enclosure is Metal by the way it’s not a plastic shell which I like a lot and it kind of Clicks in in the front like it’s got these two little

pins that go into the screw holes on the bottom and then by the connector that’s where two actual screws goes I guess I’ll use the included screws rather than digging into my own stash oh I’m getting ahead of myself one thing I’d like to do is get the serial numbers off the drives just so if they do die and I have to replace one of them I know which is which I could just call one I could just label one sand disc and one PNY and that would probably be enough but you know by tradition I go with serial numbers so I’ll stick with

that and and you know what I’m actually going to print two copies of each because the drives are going to be obscured on the bottom by the case so this way I can put one label on top of the disc and know which is which because I do want to be able to fit on the front of one of these bays and uh that’s kind of silly yes so I use that as the top label for it I’ll cut it off to oh I have all the same amount of digits as that one hey I got the same amount of digits well roughly I mean the font spacing is little the it’s not fixed with font so who knows and now I don’t care about these SSD specs in particular um I’m not concerned with performance because this is just going to be used to boot and write a few small log files too I’m more concerned with reliability and of course with ssds long-term reliability comes down to right Cycles but these drives are also not going to be written to a hell of a lot so it doesn’t matter if they’re a little old and used I mean they’ve been gently used I don’t think these are um strongly used

drives but the point here is just reliability from having a raid one just in case one dies of random causes not so much due to fatigue and there we go now I’m not going to apply the label to well actually you know what I will because I’m not worried about well no it’s GNA make this not pop out easily because it’s thicker than this and it’ll obscure the drive LED I’ll put this on the uh Drive Bay below this one because I’m only going to have two drives in this enclosure yeah why why is it like sticking out maybe I should have uh that’s as far back as the uh Drive will go in the sled it’s still it’s still proud of the ones without drives in them but I guess those can just sink deeper I don’t think that’s a problem as long as it’s seated in the connector that’s all that really matters

I’m such a spaz I put the label on the wrong I had it upside down the whole

time I wonder how many of you noticed that like could I have just edited that part out and then no one would ever know who knows now of course if I was fully populating this unit with drives uh it would be a bad idea to label all them this way and block up all the air flow you know even if they’re mostly going to be read drives it will generate some heat I think we’ll still get some air flow off these top vents yeah I’d probably figure out an alternate way of labeling for example I have an icy dock enclosure similar to this behind me well you guys can’t see it but it’s right about there yeah there it is and as you can see I just labeled all six drives next to the unit and that worked out fine now as for this again I’m concerned about the reliability and noise of these fans so I’m thinking this Dell chassis has a lot of fans in it and has pretty good air flow front to back and these drives are not going to be working terribly hard anyway oh those are just some chewy screws those are not going into metal at all I can guarantee these are going into plastic just from the feel of them not saying the whole thing is metal but like uh yeah like this whole back plate from this seam wrapping around just to here is all plastic yeah I mean they don’t they don’t feel great they don’t feel super cheap like I’m not saying they’re awful I’m just saying that the Dell server has a whole ton of fans in it and why not let them do their jobs and the reason I’m completely removing the fans is just so that there’s a bit more room for airflow without the fan blades obscuring part of that when they’re not moving so so whatever air flow chassis does pull through this um you know it’ll be easier to pull the air through I guess also it makes cabling a little easier because we don’t have the fans right up tight next to those SATA connectors I think it’s the right move for this

application and of course we don’t throw out screws ever and then these are presumably the screws for mounting it in the 5 and a quarter Drive Bay I guess I don’t know why this why are there so many screws wait what does it think these screws are for 10 pan head screws for device yeah I guess they want you to put 10 screws to mount this thing one two uh three four yes there’s eight but oh it says two oh I missed that it clearly says two for spare parts but still I I don’t know who would actually put in put this in with five and the really short for getting past like the sheet metal of the case and then actually getting a bite into this unit so I don’t know man I’m just not uh not going to use them oh my God no this is thing’s heavy now I mean it was slightly heavy before I put the drives in it but now it’s it’s freaking massive but uh so I might before I bring this upstairs to the second floor I might take out all the drives I’m sorry my uh hands are so bright oh actually yeah I’m sorry my hands are so bright but I need to up the gain on the camera to make the interior of the chassis visible to you guys because obviously I have most of my lights on the other side and perhaps unwisely not so many lights on this side but you’ll get the idea hopefully so we got these uh pcie card retention brackets it’s for tall cards they don’t rattle around uh not really necessary for me just uh I’ll probably just omit

those all right so of course up here we got the 5 and a qu Bay and this should be tooless to remove this front plate yeah there we go basically this is just a dummy bezel like a drive bay cover but really large and as you can see it has these screws that I have a bit of a standoff of unthreaded uh length and so we’ll be using these screws to mount the icy

dock

there and so it’s two on this side oh and so it’s two on this side and one on that side and now these interesting but fortunate for me these are the fine threaded screws the type you’d find on like CD ROM drives and stuff uh not the old school 5 and A4 inch Drive Bay like where they use the coarse thread screws so because fine head screws is what this uses they’re fine thread not fine

head oh that’s why the screws were so short that I came

with is these are too long the problem is these are sort of specialty screws now you can use normal screws and just not drive them in all the way but let’s see if I have a shorter version of those exact screws yeah I got more of them but the uh the threads look like they’re about the same length yeah they’re identical so uh and I’m wondering if this one that I did put in is so long that it would interfere with this bottom Drive Bay but I don’t really care because I’m not using it so the problem is these in the back are digging into the plastic bezel back here and might even yeah I don’t think I can drill deeper because it might hit the side of one of these SATA ports so that that’s not B that’s not a problem I mean back in the days of mechanical hard drives or Optical drives you’d want to make sure that your drives are locked down in a fairly sturdy fashion so that they don’t vibrate which could cause problems you want them really uh connected to the chassis quite sturdily but with ssds I mean who cares they’re not going to vibrate and even the chassis experienced vibration who gives a [ __ ] it’s not going to interfere with the operation of an SSD in theory it could could shake a port loose or something but let’s face it this computer is going to be sitting on a shelf in a cabinet and barring any earthquakes which we had recently in New York actually um it won’t be an issue I know I’ll admit it’s kind of a weird angle to show this to you but uh yeah there it is it doesn’t look half bad in the front of this chassis and uh it is a bit loose like it’ll just wiggle around a bit but it’s not going anywhere it’s not like going to pop out arbitrarily cuz it’s locked in by that screw being encumbered by this bracket as far as cabling goes I think I can take this out with the fan still in it yep these fans are modular from the air shroud but you can just pull the whole thing out that’s uh basically the gist of it mostly guides air over the CPUs and RAM but of course there are four fans in fact you can take out this entire

module and that’s what I was talking about when I said there’s good air flow through the chassis because it also has these four fairly beefy

fans we saw them earlier and oh I should point out air flows that way so when it’s situated like that in the chass see you can see this top fan does provide some suction through the 5 and 1/4 in Bay here we have the perk

H730 h740 yeah it’s just sort of sitting in there I don’t think it’s even yeah it’s not even locked down by a bracket or anything which is weird I think this is missing something we’re discovering things together because this uh the fan troud has this extra bracket coming up which holds the card in place so the card is not just freewheeling in there it’s actually a not prevented from going oh no it is prevented from going up there’s a notch there and then this lip holds it it keeps it up from Against Gravity so great so it has two SAS cables going up and over and then onto the back plane over here which of course holds the 18 drives and I think has a SAS expander under this heat sink and some beefy power cables going to it yeah I don’t know if I mentioned but this system did come with ah it’s Overexposed damn it it’s a lot of focus but 1100 watt psus 1100 watts is probably Overkill even for 18 SATA drives maybe for like 18 15K SAS drives or something it would be uh appropriate but and these uh silver CPUs that are relatively low power I think they have an 85 watt uh thermal design power yes these aren’t going to take too much power the drives will take the most power out of the whole system probably but it’s not going to be a lot but better to have over capacity on your power supplies than under capacity is my feeling so anyway yeah that’s the inside of the chassis this has uh 64 gigs so it’s got four 16 gig sticks and I I guess just for completeness the back of this is reminiscent of my Precision t79 7920 um and most of the that line in that it has four pcie slots here then the usual ports and power supplies and at the bottom four more I’m sorry five more pcie

slots so dual power supplies which for me is a must I like to put two UPS’s on any system like this um and also obviously in case one PSU dies this came with a Windows 2019 server license which I’m not going to use dedicated idra Port I think this came with Enterprise ID which is not terribly important uh four USB 3 ports two USB 2s I assume two uh 10 gig NX I I know says gigabit there but yeah they’re they’re 10 gigabit and obviously VGA and seral oh and should point out these bottom slots are labeled CPU 2 I I should imagine that means these would not be active without the second CPU populated I’ve never bought one of these I never bought a dual CPU system that didn’t have two CPUs in it so actually that’s a lie my Nas did but that wasn’t a dell and it didn’t have the same pcie configuration so yeah anyway all right and finally it’s still really too dark to see it up there but Behind These SAS cables are two SATA ports one’s already populated with a cable that just goes to the optical drive bay the other one’s unpopulated I’ll obviously stick my own Cable in there and wire it up to the back for those two drives the only thing I foresee as being an issue is do we have setup power oh yeah yeah it’s buried in there I have zoomed Us in so yeah there are the SAT uport it’s up here have the other end of that Sata cable and there is a man that was really tight there’s a retention clip in there that was hard to get to stic cables nice and long and I see a a power connector in there there we go Liberty and does it have a little extra it’s only the one though of course because this is only really supposed to have one Bay oh it also has an a small optical drive power cable uh this type yeah it says in order of function properly connect both 15 pin power connectors to the enclosure when using the device if I put the drives vertically on top of each other yeah I probably could have got away with one but you know what this has an angled connector anyway which isn’t ideal for putting in there so let me get a uh splitter all right I’ve got a lot of sat cables my SAT cable bin but here we go it’s a uh male to two females voila and then I’ll just sort of Tuck this all neatly up in there eventually

yeah it’s a locking connector that’s cool it’s longer than it needs to be but that’s okay it can just sort of curl up in there you know what it’s an absolute nightmare trying to get my big fat hands in there so since I got plenty of slack on these

cables I’m just going to connect them outside the chassis and then slide it back and it doesn’t matter which sat of connector goes to which drive but I am going to put HDD oh it’s labeled od0 which is the blue cable on the left side when you’re looking at the chassis from the front and that’s just for my own

neurosis and I guess technically it will help with troubleshooting or something yeah won’t matter it it won’t matter at all anyway there’s all the connections made up to the back of this unit all the connections we’re going to use anyway and then just gently guide the cables out of the way and click a lot of people would absolutely despise this mess and not be satisfied with it um at least for now I’m going to leave it like that make sure the front drive bay Works before I try tucking all these out of the way and to be honest with you I’ll probably leave it like this because it doesn’t matter it really doesn’t there’s still plenty of room around these cables for air flow like there’s not a high density of cables there so air is still going to be able to flow through this just fine I’m sure a lot of you are screaming at the prospect of it being left in this state but I kind of like that idea of someone out there feing over how awful this is and yet it working reliably anyway for years and years and years because what is literally going to happen inside this chassis with all these messy wires if no one disturbs it and I’ll be the only one who could possibly disturb it oh I’m a dumbass though because this does have to be tucked out of the way at least this cable does for the fan shoud to go back in place cuz of course that’s the other thing we need to ensure is that none of the fans get fouled on loose tables either like that can be bad that can be operationally problematic and I’m half kidding cuz that is why you want to keep your cables neat inside your chassis amongst other reasons anyway let’s see oh that’s actually yeah that’s got to go up and

over yep not pinching any cables none the cables are anywhere near the front of the fan which has this guard on it anyway so wonderful oh I’m an idiot this card is retained by the fan

troud and so thus concludes my physical assembly and rray builda of this 18 drive time 20 tbte massive Beast of 360 ter of raw storage um will be less once I decide how many parody drives I’m going to have in total in other words how many arrays I’m going to have in total because it’s going to be running raid six in either case and uh yeah the rest of it is just going to be loading an operating system on it I might just use a buntu I’m not sure just because it’s a long-term support and it’s just well supported and reliable and what I need ultimately from this machine is reliability I’m going to do a minimal install no gooey nothing fancy and the only things really loaded on this machine other than the really minimal OS and and tools is going to be um some scripts that I’ve already written for my other backup servers that’ll just modify slightly for this one and my backup strategy with these type of backup servers mostly consist of as I said earlier this is a client it’s not really a server technically um other than SSH which I have firewalled off so only I can access it from one of my machines down here has a private key on my machine and a password so it’s pretty secure in that regard that’s the only thing this will serve otherwise this is just a client it has an smv client an NTFS not NTFS client an NFS client and using rsync it just synchronizes other servers pulls its data down to it sometimes locally sometimes over a VPN if it’s a remote server that’s backing up and um yeah that’s basically it it it’s quite simple the only couple of mildly interesting things the backup scripts do is they retain uh for example if I’m backing up uh VM snapshots especially from offsite it will retain the last however many X number of snapshots I usually have it set to 30 so it has 30 days worth of snapshots of remote machine or a couple local machines too actually and those are just gz raw dis images taken from a snapshot when the backup begins so it’s kind of like backing up the machine in a crash state but you know 99% of the time that’s fine I do also back up my databases and other important applications at the application layer um in other words like I’ll use my SQL dump to dump all the contents out of my MySQL databases gzip those up and also get those backed up to this and other back servers uh it’s just much easier if like even though the machine is in a crash state if I can restore the whole VM image rather than having to do a a SQL Restore for example the SQL restore might take longer than just copying over a gzip file to the local server and then unzipping that and there you go so it will retain a few snapshots of whatever information I’m uh putting on it and for example with my file server at least for my main most important documents my programming projects um video projects that is the actual project files not all the raw video all of that also gets R synced over to one of these servers but then after it’s R sync successfully that night it gets then gzipped but not deleted just gzipped as a snapshot of that data and then that’s stored in a separate directory on the same server on the same backup machine the point of that being that this way if I get some kind of hor malware that either deletes or encrypts my files and then rsync decides to Oh all these files are changed and then sync over the encrypted files to the backup machine well that renders the backup machine kind of useless so I have these we’ll call them snapshots that I’ve just they’re just tar gzip of directory structures that are a point in time and I retain 30 days worth of those as well so yeah if I go 30 days without noticing uh a rans somewhere attack then yeah okay all those backups will be wiped out but for my most important data that’s how I do it I keep point in time backups uh usually daily for most of my important stuff and less important like uh virtual machine images like some of my personal servers that I don’t really modify or use that of or you know change data on that often uh those might get snapshotted weekly anyway the point is it’s not fancy but it’s reliable and so the reason for this giant ass machine being in my house on a different floor is twofold I mean one it back UPS it backs up video files which can be quite massive and be quite timec consuming to transmit over the Internet which is the other reason the inverse of that is if I need to restore the files from this machine when it’s in situ upstairs it’s going to be connected via a gigabit Ethernet connection but I could easily bring it downstairs and connect it to a 10 GB ethernet connection for the restore and I could even reconfigure this as a file server and then just access my files off of this machine and build a new backup server in the meantime you know what I mean it’s good having on premises backups like this just because it’s much faster for restores and that’s I would forewarn everyone if your backup strategy is exclusively backing up to the cloud you got to remember that in an urgent situation where you need to store your data if you have terabytes of data how long is that going to take just to download let alone like if it’s application Level backups will don’t actually put those backups onto each server they need to go on to yeah if it’s going to take you a week just to download all the data let alone restore it to various servers or whatever even if it’s just for home use like if your entire video libraries in the cloud and then you need like terabytes of it back for a project you might have to wait a couple of days I mean depending on the speed of your internet connection so I’d always recommend backing stuff up locally and remotely because of course if my house burns down if burglars actually manage to carry all this [ __ ] on their backs out of my house which is you know daunting I mean it’s daunting enough for me to manage all this stuff but yeah I mean that could happen so you have to be prepared for that too so offsite and on-site backups I think are the best move and the other thing about Cloud backups is that they’re so hard to verify like how like how do you know all your files truly exist on your cloud backup providers hard drives and they’re not just showing you you know you might log in and see here’s a list of all the files you have backed up and here’s all the versions of all those files we have backed up and oh it’s glorious look the files they’re right here but they could just be showing you entries in their database that don’t necessarily correlate to any actual files on any actual file system anywhere in their data center like not on purpose I’m not saying they’re a scam I’m just saying you don’t know how well their entire system is managed really like how often are they verifying the files that they’re displaying to you that they have logged in their database of files are actually stored somewhere because I’m assuming they abstract their file information from the actual file storage because otherwise I feel like it’ be a nightmare storing file files on all sorts of different servers and all sorts of different dis packs and then like trying to find them arbitrarily like there has to be some centralized Master list which is probably not a file system in anyway even if it is a file system how often is that file systems contents verified that exists and is in one piece you know in other words my point being you could have a catastrophic failure and go oh my God I got to go to my cloud backup provider and get all my files and then it could turn out that some or all of your files just aren’t there even though you thought they were and even if every now and then you’re smart and you go in and download a couple of arbitrary files here and there just to make sure they do exist I mean that verifies some of them but you don’t know that all of them exist unless you go and download all of them and look at them yourself which could take days or weeks depending on how much data you have stored and depending on the speed of your internet connection so at least with this first of all I know how it works I know there’s no abstraction between like the file system and the files that are stored I mean there is internally to the file system I suppose if you want to be pedantic but you know what I mean there’s not like a separate like I don’t know MySQL or couch DB instance that storing list of files expecting it to cor to uh correlate with file systems on various servers throughout their data center that’s really what I’m talking about when I’m talking about abstraction in that sense so I just feel more comfortable able having my data under my control and me knowing exactly how it’s stored and how it all functions end to end it just makes me feel better it’s more of a hassle definitely more of a hassle and more expensive in the end I mean back Blaze which is a backup provider I do recommend to friends and family and I have used it for other purposes as well um I do like back Blaze I I particularly like their transparency and their Drive stats that they publish so yeah not not not this is not a paid endorsement they didn’t sponsor this video or anything I give a crap whether you use them or not but uh generally speaking I do like them it’s just because they’re a cloud provider I don’t trust them not because they’re bad just because they’re not me but anyway point is back blaz subscription would be far cheaper than what this server costs and I’m pretty sure they offer unlimited storage to this day so in theory I could back up everything to back blazs for much much less money than this thing costs and probably for less money per month and this thing will cost a run in utility bills maybe but then again most of you probably don’t need this kind of absurd level of storage um honestly if you want something low power and quiet hook Raspberry Pi up to a couple of USB ssds and stick it in a closet somewhere you know assuming that’s enough storage for you um it’ be dead silent not use much power it could do its thing same sort of setup architecturally as I have here and you know your data is it’s in your closet and also offsite somewhere I mean back up to a cloud Prov for your off-site needs back up to a cloud provider I’m not saying don’t do that I’m just saying make yourself more comfortable and also back up locally and snapshots are very important because ransomware will murder you if you don’t have snapshots because otherwise you’re encrypted files will just get backed up and over and potentially depending on how you have your backups configured potentially overwrite your existing backups which is what would happen in my case if I wasn’t also tar gzipping all those files nightly and storing them in a time-coded timestamped file for 30 days worth of files that’s part of the reason why I need such a massive amount of space I mean right now I have approximately 150 terabytes of data here on the server behind me to back up which even with four parity drives would fill up less than half of this but there’s a lot of overhead in that I duplicate a lot of that data by keeping all those snapshots anyway I’ve done plenty of videos ranting on about backups for way too long but uh thanks for watching um maybe I’ll post a follow up to this once it’s up and running if I have any you know what if anything goes wrong I’ll probably post a Fallout video fixing this mess but uh if everything goes right then yeah this is just replacing two of my current backup servers in that closet which already have been doing their thing and running fine for I want to say one of the the one on the top shelf has probably been doing its thing now for8 years I don’t think I’ve upgraded that in a long time and the one in the bottom probably five years and they just hum away and they do their thing and those are made of very cheap parts this in theory should last a long time but we’ll see anyway I’ve been Scott uh good night why with the salute I always with the saluting I always [ __ ] salute

does anyone mind the saluting like should I keep doing that that was more of a doing of the cap the [ __ ] is no that’s a salute I did more of like a cap doing I never wear a hat I don’t know why I do that tip my Fedora that’s more like this you don’t like do you tip your now I don’t even remember do you tip your Fedora this way or do you tip your Fedora that way who knows it’s not even a thing it’s a meme I mean I think that fat actually did wear a fedora in that Meme for his own purposes I shouldn’t have said fat I’ll have to beep that out yeah you can’t make fun of people on the internet anymore though to be fair it would be inappropriate to randomly make fun of people in a video about the [ __ ] backup server right okay now I’m actually done

I repair the scrollwheel on a Razer Deathadder Left-Handed mouse after dropping a hard drive on it. Check the chapters to skip right to the solution.

This is after doing a hardware button swap on that same mouse: https://youtu.be/n00ioWfDE9k

I assemble a small 240V 30A distribution box for some future purposes.

I converted an older HP DL380 Gen8 (aka a StoreOnce 2900) from using a hard RAID controller to an HBA for software RAID. The conversion is simple, but the video is long af because I spend a lot of time discussing the “why” more than the “how”.

In this excerpt from the below video, I talk about the total cost of ownership of RAID arrays.

This describes why I created the RAID HDD TCO Calculator which helps you figure out the total cost of ownership of a RAID array, inclusive of stuff like electrical and cooling costs.

This is partially just for my own reference, so I don’t have to go down this rabbit hole again. (But I hope it helps you, too!)

The Situation

I wanted to upgrade the LITE-ON 256GB SSD in my trusty ol’ Lenovo X1 Carbon laptop to a snazzy new Samsung 960 EVO 2TB drive. I have a version of Acronis that came with a Crucial (or Kingston?) SSD, which has worked great in the past. The problem?

There was a system reserved partition at the very end of the disk, and Acronis therefore would not proportionally scale the OS partition to fill the disk; It would only scale that system reserved partition.

In a moment of errant stupidity, I said, “Ah-hah! If I make the drive a dynamic disk, that will allow me to rearrange the partitions!” (It most certainly will not.)

So I made the main boot drive a dynamic disk. I didn’t think for a second that would render the drive un-clone-able by most any software. Yet, that’s what it did.

Hence my odyssey began…

I tried EASEUS, AOMEI, Acronis, Clonezilla, creating a RAID1 array, Windows Image Backup and even dd.

The free versions of the paid tools (the former three) would not clone dynamic disks, though some claimed that the paid/pro version would. However I wasn’t going to shell out between $49 and $99 to do something that should be free.

Clonezilla had no problem actually cloning the drive, but a non-proportional clone resulted in the remaining ~1.75TB being unusable. I couldn’t create a partition on it using Windows (either in the GUI or using diskpart), and so I tried booting into gparted and creating a new NTFS partition at the end of the disk. That ostensibly worked fine, but then I got the dreaded INACCESSIBLE_BOOT_DEVICE error.

The same error resulted when doing a proportional clone in Clonezilla.

You can’t create a RAID array using a USB drive.. Or at least I couldn’t. (The new SSD was housed temporarily in a USB enclosure.)

Windows Image Backup worked to do the actual.. ahem.. backup, but when I went to restore I got some obscure error about the volume shadow copy service (?????).

And dd just caused the thing not to boot.

Requisite Disclaimer

If you make one minor mistake while doing the below, you could wipe out all the data on your original drive.

Heck, even if you do everything right, your original drive might decide to poop the bed.

So before doing any of this back up your important files to a flash drive, another SSD, a hard drive, a cloud, or 4,000,000,000,000 punch cards. Show the hex representations of each file to an android so that he/she/it can later recreate them via a keyboard with their hands but a blur.

Literally anything is better than just assuming you’ll pull this off without issue.

The Solution!

I put the new 2TB SSD in the laptop and wiped the partitions using gparted. (You won’t have to do this if you’re starting from scratch, as there won’t be any partitions.)

Then I installed a fresh copy of Windows 10 using the default settings.

This accomplished two important things:

• It created a 1.86TB (usable) partition for the OS
• It rendered the SSD bootable to that partition

When you’re installing this fresh copy of Windows, leave the network disconnected because you don’t want to get snagged into doing lengthy updates for no reason.

Furthermore, don’t bother changing any settings or doing anything other than the default procedure. Because the next step will wipe all of that out.

The Next Step

Connect the original SSD via USB (or via whatever).

Fire up Clonezilla.

Set it to expert mode.

Don’t let expert mode intimidate you. Most everything is going to stay at the default.

Select a local partition to local partition clone.

Choose the OS partition on your original drive as the source.

Choose the OS partition on the new drive as the target.

Check and check again and again that you have the right source and target selected. If you get it the wrong way around, you’ll end up with your virgin Windows install overwriting your original OS and related files.

When the option comes up, choose to clone the partition proportionally so that it fills the disk. (I’m assuming that you’re going up in size to a larger drive like I was. But either way it should work even if you’re using a drive of the same size. Smaller will not work.)

Commence the clonein’.

Once complete, disconnect the original SSD that’s connected via USB, and remove the Clonezilla flash drive.

Reboot, and you should now have a functional copy of your old system drive.

The new drive will also be marked as a basic disk and can therefore be cloned by most any software until your heart’s content.

myisamchk

If you’re reading this, it’s probably because you’re trying to run myisamchk on a large table and you want to set the --sort_buffer_size to some large value. That’s advisable, of course.

However, it seems that some versions of MySQL don’t understand a 2-digit value for that parameter.

I was trying to do the following:


myisamchk -f -r --update-state --key_buffer_size=4G --sort_buffer_size=16G --read_buffer_size=32M --write_buffer_size=32M /var/sql/myisam-temp/mastersitedb/applicants_master_search
- recovering (with sort) MyISAM-table '/var/sql/myisam-temp/mastersitedb/applicants_master_search'
Data records: 0
- Fixing index 1
myisamchk: error: myisam_sort_buffer_size is too small
MyISAM-table '/var/sql/myisam-temp/mastersitedb/applicants_master_search' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag


If you’re getting an error like that, try using --sort_buffer_size=9G. It took me about 20 minutes to figure that out (which is forever when trying to recover a DB). Hopefully this helps you. :)

Also, --myisam_sort_buffer_size is not a valid parameter, no matter how much the error message leads you to believe that it is.

I got this because it looked like a cool little project, and a neat (if tiny) gift for the missus. So that’s about the extent of my motivations. If you’re trying to put one of these together, I’m hoping that this video might answer a couple of your questions. It’s not really meant to be an educational video, though; More a demonstration of what’s involved in assembly for anyone considering a purchase. Speaking of which, if you want one you can pick one up for about 5 bucks over at banggood.com.

And in case you were wondering, this is not a paid advertisement. I actually paid them for this thing, so it’s pretty much the opposite. (If this kit were complete crap I would tell you.)

2D Version

3D Version

For viewing with Google Cardboard, Samsung Gear VR, Rift, etc.

A couple of people have rightly told me that you can just set a 3D video to 2D viewing mode, and therefore don’t need to upload 2 versions of the same video.

There are a couple of problems with simply uploading one 3D video, however:

• Because the two sides of the video are compressed horizontally, viewing it in 2D stretches one of the “eyes” to full screen. That means it’s much lower quality with half the horizontal resolution versus a regular 2D version.
• As far as I can see, YouTube defaults to anaglyph (red/blue) mode when browsing 3D videos on a regular monitor. That’s a potential turn-off for any viewer that doesn’t know about the switch to view it in 2D instead.

If I’m off base here, please let me know. I’m still learning this whole 3D creation process.

It turns out that I have more battery chargers than sense, so I built this monstrosity.

My old style programmable thermostat died last weekend, and so I rushed over to Home Depot to get a replacement. I wanted a smart thermostat mainly for its wifi connectivity, but also wanted to try out the truly “smart” aspect of it: Optimizing heating and cooling cycles to suit my needs.

They did an excellent job with the user interface, making setup really easy. And their app, though slow to connect at times, is overall well-designed and easy to use. So far I like the ecobee3, but my main concern is with reliability/longevity, and that of course remains to be seen.

I’m a leftie, but all my life I’ve used right-handed mice in my left hand. I needed to replace my old Microsoft Optical Mouse, and found the DeathAdder Left-Handed Edition. It’s the perfect size and shape for me, but they did the weird thing of switching the left- and right-click buttons.

It’s easy enough in most any operating system to swap the buttons in settings. However — at least with Windows — the buttons are only changed locally. So when connecting to other hosts via Remote Desktop the buttons revert to their hardware configuration. That’s a no-go for me, but I liked the mouse so much that I decided to mod the hardware instead.

Fortunately, it’s very easy to reconfigure the buttons in the DeathAdder. The buttons are on a separate circuit board from the sensor and control circuitry. The boards are connected by a ribbon cable, and it’s just a matter of swapping two conductors on that cable.

Razer Support

I don’t make use of companies’ customer support too often, because I prefer to solve most problems myself. Plus, I’m more often than not completely disappointed by support interactions.

Razer’s support committed one of the cardinal sins as far as I’m concerned: They didn’t actually read my original email, and replied with a cookie-cutter solution (which didn’t solve anything).

Another pet peeve of mine is that they wouldn’t address the issue on Twitter, and instead directed me to their web-based support form on their site. I’m grateful that they responded quickly on Twitter, but the useless support response took over 24 hours.

.@RazerSupport Re the left hand DeathAdder; It's great, but are reversed buttons typical? Sadly changing in sw doesn't translate over RDP :/

— Scott Dot (@SCOTTdotdot) September 1, 2016

Basically what happened was that in my support submission I explained my issue with the buttons and their not working in RDP despite changing the buttons’ purpose via Control Panel, and more importantly asked the very specific question: “Is there a way to change the buttons in hardware?”. (There’s a button on the bottom of the mouse to change profiles, so I was hoping that it had an undocumented or poorly-documented ability to do that.)

The response that support gave was step-by-step instructions on switching the buttons around in the Windows settings. And that’s it.

I wrote back to them, briefly expressing my irritation that they didn’t actually read my original question, and asking again if the buttons could be switched in hardware.

The answer was no.

And that’s why this video exists.

The folks over at Creation Crate got in touch and asked me to do a review of their monthly electronics project subscription service. I like the whole “crate” concept, and I like electronics, so figured I’d go for it.

This is an unboxing, review, and assembly of the whole thing, cut down to an unusually-short 16 minutes or so.

Overall, I like the Creation Crate, though there are a couple of minor points with which I took issue. But if you’re looking for a gift that’s educational and fun (for those that find electronics to be fun, at any rate), then you could do a lot worse. I’m kinda basing this on my own childhood, but this seems like it would be perfectly suited to pre-teens and early teens, but would be good for anyone just starting out with basic electronics and programming.

If you’re interested, they can be found at mycreationcrate.com.

And in full disclosure: The kit was provided to me for free by Creation Crate, but this is not a promotional video.

I would have made the very same video and said the same things if I’d paid for the kit, and at a value of ~$30 bucks it’s not exactly enough for me to sell myself. On the other hand, if anyone from Tesla ever reads this, I will guarantee you a good review if I get to keep the car.

This video covers almost everything you need to know about HDDs and SSDs to make an educated choice before pulling the trigger on a purchase.

Well, almost everything. This video is primarily directed at the home user trying to get some additional storage, whether that be an extra drive in their workstation, a NAS, or a full file server. If you’re trying to choose between SAS drives for a large-scale datacenter installation, then you probably know this stuff already.

As I promised in the video, here’s some links to my HDD TCO worksheet from 2013 (though I may have updated it in the interim with a couple of “new” drives). Please download the Excel version and use this for any drive that you want. If you make any interesting changes or large scale updates, I’d love to see it, so please let me know!

I also added a bunch of notes so you know what I meant with all the field names.

HDD Total Cost of Ownership Worksheet (Google Sheets – View only)

HDD Total Cost of Ownership Worksheet (Excel – View and edit)

Hopefully you’ve already checked out my other video: Project Fi – From 1996 to Nexus 6. In that video I discuss Project Fi, the Nexus 6, and the cellular industry in general. It started out with an opening shot of twelve Nexus Sixes (or Nexus 72, if you prefer).

In another blog post and video I showed off my motion control slider project. I’ve been looking for excuses to use it in particularly “motion-controley” ways, and so I decided to composite a bunch of passes of the camera over my one Nexus 6 to make make many of them.

As you can see, it’s done in front of a green screen. That let me isolate the Nexus 6 in each pass, which I then layered on top of each other in Adobe After Effects. I just used a mirror effect in Adobe Premiere to create the left side of the “wall”.

I thought that this was a good use of motion control for compositing because it would have been nearly impossible (or at least very difficult) to make each of the 6 passes of the camera happen at exactly the same speed if I did it by hand.

My green screen and the holder for the phone (just a mini light stand covered in green masking tape) weren’t perfectly uniform, either in texture or lighting, so the matte came out a little rough and you can see some artifacts around the phones.

I’ll admit that I’m a lazy man, so rather than re-light and re-shoot the entire thing I used a garbage matte to get rid of the worst of it. With the footage having been sped up 3x and the white-to-normal dissolve thing, it’s not overly noticeable in the final product. (Or at least I’m hoping that you didn’t notice it.)

If you have any questions about how I put it together, or any advice for me on how to get a better result next time, please let me know in the comments!

My office uses Google Talk for intra-employee instant messaging. This Monday all users got a broadcast message from Google saying that the Google Talk desktop client will cease working on February 15. (Though this may be an old automated notification from when Google was threatening to EOL Talk last February.)

Update (2015-03-09): They finally did kill Talk for Windows as of February 23, 2015.

Of course we can’t take the risk of Google actually shutting down our IMs, and I personally don’t like the new Hangouts Chrome app.

Moreover, we want to limit employees to only messaging other people in our organization. We also don’t necessarily want Google being a party to all of our communication. That means running our own IM server.

ejabberd is a well-known workhorse for IM. Yes, I’ve heard great things about Openfire and others, but I decided to go ejabberd nonetheless.

Environment and Goals

Our organization runs a Windows 2008 AD infrastructure that is very simple: One forest, one domain, one organizational unit. (There’s only about 15 employees.)

I had three goals for our IM solution beyond the obvious ability to chat:

• Authentication to AD
• Interchange of employee particulars (real name, email address, etc.) between the IM server and AD
• Chat contacts defined in AD

The reason for all three of these is that it’s an administrative pain to maintain a separate system and have to create all user accounts twice (something I was doing with Google). Out of the box, ejabberd fit the bill on all three with the following features:

• LDAP authentication
• mod_vcard_ldap
• mod_shared_roster_ldap

We are largely a Windows workstation / Linux server shop, which is why we use Active Directory and I’m running ejabberd on Linux.

It sounded perfect on paper, but in reality it was quite an affair to get it 95% functional.

The Challenges

Installation Woes

By default, the installation binary puts ejabberd in the path /opt/ejabberd-14.12. There are no external dependencies and no config files in etc. Everything is within that directory.

I’m using CentOS, so I followed along with these comments:

The init file doesn’t support chkconfig because it’s missing the required header:

Just adding the following to the beginning of ejabberd.init does the trick:

#!/bin/sh
#
# ejabberd	Startup script for the ejabberd XMPP Server
#
# chkconfig: - 99 10
# description:	ejabberd XMPP server

# Source function library.
. /etc/init.d/functions

set -o errexit

The numbers after chkconfig are boot-up and shutdown priorities, they may need tunning but those seem safe. I don’t think chkconfig is distro-specific, I’ll ask ejabberd developers to add this to the distribution package.

So after adding that to ejabberd.init, all that is left to do is:

cp ejabberd.init /etc/init.d/ejabberd
chmod +x /etc/init.d/ejabberd
/usr/sbin/groupadd -r ejabberd
/usr/sbin/useradd -g ejabberd -p ejabberd -r ejabberd
/sbin/chkconfig --add ejabberd
/sbin/chkconfig ejabberd on

Another issue I ran into was that I installed ejabberd as root, but I was trying to run it as the user ejabberd and so got the error:

sed: can't read /opt/ejabberd-14.12/conf/ejabberd.yml: Permission denied
sed: can't read /opt/ejabberd-14.12/conf/ejabberd.yml: Permission denied
sed: can't read /opt/ejabberd-14.12/conf/ejabberd.yml: Permission denied
sed: can't read /opt/ejabberd-14.12/conf/ejabberd.yml: Permission denied
mkdir: cannot create directory `/opt/ejabberd-14.12/database': Permission denied
./ejabberdctl: line 125: cd: /opt/ejabberd-14.12/database/ejabberd@localhost: Permission denied
sh: /opt/ejabberd-14.12/bin/erl: Permission denied

The solution there is pretty simple: chown -R ejabberd:ejabberd /opt/ejabberd-14.12

Should the ejabberd server fail to start or if it crashes, there may still be ejabberd processes running in the background. That may prevent it from starting again.

# ps -ef |grep ejabberd
ejabberd  1429     1  0 17:04 ?        00:00:00 /opt/ejabberd-14.12/bin/epmd -daemon
ejabberd  1431     1  3 17:04 ?        00:00:04 /opt/ejabberd-14.12/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-14.12 -progname /opt/ejabberd-14.12/bin/erl -- -home /opt/ejabberd-14.12 -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -pa /opt/ejabberd-14.12/lib/ejabberd-14.12/ebin -mnesia dir "/opt/ejabberd-14.12/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -sasl sasl_error_logger {file,"/opt/ejabberd-14.12/logs/erlang.log"} -smp auto start

# kill 1429
# kill 1431

Finally, I had one more issue when stopping the service:

service ejabberd stop
Stopping ejabberd...
/opt/ejabberd-14.12/bin/ejabberdctl: line 364: epmd: command not found

Another easy one, because the error message is quite clear. There are a few lines in the file /opt/ejabberd-14.12/bin/ejabberdctl which assume that epmd is in the same directory… which it is! However, ejabberdctl is being invoked by the init script without changing directory to that path.

You can deal with this in three ways:

• Change ejabberdctl to contain the full path of epmd
• Change the init script to cd into /opt/ejabberd-14.12/bin
• Add /opt/ejabberd-14.12/bin to the path

Default SSL Certificate

The first thing I did was start up a plain-Jane configuration of ejabberd and try to connect to it. I was testing it out with both Pidgin and Spark

I don’t remember which of the two clients it was, but one of them refused to connect to the server due to an SSL error. The error message wasn’t explicit, but from what I read online it appeared that an expired certificate would stymie the client. So it seems that the default PEM file included with the ejabberd binaries is invalid.

A self-signed cert isn’t a problem, so I generated one using OpenSSL and gave it an expiration 10 years hence. Not a real issue, but something that you should do right off the bat.

LDAP Authentication

Getting LDAP authentication working wasn’t much of a problem. The documentation is pretty much all you need.

Here’s a sanitized snippet of my LDAP configuration. Note that I created a user with the common name Ejabberd LDAP as an LDAP reader.

auth_method: ldap
ldap_servers:
   - "dcserver01.mydomain.com"
   - "dcserver02.mydomain.com"
ldap_encrypt: none
ldap_port: 389
ldap_rootdn: "CN=Ejabberd LDAP,CN=Users,dc=mydomain,dc=com"
ldap_password: "SomePassword0239"
ldap_base: "cn=Users,dc=mydomain,dc=com"
ldap_uids:
   - "sAMAccountName"
ldap_filter: ""

That’s enough to get LDAP authentication working.

By the way, if you want to get the ldap_rootdn (the user’s Active Directory distinguished name) fire up Active Directory Users and Computers. Open the properties of the user, click on the Attribute Editor tab, and scroll down until you find distinguishedName in the list.

mod_vcard_ldap

This should have been easy, but it was far from it.

First off, at some point the developers of ejabberd switched to a YAML-formatted config file, so most of the examples you’ll find online are in JSON. Once you get a handle on YAML it’s not a big deal to visually bounce between the two, but I’m not accustomed to YAML so there was a bit of a learning curve there. (By the way, “every JSON file is also a valid YAML file“.)

Update (2016-03-25)

Below I point out that a typo in the documentation caused me some grief. I’m leaving that in this post for posterity (or in case you’re referencing an old or archived version of the docs), however Mickaël Rémond from ProcessOne was kind enough to comment that the doc has since been fixed.

Thanks Mickaël, much appreciated!

The real problem was that I copied the mod_vcard_ldap config snippet straight from the docs on Active Directory LDAP integration. Here’s what it says:

  mod_vcard_ldap: 
    ldap_vcard_map: 
      "NICKNAME": {"%u", []}
      "GIVEN": {"%s", ["givenName"]}
      "MIDDLE": {"%s", ["initials"]}
      "FAMILY": {"%s", ["sn"]}
      "FN": {"%s", ["displayName"]}
      "EMAIL": {"%s", ["mail"]}
      "ORGNAME": {"%s", ["company"]}
      "ORGUNIT": {"%s", ["department"]}
      "CTRY": {"%s", ["c"]}
      "LOCALITY": {"%s", ["l"]}
      "STREET": {"%s", ["streetAddress"]}
      "REGION": {"%s", ["st"]}
      "PCODE": {"%s", ["postalCode"]}
      "TITLE": {"%s", ["title"]}
      "URL": {"%s", ["wWWHomePage"]}
      "DESC": {"%s", ["description"]}
      "TEL": {"%s", ["telephoneNumber"]}]}

Do you see the problem? I sure didn’t, and ejabberd was spitting out the following error in the log file:

2015-02-03 17:17:08.699 [error] <0.36.0> CRASH REPORT Process <0.36.0> with 0 neighbours exited with reason: {undefined_macro,''} in application_master:init/4 line 133

Here’s the problem: The documentation’s example uses commas when it should be using colons. That’s it. Of course I didn’t notice/know that for a few hours, so I tried every single bit of nonsense possible to reformat that section into valid YAML.

This is the correct, working mod_vcard_ldap section of my configuration. Note that I pared it down a bit because I don’t populate all of the VCard fields in Active Directory anyhow:

  mod_vcard_ldap:
    ldap_uids: {"sAMAccountName": "%u"}
    ldap_filter: ""
    matches: infinity
    ldap_vcard_map:
      "NICKNAME": {"%s": ["displayName"]}
      "FN": {"%s": ["displayName"]}
      "EMAIL": {"%s": ["mail"]}
      "GIVEN": {"%s": ["givenName"]}
      "MIDDLE": {"%s": ["initials"]}
      "FAMILY": {"%s": ["sn"]}
      "ORGNAME": {"%s": ["company"]}
      "ORGUNIT": {"%s": ["department"]}
      "TITLE": {"%s": ["title"]}
      "TEL": {"%s": ["telephoneNumber"]}
    ldap_search_fields:
      "User": "%u"
      "Full Name":  "displayName"
      "Email": "mail"
    ldap_search_reported:
      "Full Name": "FN"
      "Nickname": "NICKNAME"
      "Email": "EMAIL"

It’s not strictly-speaking necessary to define ldap_uids in this section because ejabberd will use the LDAP settings you’ve previously defined. However you should note that both the documentation and the plurality implied by the field name are inaccurate. In the main LDAP config, the ldap_uids can be a list (array, whatever):

# Correct in main LDAP section
ldap_uids:
   - "sAMAccountName"

In the mod_vcard_ldap section, it is an object/mapping/whatever. An error is thrown if you provide more than one uid field. Also, the second (%u) parameter is required.

# Correct in mod_vcard_ldap section
ldap_uids: {"sAMAccountName": "%u"}

Hopefully my configuration examples here will save you the same headache I faced.

mod_shared_roster

Oh man, was this ever a pain. There are many conflicting examples, many half-baked workarounds, and many compromises to be decided upon.

I’m unclear on whether or not this is still true in the current release, but apparently spaces in a user’s CN causes mod_shared_roster to fail silently.

mikekaganski wrote:
That’s the most trouble, because the “member” stores its members as DNs, thus if you have your users like “CN=John Doe,OU=blah,OU=blah,DC=example,DC=com”, then we’re stuck. In the better case when you have CNs of your users without spaces, you may choose to use ldap_memberattr_format_re = “CN=(\\w*),(OU=.*,)*DC=example,DC=com” (this is from the guide, I didn’t test this regex). Source: https://www.ejabberd.im/node/4826

It took me a while to find that. What does it mean? Well the overly-simplified version of the shared roster LDAP module’s alogrithm goes like this:

1. Run the ldap_rfilter query to get a list of groups that contain Jabber-able contacts
2. For each of those groups, run the ldap_gfilter query to get the group’s displayable name and member list.
3. For each distinct group member retrieved in step 2, get the user’s displayable name.

Everything went wrong for me in step 2. All of my users have their full name as their common name, as you can see below:

# ldapsearch -LLL -H ldap://dcserver01.mydomain.com -x -D 'mydomain\ejabberd.ldap' -w 'SomePassword2098' -E pr=1000/noprompt -b 'dc=mydomain,dc=com' '(&(objectCategory=group)(cn=All Employees))' displayName member

dn: CN=All Employees,CN=Users,DC=mydomain,DC=com
member: CN=Jennifer Doe,CN=Users,DC=mydomain,DC=com
member: CN=Eric Von Lastname,CN=Users,DC=mydomain,DC=com
member: CN=Kieran Wonderbra,CN=Users,DC=mydomain,DC=com
displayName: All Employees

Note that I truncated and sanitized the output above.

As you can see, all the common names have spaces in them, and so it (I believe) is unparsable. I tried variations on the ldap_memberattr_format and ldap_memberattr_format_re properties with no success.

After much hair pulling, this is my working configuration:

  mod_shared_roster_ldap:
    ldap_groupattr: "sAMAccountName"
    ldap_groupdesc: ""
    ldap_memberattr: "sAMAccountName"
    ldap_memberattr_format: "%u"
    ldap_useruid: "sAMAccountName"
    ldap_userdesc: "displayName"
    ldap_rfilter: "(&(objectCategory=group)(cn=All Employees))"
    ldap_gfilter: "(&(objectCategory=user)(memberOf=CN=All Employees,CN=Users,DC=mydomain,DC=com))"
    ldap_ufilter: "(&(objectClass=user)(sAMAccountName=%u))"
    ldap_filter: ""
    ldap_group_cache_validity: 60
    ldap_user_cache_validity: 60
    ldap_auth_check: off

Remember what I said about compromises? Well, my ldap_gfilter query isn’t really giving the module what it wants: A list of group members and the group’s displayable name. That’s why I left ldap_groupdesc blank; My query does not return the group’s name. This means that my group’s name isn’t propagating to the chat clients. The clients simply display a list of contacts outside of any particular group. I have no problem with that, because I only have the one group that contains all employees.

Here’s the output of my ldap_gfilter to contrast with the earlier example that retrieved a group’s members with just their distinguished names:

# ldapsearch -LLL -H ldap://dcserver01.mydomain.com -x -D 'mydomain\ejabberd.ldap' -w 'SomePassword2098' -E pr=1000/noprompt -b 'dc=mydomain,dc=com' 
\ '(&(objectCategory=user)(memberOf=CN=All Employees,CN=Users,DC=mydomain,DC=com))' sAMAccountName

dn: CN=Kieran Wonderbra,CN=Users,DC=mydomain,DC=com
sAMAccountName: kwonderbra

dn: CN=Eric Von Lastname,CN=Users,DC=mydomain,DC=com
sAMAccountName: elastname

dn: CN=Jennifer Doe,CN=Users,DC=mydomain,DC=com
sAMAccountName: jdoe

Again, the output above has been sanitized and truncated for brevity.

I’m now retrieving the sAMAccountName, which I’ve referenced in the ldap_useruid. The contents of sAMAccountName (e.g. jdoe) is then used in the ldap_ufilter in place of %u.

Also, note that I left my group common name (CN) of “All Employees” hard-coded in all three filters. In the ldap_gfilter and the ldap_ufilter fields you can use %g, which will be replaced with the content of the field defined in ldap_groupattr (which is “sAMAccountName” in my example). I only want to have one group which represents all ejabberd users, so this works for me.

Debugging Tools

ldapsearch

Install and run this on your ejabberd server. First off, you’re ensuring that you have connectivity between that server and your LDAP server (e.g. domain controller).

It will also allow you to independently test all of the LDAP filters in your configuration.

ldapsearch is well documented, but here’s an example of its usage as appropriate to my environment:

ldapsearch -LLL -H ldap://dcserver01.mydomain.com -x -D 'mydomain\ejabberd.ldap' -w 'SomePassword2098' -E pr=1000/noprompt -b 'dc=mydomain,dc=com' '(&(objectClass=user)(sAMAccountName=scott))'

That statement will retrieve all of the LDAP attributes for the scott user account.

tcpdump and Wireshark

OK, these are by no means the only appropriate tools out there, but they’re what I used.

In my environment, the traffic between the ejabberd server (in the public server “DMZ” network) and my domain controller (in the private office network) pass through a Linux-based router (i.e. a desktop PC with a lot of NICs). I used tcpdump on the router to capture all of the packets between the two machines on port 389 (LDAP) to a file, and then used scp to transfer the packet dump to my Windows workstation for analysis with Wireshark.

Incidentally, the reason I captured packets at the router instead of on the ejabberd server was so that I could rule out connectivity/firewall issues. The reason I didn’t run Wireshark on my domain controller is because I consider it bad practice to install anything unnecessary on my DC.

Here’s the tcpdump command to get a file that Wireshark will parse (obviously replace the IP address with that of your own ejabberd server):

tcpdump -s 0 -w ~/tcpdump.ldap.20150205-1202.pcap -nnXSvi eth4 "port 389 and host 10.101.1.57"

Also, I chose to use Wireshark rather than reading through the raw tcpdump output because it formats everything quite nicely:

Conclusion

ejabberd does suffer a bit from being long on examples but short on consistency due to its old age. It’s also developed (logically) by Linux/erlang people, and so they’re not as Active Directory LDAP friendly as I’d hoped. (I don’t blame them for that, of course. I went with a Linux-based ejabberd server for a reason.)

On the other hand, ejabberd is quite mature and extremely stable. It’s been a great replacement for the proprietary and non-AD-integrated Google Talk.

I know that I didn’t give a comprehensive how-to guide here, but I’m hoping that at the very least my example configuration snippets point you in the right direction.

As stated in the title, I am using ejabberd 14.12. I installed using the binary installer from the process one website.

Back in August I reviewed the Netgear 6100D from Sprint and followed up with a post detailing some advanced configuration options.

The Video

I also installed a flat panel 4G antenna from 4G Antenna Shop. I made a video detailing the unboxing and installation (which I just got around to editing together):

It’s my first video of this sort, so if you have any feedback please let me know in the YouTube comments or by email!

4G Antenna Shop

I didn’t get into it in the video, but overall I’d recommend 4G Antenna Shop. The cable and antenna I got were both of very high quality and definitely worth the price.

Their customer service was great; I had a couple of questions about my order, and one of their guys (Robert) got back to me within 15 minutes and was extremely helpful. They shipped really quickly, too.

I did have two minor issues, both of which I talk about in the video:

During checkout they give you the option of selecting your device so that they can provide the correct adapters to go from the cable (if you order it through them) to the device. At the time I’d ordered, they had an option for “Netgear Sprint Spark LTE”, which I thought was the Netgear 6100D. There was no separate option for the 6100D, but it turned out that they were referencing a different product, and so I received the wrong adapters. (They’ve since added the 6100D as an option.)

I chalked this up to being mostly my fault, as I didn’t know that there was another Netgear LTE device out there for Sprint Spark.

My other issue was with the packing job. Again, it’s a minor complaint because nothing was damaged, but the box arrived pretty beaten up with holes in the top from the antenna mount having poked through. There was no packing material to keep the box rigid, and the antenna and cable were just sorta rattling around inside.

Bear with Me…

Oh, and sorry if I rambled on a bit in the video. If you couldn’t tell from a lot of my other posts on here, I have an aversion to brevity. :)

I’m hoping to get some more how-to and instructional videos out there in 2015, so please subscribe to my YouTube channel!

(Hey, I’m allowed to shill for myself, right?)

Bullhorn vs. Exchange 2003

One of the companies for which I manage IT uses Bullhorn’s applicant tracking software for their recruitment workflow.

That company also uses the now-ancient Exchange 2003 for their email.

But, Bullhorn doesn’t officially support integration with Exchange 2003.

What’s involved?

First off, “integration” is a strong word. It implies that our servers will pass information back and forth and stay in some meaningfully synchronized state. That’s not the goal in this case. The integration simply consists of passing all emails that are sent and received by our recruiters to Bullhorn’s servers.

Once Bullhorn receives the emails, they’re parsed and can be viewed in the Activity Center and/or under the contact record to which they apply (using email address matching).

Exchange journaling

The integration path I wanted to take was to use message journaling in Exchange. In essence this forwards a copy of every single message sent and received on our server to Bullhorn. There are other more convoluted options, but this seemed to be the most straightforward from a sysadmin perspective.

Unlike later versions, Exchange 2003 doesn’t support journaling rules. Those would have allowed me to journal messages that were to/from a particular user or set of users.

Instead Exchange 2003 supports mailbox store-level journaling. In other words, every email to/from a mailbox on a particular store is “forwarded”. (It’s technically different than forwarding, but is conceptually similar enough for this discussion, so I’ll drop the quotes going forward.)

This isn’t a problem in my case; Our organization is relatively small, and so it was trivial to move all of my Bullhorn users to one mailbox store, and move some other users out.

Resources

I’m not going to go into depth regarding setting up journaling because there are quite a few good resources available on the subject:

MSExchange.org – Implementing Exchange 2003 Message Journaling

Microsoft Technet – How to Deploy Exchange Server 2003 Journaling as Part of a Compliance Solution

Bullhorn – Enabling the Bullhorn Email and Calendar Integration with Microsoft Exchange [2007 through 2013]

Google around; You’ll find a lot more information.

Create the contacts

The short of it is that you first need to obtain from Bullhorn your tracker address. It looks something like this:

<corpname><.corpID>@slXtracker.bullhornstaffing.com

You’ll also need tracker addresses for each of your Bullhorn users. Those look something like this:

<username>@slX.bullhornstaffing.com

In Active Directory Users and Computers create a contact for each of your Bullhorn tracker addresses. For sanity’s sake I’ll make up some examples.

Let’s say our corporate tracking address is:

recruitfast.1234@sl1.bullhornstaffing.com

And we have two Bullhorn users, each with these tracker addresses:

Malcolm Reynolds – mreynolds.rec@sl1.bullhornstaffing.com
Zoe Washburne – zwashburne.rec@sl1.bullhornstaffing.com

Here’s what it should look like when you create the corporate tracker contact:

Of course you can give the contact a first and last name that you find to be appropriate. But that works for me!

Modify the “E-mail” field, and add a new “SMTP Address” using your corporate tracker email.

Once the contact is created, go back in and edit it. Under the “Exchange Advanced” tab, check the box next to “Hide from Exchange address lists”. This will help prevent users from sending email directly to this address.

There are more advanced ways to prevent users from emailing an internal recipient. Since the risk of that happening is low and I like to have the ability to run tests later by manually sending emails, I haven’t bothered.

Now create contact records for Malcolm and Zoe. It’s the same process, though I would name them in this fashion:

Malcolm Reynolds (Bullhorn Tracking)
Zoe Washburne (Bullhorn Tracking)

By adding “(Bullhorn Tracking)” to their last name (or just to the display name) there won’t be any confusion in the future as to whether you’re looking at their user account or their contact record.

I’ve been frustrated by this myself: Depending upon your Active Directory and Exchange environment, it may take some time for the contact records to propagate to your Exchange server(s). Be patient if you don’t see them right away.

Enable Journaling

I’m assuming that at this point you have all of your Bullhorn users on one mailbox store. In my example, it’s going to be “Mailbox Store 2″.

In Exchange System Manager (ESM), find the store and open the properties.

Click on the image for a larger version.

Check the box next to “Archive all messages sent or received by mailboxes on this store”. It doesn’t say “journaling” — why make it easy? — but in this context the “archive” is the journal.

Click “Browse” and locate your “Bullhorn Tracker (Corporate)” contact.

Apologies that in this screenshot I left off the “(Corporate)” part. It’s the same contact.

Hit “OK” to set the contact, and then “OK” to save the mailbox store properties.

Congratulations! You’ve now set up journaling! (Pretty easy, huh?)

But what about envelope journaling?

Exchange 2007 and above only do envelope journaling. Exchange 2003 does not have the ability to do envelope journaling by default, and there is no way to enable it in the GUI. So is that why Bullhorn doesn’t work with Exchange 2003?

Apparently not.

The article from MSExchange.org that I posted above talks all about configuring envelope journaling in 2003. I found that it makes no difference to Bullhorn whether it’s enabled or not. Hence I don’t think that there’s a point to configuring it. (Though it couldn’t hurt, and it’s really easy!)

Incidentally, I also found a few articles relating to email journaling to a third-party compliance solution. One of them discussed that Exchange 2003 does not actually forward the full envelope over SMTP even with envelope journaling enabled. Their suggestion was to create a local journaling mailbox, then to enable user-level forwarding for that mailbox to the third-party receiver. I found that did not fix the situation either.

So are we done?

You’d think that would be that. We want to track both incoming and outgoing emails in Bullhorn, and Exchange 2003 is now forwarding both incoming and outgoing emails to your Bullhorn tracking address.

But here’s the rub: It doesn’t work for inbound emails, and since Bullhorn won’t talk to me about my integration because I use Exchange 2003 I have no idea why it doesn’t work. It should work, but it doesn’t.

Email forwarding to the rescue

To get inbound emails to show up in Bullhorn I had to configure email forwarding at the user account level in addition to journaling!

If you’re the usual Exchange administrator then this should be a cakewalk for you. Let’s skip right to the screenshot:

As usual, click on the image for a full-sized version.

Open up the trusty old Active Directory Users and Computers management console.

Locate the Active Directory user account for your Bullhorn user and open its properties.

Click on the “Exchange General” tab, and then click the “Delivery Options..” button.

In the delivery options dialog, click the radio button next to “Forward to:”.

Click the “Modify” button and find the contact for your Bullhorn user. In this case it’s “Malcolm Reynolds (Bullhorn Tracking)”.

Check the box next to “Deliver messages to both forwarding address and mailbox”. (That’s an easy one to forget!)

Click “OK” to save and close the delivery options, and “OK” to save and close the user’s properties.

And that’s it!

You should now see emails flowing into your Bullhorn Activity Center and/or to your candidates’ and contacts’ activity log under “Email”:

Sorry for all the redactions!

Here you can see that both inbound and outbound emails are associated with my candidate record!

The eagle-eyed reader may have noticed that the first email in the list has a timestamp of 1139 (11:39 AM) in the subject field, but the “Date Sent” column says 6:36 PM. This is not an issue with the integration; For some reason the Bullhorn website was extremely slow today to the point of being unusable. I’m assuming that’s the reason that the email didn’t show up in our account for 7 hours.

Testing

I created a candidate record in the system under my name, and gave it two of my personal email addresses.

I’m not a Bullhorn user, but I hopped onto one of our users’ email accounts and sent a test email (or two) to my personal address.

I also sent an email (or eight) to the Bullhorn user’s corporate email address.

Because my personal email address matched an email address on my candidate record, those emails show up in the “Activity” tab under “Email”!

If they don’t show up under a candidate or contact record, they’re probably in the Activity Center (under the main menu).

If you aren’t seeing any messages in Bullhorn, try journaling messages to an external email account that you control.

For example, if your personal email address is butterface1990@gmail.com then set up an Exchange contact just as you did with the Bullhorn tracking contact. Change your mailbox store’s journal settings to use that contact.

You should start seeing messages in that GMail account. If no messages come through then the problem is likely on your server(s) and/or network and not with Bullhorn. (As always, give it some time for that journaling change to propagate through your Active Directory and Exchange servers. Oh, and check your spam folder!)

Caveats

At some point we may start seeing duplicate emails in the system. Bullhorn appears to be ignoring inbound emails that are journaled to the corporate tracking email address. That behavior may later be fixed on Bullhorn’s end, in which case we may see the situation where both the user-level forwarded emails and the inbound journaled emails show up in our account!

No official Bullhorn support. I’ve done my best to get this integration to work. If it breaks down the line then we may just be plain out of luck.

This is not an ideal situation for a large organization. We have 3 Bullhorn users, and likely won’t be increasing that number tremendously over the next few years.

If your organization is large (that’s a subjective term, I know) then you may wind up putting too much load on your server and/or network by journaling and forwarding messages for a lot of users. You may also run out of mailbox stores! (They have size and quantity limits.)

In conclusion

Yes, integration with Exchange should work with just jounaling in place and without the need to forward emails at the user level.

And believe me, I tried every which way to get inbound journaled messages to show up in Bullhorn! Feel free to try some other solutions yourself (and if you succeed let me know), but I’m hoping that this article saves you the time. It cost me a lot of mine!

Of course, we haven’t even talked about calendar integration. That’ll be a post for another day, perhaps.