New Hard Drive in the House

coolgear firewire sata enclosure housing a 500gb maxtor maxline pro
My new file storing powerhouse.

I like to keep my eye on on the price of hard disk storage. More storage more better is my philosophy. Not only can I never tell when I’ll need an extra gigabyte or two for projects, but I’m justifiably paranoid about data failure and have become something of fiend for backups as a result. So I’ve been watching with interest as the price of 500GB drives have fallen at a steady clip since the beginning of the year.

My interest was piqued recently when I noticed that a number of external 500GB drives have been hitting the ~$100 area lately, which is nearly too good a deal to pass up. My enthusiasm was tempered though, by the observation that all the drives I found were USB 2.0 only. USB is no slouch and a fine enough protocol, but real file transfers, particularly large ones, are best left to FireWire. Unfortunately, FireWire versions seem to cost $50-$90 more than their USB-only counterparts.

So I did what I’ve done for the last 5 drives I’ve purchased: I looked into do-it-yourself solutions. Assembling an external drive from an internal one plus an enclosure is a trivial task and it helps you get exactly what you want. In the past, that approach has helped me save some cash at the same time.

Unfortunately, the gap in price between internal and external drives doesn’t seem to be as great as it once was. Mail-in rebates can often bring the price of external drives to below that of internals. Realizing that I probably would be out of luck trying to save some dough, I set out to see how much I could get within the price range for an external 500GB hard drive with a FireWire connection ($160-190).

In the end, I think I did pretty well.

Continue reading

The $200 terabyte has arrived

western digital my book

A couple months ago, I wrote about the advantages of maintaining a large digital music library, specifically with regards to cost and storage as compared to maintaining a large physical music library. One of my points was that the decreasing prices of hard drives makes it increasingly easy to store, as well as backup large quantities of high-quality music.

I even went so far as to say that “before too long” we’d see the $200 terabyte, which is roughly enough storage for 80 continuous days of lossless quality music plus a complete backup of it all.

Well, that day has arrived. OfficeDepot currently is listing a Western Digital My Book 500GB for $99 after mail-in rebate.

At that price, you might as well get two and secure all your digital media.

In defense of digital music files

Volkher at livingwithmusic posted the other day a rather thoughtful treatise against digital music files as a medium. He does a good job of bringing up all the relative shortcomings of abandoning physical media, including the effort required to encode/download and properly organize/tag files as well as the burden and cost that goes into storage and preventative backups. And he’s right on the money about picking an audio format that may or may not be around for the long haul.

It’s a valid argument; you should go read it. But as one of those “young folks” who’s been living with mp3s and related files for 10 years now, I’d like to offer a friendly rebuttal, because digital music files do have much to offer, despite the occasional hassle.

Firstly though, I’m going to side-step rights-management and other DRM-related issues. It’s quite possible to build a large collection of digital music and never touch the stuff. Plus, with all the talk lately about eliminating DRM from the marketplace entirely, it may well not be an issue in the near future.

Carrying on then, why embrace digital music? In my case, the number one reason is convenience and flexibility. Using iTunes, it only takes a handful of clicks to set up a playlist that will last all day. That playlist will only include songs that I like, ignoring ones that I might not care for. I can listen to one hundred different artists as easily as I can listen to Radiohead’s complete discography, including live shows and unofficial tracks. No need to organize or hunt for physical CDs, or interrupt the music to change discs or skip ill-favored songs.

With some extra up-front effort and Smart Playlists, I can turn my library into a self-refreshing and randomized jukebox that I can assume control of at any moment. With an iPod, I can take it all with me, wherever I go. It truly is awesome stuff.

Another reason I enjoy digital music is the physical space savings. I still have a large number of CDs, even though the vast majority of my music listening is done via iPod or iTunes. Finding a place to put all of those discs has proved challenging and, after 19 months of living at my current house, most of them are still boxed up and hard to access. That’s fine though; they can stay in the garage/closet/attic because I already have everything I need on my hard drive.

Additionally, expandability is a significant motivation for taking to digital music. As a physical collection grows, the tyranny of the shelf kicks in, which ultimately limits the collection’s size and imposes increased time-overhead on organization and media retrieval. iTunes offers no practical limit to the number of songs it can manage. Hard drive space and memory are the only true limitations (though a computer’s processor speed can become an issue, especially if there are a large number of live-updating Smart Playlists). Currently, I’m storing about 18,000 files (between my library and my wife’s) + a backup drive in the same physical space as about 6 CDs. I could double the amount of songs and hardly use any more desk space.

Of course, this digital utopia is not without its pitfalls, many of which Volkher mentioned. Number one, by far, is data security and integrity. Hard disk drives are notorious for failing, whether though a mechanical fault or corrupted disk header. And they usually fail inexplicably and at exactly the wrong moment.

A hard drive crash can obliterate a library of thousands in an instant, often with no warning whatsoever. I know; it’s happened to me on multiple occasions. By contrast, a scratched CD might lead to the loss of that CD and nothing more.

Thus, a workable, redundant backup system is necessary to protect against irrevocable and irrecoverable catastrophe. The cost and effort of doing that, of course, increases with the amount of data to be backed up. A 10GB library is easier to deal with than a 100GB.

In his post, Volkher posits a figure of one terabyte of digital music, once all is said and done and a large collection is encoded and/or downloaded. But is that a meaningful gauge? Just how much music will fit in a terabyte?*

  • At 128kbps, the bitrate of standard iTunes-purchased AAC files, one terabyte is 18,641.35 hours / 776.73 days / 2.12 years of non-stop, continuous music listening.
  • At 191kbps, the average bitrate in my library, one terabyte is 12,492.63 hours / 520.53 days / 17.1 months of continuous listening. That number is about 11 times the size of my current library.
  • Suppose you’re a true audiophile and only deal with lossless encoding, such as FLAC or Apple Lossless format. The average bitrate of all the lossless songs in my library is 728kbps, which is still nearly 3,277.6 hours / 136.6 days / 4.48 months worth of continuous audio.

*these numbers do not take into account file overhead, album art, etc. However, it seems pretty clear that one terabyte will hold a lot of music.

You’d have to own a seriously HUGE collection (roughly 4000 full-length CDs, lossless compression; 7500 CDs, extreme quality 320kbps mp3s) before a terabyte is a serious option for the working copy of your library. I know there are people who can claim those numbers, just not the vast, vast majority of music listeners.

Heck, I consider myself a respectable avid collector/listener/explorer of music and it took me a full year to listen to each and every song in my library, a library that is the equivalent of ~1400 full-length records.

Hard disk storage is cheap and getting cheaper. Practically speaking, two 500GB drives would sufficiently provide enough storage for a live copy and backup of all but the most copious of collections. Add a third for redundancy, if you’re paranoid. At $230 a piece as I write this, the cost of the drives compares favorably to all the shelving and organizational furnishings needed to manage a large physical collection, even those from IKEA.

Which brings me back to physical space savings: how does one translate a digital library into a physical space? Let’s use numbers from my library to hazard a guess. A trusty ruler tells me that a standard CD jewel case measures 11mm thick: a “CD unit” for this purpose. FreeDB tells me that the median number of songs per CD is 12 (with an average of 13). Therefore, I can estimate that each song on a CD takes up .917mm of physical space. Applied to iTunes, the 14,554 songs in my library would use up the equivalent of 1443 “CD units” or 15,872mm or 52.1 feet of “shelf space.”

That number is based on the assumption that a digital library consists entirely of full CDs. When considering partial albums and single tracks, the space savings is even greater. The single track of Nate Dogg and Warren G’s Regulate in my library actually saves me the full 11mm of space rather than .917, since I don’t have to own the full album just to have that one song.

Despite my continued devotion to the digital music scheme, I must admit that I do miss some of the concrete and tactile aspects of handling a physical record or compact disc: album art, liner notes the satisfactory “click” of snapping a disc into place and-contrary to what I’ve said above-the awesome feeling of standing back and viewing a neatly organized array of records on shelf after shelf. But at this point, for me, it’s all digital and there’s no going back.

Volkher goes on to discuss the inherent uncertainty of choosing an audio file format that may or may not be in use and supported by audio devices in times past the immediate future. And he’s got a valid point. I know from experience. Long ago, a portion of my digital music collection was in the MP2 format, which was largely made defunct by the growth of MP3 tools and players. The death knell for me was the iPod. I was dismayed when I bought my first one and discovered that it didn’t support MP2, forcing me to convert those portions of my library into something more usable.

So futureproofing is an ever-present concern. But, like the compact disc and vinyl record, there’s no reason to believe that mass-market digital formats won’t be around for a very long time. The use of MP2 was never really widespread. MP3 and AAC however have users in the tens of millions. Many people and many companies have invested a lot of resources into those formats. They’re not going to die any time soon. In fact, the patents on the MP3 format begin to expire in 2011, just 4 years away. I’d wager that individuals, corporations and open source communities will have a field day with it in short order, continuing to breathe life and support into software and hardware for decades.

Look at the passion with which gamers and code archivists continue to resurrect, port and support obsolete games. Just yesterday I ran across a 30+ year old command-line game called Super Star Trek that certainly would not be playable on today’s technology. But thanks to the efforts of a dedicated user, there is now a refreshed version for Mac OS X, Windows, Linux, OS/2 and DOS.

Likewise, I expect that in three decades time, I will fully be able to enjoy my collection, no matter what size it has expanded to. I believe that the flexible nature of software will make it easier to maintain support for these formats as they age, unlike hardware-dependent media (I’m looking at you, reel-to-reel, 8-track and increasingly, cassette tapes).

The fact that I was able to easily convert my MP2 files to MP3 is a strong argument for digital files. Just try converting a reel-to-reel tape without a reel-to-reel player.

I admire Volkher’s decision to keep his music in the real world; I know there’s no better feeling than finding an old, rare, long-sought-after gem. But for me, the future is all digital. It has its trappings, but they are easy to overcome. The rewards outwiegh the risks.

So, now if you’ll excuse me, I have a playlist to build.

mp3 waveform

Cue hard drive failure… Now

Like clockwork, each January means the end of Christmas decorations, the onset of the full desolation of winter, learning the motor-memory of writing a new year and, of course, a sudden, massive hard drive failure. Every year since 2001 it has happened to me, always to my external music drive and always in January.

It’s quite comical, if you have the right sense of humor.

Then it should not have been a surprise to me that, once again, right on schedule, my music drive crashed last night. I was politely updating my iPod, having devised a new experimental listening scheme for the new year. As it would take several minutes to copy the several thousand songs, I decided to have a quick shower.

When I returned, I found my desktop in an unusual state. iTunes was no longer running. Mail and Safari were quit as well. The three FireWire volumes that I normally have mounted (including the music one) were missing and LaunchBar’s command area was active, as if the computer had been restarted.

I had no idea what caused that state of affairs, and still don’t. But I sensed danger, so I decided to do my own restart, which went smoothly enough. Until it was time for the external drives to mount. Two of them did; one of them didn’t and I’ll let you guess which one.

Disk Utility was of no help, failing immediately. It could see the drive, but attempting repair resulted in a message similar to “The underlying task failed on exit.” Whatever the problem was, the drive’s directory looked like it was in bad shape. Fortunately there is a god whose name is DiskWarrior. This diagnostic deity has raised many drives from the dead and after a few minutes, he had raised one more, rescuing my music from binary oblivion.

From there, the iPod update went well, other than about 40 songs that did not make it back from Hades. However, having gotten accustomed to these failures, I’ve become the king of backups. Twice nightly, Synk, the handiest little backup program I know, copies my music volume and other important data to a dedicated backup disk. Some quick drags-and-drops and even those handfuls of missing files were replaced.

Though I still don’t have a clue what caused the malfunction, I gotta say that the experience really wasn’t that bad. Much better than past years. However, despite all my preparations, this is one New Year’s tradition I would rather not repeat.

And even though I didn’t have to make full use of mine, remember that lesson kids: Backup Backup Backup.