Crawford Charts New Path To Digital Storage

Crawford Media Services this year reached a technological crossroads, says company EVP and CTO Steve Davis.

The homespun RAID 6 storage upon which the Atlanta-based post production and media services company depended was coming “uncomfortably close” to failures that would require cumbersome RAID rebuilds from tape backups, he says.

“Just letting it keep going and going would have meant the system would have become too unstable,” he says.

Like over-the-air broadcasters that for years have been transitioning away from linear videotape to a variety of spinning and solid-state disks as well as digital storage tape for longer term backup, Crawford’s dependency on digital storage has skyrocketed.

In fact, Crawford, which divested its satellite systems business in 2010 while retaining its post operations, has built an entirely new digital storage-dependent business by offering broadcasters, government and private institutions as well as sports leagues a set of media management services that include digitization of film and tape libraries, long-term archiving, metadata creation, asset management and full-quality asset retrieval and delivery. 

The danger facing Crawford was if more than two hard drives failed in any given RAID group before the RAID was finished rebuilding, the only alternative was to rebuild the spinning disk storage from media files stored on LTO tapes, an unattractive option for a business focused on maintaining operational efficiency and maximum asset availability.

Crawford’s existing storage, which was based on JBODs (Just a Bunch of Disks) running the Linux Red Hat XFS file system that were configured with RAID 6 double parity controllers and three hot standby disks per JBOD, simply left the operation too exposed to failure as disks aged, he says.

The decision facing Davis and Crawford was whether to undertake a forklift upgrade of its RAID-based storage, which in the short term would fix the failing disk problem but longer term land the operation in the same predicament as disks in those arrays began reaching their end of life, or to find a new approach to digital storage that put the operation on a different trajectory.

For Davis, the solution centered on finding a system that used erasure code, a data protection technique that relies on an algorithm “that is the equivalent of a multivariable equation,” which, for instance, makes it possible to solve for two missing variables if three are known, he says.

In essence, the technique runs forward error correction on data to be stored, he explains.

“I had learned about erasure code several years ago and had spent a lot of time researching it,” Davis says. “By the time our existing system started aging, I knew we wanted to go into an erasure code-based system.”

Erasure code-based systems offered another attractive feature: small storage nodes, which typically are a 12-disk unit.

“By having a lot of nodes with relatively few disks in [them], you can spread your data in very diverse and controllable ways as you write it into the system,” Davis says.

The combination of smaller storage domains and the use of forward error correction translates into “perfectly reproducible data,” even if multiple disks were to fail, he says.

“This allows you to have a system that is not only extremely impervious to component failure, but because it actually encodes the data and rebuilds the data when you read it, any bit rot or entropy that’s crept into the system is basically canceled out by the error correction process.”

After doing some homework, Davis chose Quantum’s Lattus object storage-based system. It runs the company’s StorNext file system and uses its Artico storage appliance to “tuck the actual files away in object storage,” Davis says.

The new setup offers Crawford the protection against data loss it desired and a lower cost of storing each bit of data, he says. Savings are derived from the way the system approaches storage.

Rather than relying on bit parity across, for instance, three drives to maintain protection against loss, the system allows users to set their own spread policy that specifies how a given data payload is distributed across the system.

“That [erasure code and spread policy] is [responsible for a storage savings] in the neighborhood of half what you would have in a triple-replicated system,” says Davis. “So you gain cost efficiency.”

Maximizing unit cost savings to store data is especially important when it comes to media.

“These are such large files,” he says. They range from mathematically lossless JPEG 2000 in an MXF OP1a wrapper to ProRes, DNxHD 145, XDCAM and even MPEG-2 IMX 50, but regardless of the format, there’s a lot of data to store.

“Just a few seconds of content may not have a lot of monetary value, but it has a pretty high cost because there is just so much data.”

The new system also eliminates the need for future forklift upgrades because it is easy to add disks without being concerned about how similar a future disk is to one in the system.

“It doesn’t have to be like the other form factor in the system,” Davis says. “That lets the snake eat its tail as you replace failed drives.”

Crawford decided it could help clients manage “this sharp left turn from baseband video and audio to file-based media” eight years ago and responded with its own media management division, Davis says.

The goal was to help them migrate 50 years of legacy film and tape to the digital domain, which proved to be a successful business model, says Davis.

However, it didn’t take long to realize that if its clients were to get the full benefit of storing their media assets digitally, they would need to be able to handle large files requiring high bitrates, fast networks and lots of storage — something that at the time was beyond most of the organizations Crawford was working with.

Eight years later, Crawford has successfully helped its clients overcome those difficulties with a comprehensive digital media archive service that relies on proxy video, streaming and web browsers to allow clients to search their stored content and ftp or WAN-accelerated delivery made possible by Signiant to retrieve full-res copies of requested content.

But internally, the challenge of handling large media files and storing them safely and affordably, remains, and Crawford is banking on Lattus and its support for erasure code and the user-definable data spread policies to satisfy its requirements.

To stay up to date on all things tech, follow Phil Kurz on TVNewsCheck’s Playout tech blog here. And follow him on Twitter: @TVplayout.