Next generation of optical storage offers a big data option

For the military, information really is power. The Defense Department has never had more of it, and it’s about to get a lot more, as it expands its use of unmanned air vehicles and other sensors, and builds ever-expanding geospatial data bases, to go along with all the data it routinely collects.

The challenge is managing all that data and being able to access it, in real time when necessary. As much as great stores of data can be a blessing for military operations, it’s also a headache for leaders who have to manage it.

“Everybody’s swimming in the same problem,” said Christian Heiter, CTO of Hitachi Data Systems Federal.

Big data is often looked at first as a software problem, with the challenges of tagging data as it arrives, establishing search parameters and writing algorithms that separate the valuable information from the chaff. But there also is a hardware side to the equation. For one thing, there are the sheer bandwidth requirements of handling large data sets and getting results in real time. For another, there are the processing demands that have prompted the Army and others to look to the power of graphics chips.

And then there is storage. All that data has to kept somewhere, and some of it has to be kept for a very long time.

Optical storage could be a good way to go, Heiter said.

Magnetic storage—primarily in the form of hard drives and tape—is widely used and generally reliable, though it can be risky, because it’s vulnerable to magnetic and mechanical disruptions affecting data, and hard drives eventually fail. Tape can last longer than hard drives, but still has its limits in both duration and retrieval times. Optical storage might have its own limitations, but also has advantages for long-term storage.

Many people are familiar with optical storage in the form of commercial CDs and DVDs, which can last anywhere from a few to 20 years, but enterprise-class optical storage, using M-disc technology, is far more durable, lasting even up to 1,000 years, Heiter said.

Hitachi Digital Preservation Platform (HDPP) currently uses Blu-Ray technology, and will eventually incorporate M-disc, or Millennial Disc, which would allow those long storage times and help agencies meet the Presidential Memorandum for Managing Government Records, which requires digital formats for all permanent government records. (If the government wants to get carried away with the word “permanent,” Hitachi is working on glass-based storage resistant to heat and water and capable of keeping data for hundreds of millions of years.)

HDPP can withstand elements such as salt water and sand, which would help in the event of an environmental disaster such as Hurricane Katrina. “You can’t protect the electronics,” Heiter said, “but the media very likely would survive.” It’s even been tested successfully against electromagnetic pulses.

Although optical storage’s cost per gigabyte can be a little higher than magnetic, it’s still inexpensive and prices are steadily coming down. An agency moving to optical would likely find it more expensive with the initial migration, because of the costs of copying data over, Heiter said, but eventually would see cost savings in operations, in part because its takes less power to keep cool than magnetic drives. And because of its durability, replacement cycles also would be a lot less frequent.

When talking about storage, flash is as another option, though that might be best for short-term storage. The solid state drives used for flash are sturdy, in part because they have no moving parts, and fast. But flash costs more per gigabyte that other media and those solid state drives wear down with repeated use, making them somewhat fickle for the long haul.