Videography June 1998
Managing the Digital Data Explosion
By Sheldon Liebman
As technology continues to advance, more and more of the production process happens digitally. Initially, this phenomenon described the intermediate steps in the production process. Analog video was converted to digital, processed on computer systems and then output back to analog videotape. With advanced in digital videotape formats, it is now easier than ever to generate complete production projects in the digital realm.
In the "old days" of analog production, the ultimate result of a production was a videotape master that was cataloged and stored in a vault or library. By definition, this was offline storage if more work was needed, the videotape was found and brought online for that process. Today, more and more material is being saved using digital masters. If this term simply referred to a digital videotape format, the situation would be very similar to traditional methods.
However, the term digital master is used to describe a wide variety of materials. In the simplest form, a digital master can represent a single digital video file that contains the final version of a project or program. At the other end of the spectrum, you can have a digital master that consists of all the pieces that were used to create a project along with a description of how they should be merged together. For a 3D animation project, for example, the models, motion scripts and rendering parameters could be archived for the project since they can be used to exactly recreate the original result (assuming the original software is also available). In between these two extremes, we can have sets of video clips, compositing directions, special effects plug-ins, background graphics, and other materials that, taken together, represent a project in its entirety.
The Best Choice is Multiple Choice
With all of these choices, it becomes more and more difficult to determine the "best" method to use for archiving a single job. As a result, many companies are storing multiple representations of a project. For example, theyll keep the final rendered images AND the data that was used to create them. In addition, there may be a low-resolution version of the finished project (stored as a QuickTime movie, for example) that can be easily viewed as a proxy for the original material.
All of these changes in the production process have fueled the expansion of three critical areas of technology storage, networking and asset management. Just a few years ago, Videography was at the forefront of our industry by covering technologies like Fibre Channel, SSA, UltraSCSI and Windows NT workstations as they related to video production. At this years NAB, you couldnt walk more than 50 feet without running into an exhibitor claiming expertise in one or more of these areas.
As storage has become less expensive, more companies are offering high-speed storage subsystems with astounding capacities. Single disk drives are available as large as 18 GigaBytes (GB), enabling storage subsystems with hundreds of GB of storage. In researching some of the articles Ive written over the past year, Ive even run into facilities that have over a TeraByte (TB) of online storage available to their employees and clients. There isnt even room in this story to list all the companies at NAB who were selling storage subsystems.
Lets Move it Along, Please
Large amounts of storage make it easier to save a project in a single digital file, but manipulating that file becomes more difficult as it grows in size. This has provided plenty of growth in the area of high-speed networking. Networking technology has migrated from the mainframe world down to the PC platform. It has become so affordable that I recently installed a three station 100BaseTX network in my home office so I wouldnt have to wait as long to backup my data from one machine to another. Total investment for this network was less than $500 and I was able to get it running in under an hour.
A few years ago, the 10 MegaByte per second (MBps) network I just installed was considered reasonable for a production facility. Today, data rates of 100 MBps or higher are required (or at least desired). Also, the days of single platform facilities have ended. More often than not, the goal is to share data between Macs, PCs and SGI workstations, all of which are still used heavily in our industry. This raises the level of complexity of these networks substantially and requires that specialized manufacturers or VARs help to design and install the network being used at even a small facility.
The problem we all face is that many of the companies that sell storage subsystems or fast networking claim to have the expertise to design and deliver a complete solution. The reality is that some of the solutions are not as neat and clean as wed like. This is one of the reasons that Videography brings you real-world application stories and not just product or technology news. Next month, well present a few examples of projects that were successfully managed using high speed networks and storage systems.
Finders Keepers, Losers Weepers
With all of these files hanging around, its critical that you have a system to manage and catalog your data. Asset management systems have been developed to meet this requirement. Today, they are more accurately described as Media Asset Management systems, as most of them are designed to manage digital video files or tapes. Ultimately, asset management may encompass all of the resources available in a facility, including each piece of hardware and every software program that are present. For now, though, creating a database around video clips is enough for many of us.
In the simplest form, a facility can create a database that identifies each clip with things like a description, location, length, resolution and a list of keywords. Of course, its easy to move a digital file to a different location, to change its contents or (unfortunately) to delete it. As asset management systems grow in complexity, they can maintain links to your data so that new locations arent a problem. User profiles can also be used to eliminate the chance of changing or deleting important data files by denying "write or delete" access to most people.
Of course, the goal is to automate as much of this process as possible, which is the area in which todays media asset management systems are putting the most effort. Automatic cataloging of data is critical to production facilities given the sheer volume of the data. To this end, systems are now available that can interpret captioning data, sound and even images to automatically generate the "keywords" associated with a particular piece of video or audio data.
Where Do We Go From Here?
As mentioned above, you couldnt turn around at NAB without running into another company demonstrating a solution to one or more of these issues. Trying to cover all of these companies would take more space than we have available, so well just mention a few of them.
Last year at this time, the phrase Storage Area Network (SAN) wasnt even a part of our vocabulary. Over the past year, this phrase has been introduced to describe the marriage of large storage subsystems and fast networking technology. A new industry association has been formed around this area and lots of companies have issued statements about being "committed to SAN technology."
The question we need to answer is whether a video facility needs more than just big storage and a fast network. For example, we looked at Gigabit Ethernet technology last and discussed how this particular flavor of fast networking might not be optimal for very large files.
One company that is studying this closely is Pathlight Technology. At NAB, they introduced the concept of a Video Storage Area Network (VSAN) to specifically address the needs of video facilities. According to the company, VSAN "transparently connects each user from their own workstations directly to shared storage devices so that they have seamless access to the media they need." Traditionally, we have covered Pathlight with respect to Serial Storage Architecture (SSA) networks. With VSAN, the company is positioned to move beyond a single network architecture or platform. As part of this effort, the company has introduced products like ImageAgent II, which allows existing SCSI devices to operate in a high speed networked environment.
Other companies have developed partnerships or expanded their capabilities through mergers and acquisitions. One company that changed their focus also changed their name. MegaDrive Systems became DataDirect Networks at NAB to more accurately reflect their commitment to "cross-platform shared data access, network computing, and integrated networking services," according to company literature.
Even the biggest players in the industry are turning their attention to storage, networking and asset management. Avid Technology made announcements in all of these areas. On the storage and networking side, the company began shipping version 2.0 of their MediaShare F/C product for high performance, Fibre Channel based workgroup storage. According to the company, MediaShare F/C 2.0 can support up to six workstations and can deliver multiple, simultaneous streams of 2:1 compressed video. Avid also markets Fibre Channel storage subsystems for use with the product.
On the asset management side, Avid announced an alliance with five other companies to develop standards for digital media management. The proposed standard is called the Open Media Management Initiative (OMM) and is designed to link content creation tools with digital media management systems.
Avid has a leadership position in the creation area and is partnering with acknowledged leaders in the asset management side. Two of those companies are Virage, Inc. and Cinebase Software.
Just about everywhere you find asset management, you find Cinebase. They have created partnerships with many of the companies supplying fast networking and storage solutions in an attempt to address the entire picture. Their Digital Media Management System (DMMS) is a multi-platform, client-server application designed "to manage extremely large volumes of digital media for archival, distribution and production management purposes," according to company literature.
Cinebase supports a variety of content capture solutions (including one from Virage) that automate the process of cataloging media and entering metadata information. Once the cataloging has been accomplished, a variety of search criteria can be used to find matches to the stored information. Dozens of file formats are supported directly by the system. Cinebase even supports third party plug-ins which expand the options for indexing and retrieval of information.
For additional flexibility, the Cinebase system supports both online, nearline and offline storage. The system can also be used to translate between different media formats including MPEG, Motion JPEG, Avid OMFI AVR, D1 and others.
Virage, which has been known primarily as the supplier of a capture front-end to Cinebase, is stepping out from behind Cinebases shadow. Virages Video Cataloger is not only available from Cinebase, but is also being used as part of the AvidNews Media Browse system. The company also closed a $6 million financing round let by Adobe Ventures which will be used to accelerate the growth of Virages product portfolio. According to Carlos Montalvo, who recently moved from Cinebase to Virage, the Virage Cataloger "maps into multiple database environments including Informix, Oracle and (SGIs) Studio Central." The company also has a search engine that works with multiple database formats.
At NAB, Virage also showcased a technology demonstration that utilizes speech to text technology as part of the cataloging process. This real-time technology can be used to index satellite data, for example, as it is being fed into the system. According to Montalvo, with this new capability, Virage "watches, listens and reads video in real-time."
Another technology demonstration at NAB had most of the Videography staff scratching their heads in amazement. MATE Media Access Technologies Ltd. is an Israeli company that made their NAB debut with a revolutionary cataloging and search system. MATEs Visionary, which should be available as a product by NAB 99, uses image processing, speech recognition and artificial intelligence to catalog and search large amounts of video and audio data.
Visionary includes three different modules. The first is an AutoLogger, which "accepts analog video and audio inputs or digital media files and produces highly compacted audio-visual abstracts," according to the company. The second is a powerful Query Processor that uses multiple video search engines coupled with a database interface. Finally, the system has a Browser/Searcher module "which enables users to define, browse through and receive content events and abstracts."
What does all this really mean? Lets say youve cataloged the evening news for the past month and you need to find clips that show President Clinton answering questions from the crowd as he boards his helicopter. You can actually input a picture of the Presidents face along with the sound of helicopter blades and ask the Visionary system to find clips that have both.
You may also be searching old fashion show footage for pictures of Claudia Schiffer in a red dress. Input a picture of Ms. Schiffer, input a swatch of red cloth, and Visionary looks for images that have both. Words on a page dont do justice to this system, so be sure to look for it at next years show.
Theres a lot going on in the areas of storage, networking and asset management, and Videography is committed to helping you make sense of the evolutions and revolutions that take place. Watch this space carefully in the coming months for more information.
Back to Recent L&S Marketing Articles