Now’s The Time To Prepare For Disasters

Through a series of articles, TVNewsCheck has been commemorating the broadcast industry’s essential role on Sept 11, 2001, and reporting on its coverage of recent disasters, including Hurricane Irene, the Virginia and Colorado earthquakes and the devastating tornadoes that struck Joplin, Mo., and Tuscaloosa, Ala., earlier this year.

When all other resources fail — from transportation to public utilities — viewers count on their local TV stations to be among the last ones standing. And national networks rely on these same stations to contribute content and to get the news out to the extended community, nation and the world.

This expectation makes it essential for TV stations to have a disaster recovery plan in place to ensure they remain on the air even when calamities disrupt studios, master control operations or transmission facilities. A solid disaster recovery solution, combined with a good liability insurance policy, allows stations to maintain business continuity during a disruption. A station that fails to remain on the air when its viewers most need information can cause it to lose market share for months or even years following the event.

If your station doesn’t have a disaster preparedness plan in place, how do you know where to begin? Following are some tips for disaster recovery planning. And if you already have a plan in place, hopefully these tips will help you evaluate its thoroughness.

How do you plan for a disaster?

How do we know what type of disaster to prepare for? While we can imagine a near-infinite number of disasters, the process begins with anticipating the disasters that are likeliest to occur. For most of us, they fall into six major categories:

• Physical facility destruction through fire, flooding or other incident
• Loss of facility access
• Loss of a significant element of the technical operation
• Loss of facility personnel
• Major power outage
• Other city-wide or regional disaster

When it comes to anticipating which natural disasters to plan for, insurance brokers and federal agencies such as NOAA and FEMA are good sources for determining the likelihood a particular type of disaster may occur in your market.

How do you know how long to plan for? The nature of the disruption can determine the length of time you may be without all or part of your operation. To stay on air, you may need to outsource services to an alternative facility for a period of time, ranging anywhere from a few days to several months, to cover news desk or studio operations.

You also might need to relocate part of the operations management team to a disaster recovery facility for an extended outage. The duration depends upon how much you need to rebuild within a station, a community or even an entire region (as witnessed with Hurricane Katrina, the earthquake in Haiti and the Japan tsunami).

How much should you spend?

Since a disaster recovery solution involves both upfront investment and replacement costs required to restore station operations, you need to take both the revenue and expense side of the equation into account.

What does it costs your business to be down for an hour, a day, a week or more? How can you meet insurance provider requirements while protecting the elements most critical to your station’s success — brand integrity, advertising revenue, customer value, broadcast license, content availability, and data reliability?

You need to evaluate all these factors when calculating the cost of your disaster recovery plan. For example, while you may be able to use archived programming to cover a short-term outage, programming with advertising or retransmission consent agreements may require you to provide ongoing production or broadcasting during an extended period.   

How do you choose the ideal disaster recovery solution?

Knowing the answers to the questions below makes it easier to develop the disaster recovery plan that’s right for your station. Here are a few of the major factors you should consider when customizing your solution.

• Build vs. Buy: The most basic question in any technology decision is whether to purchase the service or build it yourself. Although most individual stations do not have the resources or capabilities to build and maintain a duplicate facility for use in case of an emergency, there may be opportunities within their station group to make the appropriate level of investment. Also, centralcasting technology — the ability to deliver a number of channels from one facility — allows stations or station groups to consider outsourcing to a third party that can provide economies of scale while addressing content and branding requirements unique to each station or affiliate.
• Shared vs. Dedicated: Some larger station groups may have the resources available to invest in a disaster recovery facility reserved solely for their own use. However, for many stations, station groups and even national broadcast networks, an appropriately shared facility can address the risk of a disaster while keeping the costs reasonable.
• Geographic Diversity: To avoid impact from a widespread disaster, local stations can benefit from choosing a disaster recovery facility located in an area geographically diverse from their own. By incorporating geographic diversity into the decision for where to build or outsource a disaster recovery operation, stations can improve the likelihood that they won’t need to move beyond plan B for remaining on the air.  
• Location: The disaster recovery facility shouldn’t just be in a market that’s unlikely to experience the same disaster, such as a hurricane or earthquake; it should also be housed in an area with low risk of any types of natural disasters. Resources, such as the disaster maps available through FEMA, can help you choose the safest location for your disaster recovery solution.
• When selecting the right location for your disaster recovery solution, you should also consider its capabilities for addressing satellite and fiber-based transport requirements. For example, it should have the ability to access the entire CONUS (CONtinental United States) satellite arc. Some operations may also require capabilities for reaching POR (Pacific Operating Region) and AOR (Atlantic Operating Region) satellites from the same location.
• A back-up facility should also have sufficient fiber capacity to meet the station or station group’s needs given fiber’s ability to acquire and deliver broadcast signals as an alternative to satellite
• Finally, an ideal disaster recovery facility market location should address such operational requirements as convenient air travel, adequate lodging, business services and accessible ground transportation.

Facility Requirements: Whether you decide to build or buy, the disaster recovery facility needs to meet your station’s specific requirements, including:

• Sufficient capacity for supporting your operation as well any other stations that will be using its services at the same time.
• Adequate security for both your content and your personnel.
• The ability for your staff to access the facility 24/7/365.
• The appropriate level of skilled personnel to support your operation.
• The required infrastructure such as power, HVAC, equipment/rack space, office space, and shipping/receiving.
• Adequate physical and electronic storage capability for maintaining the programming, interstitials and commercial advertising assets that maybe needed at a moment’s notice.
• Access to production and live event content management facilities, including studio sets and other resources that will be required for originating news and other live programming from the facility.

What activities should be included in your DR operations checklist?

A good disaster recovery plan should address the activities that need to take place in advance of a disaster; during operational readiness stand-by mode; when transitioning during a disaster; and following up after a disaster.

Ongoing pre-disaster planning activities include:

• Delivering and ingesting content for future use by the back-up playout facility.
• Preparing and delivering stand-by traffic schedules.
• Testing and verifying availability of disaster recovery equipment and systems.

Disaster readiness activities include having a formal process in place for:

• Activating stand-by mode.
• Transferring current programming, logs and commercial content.
• Defining key points of contact for go / no go decisions as well as operational contacts for managing operations during an emergency.
• Having a clear, pre-established process for transitioning operations to the DR facility.
• A clear, pre-established process for notifying legal authorities, satellite vendors and others who will need to re-route their content and communications to the disaster recovery facility .

During operations transition to the disaster recovery facility, activities include:

• Content ingest and management.
• Traffic operations and log creation.
• Master control, transmission operations and monitoring.
• Engineering and technical support.
• Feed coordination.
• Managing the transition back to normal operations.

Post-disaster follow-up activities include:

• Creating as-run log and discrepancy reporting.
• Holding a post-mortem discussion with plan effectiveness evaluation.
• Implementing an action plan for any recommended revisions to the disaster preparedness plan.

The tips I have provided above focus on the major areas you should consider when developing your disaster recovery plan. I encourage you to work closely with your risk management department, casualty insurance agent and operations team to review your current level of preparedness and ensure you are as ready as possible for maintaining business continuity despite the types of disasters that may come your way. Here’s hoping the only time you ever have to refer to your disaster recovery plan is during those periodic checks to ensure it’s current.

Richard Buchanan, a 30-year veteran in video production, management and distribution for the entertainment and enterprise markets, currently serves as vice present and general manager of content services for the Comcast Media Center, which provides network origination, satellite and fiber transport, production and post production, disaster recovery and multiplatform content management services for broadcasting and cable/satellite television networks, among others. He may be reached at [email protected].