Producing and broadcasting live TV over IP brings new operational challenges as well as opportunities, according to a group of technology executives who gathered for a roundtable discussion about virtualizing workflows. The group agreed that the industry is moving toward IP infrastructures that leverage commodity hardware and networking, embrace virtualization and provide a path to cloud deployment while leveraging existing infrastructure to lower cost and provide greater coverage and accessibility. Participants included Del Parks, SVP and CTO, Sinclair Broadcast Group; Bruce MacCormack, executive architect of media asset management and workflow transformation at CBC/Radio-Canada,and Dave Siegler, VP technical operations, Cox Media Group.
Broadcasters competing in an increasingly fragmented media landscape have been overhauling their technical architectures to achieve new cost and operational efficiencies in video transmission and master control.
Now, they are starting to make the same technological shift in their news operations to manage costs and keep pace with new digital competitors.
Doing so will be a gradual process that will require a forward looking strategy of how to leverage IP for contribution and distribution using standard internet connections and bonded cellular solutions but also thinking about how to leverage existing infrastructure for end to end IP workflows.
“To try to put a timeline on [the IP transition] is really not practical,” said Del Parks, senior vice president and chief technology officer for Sinclair Broadcast Group. “Many of us already have facilities that have not fully depreciated, No. 1, and No. 2, we’re not going to invest in new technology just because the technology exists, unless there is a positive financial need to.”’
Parks was one of nearly a dozen TV engineering and production executives who participated in a December roundtable conversation, with executives from Avid, Microsoft and TVNewsCheck, about virtualizing workflows.
Sinclair, which has been on a station-buying spree in recent years, is already a believer in virtualization. When Sinclair acquires a station, it installs a standardized IT platform based on IP switches and VMware virtualization software to establish communication with the station and support basic IT functions like email and system management. Then it looks to invest in new HD/SDI gear for news production, as necessary. All this new equipment is 3Ghz capable and can easily support 1080P HDR in the future, which, coupled with 1080P capable cameras can provide a signal that looks great when upconverted to UHD at the set in the home.
Sinclair also chose a fully cloud-based system to launch its Kidsclick daily children’s programming block last July, eschewing traditional playout gear for Imagine Communications software running in the Microsoft Azure cloud, and using LTN Global Communications providing IP delivery to its affiliate stations. Sinclair had already been using LTN’s network to deliver its digital subchannels Charge!, TBD and Comet.
“The real question is, when, and if, IP technology becomes part of our studio operations at local TV stations and the costs to deploy it,” said Parks. “Or how much of that migrates to the cloud, whether it’s a personal, private cloud or the public cloud. So I think we’re very, very early on IP, and I think it’s years to go, miles to go, before we’re fully IP. For someone who’s in 85 markets, that’s not an overnight, snap your fingers thing.”
Sinclair did take a big step toward virtualized news workflows in December 2015, when it signed a landmark 10-year deal with Avid to standardize on Avid’s IP-based MediaCentral editing and content management platform across all 64 of its news producing stations. As part of the enterprise-wide contract, Avid will provide editing software, training and support for all of the Sinclair news stations. It will also create a content management system that stores proxy versions of each station’s stories in the cloud to facilitate easy sharing amongst the group.
Sinclair is about two-thirds of the way to completion in converting to the Avid software systems, which run in a virtualized environment at each station on common off-the-shelf IT hardware. Parks said the next step is to develop the media asset management (MAM) system. Sinclair currently shares about 500 stories per week, through a mix of FTP (file transfer protocol), Avid systems and a Masstech product, Emerald for News
Competitive Challenges Drive Move to IP
Bruce MacCormack, executive architect of media asset management and workflow transformation for Canadian public television and radio broadcaster CBC/Radio-Canada, came to the broadcast world after stints in the newspaper business and the telco industry. He sees another industry wrestling with change and new economic realities. He points out that IP isn’t new, as the telecommunications industry made the shift to IP 10 years ago. And he thinks it’s high time that broadcasters embrace it, given the pressure being applied by digital competitors.
“Your point about it not being fully depreciated?” said MacCormack to Parks. “That may not matter. Because when other competitors come in, and start bringing into your market functionalities based on IP, you have a choice: do you want to compete, or do you want to get out of the game? So the phone network had to rapidly depreciate stuff, and write some of that stuff off— because if they stuck with that [technology] for another 15 years, they weren’t going to be offering a product in the market anymore.”
His first big project at CBC/Radio-Canada has been creating an enterprise-wide media asset management (MAM) system to track content from its English- and French-language television and radio networks and all of their digital platforms. The goal is to preserve the broadcaster’s vast archive of material—it has about 25 million items in its database—while giving reporters and producers easier access.
While the subtleties of video are complex, MacCormack sees the progression of IP going the same way as it did in the telecommunications industry. The first phase of IP is “doing what you already have,” replacing current functionality with IP-enabled tools. Then the second phase of IP is “doing things you can’t do otherwise,” such as offering your customers new features that can generate revenues.
Showing a direct impact to the revenue line will be necessary for many broadcasters to make a major investment in IP, noted Dave Siegler, vice president of technical operations for Cox Media Group.
“Look at what we’ve done with these 40, 50-year old facilities we all have, we’ve bolted all this stuff on,” said Siegler. “It’s hard [to go IP]. And just for the sake of doing IP, it’s hard to sell that to the CFOs. ‘What am I going to get back from this investment? What do we get from going to IP, what do we get from a new CMS?’ They don’t always care about being modern, or the new thing. I think you’ve got to have certain business cases [to make the shift].”
Tribune Broadcasting has already embraced virtualization and saved significant money in doing so, said Hank Hundemer, Tribune Senior VP of Engineering. The station group has replaced proprietary broadcast equipment at its biggest stations with common personal computer hardware that one could buy at Best Buy, said Hundemer, who described his job as “I write software.”
“The newsroom video server that keeps WGN-TV [Chicago] on the air is a hundred thousand dollars worth of hardware,” said Hundemer. “Every single day, [at] KTLA [Los Angeles, it’s] a hundred thousand dollars worth of video server hardware. That’s it.”
Sinclair’s planned acquisition of Tribune Broadcasting is under review at the U.S. Department of Justice.
Catching Up to Changing Skillsets
Since salaries are a large part of any broadcaster’s yearly operating expenses, adopting virtualization and cloud-based services can also help broadcasters realize significant cost-savings by cutting down on the number of full-time staffers who are actively managing gear. But going IP brings new complexities, warn broadcast technologists, particularly in terms of the latency that is inherent in cloud-based services. For example, the level of latency that might be fine for a cable network playing prerecorded programming may be untenable for a live broadcast news application.
More important, the new workflows may require fewer people at a broadcast facility to manage them, but those personnel need skill sets that are in short supply at many local stations today. Some stations have engineering staffs that are just learning about virtualization. To replicate the necessary training and new hiring across multiple stations in a large group may prove too time-intensive and expensive.
Instead, some station groups are thinking about not only virtualizing workflows, but centralizing them in a data center where they can be managed by a small team of highly skilled IT professionals.
“These systems are becoming more and more complex,” noted MacCormack. “You can’t have distributed complexity without distributed skills. And when you have a shortage of skills, you cluster your skills, and you bring the complexity to them.”
CBC/Radio-Canada has been reorganizing its overall technical operations for the past two years. A new business unit, Media Technology and Infrastructure Services (MTIS) was established to remove duplication between CBC/Radio-Canada’s facility in Toronto, where it originates its English-language networks, and Montreal, where it runs its French-language networks. And instead of separate IT and engineering groups within CBC/Radio-Canada, technical staff is now being grouped into “platforms” and “applications” groups that serve both the English and French networks.
“Our job is to get the workflows to be the same,” said MacCormack. “So we started with the MAM. The MAM was viewed as a technology project, but it really was a change management project to align six different sets of workflows into one common file-based lifecycle and one common flow. And once we get that right, we will change the edges to make it English or French, to make it digitalTV or radio.”
Scaling Resources To Immediate Needs
One of the benefits of virtualization is the ability to scale technical resources up and down to match the ebb and flow of the news cycle. For example, during coverage of a major disaster like a hurricane a network’s ingest load might spike dramatically for a week, then just as quickly return to normal. Being able to use a common hardware platform to do multiple tasks allows a broadcaster to quickly add resources like extra transcoding for a short period of time.
For Tribune, virtualization and IP connectivity allows it to remotely control and supplement a station’s news operations, which can help in instances when stations are short-staffed. Hundemer recalled a Sunday afternoon when one of Tribune’s smaller-market stations was hit with a breaking news event when the station was broadcasting NFL coverage, when it had a skeleton news staff in place.
“Breaking news can happen anywhere, any time,” said Hundemer. “So you drop a breaking news event in the middle of Fort Smith, Arkansas on a Sunday afternoon, the virtualization that we have in our software allows editors to dip in, get access to content, create graphics, and get the machine going so that the local market can deal with the breaking news event and not have to worry about staffing up; ‘We’re calling in the cavalry!’
It’s not that we don’t have the cavalry. It’s just that it’s a Sunday afternoon, and they are not there. We would normally be in CBS football…the fact that all of the data, all of the video was available without regard to location, allowed me to kind of get some production going that I otherwise wouldn’t be able to get going until the bodies got in the building.”
Some broadcasters who haven’t yet virtualized are thinking about skipping the virtualization step and just moving directly to the cloud, said Scott Bounds, media industry lead for Microsoft. He recalled a recent conversation with a large broadcast customer.
“He said a year ago his management was buying servers, ‘We’re going to rebuild the rack room, we’re putting servers in’,” said Bounds. “And this year they came to him and said, ‘How are we going to move all this stuff to the cloud?’ So within the space of a year they went from servers to cloud-based [architecture], and the mentality is changing really fast.”
Taking advantage of off-premises cloud services can dramatically increase capabilities on a temporary basis, without a broadcaster having to make a significant capital investment. Instead, the financial model shifts to an emphasis on operating expenses, with a broadcaster simply paying for consumption.
“Think about what we do with NBC with the Olympics,” said Bounds. “They used to buy all these racks of hardware. Now we spin up 200 channels for online streaming in an afternoon. You just spin those things up when you are going to turn them on.”
“I think the difference between cap ex and op ex is changing, and I think op ex models allow you to be a little bit more creative with timing and when you do things,” added Alex Keighley, Avid senior director of strategic accounts. “ It’s ‘Hey, that project that I was going to do next year was going to cost me a million bucks. So I am just going to do it this year because it’s going to cost me whether I do it or not, right?’”
Can Virtualization Make Maintenance Easier?
Maintenance has always presented a challenge in the broadcast industry, where stations are expected to be on the air on a 24/7 basis. And maintenance has not necessarily gotten easier with today’s software-based broadcast systems. All of the executives at the roundtable shared stories about the difficulties of performing software upgrades and patches, particularly when products from multiple vendors are networked together. They also agreed that in today’s multiplatform, “do more with less” world, no broadcaster really has the time for a “forklift upgrade” where a system is completely replaced.
“We have got three 24-hour channels going, and it’s a centralized production facility for three control rooms and four studios,” said Danny Kischel, VP of business services and special events for New York-based Regional News Network. “I mean, we are going 5 AM till midnight. So those maintenance windows are near-impossible to find.”
Virtualization may help address part of that challenge by allowing part of a broadcast facility to go offline while its core functions are performed on another piece of hardware. The functions of an entire control room could potentially even be moved to a virtual environment at another location.
In that vein, Robert Lawson, director of technology, news editing for CBS News, says he is considering taking some of CBS’ Avid systems, such as Interplay production asset management, to a virtual environment in order to make regular maintenance easier.
“Last year I had six hours to upgrade one Interplay work group—six hours,” said Lawson. “I used to have two weekends during March Madness where I could shut everything down, but that went away. So that’s my big goal for the virtualization, is to fit within that window and upgrade, and fail back if something goes wrong. Because I don’t have those forklift upgrade days that I used to have.”
Parks said that maintenance and upgrade considerations were the primary driver in Sinclair deciding to virtualize its Avid newsroom systems and get all of its stations on the same version of Avid editing software. Sinclair has installed three different virtual server configurations—a small-market, medium-market and large-market—depending on the station, and is about two-thirds of the way to complete deployment.
“Part of our deal is we upgrade software once a year, and I think that’s a lot easier to do in a virtual world,” said Parks. “Most importantly, it gives us consistency.”
Deployment across all the Sinclair stations should be done in 2018, and then Avid will begin software upgrades on the first group of stations. The upgrades should be a smoother process than the initial installation, said Avid broadcast solutions specialist Wade Klassen, as in relative terms it is easier to upgrade from Version 3.6 to 3.7 than it is to go from 2.7 to 3.7.
“It becomes a matter of making that process as quick and painless as possible and that’s where the virtualization comes in,” said Klassen. “Where can we leverage the virtualization to help with those efficiencies? When we designed the Sinclair systems there was a lot of thinking about how much VM [VMware] resources do we really need, and we went back and forth about the fact we aren’t going to build a system that was that big. It needs to be thinner, more efficient, and easier to work with, and that was all part of that design process. So this year we will start going through the whole cycle again, and hopefully start to realize those efficiencies.”
After taking traditional broadcast functions to an IP platform the next step is to take advantage of all of that computer processing power and high-speed connectivity to deliver new features. The roundtable closed with a lively discussion about some of the future capabilities that IP could enable, such as using artificial intelligence to deliver targeted content and advertising down to the level of individual viewers.
Artificial intelligence could be used to improve news production, said Bounds, noting, for example, that algorithms might be created to detect “fake news” content like a bogus user-generated video. In partnership with Avid, Microsoft already offers “cognitive services” that can help journalists better sort through the reams of content that come down the digital pipe. Microsoft’s cognitive services can also help with functions like broadcast compliance, automatically flagging content that might be inappropriate for international distribution to certain countries.
Such advanced capabilities could be just the beginning of a brave new world for broadcasting, said MacCormack.
“IP unleashes the ability to have intelligence where there was no intelligence before,” he said. “So systems and machines can start querying things and when you start putting that in combination with AI [artificial intelligence] and machine learning, you get a whole different industry starting to emerge here.”