Broadcasters Speeding Up Climb To Cloud

Broadcasters have been moving to both virtualization and cloud technology for several years as a way to standardize functionality across large station groups, achieve new efficiencies in production and master control workflows, and perhaps even save on real estate, power and maintenance costs.

The COVID-19 pandemic has only accelerated that shift, according to top broadcast and technology executives who gathered last week for a TVNewsCheck virtual event.

NBCUniveral Local’s Trajectory

The NBCUniversal Local Stations were well down the path of virtualization before COVID-19 forced many broadcasters to evacuate their facilities in March 2020 and rapidly switch to new remote production workflows. The group first announced a major virtualization initiative at TVNewsCheck’s NewsTECHForum back in December 2018. And by September 2019, NBCU had launched a virtualized news production scheme at KBLR, the Telemundo station in Las Vegas, with the hardware located 1,200 miles away at a remote data center in Dallas and connected via low-latency fiber links.

Matt Varney

The model for KBLR and a few other Telemundo stations running their news through the data center was to take a “lift-and-shift” approach and move applications previously run on proprietary broadcast hardware to virtual machines connected by IP networking, said Matt Varney, VP media technology, NBCU Local. By doing so, NBCU achieved about a 95% virtualization rate across its production environment.

While the technical achievement of the Dallas data center was significant, NBCU saw it as only an initial step in an eventual move to the public cloud for much of its live production infrastructure. The next step was to “dematerialize the data center” by taking the applications running on the VMs and turning them into services running in the public cloud, Varney said. And that migration has sped up dramatically due to the pandemic, as the NBCU stations now plan to move as much of their live production workflows to the cloud as possible over the next three years.

“As far as what COVID did for us, it was the great accelerator on the vendor side more than anything,” Varney said. “As we moved from the beginning of the pandemic, where we wanted these types of technologies but some of them were a little bit out of reach, the work by necessity happened over the last two years to where these technologies are real.

“Now it’s really about operationalizing them and building the proper business models and support models around them, rather than having a discussion of whether they’re possible,” he said.

COVID A Change Agent, Avid Says

Editing, graphics and content management vendor Avid has seen a similar jumpstarting of new cloud technologies across its customer base, said Craig Dwyer, VP global cloud and SaaS practice, Avid.

Craig Dwyer

“Sometimes we talk about COVID-19 being the digital change agent,” Dwyer said. “It really drove a lot of decision-making imperatives, where we had a lot of dialogue with clients who were considering, planning, doing POCs — and they immediately jumped into working out how to adopt this at scale.”

Avid first focused on an “on-demand” platform, with SaaS-based applications to allow customers to remotely spin up editorial functions, storage and file acceleration to upload. Dwyer said that cloud-based offering has been particularly valuable for some of Avid’s bigger clients, who quickly created virtual post-production environments they’re still using today. But he has also seen a lot of adoption of virtualization, sometimes facilitated by virtual desktop infrastructure (VDI) software that allows editors to remotely access on-premise hardware at a station or studio.

“A lot of clients have on-premise facilities, and they just wanted to run from their own data center, so we gave a lot of support for folks doing that,” Dwyer said. “In fact, we granted a lot of clients extra licenses to ensure that they would be able to connect remotely. That was a big success.”

Across Avid’s entertainment customers, there are a lot of editors working from home now and they like it, Dwyer said. He cited a recent survey in the U.K. where 70% of editors said they weren’t going to come back [to the office] full-time. He noted that in many cases those remote workers have moved out of cities to other locations.

These changes have created “a new hybrid world” where producers and editors might meet in person to collaborate on the start of a story or show, but then go back home to do the bulk of the work, Dwyer said. Remote collaboration tools, such as “over-the-shoulder viewing” software that lets two people in different locations view the same edit screen, is key.

“One of the things that we still need to work on is the third monitor, kind of a hi-res monitor, and there’s still some challenges to really achieving that,” Dwyer said. “But again, up until recently it was almost binary. The editor needed to physically be in the edit suite to access all the monitoring and have a full experience.

“We’re now seeing there are a lot of workflows that can be accomplished remotely, and maybe they’ll just come in for a screening or a viewing or a review,” he said. “And they can be super-productive and work around their own schedules a lot more flexibly than they did in the past.”

Sinclair’s Big Plans

One of Avid’s big news customers that has already embraced virtualization is Sinclair Broadcast Group, which is midway through a 10-year deal it signed with Avid in 2016 to virtualize newsroom functions across its stations. But the newsroom isn’t the only place where virtualization has big benefits, said Walid Hamri, Sinclair senior director, media systems.

Walid Hamri

“When everyone thinks about virtualization, they talk just about the video flow,” Hamri noted. “But it’s actually much more than that. You can have programmers working remotely, you can have traffic people, you can have a lot of the pieces that make a whole chain, all the way to the airchain, going remote. That’s what we’ll be looking at, that every piece of the workflow we’re trying to either bundle the function or have a centralized architecture where a lot of people can jump in.”

Sinclair was an early adopter of public cloud technology to handle playout for its emerging (multicast) networks as well as its OTT programming. The company has spun up workstations directly on the cloud to allow remote control functionality for master control operators, focusing on making the user interface of a “soft panel” control with mouse and keyboard feel just as comfortable as if they were pushing a button back in the station. Now Sinclair is looking to move other existing workflows, like live production at stations, to the cloud where it makes sense.

But the company’s goal isn’t to simply lift-and-shift, but to reinvent workflows where possible.

“That’s the path we’re taking,” Hamri said. “It starts with emerging networks, which is an outside, easy workflow to put on the cloud, and it’s moving to stations, basically hubs, and remote productions. The more challenging one, which I think will come at the end, will be the sports networks.”

ABC Owned Stations Realize Possibilities

The ABC station group tackled COVID’s early days by standing up a lot of “VM clusters” and using remote desktop software to access broadcast applications, said Brandon Carleton, executive director, technology for the ABC Owned Stations. To minimize risk, the ABC stations placed strict limitations on what systems they could connect to. Allowing any type of remote access to the production environments that broadcasters traditionally go to great pains to protect requires a new mindset.

“To stand that up, it was a mental shift in trying to keep a closed environment within your own network to opening it up to people from home,” Carleton said.

Brandon Carleton

Many of those remote workflows are still in place today, he added, “because the end date to this keeps shifting right in front of us.” That includes a director for one ABC station who uses Ross OverDrive production automation software to remotely direct daily news from her home on an ongoing basis. The system uses  a VPN and PC over IP (PCoIP) technology to handle monitor delays and KVM technology to drive controls at the station with the least latency possible, and includes a backup plan which allows a staffer at the station to jump in and take control if the director loses connectivity. It has been successful, and that director hasn’t come to the station since COVID hit.

“It’s a real interesting shift, because we never really thought to do that, and we didn’t even really believe it was possible until we got into it and started dissecting it,” Carleton said. “But it’s a real great example of those workflows that we have historically thought, ‘You absolutely need a director at the station.’ How can you do this? How can you have them do it? And it helped us to look through all the other work functions. What are the other things we thought couldn’t happen, and what’s really possible?”

While Carleton is a believer in remote workflows, he doesn’t think it always makes economic sense to run them through the public cloud. That’s why the ABC group leans toward a private cloud approach that leverages existing hardware at ABC stations and Disney data centers. Carleton added that the hardened infrastructure and robust disaster-recovery plans at broadcast stations — KABC Los Angeles has three generators — shouldn’t be discounted in the rush to the cloud.

“I’ve been in this business a long time, and engineers are paranoid people,” Carleton said. “We want to make sure that when an earthquake hits and we have no connectivity to the outside world, when a flood hits, when a hurricane comes through Texas, when there’s a massive snowstorm and there are outages and you lose connections — how are you going to inform the public, and how are you going to produce your product and get something on the air?

“For a lot of us, there’s a sense of calm that comes with knowing that even if you’re broken off from the outside world, you can continue to create.”