Mapping The Path For Post-Pandemic Production Changes
The COVID-19 pandemic led to a dramatic shift in TV production as broadcasters figured out how to deliver news, sports and entertainment amid new remote workflows and a slew of social distancing requirements. As TV production and technology executives look ahead to a post-pandemic new normal, some of those changes proved to be only temporary blips while others have already become permanent workflows, according to top production and technology executives who gathered for a TVNewsCheck virtual event.
Lessons Learned At Fox
A move to distributed workflows was essential for all types of television production during the early days of COVID-19, and syndicated television was no exception. For its entertainment news show Dish Nation, Fox set up TVU bonded cellular systems at hosts’ homes while the new streaming channel Fox Soul relied on videoconferencing solutions like Zoom. That let Fox produce four to five hours of live programming each day, said Stephen Brown, EVP programming and development for Fox Television Stations.
For the game show 25 Words or Less with host Meredith Viera, the workflow was far more complex. Brown’s team created a virtual set environment, with Viera hosting the show from an impromptu studio set up in the basement of her home outside New York City and celebrities and guests in a Los Angeles sound stage playing the game while sitting in COVID-safe “pods” and being shot by robotic cameras.
“It looks like The Brady Bunch playing the game,” Brown said. “What we found accidentally was that this was a better way to watch the game. Because when you’re cutting live, the director has to choose which reaction to go to, and we have seven characters on at the same time. Now, you can see all seven characters and see their reactions to how their opponents are playing, how they’re bidding, and how the game is actually played. So, we stumbled into what we think is a better experience for the viewer.”
For Divorce Court, the audience was virtual so Fox used Skype to bring in the 40 audience members so they could listen to and react to the litigants presenting their case. And when You Bet Your Life began production last summer, the resurgence of COVID-19 required the audience and all participants to be in masks, with the exception of host Jay Leno.
“We scrambled and we learned a lot, and we used this new technology, in some cases like Fox Soul, to make better programming,” Brown said.
It would seem that such remote production techniques might result in significant cost savings for Fox, but Brown said that is not the case. While using robotic cameras and shotgun mics cut down on operators, those savings were quickly wiped out by the high cost of COVID testing for both the talent and the audience, which can run from $500,000 to $1 million per season, per show.
While Fox is embracing the use of distributed techniques and IP-based technology like the NDI (Network Device Interface) transport protocol for efforts like Fox Soul, Brown is also looking forward to a return to traditional production in the studio when the pandemic is finally over.
“I think for talk and game [shows], you need that human interaction,” he said. “That’s what we crave. I’m looking forward to going back to that but using a lot of the things that we’ve learned and the tools we’ve started to accumulate to our benefit.”
To keep pumping out its news programming, Vice Media shipped LiveU bonded cellular systems to hosts’ homes and sent prompters to iPads via Google Hangout videoconference sessions. It also dramatically expanded its use of Unity IP-based intercom systems, which for editorial production is now favored over traditional systems.
“It’s been adopted so heavily that it is our new norm for folks producing remotely,” said Joe LaMonica, Vice Media VP, broadcast engineering and operations.
Vice also turned to new cloud production tools, running the vMix software-based production switcher in the Microsoft Azure public cloud. Vice was already doing a small POC with vMix before the pandemic, LaMonica said, and ramped up its use of the system when COVID hit. Vice used vMix to handle remote contribution for staffers who didn’t have bonded systems and as a way to record iso camera feeds via the Azure cloud and then move them into Vice’s LucidLink cloud-based storage system for post-production work.
“That’s going to stick around,” LaMonica said. “We use Vmix — whether it’s bare-metal machines on-prem or in an Azure environment — a lot, whether it’s a purely digital product or to contribute to the TV stuff.”
Vice has leveraged its on-premise workstations through established remote desktop software like Teradici and TeamViewer as well as Parsec, a remote software tool originally developed for gaming. For high quality remote monitoring for post-production staff, Vice has relied on the SRT (Secure Reliable Transport) IP transport protocol, which can deliver better video quality with low latency compared to a web RTC Zoom call. And its talent and production staff have learned to navigate the latency of cloud production, which LaMonica said has improved dramatically over the past year.
“All across the board this thing was a major shock to the system,” he said. “And we’re coming out of it with skills that are going to benefit us going forward, for real.”
Fox Sports, The Vault And Human Resources Changes
Fox Sports had to rely on remote integration (REMI) and centralized production techniques when it restarted major sports coverage in 2020 due to social distancing restrictions. For example, Major League Baseball initially allowed only 30 production staff at a game venue when it resumed play that summer. But the network has found long-term benefits to both its coverage and its talent’s work-life balance.
One of those workflows that has become permanent is The Vault, a remote replay facility at Fox’s network center in Los Angeles where up to 40 EVS replay operators and other staff work in pods that connect to different mobile units to support live coverage of MLB, college football and NFL games. SRT is a core technology for the Vault, using Haivision encoders, as are IP intercom systems like Unity.
“We’ve got everything from graphics to replay to the ADs [assistant directors] and support people working in that facility,” said Brad Cheney, Fox Sports VP field operations and engineering. “To the point where during the World Series this past weekend, we had overflow and were shift-scheduling people into those slots and into those rooms. We’ve gotten to max capacity. That just proves that it didn’t go away when we went back [onsite].”
Fox also does centralized productions, which it calls “Home Run Productions” or HRPs, from multiple control rooms in Los Angeles, at its broadcast center in Charlotte, N.C., and at the Big Ten Network facility in Chicago. The majority of its college football games are being produced as HRPs this fall.
Using the same people consistently to work on the same sports improves Fox’s coverage, Cheney said, particularly when it comes to creating specialized replays like the view from a “pylon cam” for football. And in many cases, it has created a more comfortable working environment for operators than being jammed into trucks, where Fox often was fitting five people into a four-wide space. The same applies to graphics operators and editors who are now working from home for Fox.
Centralized production also helps manage the continued challenge of travel during the pandemic, a major issue for Fox now that airlines have greatly reduced their schedules compared to 2019. Across all of its sports, Fox typically has more than 1,500 people working across the country on a fall weekend, and it had over 2,000 working during the World Series. When a major airline ran into weather problems that snowballed into staffing issues, some of those people were left stranded for a day or more.
“To move them in and out into these different places around the country is just not as easy as it used to be, to be honest,” Cheney said. “The flights are all full, the cargo planes are all full, everything’s full out there.”
Fox made heavy use of remote commentary during the early days of COVID production due to staffing limits, and that capability has also persisted where it makes sense. Cheney gave an example where one of its main baseball analysts had a conflict between a Saturday game and his daughter’s college graduation and wanted to do both. Using SRT and other remote IP commentary tools, the analyst was able to attend the graduation and then go to a nearby hotel that night to call the game.
In the past, that analyst would have missed working the game, said Cheney. Besides offering better work-life balance, the new IP tools also allow remote contribution or live commentary at a much higher video and audio quality level than the early days of war correspondents reporting via grainy video from their cellphones.
“We’ve found a way to finally bridge that gap,” Cheney said. “You can do something that’s extremely high quality if you want to and use the public internet to do so.”
Cloud Improvements And Vizrt
Technology vendor Vizrt would certainly agree. The company was testing some cloud workflows before the pandemic but made a rapid pivot to the cloud once COVID hit, leveraging its software-based graphics and content management applications and the NDI video transport technology it gained with its 2019 acquisition of NewTek. Today, Vizrt has major news and sports customers as well as big social media companies “that are doing productions 100% in the cloud with our tools,” said Brian Olson, Vizrt sales director, U.S. group stations.
Switching and transport were two big stumbling blocks to doing live production in the cloud, but those problems are getting solved, said Olson. To make it easier to get video into and out of the cloud Vizrt has developed a new version of the NDI protocol, NDI 5, with a feature called “NDI Bridge” that is designed to form a secure connection between any NDI networks, regardless of location. The company hopes it will be an attractive alternative to SRT, which many of its customers use for video transport today.
“NDI Bridge, what it does is you’ve got three quality settings depending on your bandwidth,” Olson said. “You select the one that works the best, and even over public internet you can get frame-accurate video from place to place. When it does get to where it needs to be — whether that’s the cloud or a facility — it goes back down to normal NDI. So, people on the LANs see their cloud and remote video sources — they look like local video sources, and you can see those in all your applications that are NDI-capable. That promises to be really interesting.”
Olson noted that many of the distributed production workflows broadcasters are using today are still based around a traditional facility with on-premise hardware, which means they are still bearing significant real estate, power and maintenance costs. He said the real benefits of distributed workflows will come when more broadcasters take their live productions completely to the cloud.
“You can have your entire production team completely distributed in those scenarios,” Olson said. “They can be in different countries and still participate because you can distribute parts of the workflow.
“You can still have your audio guy, your TD, replay, tape playback and graphics, and you can have as few or as many people involved in the production as you want to,” he said. “But again, they’re home and they’re not traveling. And if you’re setting up a virtual production that doesn’t have a physical plant or even a mobile OB truck. There’s got to be some cost savings ultimately wrapped around that.”