Broadcasters seeking more efficiency and flexibility have been gradually moving from HD-SDI to IP networking across various workflows, from playout to production. A few major networks and big-market stations have even built brand-new plants based on the SMPTE 2110 IP networking standard, with IP routing and COTS hardware all running under software control. And more recently, the growth of public cloud platforms and improvements in IP connectivity have led some broadcasters to take key functions off-premise and run them completely in the cloud.
But transitioning to IP is not always easy, as big technology changes also come with new staffing challenges and network security concerns, said top technologists from the broadcast and vendor communities who gathered last week for the TVNewsCheck webinar IP Networking State of the Art. The move to IP can be particularly daunting for smaller stations that don’t have the resources to replace aging HD-SDI gear in one fell swoop.
‘Evolutionary Path’ For Most
While “unicorns” like Fox and CBC/Radio-Canada have built brand-new all-IP plants based on 2110, said Paul Briscoe, chief technologist for IP monitoring vendor TAG Video Systems, most broadcasters need to find “an evolutionary path.” That may mean using gateways to bridge legacy HD-SDI gear with new IP switches as they spread their investment over a period of years.
“Customers at any point of the spectrum are looking at IP and 2110 as what they have to do,” Briscoe said. “And for some it’s a very, very long climb from an existing infrastructure where they’re adding one 2110 studio today, or maybe they’re putting in a [IP] switch in the middle but keeping their SDI studios. For others who can make the big jump, it’s a lot easier.”
Fox Tests IP’s Mettle
Fox made its big jump in the wake of the Disney deal, by creating a new content hub in Tempe, Ariz., that combines 2110 on-premise hardware with Amazon Web Services public cloud technology and feeds all of its broadcast and digital streams. While the Tempe facility has been operational for several years, Fox has just begun to extract the full value of end-to-end IP technology from production to contribution all the way through to distribution to consumers, said Paul Cheesbrough, CTO and president of digital for Fox Corp.
Fox’s broadcasts of the 2022 FIFA World Cup and the 2023 Super Bowl showed how IP can handle big events with “large footprints of both contribution and distribution,” Cheesbrough said, with Fox delivering 4K HDR streams of both events in addition to its 720p SDR broadcast feeds.
For the Super Bowl, Fox was able to go IP all the way from the cameras at the stadium, through its mobile production trucks and 2110 “flypack” systems, and then directly into its Tempe facility. That facility is hybrid, Cheesbrough explained, with an internal infrastructure based on 2110 and then a direct interface to the AWS cloud; the facility actually sits on the AWS network backbone. Fox’s feeds stayed IP all the way down to its consumer apps as well as its third-party distribution partners, including traditional cable and satellite operators as well as virtual MVPDs.
“By doing that we extracted a couple of really big benefits that were highly visible to the consumer,” Cheesbrough said. “One was close to zero latency against the broadcast feed. Our streaming operation was within one second of the actual broadcast feed, which has never been heard of before. We were 17 seconds ahead of cable, and almost a minute ahead of the other third-party platforms that carry our content.”
The low latency was particularly impressive given the Super Bowl’s scale, with Fox hitting close to seven million concurrent streams.
A second big achievement was maintaining a high level of picture quality for its 4K HDR product. Cheesbrough said that came from a combination of using JPEG-XS compression for contribution, minimizing the hops between the camera and the consumer and optimizing the quality throughout the network. That included doing dynamic color correction with distribution partners like YouTube TV and optimizing images for different brands of connected TVs. Cheesbrough was very happy with the feedback on the 4K picture’s quality, particularly for the Fox apps.
“We carried the Super Bowl in 4K, the same with the World Cup, and about 20% of our streaming audience consumed that in 4K,” Cheesbrough said. “So, that number is pretty significant now in the U.S.”
Full Flexibility
Virtualized production specialist Nevion has seen a shift in priorities since it began working with IP about eight years ago, said Nevion chief technologist Andy Rayner. Broadcasters were initially looking to use IP in facilities as a “temporal- and spatial-resolution-agnostic connectivity for the media flows” as defined in 2110, he said, but that in itself didn’t give them the full flexibility of IP networking.
Over time improvements in compression technology like JPEG-XS and the development of new tools in the “control plane” of IP, such as the Advanced Media Workflow Association’s NMOS (Advanced Media Workflow Association-Networked Media Open Specifications) protocol for communicating with and controlling remote equipment, have paved the way for fully distributed live productions.
Rayner said that Nevion now talks about “the three Ps” in a production: the people, the processing and the places [or physical infrastructure, like cameras at a sporting event]. And those three Ps can be located anywhere, given today’s low-latency compression and the federated control of devices across an IP network.
“We’ve done a Pan-European solution where there are two big data centers where all of the processing is housed, and then all of the actual end users are in whatever country, long-hauling with JPEG-XS,” Rayner said. “We’re also looking at global solutions, having centralized locations where things are done. The distributed nature of the deployments we’re doing now is the difference, and that’s leveraging genuine cost savings for content producers because of the flexibility that brings them.”
An Early Bet On IP At CBC/Radio-Canada
CBC/Radio-Canada’s plans for a new headquarters in Montreal that would feed all of its French-language television and radio programming began before ST 2110 was even ratified as a standard, said Francois Legrand, senior director, capital project management, governance and engineering, core systems, for CBC/Radio-Canada.
“We worked hard in the process to make sure that we were betting on the right technology and that it would eventually become usable for a real facility,” Legrand said.
The fact that audio travels separately from video in 2110, but with a means to still synchronize them together, was very appealing for a CBC facility where radio is as important as television. So was the ability to operate all of the equipment on the same infrastructure and network without regard for whether the content was bound for radio, TV or digital.
The real gain of 2110 isn’t going IP alone, Legrand said, but the move to a software-defined infrastructure. Instead of defining a production facility by how the SDI equipment is connected, IP now allows the dynamic allocation of resources, including common off-the-shelf (COTS) hardware, under software orchestration.
“The IP for us really enabled the sharing, the pooling, of the resources,” Legrand said. “So, one control room is no longer tied to the equipment that is attributed to that control room, but it can use whatever is available at that time on the network. And it’s truly dynamic. It’s been a tough journey to get there, but the facility is now fully on-air for every media. Everything we do in Montreal is now in IP.”
Other broadcasters often remark that the CBC was lucky to be moving into a brand-new building as it switched to IP, Legrand said. But in fact, the CBC didn’t make a hard switch from the old building to the new plant, but instead gradually transitioned over a period of several months.
At first the new IP facility was only an extension of the older SDI facility, with IP studios in the new building but SDI playout facilities remaining in the old building.
“We had to carry the signal between both systems,” Legrand said. “We had two different brands of intercom provider, two different brands of a lot of things, and it all needed to be integrated.”
At a certain point CBC began to have many more services in production out of the new facility, at which point the older facility then became an extension of the new building. Eventually CBC was able to cut the ties between both and decommission the legacy plant.
“It’s never truly a greenfield project,” Legrand said. “You always have some kind of legacy to interface with, you have older equipment in your tech [stack], you need a lot of conversion to adapt the old with the new. But at the end, yes — you end up with a new facility and new equipment and much less conversion, and you can say it’s a true IP facility. But it took a lot of planning to get there.”
Integration And Personnel Challenges
CBC’s engineering team decided to do its own integration of the new plant instead of relying on an outside contractor. That was a great opportunity to build expertise, Legrand said, but also meant “we were at the forefront of all the problems. We were the one responsible to make sure that Vendor A would interface with Vendor B.”
Fox, which started its Tempe project several years after CBC’s IP migration, was fairly lucky on the vendor front when it came to integration, Cheesbrough said, with major support coming from AWS and Evertz engineers. He said most of the “teething issues” in Tempe came down to hiring a new team. While Fox was able to bring 40 broadcast technology veterans from Los Angeles to Tempe, it also hired around 200 new personnel. Most of them were software engineers who needed months of training in broadcast-specific skills to get them up to speed to handle a big event like the Super Bowl.
“It gradually became less of an issue over time, but it was definitely a little bit bumpy to start with,” Cheesbrough said. “As for 2110, the technical skills availability of that expertise is super-limited, certainly in the U.S. where we are. Those skills are in high demand and they’re hard to hold onto, they’re fairly transient. That is one of the things to bear in the back of your mind from a risk point of view. If you’ve got those skills in those areas, really hang on to them and look after them, because that can be a risk factor in this.”
Managing Security Risks
Another obvious risk of going all-IP is network security, said Mike Kelley, chief information security officer for The E.W. Scripps Co., whose job is to protect the broadcaster from escalating threats like ransomware.
“As you move from an SDI to an IP perspective, you have to understand that brings new threats,” Kelley said. “Anything that’s connected to a network becomes two things. One of them is a target of opportunity, because if it’s connected, it can be seen. The other is it can be an actual threat within the ecosystem, because if it is compromised, it can hit other things in the network.”
Transitioning from SDI to IP requires a shift in mentality from both an engineering perspective and a security perspective, he said. A big challenge in making the move is managing legacy broadcast equipment that is not prepared to be connected to a network, which is where the “vendor ecosystem” comes into play.
“We’ve taken these systems and now we’ve network-connected them, and there are some things that these technologies have been using before that are legacy protocols that are vulnerable to things like ransomware,” Kelley said. “And this is where we have to bring those vendors into the equation and conversation, that we need to make sure these are secure. A lot of them, I’m seeing positive signs that they’re moving toward building more secure product. But we still have to deal with this hybrid infrastructure, because we can’t forklift everything all at once.”
The full implementation of IP through a software-defined infrastructure will allow broadcasters to replace broadcast-specific gear with modern COTS hardware that is designed to be run through vulnerability scanners, Kelley said. It also makes it possible to implement things like zero-trust security. Instead of building firewalls around a network and then trusting every device connected on the network, zero-trust means that every device within a network is authenticated to be secure before gaining access, on a continual basis, allowing broadcasters to isolate individual workflows.
“I’m excited about the future, I think there will be some big benefits,” Kelley said. “It’s just going to be a bit of a challenge as we move from the hybrid where we are today into this new standard.”
16 May 2024
23 May 2024
Comments (0)