For TV Stations, IP Transition Moves At Tiptoe Pace

While there is a lot of buzz today about moving broadcast applications to the public cloud, the reality is that physical broadcast facilities aren’t going away any time soon, particularly at the local station level. But most of those plants do need updating to effectively handle today’s IP world, particularly stations with aging HD-SDI equipment and outdated software systems.

Local broadcasters are transitioning to IP but doing it slowly, faced with the cost and complexity of new technology such as IP routing systems based on the SMPTE 2110 standard. Beyond the upfront capital investment and integration costs, they often have to retrain existing staff, or hire new ones, to manage software-based systems that are very different from traditional hardware. But an eventual move to an IP infrastructure is inevitable, said top engineers and technology vendors who gathered last week for the TVNewsCheck webinar The IP Transition and Local TV, moderated by this reporter.

So far, Fox Television Stations has only transitioned one station to IP, building a brand-new plant for WTTG Washington with a 2110 routing infrastructure. Across the rest of the group, the big technology priority has been moving away from “traditional bare-metal servers” to a virtual hosted infrastructure, running VMware software on on-premise computers, said Paul Capizzi, Fox Corp. chief information officer and SVP enterprise technology. That standardization effort has significantly collapsed the hardware footprint at the stations, improved backup capabilities and set the group up for a scaled rollout to 2110 and video over IP transport at other stations in the future.

“Currently there is no demand for 4K at the local stations,” Capizzi said. “But with that said, we need to start thinking, how do we futureproof our design? What does an SDI gateway look like that we could start deploying and have a scalable approach as all the other technologies are refreshed, plug it in so we can support both SDI and 4K?

“We’re making a ton of progress there,” he said. “We’re also focusing on the infrastructure, so our connectivity coming into the space as well as the network laid out across the environment.”

Sony is seeing the biggest usage for 2110 at the network level, particularly in mobile production trucks used for high-end sports production, said Deon LeCointe, Sony Electronics director of networked solutions, imaging products and solutions – Americas. Most new trucks being deployed are all 2110, as are many newer stadiums and arenas as well as studio facilities for regional sports networks. But the picture is different at the local station level.

“At the call-letter stations we’re seeing more of a cautious approach,” LeCointe said. “I know of a handful of stations that are trying to move toward IP, but what we’re seeing is a lot more of our customers are buying IP-ready gear.”

While IP contribution, such as bonded cellular systems, is widely used across TV stations, most on-premise infrastructure remains HD-SDI. LeCointe noted that compared to previous technology transitions, such as the analog-to-digital conversion or the move from standard-definition to HDTV, there isn’t any “trigger point,” such as a government regulation or industry-driven mandate, to make the shift to 2110. That, combined with the cost premium to SDI that 2110 technology represents, explains why most stations are waiting on the sidelines.

“IP doesn’t represent such a mandate,” LeCointe said. “So, for a local station that is able to deliver the news with their existing SDI infrastructure, there’s no real push for them to go in that direction.”

In the playout and automation market, today 2110 is a “small part of the conversation,” said Shawn Maynard, Florical Systems SVP-general manager. He said that his customers are looking more into virtualization and hosting playout applications in the cloud than investing in 2110 infrastructure today. “We’re definitely seeing more cloud conversations.”

Maynard noted that beyond cost, finding personnel with the expertise to manage IT infrastructures remains a stumbling block for local stations. That is despite the fact that they have been steadily hiring more staff with a deep IT background since broadcasters began shifting to software-based tools some 20 years ago.

“At this level, we still have a big gap in knowledge base for on-site personnel,” Maynard said. “There are great video engineers, but not great IT technicians who can figure it out and map it. So, there are serious challenges on personnel.”

For its part, Hearst Television is systematically working to gain IP experience among its corporate and station staff before it makes any big investment in 2110 hardware, said Stefan Hadl, Hearst Television SVP, broadcast engineering and technology. The group has already done small proofs-of-concept of 2110 routing at two of its stations (WLWT Cincinnati and WESH Orlando), which were designed to build the group’s knowledge base as it looks to gradually refresh equipment across its stations and then efficiently maintain it over time.

The POCs were “a great learning experience,” Hadl said, as Hearst engineers learned how to build a red/blue network and got familiar with the complexities of Precision Time Protocol (PTP).

“This isn’t a software, or a one piece of equipment replacement, that you’ve got to learn and understand,” Hadl said. “This is an infrastructure — this is a station infrastructure that you’re changing from what we have now in an SDI plant. It’s a major lift.”

The POCs involved creating a monitoring wall that ran on 2110 while leaving the stations’ core HD-SDI operations untouched, which represented a low-risk way to experiment with IP.

“We built a monitoring wall because that was the low-hanging fruit,” Hadl said. “If you lose the monitoring wall it’s not going to take you off-air.”

Hadl is excited by the possibilities of 2110 but noted that HD-SDI still works very well today. Looking into the future, however, IP is likely where Hearst is heading.

“When you do a major technology change or you have to do a technology refresh, you have to be thinking about IP,” he said. “I don’t know if we will be doing that immediately. But we are going to be making technology changes in some of our markets that are going to be at that point of, it should all be done, and it should all be done in a 2110 environment. If I had a greenfield opportunity, it probably would be a 2110 environment.”

The CBC had just such an opportunity with its network center in Montreal, where the Canadian public broadcaster was relocating to a new space that was less than half the size of its previous home. The CBC set out to create an all 2110-facility mainly for the scale to tackle a variety of multiplatform production, including its radio broadcasts. But in doing so it was also able to take advantage of the smaller physical footprint of IP gear, as well as the ability to repurpose virtualized computer resources for different applications throughout the day. That is a major difference from the dedicated hardware of the past, said Francois Legrand, CBC senior director, capital roject management, governance and engineering, core systems.

The data center in the CBC’s new plant is one-third the size of the data center in its old HD-SDI plant, which had some 500 racks of equipment. And only one-third of the new IP-based space is actually being used today, with the remaining two-thirds ready for expansion.

“So, we have way more density with IP equipment and software-based equipment than we ever had before,” Legrand said.

The CBC had early interoperability problems at the transport level, particularly exchanging signals between a “software sender and a hardware receiver,” Legrand said. He considered those to be “growing pains” that have mostly been resolved. In addition to uncompressed 2110, the CBC has also been experimenting with compressed formats such as NDI for certain workflows, which can represent significant cost savings.

Legrand is now tackling a project to convert the CBC’s English-language network center in Toronto from its existing HD-SDI infrastructure to IP.

“It’s a bigger facility than Montreal, and we’re not going into a new facility,” he said. “So, we have to build while maintaining the current operation on air and rebuild everything in IP.”

While the CBC has learned a great deal about managing a 2110 plant, Legrand said the technology is still far from “plug-and-play.” For example, he explained how a small PTP problem during a recent network update in Montreal interrupted the CBC’s radio broadcasts for several minutes, despite the facility’s completely redundant red/blue architecture.

But the biggest difficulty to 2110, Legrand said, remains finding technical staff who truly understand how IP works.

“We have lot of engineers within our station who are good support people and have a pretty good idea of the end-to-end SDI workflow,” he said. “Nobody truly understands end-to-end 2110 workflows. There are so many components that need to go well so that the system goes well. This is a true challenge. Although we have a lot of experts, we never have enough of them.”