The Transition To IP Blurs The Lines Between Engineering And IT

The relationship between broadcast engineering and information technology at TV stations and networks has evolved over the past decade as the core technology used for on-air operations has changed.

The gradual shift from dedicated broadcast equipment to common off-the-shelf (COTS) computer hardware made the IT department an essential partner of broadcast engineering, and the ongoing transition to IP networking and public cloud technology is only increasing the two groups’ interdependence, said top technology executives who gathered for a TVNewsCheck webinar last week.

Some station groups still have separate engineering and IT departments, while others have a single technology department that blends engineering and IT. But whatever the organizational structure, today’s broadcast technology often requires expertise in both disciplines, which is driving staffers in both engineering and IT to learn new skills.

Meanwhile, broadcast vendors meet with both engineering and IT in specifying and selling new systems and continue to work with both sides through the support process, as well as interfacing with other vendors to troubleshoot today’s complex networked systems.

Jeff Birch

Jeff Birch, VP of engineering, CBS Television Stations, said that while CBS at the corporate level still has separate IT and engineering groups, the distinction between the two is blurring and “eventually that line of demarcation will go away.” At the stations, IT is already part of the engineering group, as there has been a merging of the television broadcast domain and “what used to be the desktop IT guys.”

“More video is flowing as data now than as discrete audio and video, so the two groups have to work together as one unit,” Birch said during the TVNewsCheck webinar How We Work Now: Engineering, IT & Optimizing the IP Transition.

IT staffers at the Fox stations have evolved their skills as IT has taken on a bigger role in broadcast operations, including the virtualization of many functions on COTS hardware, and IT and engineering now work closely together in a strong partnership. But the two groups are still separate and there is currently no intention of merging the two, said Paul Capizzi, CIO and SVP of technology, Fox Corp.

Paul Capizzi

“The technology is driving us to continue to work more together and to look for more modern ways to deploy and to secure our systems,” Capizzi said. “But there is no talk to combine them.”

Hearst Television has taken a different tack, blending its IT and engineering organizations at each station into one technology department that reports to the director of engineering. There are IT specialists within the department who oversee certain IT technologies, said Joe Addalia, director of technology projects, Hearst Television, but that is no different from in the past when certain individuals in engineering might manage RF or studio systems.

“The crossover between what we once called pure IT, now we could call it ‘broadcast IT,’ but you’re really just talking about technology,” Addalia said.

Joe Addalia

Hearst is very standardized in the technology it has deployed across its stations, which makes assigning responsibility to manage individual systems within the technology department a straightforward process.

“Our corporate team can provide the right guidance to the stations to install gear in the right place, and then it falls underneath the appropriate category,” Addalia said. “We manage our technology sectors by standardization.”

As a software company, Vizrt sees the IT department getting involved in every broadcast sale, particularly if its software is going to run on a virtual machine [VM] or in the cloud, said Bob Valinski, sales manager, Vizrt Americas. But there are still challenges in getting engineering and IT to work together.

“They need to do security risk assessments in some cases,” Valinski said. “And what I’m running into is the traditional broadcast engineers aren’t thinking IT yet, so they don’t bring in the IT department initially, and then we get caught just as the project is about to launch where a security assessment needs to be done.

Bob Valinski

“From the vendor side, we need to ask up front now with the broadcast engineer, does IT need to do a security assessment?” he said. “And how long is that going to take? Because sometimes it takes them two to three weeks to do an assessment on the software, which can delay the project if we don’t plan ahead for that.”

New Emphasis On Security

With recent ransomware attacks impacting big groups like Sinclair Broadcast Group, cybersecurity is a “top of mind” for all broadcasters, Addalia said.

“There isn’t a piece of gear that comes into a facility today that doesn’t attach to the LAN,” he said. “So, the most important piece going forward is to make sure that’s installed and managed properly from a security point of view and also a corporate governance point of view.”

Fox’s information security department is a “strong partner day to day” within the Fox stations, Capizzi said, and works with vendors to keep up to date on software patches and communicate potential impacts to broadcast operations across IT and engineering. Capizzi said the traditional broadcast vendors have also done a better job of addressing security by involving their top technology executives and hardening their systems as much as possible.

“There’s a big change in mindset,” Capizzi said.

At one time CBS’s broadcast engineering and information security departments “were at each other’s throats,” Birch said, but that relationship has improved dramatically as the two groups better understand each other’s needs. Now infosec is a “really good partner” that is brought in at the beginning of any technology project. Like Capizzi, Birch has also seen a new focus on security from vendors.

“If you go to the manufacturers, three or four years ago a lot of vendors were saying you can’t run antivirus on our machines, it will corrupt the system, it will slow it down,” he recalled. “They’ve all gotten religion, and I don’t think there’s a vendor out there now who won’t work with you to run some kind of preventive software on their machines so that they’re not the root cause of your system going down.”

Beyond security issues, Birch said that CBS has become increasingly dependent on vendors to help manage today’s complex IP-networked systems. The days of handing an engineer a thick manual and a spectrum analyzer to learn a new piece of equipment, which he could then maintain for decades, are long gone.

“With the density, the way things are packed, you can’t troubleshoot locally, and a lot of it is living, breathing data, so we’re reliant on the vendor to be able to ‘dial in,’ if you will, and help us troubleshoot where the issue is,” Birch said. “And a lot of times it isn’t the equipment per se. It’s the networking that we did to connect that device to another device, and that’s where the logjam is.”

From its end, Vizrt also finds that supporting its products has become more complicated in today’s IP-based facilities. Valinski said that today the vendor is being called on to help support entire systems, not just individual products. That has changed Vizrt’s support structure, with big accounts now being assigned technical account managers who maintain extensive documentation of the installed software and connected systems in order to speed troubleshooting.

“It’s not like the old days when we would go put in a graphics system, hook up a monitor to the BNC cable and say there’s an output,” Valinski said. “In a 2110 system, the Sony switcher, you could be pushing a button and say I’m not getting a key signal. But it might just be that the tables are wrong in the 2110 router. And it’s not our problem — but we have to work with two or three vendors to solve a simple issue like that.”

Learning New Skills

Training can be a way to foster collaboration between engineering and IT staff as it allows them to both learn each other’s technical skills and also appreciate differences in culture.

“Training on both sides of the house is key,” Capizzi said. “As we look to new platforms like virtualization or IP routing, we ensure that both the engineering and the tech [IT] team, where those responsibilities will lie, have that skill to maintain it.”

Hearst started a training program for employees over a decade ago called “IT for the Broadcast Engineer,” educating traditional engineers on day-to-day IT issues that can sometimes impact the newsroom and other production areas at a station and giving them the skills to solve them.

Today, most of Hearst’s new technology hires are coming in with strong IT expertise, Addalia said, “but they really don’t know anything about television broadcasting.” Therefore, the group is starting a new training program that will be the inverse of its original effort and focus on the basics of on-air operations called “Broadcast Engineering for the IT Specialist.”

CBS has been fortunate to have a new crop of engineering personnel over the past five or six years who were “already IT-savvy to some degree,” Birch said, and the group has provided them with additional IT training and exposure. CBS has also had several IT staffers gradually gain broadcast expertise and then shift to the engineering side of the house, even taking the top job in a few instances.

“Probably our greatest success is, in a couple of stations, we took the lead IP people and they become the assistant chiefs [of engineering] or in some cases, they became the chiefs at the station,” Birch said. “Because over time they learned the culture and then they were able to merge both worlds. Having that tremendous intimacy with the IT side, they learned the TV side, they learned what a remote was, and what going on the air at five o’clock really means. And then they were able to pull it all together.”