TVN Tech | What Starts In Vegas, Doesn’t Stay There

Following through on a technology course it publicly charted late last year, the NBCUniversal Owned Television Stations group has created its first virtualized architecture at KBLR, the Telemundo station in Las Vegas (DMA 39), with a new IP-based facility that is almost completely driven by equipment housed in an off-site data center.

KBLR is the first NBCUniversal-owned television station to have its news/production infrastructure hosted remotely in an off-site data technology center the station group has built in Dallas, home of sister stations KXAS and KXTX. KBLR’s new 22,622 square-foot, fully-automated facility has a traditional studio, newsroom and control room but its technical guts are located 1,200 miles away and connected via low-latency fiber links.

The NBCU station group’s virtualized architecture for KBLR, first announced by President Valari Staab in her keynote address at the TVNewsCheck NewsTECH Forum last December, is only one piece of the 42-station group’s overall strategy in converting to IP technology.

WCAU-WWSI Philadelphia began broadcasting last October from a large, state-of-the-art IP-based plant in the Comcast Center, and the NBCU group is currently building another large IP-based facility in Boston that will accommodate four separate NBCU businesses.

But building new IP plants with on-premise hardware doesn’t make economic or operational sense for the smaller Telemundo stations and their leaner engineering staffs, and that is why the NBCU station group has built the new data center in Dallas. This “private cloud” or “production cloud” approach is certainly not the first time the NBC stations have consolidated technology at one site — the group already runs a centralized master control facility in Denver and the “Arthouse” centralized graphics facility in Dallas (also co-located with KXAS-KXTX).

The distinction here, says Brad Plant, director of technology and operations for NBCUniversal Owned Television Stations, is that KBLR’s production is still controlled locally in Las Vegas. It’s just that those commands are largely executed by hardware sitting in Dallas.

“As far as the equipment being in Dallas, it’s transparent,” says Plant. “The operators sit in the control room, the talent sits in the studio, and if they didn’t know this going in — that the equipment was 1,200 miles away — they’d never know. That was our goal from Day One — to make this transparent for the operations.”

“It’s really seamless,” adds Ramiro Lopez, president and GM of Telemundo Las Vegas.

That seamless operation is achieved by using corporate parent Comcast’s national fiber network to connect KBLR to an SMPTE 2110-compliant hardware backbone in Dallas, which performs all production operations and transmits video back to Las Vegas using ultra-low-latency JPEG2000 compression. Plant explains that the NBCU station group has worked with Comcast to create a nationwide “mesh network” that connects to the owned stations and other facilities, such as the master control center in Denver. Each station is provided with two 10-gigabit interfaces (two fully geo-diverse circuits) to the mesh network, which then has a 100-gigabit connection into Dallas.

While Plant won’t say what the end-to-end latency of the private cloud system is, he does confirm it is well under a second. The only significant local piece of hardware in Las Vegas is a broadcast-quality shading monitor with a quad-split display so that, when necessary, operators at KBLR can have real-time pan/tilt/zoom/focus control of the robotic cameras using a manual joystick.

“The goal was for an operator to press a button on his APC system in Las Vegas, have that control message sent to Dallas, have the APC system tell the switcher what to do, have the switcher do it and have the multiviewer build the mosaic,” explans Plant. “That mosaic output then gets encoded using a J2K ultra-low-latency codec, that traverses the LAN, comes back to Las Vegas, gets decoded, hits an HDMI converter and displays in the control room. And from the moment they hit that button, they shouldn’t notice any real delay compared to a station that had on-prem equipment.”

While the NBCU group installed an IP routing system based on Cisco and Grass Valley hardware at the WCAU-WWSI plant in Philadelphia, Plant won’t disclose which vendors are being used in the Dallas data center, saying only that some are the same and some are “new strategic vendors.” He says the Dallas plant uses a similar 100-gig spine-and-leaf architecture running on COTS (common off-the-shelf) hardware, but that there is a significant difference in the extent that the SMPTE 2110 standard is fully utilized.

“We’ve built a couple IP facilities now, but for the most part they’re still racked in a lot of gateways with traditional SDI infrastructure,” says Plant. “This was the first one where we really pushed for a complete IP facility with native ST-2110 end points. There’s also a big push for virtualization, not only in control operations but in media as well. We’ve virtualized our real-time graphics playout, which is big, with 2110 coming out of COTS hardware.”

Like the WCAU-WWSI facility in Philadelphia, the KBLR plant in Las Vegas doesn’t have any physical switcher panels or audio boards in the control room. Instead, it uses touchscreens that run a virtual switcher panel and other Web GUIs (graphical user interfaces) that allow manual intervention with systems like graphics.

At the heart of the production operation is Ross Video’s Inception multiplatform newsroom computer system, which links to an automated production control (APC) system that runs switching, graphics, clip playout and all of the studio technology including robotic cameras, lighting and LED arrays.

“We’re using virtual desktop or remote desktop systems to give control to the local users in Las Vegas,” says Plant. “Our more traditional FPGA and DSP-based infrastructure, like production switchers and audio mixers, they’re still located in Dallas, but they provide HTML Web UIs for manual remote interaction for the operators at KBLR. Those are mostly under APC control, but in times that you need manual intervention, such as building effects, everything’s HTML 5 Web UI-based. The control technology in Las Vegas is very lightweight.”

Lopez says that KBLR’s new home at 6380 Polaris, in the southern part of the city near McCarran International Airport and the new stadium for the NFL’s Raiders, is a significant upgrade in both capabilities and space. The station’s previous operations were split between a downtown studio on Fremont Street and a small newsroom and office space hosted by Sinclair station and NBC affiliate KSNV in the northern part of town (KBLR first leased space from KSNV when it began producing local news in June 2014). Lopez estimates the previous space dedicated to KBLR’s news operation at KSNV was only around 1,000 square feet.

The new KBLR facility includes an open-space newsroom with dedicated “pods” for different job functions, similar in design to the Philadelphia facility, and a large studio with a 7-foot x 12-foot LED array, updated anchor desk, robotic cameras and a dedicated weather center. The new studio went live Monday with the station’s 5 p.m. newscast.

KBLR, which has around 40 staffers, delivers over 10 hours of live and locally-produced news programming to Southern Nevada’s Hispanic audiences every week. Personnel went through an intense six-week training schedule to learn the new production systems that came with the new facility, including two weeks of newsroom training, two weeks of control room training and two weeks of rehearsals.

So far, so good, says Lopez, who says the new tools are much faster than KBLR’s previous technology.

“We’re still getting used to the systems, but everything right now is moving  along very well and as expected,” he says. “It’s great to be in this new facility.”

The NBCUniversal Boston Media Center will the next IP facility to go live for the NBCU station group, but Plant won’t say when other stations might join KBLR in adopting a virtualized architecture. However, it is clear that the Dallas data center is meant to support multiple stations in the future.

“Our goal with the production cloud project in Dallas was to be station- agnostic,” says Plant. “We can roll up and say plug your cameras in here, plug your robotics in here, plug your monitors in here. So we’re prepared to support whatever [a] station decides to implement.”