It’s Game On For Virtual Sets In 2018

Virtual sets for news, sports and entertainment production have been around for over 20 years, but they haven’t seen much use in the U.S., particularly by local broadcasters.

TV stations have complained about the complexity of integrating green screens, real-time 3D graphics and sophisticated camera tracking systems, while others cited workflow issues with legacy newsroom computer and automation systems.

In the early days, multi-camera systems were also prohibitively expensive for most local stations, running $300,000 or more. And for many broadcasters, virtual sets typically just didn’t look real enough for everyday use.

But virtual set vendors predict big changes in the U.S. market in 2018, as they roll out new systems that harness the rendering power of video game technology to deliver photorealistic virtual environments.

They also point to improved camera tracking systems, and even “trackless” technology, that reduce the cost and complexity of setting up and maintaining a virtual set.

And with early virtual set companies now owned by broadcast conglomerates like Avid, ChyronHego and Ross Video, better integration with legacy news production tools could also become reality.

“There is an awakening going on,” says Gideon Ferber, director of product management and business development for Ross Video’s virtual set business, which has seen a recent uptick in sales.

The Contribution Of AR Tech

What has helped prime the virtual-set pump, says Ferber, is the broad popularity of augmented reality (AR) technology, where a 3D graphic is keyed onto live video through the production switcher before being output for air.

AR, which doesn’t require a green screen, is cheaper and easier to integrate than virtual sets and has become commonplace in news and sports coverage. Many of the same companies selling AR also offer virtual set systems.

“What we’ve seen in the last two or three years is a pretty big rise in demand,” says Ferber. “It started with AR more than VS, as that is easier to integrate — you don’t need to replace your physical set, you just enhance it. But in the last year, there has been pretty big demand for a full VS solution, as the industry is moving to the Unreal game engine.”

What Ferber is referring to is Epic Games’ Unreal Engine, first created in 1998 and since used to power popular photorealistic games for game consoles such as Sony Playstation, Microsoft Xbox and Nintendo Switch as well as computer and mobile platforms.

With thousands of developers worldwide developing code for Unreal, the rendering power it can deliver surpasses the much smaller world of broadcast virtual sets and results in more realistic virtual environments.

More Powerful Rendering

Virtual set vendors have been getting more powerful rendering thanks to better hardware performance from their CPUs (central processing units) and GPUs (graphical processing units), says Olivier Cohen, senior product manager for virtual solutions for ChyronHego and former CEO of virtual set firm Hybrid, which ChyronHego acquired in 2016. But they still lag behind the images created by game engines like Unreal.

“None of the broadcast companies have the horsepower compared to companies using rendering engines for multiple applications,” Cohen says. “With open-source code working on a licensed model, you have a lot of developers building on top of the engine. The power is way bigger, way better.”

The challenge is to take that photorealistic rendering capability from the gaming world and integrate it into a broadcast workflow that includes things like timecode, character generators and the Media Object Server (MOS) protocol, the XML-based system used to link newsroom computer systems with graphics systems and video servers. And vendors want to give existing customers an upgrade option to their existing virtual sets.

At this year’s NAB Show, Ross, Vizrt and Turkish firm Zero Density all demonstrated integrations of their systems with Unreal, while ChyronHego and Avid say they will have Unreal Engine integrated into their platform in 2018.

Spanish firm Brainstorm Multimedia, whose InfinitySet is used at CBS by syndicated newsgmagazine Inside Edition and by Tegna, also supports Unreal. Next month Brainstorm will be installing its first Unreal-driven virtual set at LA Castle Studios where it will be used for feature film and episodic TV production.

Brad Rumler, Brainstorm’s VP of sales for the U.S. and Canada, says that cultural differences have somewhat explained the slow adoption of virtual sets in the U.S. compared to Europe and Asia.

“I’m Australian, and there’s the same sort of mindset toward virtual sets in my country as in the U.S.,” says Rumler. “Clients want to see it look realistic; they don’t want a computer game-looking set. But I think the next 12 months will be a really big game changer, because of the integration with Unreal Engine. It’s hyper-realistic.”

Integrating Unreal

Vendors are taking different approaches to integrating Unreal. Some like Zero Density, whose virtual set is used to produce a daily children’s show for Groupe Media TFO in Canada, are running completely on the Unreal Engine. Others seek to combine a separate GPU for Unreal with their existing graphics engine in a unified virtual set product.

Ross and Vizrt have both taken the combined approach, with Ross creating an Unreal-based system called Frontier that works side-by-side with its Xpression system (customers include TV2 Norway and Turner) and Vizrt creating a plug-in for Unreal into its Viz Virtual Studio (used by The Weather Channel, CNN and the BBC).

“Many of the features that the game engine can deliver we can deliver on our own engine, but there are a massive number of people working on game engines and we want to take advantage of that,” says Eduardo Mancz, Vizrt VP of sales for the Americas. “So the combination of both makes sense.”

“We keep developing our engine with better features, and we’re constantly doing that through our own R&D,” adds Mancz. “Just relying purely on the game engine, it’s very hard. It’s not their core business. We can’t go to Epic Games and say we need codec support; we’re a tiny industry compared to their industry. That’s the reality.”

Avid is also using a combined approach for integrating Unreal into its existing Maestro virtual set product, which counts Global TV in Canada and NBT in Thailand as customers.

The Unreal product, which will hold two rendering engines, should be ready in mid-2018, says Haim Halperin, Avid senior principal product manager for virtual, augmented and tracking technology. “Our solution will be a tight integration between the Unreal engine and our existing machine, with data-driven graphics coming from our engine,” he says.

“The background of the Unreal Engine provides very powerful graphics, but when you go to change content for news or sports, that is harder to be done on the Unreal side; it’s a little more complicated. So, our own engine provides a complete solution, and we can take advantage of Unreal for the backgrounds,” Halperin adds.

Improving Camera Tracking

Virtual sets have traditionally used camera movement data that is tracked in conjunction with a real-time rendering and compositing system to create a realistic image as news talent walks around in front of a green screen.

The two most pervasive tracking technologies are mechanical, where a tracking head and lens encoder on each camera communicate with a dedicated compositing server (vendors include Vinten and Ncam); and optical, based on motion-capture technology that uses special LED lights on each camera to shine on small reflective markers placed on the studio ceiling or lighting grid (vendors include Mo-Sys and Stype).

The cost of tracking equipment has come down considerably, with the hardware per camera generally running in the $20,000 to $30,000 range compared to $100,000 per camera in the early days of virtual sets (rendering engines run around $50,000 to $75,000). Vendors also say that setting up cameras with the latest tracking hardware is significantly easier than it used to be.

“In the past we were using mostly mechanical tracking, and the downside was that every time you would reposition the camera on set you would have to recalibrate, and some were not as precise as we needed to be,” Cohen says.

ChyronHego, which counts TV Azteca in Mexico, Raycom Media and Tegna among its North American virtual set customers, has had good luck with optical tracking systems like Mo-Sys’ StarTracker.

“It’s creating good results, and it avoids the problem of recalibrating,” says Cohen. “It always tracks.”

ChyronHego, Brainstorm and Ross also offer “trackless” systems that don’t require any investment in third-party tracking.

Trackless, which is generally aimed at the more cost-conscious end of the VS market, not only eliminates the cost of tracking hardware, but also the cost of camera operators and the time spent calibrating the cameras.

Not using tracking wasn’t an option unless the talent planned to stand completely still, as a presenter would appear to sink into the floor if they walked forward or levitate off the floor if they walked backwards.

“Without tracking, they were essentially a flat 2D sticker placed into a 3D set,” says Brainstorm’s Rumler. “But now we turn the talent into a 3D volume and do feet tracking [in software], matching the respective shadows and reflections.”

The Importance Of Design

Besides rendering and tracking, the other key to making a virtual set appealing to viewers is the design of the actual environments themselves.

Vendors say that customers take different approaches. Some with robust graphic departments take on most of the work in-house, while others farm it out to virtual set design specialists like Full Mental Jacket or Reality Check Systems.

Ross has its own design arm called Rocket Surgery and has also partnered with FX Design, a leading designer and integrator of hard sets, to help create virtual environments for stations.

Montreal-based software specialist firm Astucemedia, which handles real-time graphics design and data integration, has been consulting on augmented reality and virtual sets for several years.

The company does everything from integrating real-time data, such as financial, sports, or weather info, into virtual sets to designing virtual set environments through its creative services arm. It works with several leading virtual-set vendors including Vizrt and Ross on behalf of broadcast clients such as ABC News and the CBC.

To help clients visualize its designs, Astucemedia has created a “virtual set laboratory” in its Montreal headquarters, which includes a chroma key wall, a broadcast camera with tracking on a jib, a multi-viewer and switcher, all being fed into a graphics system.

Astucemedia CTO Thomas Desmeules says the reason for building the lab is that virtual set environments are “hard to play with” on just a computer screen, and this way it can show clients a life-size simulation of what their virtual set or AR graphics might look like. He adds that the lab is also important in demonstrating virtual set technology to the news talent who will be using it.

“Getting presenter buy-in is important,” he says.

Of course, the most important person to convince when selling virtual set technology is the viewer sitting home on the couch.

Victor Murphy, director of technology for Tegna’s WTLV-WJXX Jacksonville, Fla., says that when NBC affiliate WTLV decided to launch a virtual set in 2016 for Olympic lead-in programming, he knew it was a success after receiving positive feedback from one local viewer.

“My neighbor said it was ‘badass’,” recalls Murphy. “He’s a contractor, and he actually built the drywall for my green-screen wall. But he didn’t know that was the green screen he was looking at. I said, ‘You built that wall for that green screen.’ ”

Murphy says the set’s optical tracking system has worked well. He would still like to see some workflow improvements that make driving the set through a newscast a more automated experience than the current setup, which is “still very manual” and requires a dedicated operator on a busy news day.

WTLV-WJXX has kept using its virtual set (then Hybrid TV, now ChyronHego) in conjunction with its physical sets for its local newscasts, with about 30% of its news programming originating from the virtual set and 70% remaining on the hard sets.

The station does morning news and traffic from the virtual set, and often does its lead-off story for the evening news there as well. It has used the VS technology to show anchors interacting with everything from a shark to a school bus.

Says Murphy: “You can create an environment that helps you tell a story you can’t tell on a physical set.”