The experienced system integrator shares his hard-earned advice on how to assemble a digital TV station that can deliver full-blown HDTV and on where some of the pitfalls lie.
Who knew that closed captioning is one of the biggest headaches involved in upgrading TV stations from analog to digital?
Karl Paulsen, for one.
As chief technology officer for Azcar Technologies Inc., a leading broadcast system designer and systems integrator, Paulsen has run into the closed captioning problem and many others that plague TV broadcasters as they make the final leap into the digital age.
And experience is teaching Paulsen the fixes. Right now, he and his Azcar associates are deeply involved in building a new TV station in Pittsburgh for Cox Television’s WPXI.
In this edited interview with TVNewsCheck, Paulsen shares some hard-earned advice on assembling state-of-the-art TV stations— from ingest to antenna—that will allow efficient, multichannel digital broadcasting.
When you first go into a TV station as it prepares to upgrade to HD, what do you look for?
We typically will look at the work flow. We’ll look at production, news, master control switching, automation and even traffic. All segments of the operations must be reviewed to make a complete assessment.
Affiliates are set with either 720p or 1080i and while that’s not a gigantic difference you have to be sure that you can cross-format material that comes in at 1080i and is going to be released to air at 720p or vice versa.
The technology to do HD now is pretty much set in place with the exception of 1080p, which, while unlikely for broadcasters in the short term, is certainly emerging faster than we all would have preferred.
One of the first things you may need to address is whether you’re going to take a traditional baseband approach to content distribution or whether you’re going to go to a file-based system.
What are the issues that you have to consider with those two approaches?
Content from the distributor to the station is headed toward satellite delivery, but not necessarily in real time. Distributors like Pathfire, DG and Vyvx send commercials and programs as files to their cache servers located in the station. Traditionally, that content was played out from the cache in real time to either videotape or recorded into other servers as baseband SD video. That model is rapidly shifting from real time baseband recordings to non-real time file transfers between the cache servers and the local video servers’ storage systems.
How you engineer your plant for the future is in part dependant upon how you intend to handle these sources of media. If you take the traditional approach, you must plan for a lot of baseband video subsystems. If you take the file approach, you change from a video infrastructure to a network infrastructure. Those are fundamental changes in what the broadcaster has been used to addressing. It affects servers, automation, distribution, traffic and personnel—both in operations and engineering/IT.
How do you see these changes affecting HD program delivery?
The dynamics of getting HD programming to the viewer, through the station, potentially force the hand of a file-based delivery structure. While live network programs make up a part of the broadcast day, we ask: what about all the other possible HD content that must be fed and recorded for later playback?
Knowing it is a growing concern. We look forward into how the content providers will deal with this, and how the broadcaster must adopt. Real time delivery is an option. MPEG2 file delivery is another.
But now I’ll throw another curve ball at the game. Is your programming going to be delivered in MPEG2 HD or is it going to be sent in H.264? H.264 takes less bandwidth and less satellite space than MPEG2, but it complicates things due to a lack of hardware availability.
For delivery of content in other than MPEG, you may need to replace or upgrade the cards in your content delivery cache server or your play-to-server, or both. In this case, the distributor may be sending a new server product out. That’s probably going to mean you can only make that transfer from the cache server into a video server as baseband video because file level translation from H.264 back to MPEG2 may not yet be available.
With HD, TV stations are moving away from CRT monitors and toward flat-panel LCD or plasma monitors that have the resolution to reproduce HD. I imagine that’s a big budget item for TV stations.
It varies all over. I mean some people will buy a $1,200 monitor, some will buy a $500 monitor and some may spend $3,000 to $8,000 for an LCD monitor because that’s what is available right now.
And the strange thing about it is some people are staying with the more traditional professional space monitors from Marshall, Sony and the like, but, in other cases, they’re literally going to the Internet and buying them there.
And that’s the difference between the thousands and the hundreds of dollars.
Yes, and the display size obviously is another component.
Is there something broadcasters should look out for in purchasing monitors?
Yes, signal compatibility, the proliferation of HDMI [High-Definition Multimedia Interface] as the only input, particularly on consumer-level monitors, rather than the traditional DVI digital video interface. With copy protection now carried in the HDMI cable system, getting the signal from the source to the display gets a little more complicated.
If that’s the only way we can get a signal into the monitor, you must be aware that the copy protection may not allow a signal that’s been generated internally by the broadcast facility to be displayed on that monitor. We actually ran into that in a project here in Pittsburgh when consumer level displays were provided by the broadcaster for the master control environment.
How do you get around that?
Trial by fire, I think, is the best way to say it. Right now, if you’re going to use general consumer displays, you may need to find a device that can enable the display to produce video while defeating the copy protection.
You have to understand what HDMI does. HDMI is compatible with HDCP digital rights management technology. So let’s say you have a HD-DVD player. It sends a signal out to the monitor and the copy protection system inside the monitor says whether it’s OK to play this DVD. Or, it might say that you can look at it five times or copy it once.
So it essentially reads the DRM instructions. The fear is that this would somehow, at the worst possible time, shut down the monitor inside the TV studio.
This kind of problem has been observed with the multi-image displays generators now employed in control rooms. We had a customer who bought these beautiful monitors that the general manager loved. We put them in the control room only to find out that we couldn’t get a picture on them. Now we obviously found ways around it, but it was additional hardware that we had to buy in order to get them to work.
How do you get your play-out servers ready for HD?
First question, does your server have an HD codec in it?
So, now you call the manufacturer or your representative and you say, I have this model that I bought two years ago and now I have to play HD out. What do I do? And they may say, we don’t make anything that’s compliant for that vintage video server.
So do you go out and you buy another video server? Or, were you lucky enough to buy a product that gives you the ability to put in an HD encoder or decoder.
Now, that’s not the case with products made in the last two years. They all have an HD-upgrade path in some form. Depending upon the manufacturer, some may utilize a stand-alone box for encoding. When buying new servers or add-ons to existing servers, you can buy HD or you can buy SD or you can buy both. But if you have nothing today, you have to have a means to get an HD signal into the server and have a way to get a signal back out.
So if my servers are more than a few years old, I may be facing a big bill there.
Possibly. There are a number of different ways to get to HD, but let’s go one step further. Up to now you’ve been recording SD material in a world where servers and disk storage was costly. You probably decided a long time ago that 8 megabits a second was perfectly fine for doing an SD record and playback. It gave you all the space you needed.
So now you’ve got HD content which needs to be recorded at 40 to 50 megabits a second because of HD’s higher resolution. Guess what you’ve just done to your storage content? You’ve reduced its capacity and now you have to add storage. In those cases where you have a self contained video server with integral storage, you probably can’t expand it any further. You’ve already maximized it’s storage with SD at 8 megabits, but you now need to have a lot more.
There are vendors that allow you to literally bolt on storage and turn on the switch, but for others you may find that you have to replace a significant part of your storage with faster, higher capacity hard disk drives in order to get the through-put for HD content.
With HD, you’re pushing data four or five times faster than with SD. You have to increase the amount of storage to maintain what you have and then you still need to get that new HD content onto the server as well. Again, this is not always easily done.
OK, we’ve upgraded our servers and where do we go next? How about the master control air chain?
In many cases, most facilities have just continued to use their SD master control switchers. They use a two-by-one switch so they can switch between up-converted SD programming and the pass-through HD programs. Well, that’s all fine and good.
Now, the question becomes, how do you simultaneously manage SD and HD signals and what do you do about the digital subchannels that most likely will be part of your DTV signal?
In some cases people are saying for now they will just buy a new HD master control switcher, set that right next to the SD master control switcher and run the two for the most part in parallel. That’s one approach.
Another approach is to build on an HD master control and take the output of the HD signal and do what they call a center-cut down-convert where they’re taking a four-by-three portion of the image, turning that into SD through a down-convert process and then feeding that to their NTSC transmitter.
Are there any other pitfalls on the path to HD that we should know about?
EAS must now be included for each program stream and HD closed captioning becomes yet another challenge. In every project that we’ve done and every broadcaster that I’ve talked to, it’s turned out to have been one of the bigger headaches we’ve had to deal with.
Really? Closed captioning?
Yes. The 608 closed captioning for analog SD uses line 21 of the analog signal. Well, in the digital domain, there is no line 21. Digital closed captioning, referred to as 708 captioning, is actually contained in ancillary data. We’ve found there is HD programming that is distributed without any closed captioning. So, devices in the chain that do up-conversion or cross-conversion or down-conversion or whatever must be able to translate between 608 to 708 or visa versa.
What happens if the devices don’t do the translation property?
You won’t see any text at the receiver. There’s just no closed captioning, which in some instances makes you in violation of FCC rules.
It’s just nothing?
Nothing’s there. There’s no captioning data there so the receiver at home doesn’t see it.
To prevent that from happening, broadcasters literally have to watch their signals in both the SD state and the HD state and they have to install more hardware to be sure that the signal is being properly translated throughout the chain—the ingest point, the play-to-air point and at the off air receive point. All told, in a complex facility that has several program streams in the DTV multiplex, and does live HD production, closed captioning and EAS additions, you can add upwards of $30,000 before it’s all done.
You would have thought that manufacturers had this all worked out.
In theory, they do. But every once in a while, you’ll find a piece of hardware that’s not compatible with another. Data translation is not an exacting science and can affect all areas of the digital transmission or production chain. Transcoding between compression formats and file translation between servers are prime examples of where things can go wrong.
In the last six months, we’ve been involved in projects where some components didn’t work, even in cases where the vendors said publicly in their print and specs that they would. It can take weeks to get all the systems in sync. My suspicion is that the users and we system integrators have become the guinea pigs that must go find this out and get all these systems to talk to one another properly.
What are the issues surrounding automated playout?
If you’re doing an HD signal and an SD signal, there are points in the day where the SD programming may differ from the HD programming so each channel must now have an automation channel and playlist. That rolls all the way back up into traffic. Traffic may be putting out two or more logs—one for each SD subchannel and one for the HD channel, plus the NTSC channel.
If you’re adding more server channels you have to have more control so that’s an add-on. You might be at the ragged edge of number of ports available on the automation server so you might have to expand that.
Many automation systems that have been in service for more then five years only had a single channel playlist, and must now deal with more. Other concerns include how to update an aging [Windows] operating system and how to deal with network security inside and outside the facility.
But the newer systems can handle multiple play lists.
They can, but they’re at an incremental cost. There are also other software additions and in some cases hardware additions, including screens and keyboards and servers and the like.
Is this how you handle multicast channels like NBC Weather Plus?
NBC Weather Plus is a very good example. You can buy an entry level system that just delivers the national weather and populates your local area weather during the 15-minute wheel slot. Or, in the case of [Cox’s] WPXI, they’re playing back weather segments that they have prerecorded and inserted every two hours or so. This portion is actually being run by an automation channel that has to stay in sync with the NBC wheel.
Graphics is another important area. Up-converted graphics just don’t cut it on a HDTV display, do they?
But native HD graphics aired in HD look stellar. We now have the tendency in broadcasting towards branding engine solutions. Have you heard that terminology? The WB used to be famous for this with all these animated promos popping up at the bottom of the screen. Changing the text or graphic information dynamically throughout the day is now routine, and easily accomplished with real time branding engines.
All that material placed over the programming is being generated live by some graphics system. The push now is towards heavier graphics branding on the main and the subchannels, with the intent to attract the eyeballs with better images and more of them. More important, these new capabilities can provide additional sources of revenue on main and subchannels with only a few infrastructure changes.
How are broadcasters handling graphics for their pass through of the network HD programming?
A lot of them have just bought a single downstream HD logo inserter. You have to be careful you don’t insert the HD logo over the upconverted SD logo, so in multi-program stream branding, separate SD and HD logo generators are integrated.
Upconverted SD graphics over the HD channel are not the best thing to do for a sales standpoint, especially if you’re intending to gain new revenue those graphics or if it’s much more then just a lower third logo.
Even the time and temperature inserts put on in SD and moved up to HD can become marginal by the time it’s been received off air at the cable company, the bits are shrunk again, and then down-converted for presentation at your home converter box.
How about the transmission chain?
As they get closer to the transition, stations are probably going to install a second DTV transmitter, which will probably become their primary given that the first one’s been on the air for five years or so.
Once they no longer need the tower space that’s now occupied by the analog antenna they may wish to put up a better or different kind of DTV antenna, which becomes either a new prime or a backup antenna.
The broadcaster may also wish to explore new technologies aimed at mobile or distributed digital services or even HD streaming to the Web. The opportunities are many.
I’ve been told that stations are going to have to do a lot of work on their towers, adding new antennas or moving them around.
Particularly if you intend to change to a different frequency on Feb. 17, 2009, when they shut the analog signal off. In your interview with [Cox’s] Sterling [Davis], he talked about having to make an instantaneous change of frequencies in San Francisco involving another station.
That’s something they have to work out.
Right and there are actually still stations, some of which we represent, that are working through the Canadian issues right now. We know of some that have only recently been issued a DTV construction permit.
We haven’t talked about HD production. But including it and everything else we have talked about in making the conversion to digital HD, what’s the bottom line? What will it cost a station?
You know, if you want your entire facility to be HD, including studio production, surround sound, cameras, news and editing, master control and all the associated HD-encoding, transmission and monitoring, anywhere from $4 million to $8 million if you’re starting from scratch.
So, you’re saying, starting from scratch, $4 million to $8 million. That’s a big range. What are the major variables?
Well, first off, nobody now is “starting from scratch.” Most facilities have already bought transmitters, antennas and such. Cameras, news upgrades, servers and automation can be big-ticket items. But when you consider doing more then one channel, originating HD programming, intending to keep part of your facility as SD, the rest of it HD or visa versa, there are many variables that can change this dollar equation.
And all this money is on top of all the money stations have already spent on digital transmission.
Yes, but it’s money you have to spend to stay competitive. Look what just happened to us. We were in a station in a major market building a new production control room when another station in the market—an O&O—announced that it was going to do HD news. Our customer stopped us and switched us into a rebuild of their existing SD news control room that we’d put in only two years before. We turned it into HD just as fast as we could do it.
In a broader perspective, it’s not just all about HD. This is the new digital television era and the TV station has the opportunity to rejuvenate itself through good marketing and careful planning.