IMAX: Measuring the Impact of HDR

Faced with intense strain to scale back losses, streaming corporations have pivoted in the final two years to an obsessive give attention to value slicing, loss discount and profitability. At the second, solely Netflix is worthwhile. But these monetary imperatives have helped a number of main streaming choices—notably Warner Bros. Discovery’s Max, NBCU’s Peacock and Disney’s direct-to-consumer choices—cut back losses and enhance their inventory costs.Unfortunately, the conflict on pink ink has additionally include some prices, most notably worth hikes, elevated churn, and in some circumstances poorer high quality video. Netflix, Amazon and Max, for instance, have all made it considerably costlier for shoppers to entry higher high quality 4K excessive dynamic vary (HDR) video, a transfer that has pushed tens of millions of subscribers into tiers with second-rate video high quality. In this interview with TV Tech, Dr. Adul Rehman, the chief product officer at an IMAX, explains that streamers don’t have make “lose/lose” decisions between slicing prices and sustaining excessive video high quality as a result of 4K HDR will be delivered at scale in an economical method. This give attention to higher high quality video, can in flip, additionally drive subscriber progress and assist stem the hefty advertising and different prices created by tens of millions of subscribers churning out and in of their subscriptions every year. All that is half of a wider effort by IMAX to convey the high-quality pictures it has lengthy been related to in the theatrical movie business into the residence. In 2023 12 months, as half of that technique, IMAX acquired streaming expertise supplier, SIMMWAVE, the place Rehman was the founder and CEO/CTO. Since then, IMAX has been investing closely in applied sciences to enhance streaming video and at IBC 2023 launched its StreamSmart resolution.The following interview has been edited for readability and size.TVT: In a common method, what are some of the key issues that you simply’ve been specializing in at IMAX to enhance video high quality and the way does HDR match into that?REHMAN In phrases of shopper expertise, an essential objective is to assist storytellers inform their tales in the absolute best method. In different phrases, assist protect inventive intent of the content material that delivered as audio and video and different elements of video streams in the residence.For that, we now have developed expertise that may actually measure human expertise very precisely at scale in phrases of the method people understand content material. We can assess that in phrases of how the content material is delivered and served on a particular machine. We may give it a quantity in phrases of how shut it’s to the inventive intent from zero to 100. The increased the quantity, increased the nearer the content material is to create your intent.We use that measurement expertise to optimize video experiences and to ensure the buyer is pleased with the expertise that they are delivering.Then we assist them by offering merchandise and options to protect high quality and to shave off all the pieces that is not wanted.Because it is pushed by deep science and an understanding of how the human visible system operates, we’re capable of assist them cut back prices, in the tens of tens of millions of {dollars} usually talking, whereas not impacting high quality.The different profit is in use circumstances associated to delivering premium expertise in the residence. You can suppose of it as offering an IMAX high quality expertise utilizing the same quantity of bits to the typical bitrate budgets. TVT: In phrases of that visible expertise, how do you see the adoption of HDR?REHMAN: In phrases of the libraries which might be on the market, particularly in cloud-based content material, the proportion of content material that is out there in excessive dynamic vary [HDR] is pretty restricted.Generally talking, it’s lower than 10% and will be considerably lower than that, in phrases of the proportion of content material is obtainable in excessive dynamic vary or HDR.But the consumption I imagine is far increased. This restricted set of HDR property or content material are consumed far more broadly than the relaxation of the library.Because you’ll be able to ship extra particulars, spotlight particulars and shadow particulars, and there is the wider coloration spectrum that’s natural and represents pores and skin tones accurately, HDR provides creatives the potential to inform the story that they need to inform in a way more highly effective method.While HDR permits creatives to inform the story in a extra highly effective and versatile method, the method it’s carried out from content material creation all the technique to playback is essential.When we have a look at human notion, dynamic vary is a crucial component of that. We imagine in case you can not measure one thing, you can not optimize that have. If you’ll be able to’t measure it, you are simply taking pictures in the darkish. The expertise that you simply use may go 80% of the time or 60% of the time or 90% of the time, primarily based in your expertise or subjective analysis. But till you’ll be able to measure it at your scale, you can not actually make choices that may permit that content material to be delivered and skilled as anticipated at scale.We began engaged on measuring how people expertise content material about 16 years or so in the past. We studied the affect of processing compression. We studied the affect of gadgets and the way they do tone mapping as a typical operation to suit the sign inside the machine capabilities. Not all the content material is created for 1000 nits, not all gadgets can do 1000 nits of brightness. The machine will [adapt] utilizing the requirements like Dolby Vision and HDR 10 Plus and HDR 10 and others like.If you do that operation with some options, you might cut back or take away some particulars from the content material. One methodology could take away particulars from highlights after which the brighter elements do not look so good as a result of they’re flashed out. Or it might take away particulars from the darker areas. The query is how does that affect the storytelling and human notion? That’s the objective or our imaginative and prescient science and what we measure. We are ready to do this very precisely. We launched that functionality in our instruments in 2017.Since then, it has been utilized by our clients not solely to measure successfully what they’re delivering in phrases of HDR by measuring the efficiency of totally different encoders and numerous choices that they need to make. But it additionally permits them to dynamically optimize the bitrate requirement for HDR content material on a section by section foundation and or scene by scene foundation.Obvously, you can not do this except you’ll be able to actually measure how people expertise the content material. Our metric and imaginative and prescient science can replicate the human goal analysis very precisely.If you aren’t capable of measure extraordinarily precisely, then you might suppose that you simply’re protecting the high quality at the identical degree whenever you aren’t. If you could have an automatic method that may’t actually understand the points with the content material that appears high-quality at 14 Mbps, then you might inform the automated method to maintain taking place and you might find yourself at eight or six Mbps, the place you find yourself quite a bit of options like coloration banding and so forth so forth.We have a product StreamSmart that may precisely measure content material in an automatic method. It supplies customers with a technique to make choices on the foundation of human notion and the rules we’ve been speaking about that may enhance high quality and get monetary savings. TVT: What about HDR in stay streams and stay manufacturing? How do you see the adoption of that in sports activities and different stay streams?REHMAN: The pattern is clearly in the direction of increasingly more adoption of it. But the progress of HDR content material on the stay facet is considerably slower than the file-based facet. That’s attributable to the challenges that come from stay manufacturing and placing a linear channel collectively.For instance, if I particularly discuss a stay manufacturing, reasonably than an entire linear channel, doing a particular sport is less complicated comparatively talking, than doing a number of occasions at the identical time.There are a quantity of challenges for doing a particular sport. If you could have a particular digital camera setup, you want all these cameras to do HDR. HDR clearly requires increased bandwidth. So you have to have hyperlinks from that digital camera to the truck which might be very excessive bandwidth. Then you have to ship that HDR feed each step of the method and spend extra bits.Bandwidth is definitely a problem in the case of content material manufacturing as a result of the bodily wires that you have to put in place from cameras all the technique to the monitor or IP infrastructure that should help the 10s of gigabits per second per feed which might be coming in from digital camera so you’ll be able to combine them collectively and produce the stay sports activities.But that’s not that large of a problem. Another problem happens if you find yourself working with a wider coloration gamut and far wider dynamic vary from scene to scene. Then you must be sure that they match if you find yourself mixing them collectively. If the cameras are very totally different, in the event that they’re capturing a really totally different vary of brightness, then shoppers aren’t going to love that.So you want uniformity. Consistency is crucial. You do not need to go from digital camera to digital camera, the place the coloration altering from one kind of coloration gamut to a different kind of coloration vary. That is difficult, however that’s nonetheless simple to unravel.Adding adverts into the combine creates additional challenges. Now these adverts could or will not be out there in HDR. So you have to create some uniformity between the adverts and the stay stream.That’s whenever you use automated approaches to match the feel and look. But as a result of many of these approaches are restricted in phrases of human notion perspective, you find yourself creating a unique feel and look as you go from the stay stream to the adverts and so forth so forth. So that is a problem.Then, when you’ve got a linear channel and also you go from watching soccer and to hockey, you could have points with how the colours would match or not match and the way the digital camera setup was put in place and the relaxation of it. So these are the challenges in the case of stay and linear content material. TVT: How does your StreamSmart resolution handle these challenges?REHMAN: The objective is to assist not solely assess the feel and look of the content material—whether or not it matches or not and whether or not we’re overlaying the proper brightness vary and stuff like that. The objective can be to compress that significantly better.There’s there is a debate on the market about whether or not it is higher to do UHD SDR commonplace dynamic vary or HD, excessive definition HDR.With HD HDR, you could have much less element in phrases of decision extra dynamic vary in phrases of brightness and coloration and stuff. With UHD SDR you could have extra decision however much less dynamic vary.Based on our expertise you are able to do each. You can profit and ship the highest high quality when you could have all the pixels UHD 4k and excessive dynamic vary inside the identical bandwidth. You can provide the full expertise. Instead of selecting from a lose-lose scenario, the place you’re both shedding dynamic vary or shedding decision, you’ll be able to have each with our resolution. TVT: There are nonetheless a range of HDR codecs. Where do you see the totally different ones being adopted and their growth?REHMAN: I feel we are going to see these evolve over time as expertise grows each step of the method.At this level, in case you have a look at the file-based facet, the format that is mostly used is Dolby Vision. That’s the most well-liked method for a quantity of causes.On the stay facet. It’s HLG [Hybrid Log-Gamma] or HDR 10. HLG has a sure simplicity in the method it handles meta information and stuff like that. That goes again to my file and stay primarily based comparability or complexities which might be concerned in that.  Dolby Vision and HDR 10+ will let you change the feel and look of the content material from coloration and brightness perspective from scene to scene. That extra complexity is less complicated to deal with on the file primarily based facet than on the stay facet. ,If there’s a resolution on the stay facet that may ship the advantages of scene-by-scene adaptation and do it in a easy method from an implementation perspective, you may see a unique method in phrases of stay facet of issues.But I feel on the stay facet, issues are nonetheless in the air. How do you are taking the subsequent leap in phrases of adjusting coloration and dynamic vary on a scene by scene foundation? On the file-based facet, it is a lot simpler to do.So there are a variety of challenges that exist in the case of utilizing HDR codecs. And as these codecs evolve and resolve little issues, you will notice that the composition, what codecs are on the market and what are most well-liked goes to vary over time. Especially in the stay facet of issues.TVT: In phrases of new high-profile content material, large dramas, and so on. is HDR now extensively adopted? Or is there nonetheless some hesitancy to shoot with HDR?REHMAN: It’s a query greater than anything I feel of value. So if that premium high quality is a component of the storytelling, and creatives have the finances and time to spend on that, then I do see that individuals are creating content material in HDR and that is solely going to extend.I feel HDR goes to change into far more prevalent each step of the method. new gadgets and experiences which might be popping out to VR gadgets and different gadgets like Apple Pro Vision. So you will notice that it’s going to change into increasingly more commonplace for folks to devour excessive dynamic vary content material, which can feed this loop of creating extra content material. It will simply take a while as a result of quite a bit of content material just isn’t even out there in UHD.…[That’s] on the streaming facet. If we’re simply purely speaking a grasp for premium content material, that’s going to be 4K HDR. That is desk stakes proper now. You need to have HDR at this time. If you’re not doing HDR, then I do not know what you’re doing.On the theatrical facet, there are particular movies which might be taking it even additional. Oppenheimer was finished utilizing IMAX movie cameras. That is pushing it farther in phrases of decision considerably and side ratio. There’s quite a bit of work in the feel and look that IMAX truly been doing that’s pushing it even additional. But 4K and HDR are desk stakes at this level. TVT: How would you describe the distinction between how IMAX’s StreamSmart helps encoding and workflow, and what’s the distinction between that and say, per title encoding or content material conscious encoding utilizing some of the different strategies which might be on the market similar to VMAF? How is the IMAX imaginative and prescient science differentiated from these?REHMAN: So I’ll cowl the encoding in the first half after which discuss different high quality metrics individually in the second half.For the content material we’re encoding, the precept is that if a particular encoder finds a particular piece of content material simpler to encode, and fewer attempting, then the encoder will spend much less bits on that.That’s usually the precept. In different phrases, you’re saying that permit’s measure the complexity of content material and if the content material is much less advanced to encode, then we’ll spend much less bits. And if it is extra advanced, then lets ship extra bits. You can clearly apply that to choosing decision as effectively. At a particular bitrate what’s the proper decision to select?In phrases of IMAX, what we’re speaking about here’s a high quality pushed metric allocation. It is additive to Content Aware encoding. Yes, if the content material is much less advanced, then, let’s spend much less quantity of bits. But how a lot much less will we go down from extra advanced content material? Is it eight Mbps?  Is it six or 4 or three?That’s the place we are available. I’ll provide you with an instance. The content material encoding method out there in FFmpeg primarily based encoders known as CRF, Constant Rate Factor. You can set the CRF worth at 25, at 28, at 30, at 16. Where you set it at the finish of the day, that is primarily based on folks’s expertise and stuff like that. Now, as an example primarily based in your expertise and no matter measurement you could have finished, you need to set it for top of the range, with the decrease the quantity which means the increased the high quality.So as an example you set it to a CRF 18. But how are you aware that proper for that particular scene? How are you aware that 19 or 20, or 25 is not going to ship the identical high quality ranges?That is the place we are available. We can say for this scene, you do not want 15. We can say that for this scene, CRF 22 offers you the identical high quality at three megabits per second as CRF 15 or 18 would.So it is high quality pushed method sitting on prime of content material consciousness encoding. That is what we’re utilizing.Now in the case of VMAF [Video Multi-method Assessment Fusion], they [Netflix] in a latest weblog, they talked about the use of VMAF for optimizing HDR content material. And they talked about extending the potential of VMAF [from SDR] to measure HDR content material high quality and enabling VMAF to measure HDR content material high quality.At this level, it isn’t publicly out there. The solely group that is aware of if [WMAF] can or can not do it’s Netflix. At this level they’re the solely ones who know this as a result of it is inner and that is what they use it for themselves. So, I do not know if it is good or it isn’t adequate. They clearly suppose that it isn’t adequate as a result of it has been tweaked for inner use for HDR. But they haven’t made it publicly out there.I used to be stunned to see that some of the graphs that they put of their weblog, do not even have numbers on it. They do not even really feel snug sufficient to say that the rating is 85 or rating is 90. So that is kind of the readiness degree [for VMAF] at this level.[In contrast], we began engaged on in 2008, 2009 in our analysis lab and our HDR work may be very very extremely cited.First of all, VMAF does not help HDR 10. So it isn’t relevant comparability. The second factor is that VMAF is constructed as a mixture of metrics. So they collected a bunch of metrics which is ok. If any person needs to make use of it, you accumulate 4 or 5, six totally different metrics from totally different papers after which fuse them collectively.That’s what the F phrase stands for in VMAF. It’s for fusion. You practice some easy machine studying mannequin and hope that one of the metrics goes to do the proper job at the finish of the day. But what if none of them do their job?Keep in thoughts that these metrics are essentially SDR metrics, not HDR. When VMAF comes out with HDR capabilities, I would wish to see if they’ve modified some of these metrics to make them HDR succesful or if it’s a completely brand-new method. That stays to be seen.My private opinion is that after I go to see Netflix content material, I see banding. I see a quantity of different points.So the query is: the place’s that coming from? Is it from the use of this metric to scale back the bitrate or not? Can the estimate [based on VMAF] actually be trusted to ensure and completely assure that the high quality just isn’t being compromised? If they suppose that VMAF can do this, then 100% They ought to use it. If they suppose they can not, it may possibly’t then we’re right here to assist themTVT: What has IMAX’s acquisition of SSIMWAVE Wave meant for the growth of the options you’re speaking about?REHMAN: It’s meant a number of issues. One is assets. Obviously, as a younger firm beginning out we had a particular degree of assets out there. With this we now have extra assets out there. IMAX is strategically eager on investing and bringing IMAX providing exterior of theaters as effectively. They need to make investments and ship on increasing on the enterprise alternatives exterior of that theatrical atmosphere.Another is validation and their perception in the expertise and the folks. Having the funding and IMAX model, means one thing to shoppers. Those issues have clearly helped us with extra assets and attain.TVT: The streaming business is at a pivotal level the place they’re underneath monumental strain to scale back their loses, which signifies that along with controlling prices, they’ve to draw subscribers and cut back the churn—the quantity of subscribers who’re dropping their subscriptions. How do you see your self becoming into that strain to scale back prices and churn?REHMAN: At the finish of the day, streaming corporations and media corporations are usually telling traders that profitability is the key goal. We can assist them attain that objective quicker by serving to them preserve their shoppers with higher high quality content material and cut back the value of streaming content material to allow them to hit profitability quicker.TVT: I do know you’ll be able to’t be too particular about your roadmap going ahead, however what are the main issues you’re specializing in to additional enhance your providing?REHMAN When it involves StreamSmart and comparable applied sciences, we proceed to spend money on a number of sides to ensure the inventive intent is preserved.On the measurement facet, proper now one of the challenges is for instance is that quite a bit of content material that has both digital or movie feel and look that isn’t preserved. It’s type of butchered by the time it will get to the residence.And our objective is to assist protect and measure how effectively that inventive intent is preserved. So, we can be launching in the close to future, a metric to see how effectively it’s preserved. That is one thing a quantity of folks in the business are trying. They can use a metric like that as a result of the look, a particular type of movie look, is a crucial half of storytelling for them.So that is only one instance on perceptual measurement facet of how we’re working to ensure we will measure the preservation of inventive intent, higher and higher each step of the method.On the StreamSmart facet, we are going to proceed pushing the envelope in what customers will be saving in deployments at scale.On the content material facet with the identical spirit of preserving inventive intent, we’re bringing in contextual parts [to focus] on what scenes are extra essential to storytelling. How can we incorporate AI applied sciences into some of that?Some of that we’re engaged on nonetheless early days however we’re actually excited at what we’ve been capable of obtain and produce to StreamSmart in the close to to mid time period.We are additionally working arduous on the stay linear facet as effectively. In the business, the file-based facet is forward in comparison with the stay, linear facet. So, on the line linear facet, we’re working to deploy it far more broadly and to create a kind of a cookie-cutter resolution that may be deployed throughout differing kinds of approaches stay channels.

Recommended For You