Exclusive: HBO Max Used Six Months of Testing, Weekly War Rooms to Keep ‘House of the Dragon’ from Crashing
When it comes to the Seven Kingdoms of Westeros, there are many things to be frightened of, from fire-breathing dragons to familial backstabbing to disorienting time jumps, but for one group of invested viewers, there was something far more terrifying to be on the lookout for when the “Game of Thrones” prequel series “House of the Dragon” aired its first season on HBO and HBO Max this fall… service interruptions.
Episodes of “HotD” averaged roughly 29 million domestic viewers during its first season, with over 70% — or over 20 million households — watching on HBO Max. Despite the influx of streaming traffic to Warner Bros. Discovery’s premier platform, there were no reports of widespread (or even not so widespread) outages related to any of the episode broadcasts.
This stands in stark contrast (pun intended) to the 2019 final season of “Game of Thrones” when HBO Max’s precedent streaming services HBO NOW and [HBO Go] routinely ran into problems keeping up with demand from viewers. The series “GoT” finale specifically led to significant streaming issues as HBO NOW crashed just after the final episode began airing.
Despite numerous reports from domestic HBO Go subscribers about the app crashing during the “Game of Thrones” finale, a representative from WBD told The Streamable that internal crash reports only saw outages from HBO GO in Latin America, which was a different technical platform at the time than HBO GO and HBO NOW in the US.
However, having learned from those mistakes and the subsequent three and a half years of streaming ups and downs, the team at Warner Bros. Discovery came into the “House of the Dragon” inaugural season with a robust strategy for preventing and/or addressing streaming concerns in real time that involved months of testing, war rooms, and constant tinkering and improving.
“Preparation for something as big as ‘House of the Dragon’ starts like six months before,” WBD’s Chief Technology Officer for Streaming Avi Saxena told The Streamable. “We do a lot of events on our platforms, like the Olympics in Europe, and every time we have a very big tentpole event, we start like six months before to make sure the teams and the platform are ready for it.”
At the time of the “GoT” finale, Warner Bros. Discovery was not in charge of HBO and HBO Max was still over a year away from launching, but rightly presuming that the “House of the Dragon” premiere would be the biggest event in the short, two-plus year history of the platform, WBD’s teams came together for a special operation that they affectionately called “Project House of the Dragon Reliability” or “Project HODR” for short… a code name that will likely stir up complicated emotions for many “Game of Thrones” fans.
“The team came up with that name, and as you said, it’s so appropriate for the project,” Saxena said. “I think these project code names are very important to get the teams excited to rally behind the project.”
During the six months of Project HODR leading up to the series premiere, Saxena’s team worked to ensure that no matter where viewers were watching “HotD,” or on what devices they were watching it, they would have a viewing experience that lived up to their expectations.
Saxena said that the process of scaling video delivery so that audiences have an uninterrupted experience from the second they push play until the end of the episode comes in two main pieces in the media world: attempting to predict traffic and load testing.
“We project how much traffic we’ll see on what kind of devices, in what geographies,” he said. “Then we go and make sure that in each of those geographies, we have our points of presence; our content can be delivered and have really great quality.”
While that seems like a fairly straightforward concept, as the proliferation of streaming devices has continued to explode in recent years, attempting to account for playback issues on all types of devices and operating systems has grown to be a rather tall task. So, Saxena and WBD’s engineers turn to some outside help to make sure that they can simulate — and in turn, troubleshoot — potential problems across as many different devices that audiences will be watching on as possible.
“There is a huge variety of devices. If you look at the standard platforms there is iOS, there’s Androids, Roku, there’s Fire TV,” Saxena said. “But … Roku has 20 versions of Roku out there, and there are some people who are still running Roku software from 10 years ago. And you can only test so much because we just cannot replicate every single device. So we use many device farms where we have all the major devices available to our engineers.”
WBD’s CTO explained that the device farms take HBO Max’s software and run it across every potential device and operating system combination that they possibly can. Then, for the ones that prove the most problematic, the Warner Bros. Discovery teams adapt their software to be more resilient to the specific issues encountered during the test.
From there, they turn their attention to making sure that their systems are able to handle the volume of viewers that will be tuning in on a weekly basis.
“Load testing basically creates the simulated load on our services,” Saxena said. “Every time you run load [testing], you identify a bottleneck, [or something that] did not work. We go back, fix that, rerun it. Go back, fix that, rerun it. And this exercise continues until we successfully run the test end-to-end for all the services.
Not only is this process a technological one, but it is also incredibly customer service-focused. WBD knows that despite there being countless potential issues when watching something via streaming — many of which the content provider has absolutely no control over — viewers expect their experience to be just as seamless and painless as if they were watching via a traditional TV provider.
Saxena noted that in the video-on-demand business, there are numerous pain points for consumers that are outside of video quality and delivery, like having to download the app or being signed out of their account. However, as each episode approaches, especially for a season premiere, HBO Max’s systems start taking on traffic about 10 minutes before broadcast, and that can impact what viewers see once they’re signed in and are ready to stream.
“One of the biggest things we have been working on, as people leave their [traditional] TV and come to streaming services, one of the key things they bring with them is expectation,” Saxena said. “When I turn the TV on, I expect on everything to be working. People have done that for 50 years and it has worked every single time. Now, on streaming, we want to have that exact same expectation, but when we have these events and the traffic is ever-increasing, you need to be ready.”
While U.S. customers think of Warner Bros. Discovery specifically as the streaming home of HBO Max and discovery+, internationally, the company is a major leader in live sports, as the European rights holder for the Olympics, major soccer, tennis, and cycling events, and much more. So while domestically WBD is thought of in streaming terms as an on-demand provider, globally, the company has years of high-volume experience with live, tentpole events.
With that experience in mind, Saxena and everyone else working behind the scenes were prepared for the type of event “House of the Dragon” would be. However, no matter how much preparation is done ahead of time, things can — and often will — still go wrong in the moment. So, to make sure that they could address any unforeseen issues as quickly as possible, WBD held Project HODR war rooms for every episode of the season.
These war rooms would assemble leaders from across all of the departments inside the company that could possibly be needed in the event of some sort of service disruption. While they could all have been connected virtually from offices in New York, Seattle, India, or anywhere else around the world, Saxena explained that being together in person allowed the team to make decisions immediately and get a fix into the field with as few delays as possible
“What the war room really affords us is a very quick response,” he said. “We can make decisions, because everybody from me, the CTO, to all senior engineers and leaders from different parts of the product are in the room in case we run into a situation … We can very quickly make decisions and apply patches or make whatever corrections we need to make.”
Saxena explained that, in the war room, there is a bank of monitors that display all of the key metrics about the platform and the specific systems. They show the teams in real-time how every device is performing, how playback is running in different regions, what the latency and rebuffering rates are, and more.
In the room are representatives from all of the key systems who handle everything from login to payments, from video playback to customer service. And while they are not actively, in the moment doing anything to make sure that the individual broadcasts of “House of the Dragon” actually make it to air, they are there to respond within seconds in case something goes wrong.
“People were observing how things are working in case something needs attention,” Saxena said. “Then, there’s a connection with the customer service team, which is getting real-time feedback from what customers, ‘Hey, in this region in Colorado, I’m seeing more buffering and latency.’ So this is to improve the response time to real customer feedback through customer service or social media chatter or our observation of the metrics that we proactively track.”
Fortunately for viewers and WBD engineers alike, the war rooms ended up being pretty boring, according to Saxena. Because of the legwork done ahead of the broadcasts, there were not any major problems, which was incredibly important coming off of the “Game of Thrones” finale outages.
Saxena said that the failures in 2019 showed the team the value of having everybody from their content delivery and networking teams on standby for this type of tentpole event and how important it is to have every tool available to them in order to be able to respond at a moment’s notice.
“Every time we have a major event, the team gets together and tries to understand what really happened,” he said. “We take a follow-up action … for the issues we uncover, and with HBO Max, there’s been a massive technology change [compared to HBO NOW] … but we do carry these learnings through all the major events.”
So, for some consumers, the question might be, “If things went so well for ‘House of the Dragon,’ why not run HBO Max this way all the time?” And while this “make the whole plane out of the black box material” idea makes sense, Saxena explains that the investment required to keep the platform running during a major tentpole event, especially in terms of resources, would not be financially viable 24/7, underlining how important something like “House of the Dragon” is to the streamer.
“If we continue to run our infrastructure at the scale for the ‘House of the Dragon’ finale, it’ll cost the company lot of money and a lot of wasted resources,” he said. “So, when you design our services … we design with the mindset of we can dial it up and down very easily. ‘House of the Dragon’ is something that we know is 9 p.m. Sunday night. But there other are things we don’t know … like when a video runs on CNN. You don’t know when the next breaking news will happen.”
While HBO Max doesn’t generally need the same type of bandwidth to run the service that it does on the nights that it streams a major premiere or finale, the measures of success remain the same across the board for those behind the scenes. First and foremost is the video experience that customers have, and that starts with how quickly a video begins playing once a viewer clicks the play button.
“Video start time cannot be more than two seconds,” Saxena said. “If it’s taking more than two seconds, the customer is going to be irritated and probably leave. So for each one of our metrics across the entire system, we have hundreds of those … [and] we have automatic alerts set up on those levels. If the video start time ever exceeds two seconds, there is an automatic event created, posted right in real-time saying, ‘Hey, there is an issue.’”
From there, WBD engineers can troubleshoot the problem and hopefully prevent an isolated incident from becoming a widespread problem. Another way that the company measures success is in viewers’ satisfaction with the video quality. While consumers want the entire experience in an app to be an impressive one, they will forgive a lot of other issues if the video quality is good and it doesn’t start and stop throughout.
“Customers want to play video,” Saxena said. “They might want to see the graphic of the show, but if I have to pick between whether the customer’s going to see the graphic or play the video, I’ll always lean to saying, ‘Hey, let’s just make sure the customer has a good video playback experience. That’s the product we sell video playback, not the browser, not the search.”
That product — as well as customer expectations — will not change as the WBD streaming products evolve, according to Saxena. When the company merges HBO Max and discovery+ in early 2023, there will undoubtedly be major changes to the platform and user experience, however, the goals of the teams behind whatever the unified platform ends up being will remain the same.
“Customer expectations when it comes to [a merged platform] is the same,” he said. “A customer doesn’t care whether they’re watching ‘Fixer Upper,’ or [a] 30 minutes episode, or a movie, which is three hours. They wanna find their video, they wanna play their video. They’re not saying, ‘I’m watching a movie, so I want to wait for 10 seconds to start versus ‘Fixer Upper,’ so I only want to two seconds.’”
While that might be true for the entertainment content that will eventually be housed on the singular service, looking forward to what might be coming to the WBD streamer after that, Saxena does see some leeway. With the Discovery brand being a major leader in sports streaming internationally, and WBD having a number of major domestic sports rights — primarily through Turner Sports — it seems to be an eventuality that there will be live sports on their domestic platform sooner or later.
So while the goal will remain to deliver the best possible experience and video quality possible, Saxena does concede that sports fans have different expectations than almost any other viewing demographic.
“With online sports, I’m actually willing to take a lower rate, a lower quality video, but I want it in time if I’m watching with my friends, or I’m watching sports on social media, or I’m betting on the game, or I just wanna keep up with the game,” he said. “And we pay a lot of attention to that when it comes to live sports … I think we are trying to separate some of our metrics by that. Thinking what are the metrics for live sports. Customers cannot afford an [app] update or a delay on live sports.”
As it did from the “Game of Thrones” finale to the “House of the Dragon” premiere, it is clear that Warner Bros. Discovery is taking each tentpole event as an opportunity to learn and improve its product. The face of the company, and the entire streaming landscape, is changing rapidly with an ever-increasing level of uncertainty just across the horizon.
Whether the next big thing for the company’s streaming service is the integration of live sports or something else entirely, Saxena and everybody else behind the scenes will undoubtedly be working to bring the video to you in two seconds or less at the highest quality possible, and they will probably be doing it with a fun, but bittersweet, code name as well.
Max
Max is a subscription video streaming service that gives access to the full HBO library, along with exclusive Max Originals. There are hubs for content from TLC, HGTV, Food Network, Discovery, TCM, Cartoon Network, Travel Channel, ID, and more. Watch hit series like “The Last of Us,” “House of the Dragon,” “Succession,” “Curb Your Enthusiasm,” and more. Thanks to the B/R Sports add-on, users can watch NBA, MLB, NHL, March Madness, and NASCAR events.