How HBO Max prevented crashes during House of the Dragon

When the final season of Game of Thrones was airing in 2019, HBO Max wasn’t around yet. Instead, HBO had a couple of streaming predecessors: HBO Go and HBO Now. But there was a problem: due to the huge demand for the final episodes, both services crashed on the regular, and there was a question about whether HBO Max would experience similar problems when the Game of Thrones prequel series House of the Dragon premiered this past August.

House of the Dragon was a huge hit, but service outages didn’t happen. By and large, things seemed to run very smoothly, which wasn’t by accident. Behind the scenes, Warner Bros. Discovery made a very intentional plan to make sure people were able to watch House of the Dragon without interruption, as chief technology officer Avi Saxena told The Streamable. “Preparation for something as big as House of the Dragon starts like six months before,” he said. “We do a lot of events on our platforms, like the Olympics in Europe, and every time we have a very big tentpole event, we start like six months before to make sure the teams and the platform are ready for it.”

In the cast of House of the Dragon, the tech team started a special operation they affectionately called “Project House of the Dragon Reliability” or “Project HODR” for short (say it out loud and you’ll get it). “The team came up with that name, and as you said, it’s so appropriate for the project,” Saxena said. “I think these project code names are very important to get the teams excited to rally behind the project.”

How Project HODR kept HBO Max from crashing during House of the Dragon

As for the nuts and bolts of how this all worked, there was a lot of preparation. “We project how much traffic we’ll see on what kind of devices, in what geographies,” Saxena said. “Then we go and make sure that in each of those geographies, we have our points of presence; our content can be delivered and have really great quality.”

Making sure everybody has a good streaming experience can be tricky because of the sheer number of streaming devices out there now, but the HODR team was ready for that too. “There is a huge variety of devices. If you look at the standard platforms there is iOS, there’s Androids, Roku, there’s Fire TV,” Saxena said. “But … Roku has 20 versions of Roku out there, and there are some people who are still running Roku software from 10 years ago. And you can only test so much because we just cannot replicate every single device. So we use many device farms where we have all the major devices available to our engineers.”

And then there are the stress tests. “Load testing basically creates the simulated load on our services,” Saxena explained. “Every time you run load [testing], you identify a bottleneck, [or something that] did not work. We go back, fix that, rerun it. Go back, fix that, rerun it. And this exercise continues until we successfully run the test end-to-end for all the services.”

House of the Dragon war rooms

But that was all done before the season actually started airing. For the run of the first season, WBD put together Project HODR war rooms that would meet for every new episode. Leaders from all departments would assemble together in a room to analyze analytics as they came in and respond to localized problems quickly.

“What the war room really affords us is a very quick response,” Saxena said. “We can make decisions, because everybody from me, the CTO, to all senior engineers and leaders from different parts of the product are in the room in case we run into a situation … We can very quickly make decisions and apply patches or make whatever corrections we need to make.”

"People were observing how things are working in case something needs attention. Then, there’s a connection with the customer service team, which is getting real-time feedback from what customers, ‘Hey, in this region in Colorado, I’m seeing more buffering and latency.’ So this is to improve the response time to real customer feedback through customer service or social media chatter or our observation of the metrics that we proactively track."

As it ended up, the preparations the Project HODR team made beforehand were so effective that the war room sessions ended up being pretty boring, but hey, at least they got to live out their Game of Thrones-esque military planning fantasies.

Why HBO can’t have Project HODR on the job all the time

Hearing all this, the question arises: if this strategy is so effective, why not run HBO Max like this all of the time? According to Saxena, it’s all about picking your battles. “If we continue to run our infrastructure at the scale for the House of the Dragon finale, it’ll cost the company lot of money and a lot of wasted resources,” he explained. “So, when you design our services … we design with the mindset of we can dial it up and down very easily. House of the Dragon is something that we know is 9 p.m. Sunday night. But there other are things we don’t know … like when a video runs on CNN. You don’t know when the next breaking news will happen.”

Overall I’m pretty impressed by how confident and competent Saxena sounds. There are some streaming services that don’t seem able to run effectively even when no one’s watching (looking at you, Paramount+), so I give gives to the HBO Max folk for keeping the lights on even when there’s a huge strain on electricity. And it sounds like Saxena and his team are prepared to keep at it.

“One of the biggest things we have been working on, as people leave their [traditional] TV and come to streaming services, one of the key things they bring with them is expectation,” Saxena said. “When I turn the TV on, I expect on everything to be working. People have done that for 50 years and it has worked every single time. Now, on streaming, we want to have that exact same expectation, but when we have these events and the traffic is ever-increasing, you need to be ready.”

To stay up to date on everything fantasy, science fiction, and WiC, follow our all-encompassing Facebook page and sign up for our exclusive newsletter.

Get HBO, Starz, Showtime and MORE for FREE with a no-risk, 7-day free trial of Amazon Channels