It’s early on a Wednesday morning. In the world of sports, Monday Night Football has passed, and Thursday Night Football is more than a day away. Traffic on the sports merchandise e-commerce site Fanatics is moderate as fans across the globe attend to other matters. 

For such periods, the infrastructure behind Fanatics has been scaled down, resting on 36 Amazon Web Services servers that, combined, can process about 400 orders per minute. 

But on game days — or Black Friday, or after a particularly successful email campaign — that number changes rapidly as fans flock to the site. During those rush periods, Fanatics has to scale up its e-commerce platform to accommodate the influx in traffic seamlessly. 

Dubbed “hot market” days, sports merchandise, like jerseys, ball caps and even pajamas, become commodities coveted by fans eager to show their home team pride. It’s then, VP Software Engineering Sashanka Vishnuvajhala said, when orders can increase from 400 orders a minute to 5,000. 

“And it stays like that for some time,” Vishnuvajhala said. 

Needless to say, those 36 AWS servers aren’t sufficient to meet increased demand. 

To find out how Fanatics plans for such events, Built In Colorado spoke with three members from various levels on the same engineering team: Vishnuvajhala, Software Engineering Manager Supreya Tosakulwong and Software Engineer Ashley Hutson. Together, they filled us in on what it takes to build a scalable platform capable of supporting Black Friday levels of traffic on any given day of the year.

What’s Scalability Got to Do With It?

While the outcome of any game is left to fate, knowing when to scale systems up isn’t entirely up to chance.

“We’re in a pretty predictable business due to the fact that we know in advance when there’s a game coming up and that we’ll have high traffic,” Software Engineering Manager Supreya Tosakulwong said. “For example, as soon as someone wins the Super Bowl, we know the number of orders will increase dramatically.”

In those situations, Tosakulwong explained, the tech team prepares for the influx of interest by scaling their servers ahead of time to increase the number of orders the platform can handle. That way, when the shopping surge hits, the systems scale up automatically. 

“Based on the forecasting, we have an idea of how much we’re going to need to scale for the expected amount of traffic,” Software Engineer Ashley Hutson said. “That’s the main thing my team has to do: Make sure the scale meets the recommendations for the forecast. Then, during the event, we just have to monitor the systems to make sure everything is healthy and working together.”

Those recommendation forecasts don’t just come out of thin air. Vishnuvajhala credited Fanatics’ dedicated business management team that keeps a watchful eye over the world of sports — from entire leagues down to individual star players.

“They keep tabs on games and have algorithms running to figure out when we should expect certain increases in traffic, which are laid out into a forecast,” Vishnuvajhala said. “It would be very difficult to keep track of everything manually as Fanatics goes international and we add tennis, soccer, cricket and other leagues.”

Using the specific forecasts, the tech team can draw on solid information to make their move in preparation for events.

“We use microservices,” Tosakulwong said. “So based on what the performance team tells us to expect, we need to decide which applications need to be scaled up. We try our best to support that increase, and sometimes we do make adjustments when the traffic is a little more than what we were expecting.” 

If only everything in the world could be prepared for and scale automatically, but sometimes the unexpected happens. When Arizona Cardinals wide receiver DeAndre Hopkins made an improbable game-winning catch against the Buffalo Bills in mid-November, his jersey — and the Air Jordan brand from his receiving glove — saw a spike in attention. 

“There are scenarios we cannot predict,” Vishnuvajhala said. “So we need to be smart about our scale up and scale down scenarios. We need a system that can quickly transform itself based on traffic, the velocity of keywords and a few other dimensions.” 

The system’s ability to scale with little effort from engineers is impressive — but is it really necessary? Why doesn’t the team simply scale up the system and leave it as-is and ready to accommodate any amount of traffic?

“We’re an AWS shop,” Vishnuvajhala explains. “If we were to scale up and remain at an elevated state, we’d be spending a lot of money.”

Choosing the Right Tools to Ensure Scalability

About halfway through a routine team meeting, Tosakulwong noticed her team wasn’t satisfied.

System performance was good. Could it be better? 

The situation may seem telling of a frustrated workforce, but Tosakulwong noted that this sentiment isn’t a sign of unhappiness — it’s a point of pride. Sure, their tech was working well. But everyone wanted to ensure they were building the best tech possible.

“We always ask, ‘Is this the best we can do?’” Tosakulwong said. “It’s never, ‘Oh, this is just the way it is.’ I take it as a personal challenge to have the best applications and tech.”

While the engineering organization primarily runs on Java, Go, Node.js, R and .NET, there’s always room for experimentation. Vishnuvajhala is pleased with the libraries, utility functions, logging and tracking capabilities they’ve built, but he said there’s opportunity to experiment with and implement new tools — if they pass rigorous testing.
 

It’s never felt like there’s a wall up or that because something has always been done a certain way we have to keep doing it.”

Huston agreed: “There are all kinds of situations that come up where someone will point out something they don’t think is working well, or that there’s a new technology they’re excited to use. It’s never felt like there’s a wall up or that because something has always been done a certain way we have to keep doing it. We’re not stuck in a box.”

For example, Tosakulwong’s team, knowing the company was moving toward open-source technology, decided to give Scala a whirl. It worked, she said, but it didn’t quite live up to the team’s vision of being the best. Then, someone brought up .NET Core.

“We decided to try it out, and it worked,” Tosakulwong said. “Now, this application is being written in .NET Core and is performing well. I think it’s a perfect example of trying different languages and going with what works best.”

A Company With ‘Tremendous Opportunity’

Back when Vishnuvajhala worked at Google, he spent six months with one goal in mind: build the world’s most efficient drop-down menu. 

Certainly a laudable achievement, but he notes it was a pretty narrow focus. At Fanatics, he’s found engineers enjoy more diversity in their projects.

“You can actually grow horizontally here,” Vishnuvajhala said. “You’re able to work on several domains — not just a drop-down menu, but on services and the front end, reporting how users are coming into our system.”

Earlier this year, the organization’s three engineering teams consolidated into one, increasing the ease with which engineers can move laterally and explore different tech.

To help everyone fill their knowledge gaps, Tosakulwong led sessions with the newly unified every day for a week.

“I’d ask people to go over their projects at a high level,” Tosakulwong said. “If there were areas where we needed more detail, we’d break out into sections. Then, we put all of these talks into our video knowledge center where anyone can go in and watch the sessions. We wanted to make sure everyone was comfortable with everything in our new domain.” 

The willingness to learn and help one another hasn’t dimmed since the team retreated to their homes for the pandemic.

“This is a really helpful team,” Huston said. “We’ve been really collaborative, even as we’re all remote. If I have a question, I can just ping someone to jump on a video call for 20 minutes to talk through it. That’s how I’ve learned a lot of this new tech.”

Whether working with open-source languages, helping a team member out with a solution or knocking down silos, Fanatics’ emphasis on collaboration has allowed it to better support fans worldwide feel connected to their communities through sports, in peak traffic and during lulls. 

For Vishnuvajhala, this feeling of togetherness culminates in a feeling of potential and possibility — for the individual and the collective engineering organization. 

“There is tremendous opportunity here.”

MORE ON FANATICS’ TECHSports Are Unpredictable. Here’s How Fanatics’ Tech Team Stays Ahead of the Game.