Conventional wisdom would suggest that you close your data centers and move to the cloud, not the other way around, but in 2016 Dropbox undertook the opposite journey. It (mostly) ended its long-time relationship with AWS and built its own data centers.
Of course, that same conventional wisdom would say, it’s going to get prohibitively expensive and more complicated to keep this up. But Dropbox still believes it made the right decision and has found innovative ways to keep costs down.
Akhil Gupta, VP of Engineering at Dropbox, says that when Dropbox decided to build its own data centers, it realized that as a massive file storage service, it needed control over certain aspects of the underlying hardware that was difficult for AWS to provide, especially in 2016 when Dropbox began making the transition.
“Public cloud by design is trying to work with multiple workloads, customers and use cases and it has to optimize for the lowest common denominator. When you have the scale of Dropbox, it was entirely possible to do what we did,” Gupta explained.
Alone again, naturally
One of the key challenges of trying to manage your own data centers, or build a private cloud where you still act like a cloud company in a private context, is that it’s difficult to innovate and scale the way the public cloud companies do, especially AWS. Dropbox looked at the landscape and decided it would be better off doing just that, and Gupta says even with a small team — the original team was just 30 people — it’s been able to keep innovating.
While Dropbox wouldn’t share exact numbers around the current size of the team, it claims it’s still running leanly compared to what it’s achieving. Gupta says that he doesn’t want to understate how difficult this transition was, and continues to be, but he says that with the right set of people, and the right level of automation, it is possible to do what they’ve done without a mega staff to make it work.
In fact, the challenge hasn’t been finding ways to innovate, it’s been choosing the right ideas so you aren’t spreading the team too thin. “The challenges we have found is how do we make sure we are betting on the right technology. We don’t have the luxury to try 50 ideas. We have to decide of those 50 ideas, which five do we want to explore to drive innovation. That goes back to the talent you have on the team,” he said.
Controlling your own destiny
Principal Engineer James Cowling says a move like this took careful consideration. “One way of thinking about that was thinking about why you would or wouldn’t own your own infrastructure. It’s hard and takes a lot of work. You have to ask if it would limit your ability to be agile. We have continued to be agile through foresight and investment,” Cowling explained.
He said a primary reason that the company had to take control of its hardware was the shifting needs of its software. Dropbox recognized that its business was changing. “We expanded from file sync and share to a content-based collaboration company. This meant we were shifting from storing files to providing a platform to interact with them,” Cowling said.
He said that resulted in fundamental changes in how they architected the platform because it changed the way users access files, and even has had an impact on the underlying content. It required richer metadata, the ability to have previews and other functions that weren’t really necessary when you were just storing files in Dropbox.
Innovation happens
Moving to its own data centers has forced the engineering team to think more about increasing efficiency to keep down costs. If the goal is to find the best innovations from the vast pile of ideas, it’s not always easy to know if you chose well until after the fact, but so far, the company seems to have made wise decisions.
One of those ideas was using Shingled Magnetic Recording disks (SMR for short). Cowling says this storage idea came from the team as they were completing the initial transition from AWS. “Some guys on the team went off and started thinking about what is the next step. What’s the big bold change that will improve us,” Cowling said.
What emerged was SMR, which has high storage density and a lower price point. Moving to SMR gave Dropbox the ability to do more with less, increasing efficiency and lowering overall costs — an essential step for a company trying to do this on its own. “It required expertise obviously, but it was also exciting to bring a lot of efficiencies in terms of cost and storage efficiency, while pulling down boundaries between software and hardware,” he said.
Later, the company decided it needed a cold storage option, recognizing that the typical user behavior was to move files into Dropbox, and then rarely retrieve it again. This was a case of coming up with an idea and struggling to get it to fruition, but once it was done, it provided a way for the company to use its storage resources much more efficiently.
Nothing is certain
Cowling says that even after moving from the cloud back to its data center, they don’t have rigidity about their approach and will use the public cloud when it makes sense to do so. “The other thing that’s interesting, we will choose to go the hybrid infrastructure route where it makes sense for our users. There’s no hubris there. We announced storage for Japan and Australia where we partnered with Amazon S3, and deployed quite fast without having to wait for the time to ramp up it would have taken to build it ourselves,” he said.
Gupta says a move like this can only work if you hire the right people and trust them to do the right thing. “We hire smart people and empower them to experiment and trust them to know which way to go.”
Even while the company continues to grow and develop the platform and the set of services that runs on it, it mostly manages its own hardware inside its own data centers. It’s not necessarily the way conventional wisdom would suggest you go, but it seems to be working for Dropbox.