Diseconomies of Scale at Google

What was previously Google's biggest strength might prevent it from innovating in the future.

Google technology used to be years ahead of the industry. In 2004, Google released a paper on its proprietary MapReduce algorithm, the system that powered Google's massively parallel web crawling infrastructure. The previous year, Google had released a paper on its proprietary Google File System, which worked hand-in-hand with MapReduce. No other company was operating at Google scale.

But the industry always catches up, eventually. In 2006, two engineers would use those papers as a blueprint to create an open-source version of both technologies, Apache Hadoop and HDFS. They quickly became the industry standard - spawning huge companies like Cloudera, Hortonworks, and Databricks. Google's internal implementation was similar but incompatible. Not only had Google failed to commercialize the technology, but it now maintained a completely different codebase. This made it difficult to hire talent, expensive to keep up with improvements, and created a divergent basis for future innovation.

Avoiding the MapReduce/Hadoop situation was the initial rationale for open-source projects like TensorFlow and Kubernetes. While open-sourcing internal Google technologies has been wildly successful in both cases, Google is still filled with bespoke proprietary technology. Everything works differently at Google: building software, communicating between services, version control, code search, running jobs and applications, testing, and everything in between. Ramp-up time for new engineers continues to increase. Engineers aren't able to use off-the-shelf software because it won't work with internal technologies. Technologies that were years ahead are now only months.

These are the patterns of strategic disruption: a company like Google acts rationally, building its bespoke technologies as a competitive advantage. This series of rational decisions unexpectedly creates an opportunity for startups to move more quickly, take advantage of the current ecosystem, and eventually disrupt. While Google still has some of the best internal technology, it is on a parallel, but different, path than the rest of the world. Cloud computing has accelerated this divergence. Engineers may choose to work at companies where they can build non-firm-specific skills to further their careers. I believe that Google understands these issues, but the inertia may prove to be too great to overcome, even with the best effort.