This space has been silent for almost four years. What happened? Was I on a long vacation? Did I have nothing to share? Or was I just really busy? In the coming weeks I’m going to be adding some new posts, but before I do so I’d like to fill in the gap just a little bit.
It has in fact been a busy time, both personally and professionally. When I last posted my wife and I had no children, whereas now we are blessed to have two. Those of you who have raised children you know it’s both amazing and incredibly demanding. Those of you who might do so in the future should mark it down as something to look forward to doing when you have a lot of extra time.
Now for the professional highlights. I earned my MS in Computer Science from UC Berkeley in 2016, submitting the ReStream project as my thesis. Berkeley is a wonderful community, I came there to learn, and I’m happy to say that I learned a lot. Still, my thirst was for not quenched.
In 2017 I returned to the department, beginning a PhD that I’m still actively working on. What motivated me then is a question that’s been bugging me ever since I faced the challenge of scaling Tagged, a social network that I co-founded. I still couldn’t understand: why is programming at scale is so hard?
To most professional programmers it seems obvious. When you need to operate at scale you need a lot of computers. When you have a lot of computers you need to hook them all together, keep them all running well, and think about how they all should interact.
Moreover, prominent Silicon Valley companies such as Google have for years advertised their scale to attract top technical talent. As they tell it, scale not only multiplies the impact of your work, it also presents challenges that only the smartest engineers can solve.
This point of view resonated with me—I had hired very bright people who wrote lots of clever code that was used by hundreds of millions of end users. Still, I grappled with something fundamentally unsettling: the products that we were building just weren’t all that complicated. I didn’t work on Twitter, but viewed it as a prominent example. The original version was built in just a few weeks by a small team, but the company would employ hundreds just to ensure its ability to scale.
I will reserve a fuller exploration of this puzzle to a future post. For now I merely mention that others have also questioned the conventional assumption that scale has to be hard. Notably, serverless computing has emerged as a layer on top of the traditional cloud that simplifies its programming and operating model. Rather than working with virtual servers, programmers simply upload their code, hook it up to an API link or other trigger, and pay for the time that it runs. Launched in its contemporary form by Amazon Web Services in 2014, serverless has seen an explosion of interest in industry and academia.
It took me several years to come to the realization that serverless computing offered the most direct answer to the problem I was focused on. Programming at scale doesn’t have to be hard, it’s just hard when you need to deal with the complexity of lots of servers.
In a nutshell, I’ve been working on understanding and improving serverless computing, even before I knew to call it that. Serverless in its present form remains practical for certain limited applications, and figuring out how to bring the simplicity of programming at scale to a broader range of problems is a fascinating challenge. I look forward to sharing some of the ideas that I’ve been working on.