While I was in college I made a prototype of a chat website. I wrote the long polling server in Erlang because it seemed like the ideal language for the task and also as a chance to have fun learning a new language.
I then worked on the initial Facebook chat prototype as part of a hackathon project in early 2007, along with Rebekah Cox and Ari Steinberg. I reused some of the Erlang code from my personal project. When we decided to make it an official project and productionalize the server code, the team decided to stick with Erlang and took over the code from there.
The rationale was something like:
The last point, lack of expertise, is a real weakness of the language, but it was outweighed by other factors at the time. Ultimately it contributed to Facebook's decision to move the service to C++, though even with the knowledge that it would eventually be rewritten, I think it was the right choice for the initial version because of the limited resources we had at the time. It was probably also the right choice for WhatsApp for the same reason.
See Ben Maurer's answer to When did Facebook switch away from using Erlang for Facebook Chat? for more details about why Facebook later switched.
At the time Facebook switched away from Erlang, the chat system had numerous reliability issues. The Erlang servers (which were used to provide long-poll functionality for browsers) caused some, but certainly not all of the problems. One problem we found with Erlang was that some of the abstractions Erlang had that allowed one to transparently have a distributed system across multiple machines actually caused reliability problems -- one server in the group would fail and cause cascading issues. Secondly, we found a number of instances where Erlang wasn't scaling well to multiple cores.
I don't think these issues are necessarily fundamental flaws of the Erlang runtime. It's quite possible that there was tuning that would have made Erlang work better for us (in fact, in our investigations we found a few tuning options that helped us).
However, Facebook has a number of useful abstractions in C++, making that a logical replacement. The server we ended up writing uses the same HTTP server that we use to balance all incoming requests to Facebook's website. This server is actively maintained -- for example, we spend time adding SPDY support and making SSL faster. Our open source Folly library provides great C++ abstractions for writing servers. We're able to reuse our high performance Thrift library.
In the end, I believe being able to reuse internal abstractions provided a substantial advantage. Many of these abstractions weren't available (or were in much more primitive states) when the initial Erlang service was written. Thus I believe Erlang was likely the logical choice when the chat service was first written (but I was not around then, so can't speak as to the merits of the choice at the time)
2011 - I have run up against the GIL in server side programming in almost every instance where I need something to scale to millions of concurrent users on multiple core machines. Python is great for command line tools and things that don't need true concurrency to extract every last bit of performance from a given piece of hardware. But for things that really need to squeeze everything out of something like a Sun T2000, you don't want to write anything in Python, it will be a operational maintenance nightmare running 32 separate processes and trying to management them all. I abandoned Twisted in favor of Erlang a few years ago, Python just doesn't cut it in the large scale concurrency space. The transparent distributed nature of Erlang means it scales horizontally as well as vertically.
October 20, 2012 Erlang is a twenty-five-year-old programming language that has yet to win a popularity contest, and almost certainly will never win any medals for speed, let alone any tiaras for syntactic beauty. The language is slow, awkward, and ugly. Refactoring Erlang code is a pain. Yet for almost five years, I have spent a large chunk of my free time programming in Erlang; at this point I've spent well over a thousand hours with the language. I've used the language to write, in rough chronological order, a CSV parser (don't laugh, I said chronological order), a template compiler, a object-relational mapper, a rich-text parser, an image resizer, a language pre-processor, a web framework, and a distributed message queue. What follows are my impressions of the language compared to other languages that I've used professionally (C, Java, Perl, PHP, Ruby, Objective-C, and JavaScript). The good news about Erlang can be summed up at this: Erlang is the culmination of twenty-five years of correct design decisions in the language and platform. Whenever I've wondered about how something in Erlang works, I have never been disappointed in the answer. I almost always leave with the impression that the designers did the “right thing.” I suppose this is in contrast to Java, which does the pedantic thing, Perl, which does the kludgy thing, Ruby, which has two independent implementations of the wrong thing, and C, which doesn't do anything. Take garbage collection. When it's time to collect garbage in other languages, the entire system has to stop while the garbage collector runs. This approach is perfectly fine if your computer program is supposed to run once, write some output, and then quit. But in long-running applications, such as desktop, mobile, or server programs, this strategy results in occasionally frozen UIs and slow response times. Erlang programs, on the other hand, can have thousands of independent heaps which are garbage-collected separately; in this way, the performance penalty of garbage collection is spread out over time, and so a long-running application will not mysteriously stop responding from time to time while the garbage collector runs. Or take string concatenation. If you pop open the implementation of string concatenation in Perl, Ruby, or JavaScript, you are certain to find an No matter how blocking and concurrent your application logic is, it is impossible to make a blocking network call in Erlang, or to spawn multiple OS processes. This design decision makes it so that an Erlang server will never crash the operating system. Having lost many nights of sleep to overloaded operating systems at a previous job, I believe that Erlang's concurrency design is correct. I mentioned that refactoring Erlang code is a pain. Fortunately, in my experience it is rarely necessary to refactor Erlang code in the same way the object-oriented code needs refactoring from time to time. In Erlang, each function is passed all the information it needs, and you get a compiler warning if it was passed any information it doesn't need. In some sense, refactoring is integrated into development; it is not a distinct activity requiring bountiful test coverage and several pots of coffee. Refactoring Java or Objective-C code usually becomes necessary because too many instance methods have been added to a class, and the developer must spend time figuring out which methods require which instance variables and how to best cut the carriage in half. This is simply not a concern in functional programming; moving a function to a different module requires very little hand-wringing and virtually no effort. “Refactoring” Erlang usually consists of breaking large functions down into smaller functions. There is not much mental effort involved; however, due to Erlang's syntactic peculiarities, it can be tedious converting anonymous functions to named functions. Perhaps a clever IDE will eliminate this tedium one day. All data structures in Erlang are completely transparent. Knowing nothing about the library you are using, you can always inspect the contents of data structures at run-time. This feature greatly aids in debugging, and is a boon to old-fashioned hacking. It is easy to manipulate undocumented data structures in order to implement functionality that the original library author did not intend. Unlike object-oriented programming, you never need to worry about the original author renaming variables and breaking your subclass code; as long as the underlying data structure remains the same, your modifications will continue to work in Erlang. I find that the transparency of data structures in Erlang makes programming much easier. In object-oriented programming, I am always worrying about what to name things; in Erlang, it usually doesn't matter, as the data structure is half the interface. If you have never programmed in Erlang, you probably have no idea what I am talking about. And so we come to the bad news about Erlang: the language's benefits are back-loaded. That is, most of the language's benefits can only be appreciated after several years with other languages followed by several years with Erlang. It is certainly not a language for beginners. The syntax is strange to programmers hailing from the C diaspora. Functional programming is tough, and Erlang doesn't put any sugar on the pill. The graphics toolkits are primitive, and there are no fill-in-the-code computer games such as are found in introductory Java courses. Reading any non-trivial Erlang code requires a firm understanding of recursion, a kind of abstract thinking that many people find difficult. Erlang is also lacking in libraries compared to other languages; in my experience, for any given task, there is zero, one, or at most two Erlang libraries available for the job. I am perhaps alone when I say this, but I actually like the fact that there are not many Erlang libraries available. If I need something done, I have the excuse to do it myself, and I often make discoveries that I would not have made otherwise. It sounds dumb but it is true. I can feel productive because I am doing something that no one has done yet, and along the way I have the freedom to try new approaches and make real innovations. I have learned more in the course of developing Erlang libraries than I ever learned stitching together other people's Ruby or C code. I program in Erlang purely for the enjoyment of solving problems and sharing my discoveries in well-engineered applications. To a seasoned hobbyist programmer like myself, the only truly bad news about Erlang is that it is slow. For the server applications I have written, the speed of the language has not been an issue; the extra cost in CPU was more than made up by Erlang's correct handling of garbage collection, network I/O, and string concatenation in a concurrent environment. In the language of complexity analysis, Erlang programs tend to have a large constant out front but excellent asymptotic properties. For the programmer who wishes to write fast programs using Erlang — the sorts of programs that start, run, write some output, and exit — there is hope on several fronts. A native-code compiler is available, and according to numerical benchmarks, it makes Erlang programs faster than Ruby, Perl, and PHP, albeit slower than Java and JavaScript. There is talk of a just-in-time native-code compiler, which might provide further improvements to execution time by gleaning information from the code execution itself and making appropriate optimizations. Finally, brave souls can write computationally intensive code in C via a NIF, with the important caveats that C code will block the Erlang scheduler (potentially negating Erlang's concurrency capabilities), and that parallel C code is famously difficult to write. My own choice for writing fast programs in Erlang is an alternative technology which affords all the benefits of C code without compromising the integrity of the Erlang run-time. It is OpenCL, a C-like language introduced by Apple in 2008. Like Erlang, OpenCL easily takes advantage of all the processor cores on a given machine; unlike Erlang, OpenCL programs are very fast. In fact, OpenCL programs are usually faster than C programs, as OpenCL programs can take advantage of a processor's vector capabilities in a way that normally requires hand-tuned assembler code. OpenCL programs can be compiled and executed directly from Erlang code; in my view, it is a perfect technology for performing computationally intensive tasks (that is, running those inner loops) inside of a larger Erlang program. By way of disclaimer, I have not actually used OpenCL inside an Erlang program. As I said, speed has not been a problem in the Erlang programs I've written. I do have some first-hand experience with OpenCL and have been quite pleased. I wrote a map projection library in OpenCL which is about 5 times faster than the state-of-the-art Proj.4 library (written in C). I also wrote an OpenCL library for doing multivariate statistics; I haven't benchmarked it against existing libraries, but I suspect it is faster by a similar margin. There are some peculiarities in writing OpenCL code, but it is my hope that one day all the world's tight loops will be rewritten in OpenCL and invoked from large programs written in Erlang. |
A core component of Mixpanel is the server that sits at http://api.mixpanel.com. This server is the entry point for all data that comes into the system – it’s hit every time an event is sent from a browser, phone, or backend server. Since it handles traffic from all of our customers’ customers, it must manage thousands of requests per second, reliably. It implements an interface we’ve spec’d out here, and essentially decodes the requests, cleans them up, and then puts them on a queue for further processing. Because of these performance requirements, we originally wrote the server in Erlang (with MochiWeb) two years ago. After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel. Given how crucial this service is to our product, you can imagine my surprise when I found out that this would be my first project as an intern on the backend team. I really enjoy working on scaling problems, and the cool thing about a startup like Mixpanel is that I got to dive into one immediately. Our backend architecture is modular, so as long my service implemented the specification, I didn’t have to worry about ramping up on other Mixpanel infrastructure. Libraries and Tradeoffs The first thing to think about is the networking library and framework to use. This server needs to scale, which for Python means using asynchronous I/O. At Mixpanel, we use eventlet pretty widely [1], so I decided to stick with that. Furthermore, since the API server handles and responds with some interesting headers, I decided to use eventlet’s raw WSGI library. Eventlet is actually built to emulate techniques pioneered by Erlang. Its “green threads” are pretty similar to Erlang’s “actors.” The main difference is that eventlet can’t influence the Python runtime, but actors are built into Erlang at a language level, so the Erlang VM can do some cool stuff like mapping actors to kernel threads (one per core) and preemption. We get around this problem by launching one API server per core and load balancing with nginx. Another thing to think about is the JSON library to use. Erlang is historically bad at string processing, and it turns out that string processing is very frequently the limiting factor in networked systems because you have to serialize data every time you want to transfer it. There’s not a lot of documentation online about mochijson’s performance, but switching to Python I knew that simplejson is written in C, and performs roughly 10x better than the default json library. Finally, we use a few stateful, global data structures to track incoming requests and funnel them off to the right backend queues. In Erlang, the right way to do this is to spawn off a separate set of actors to manage each data structure and message pass with them to save and retrieve data. Our code was not set up this way at all, and it was clearly crippled by being haphazardly implemented in a functional style. It was quick and easy for me to implement a clear and easy-to-use analog in Python, and I was able to enhance some of the backing algorithms in the process. It might not sound “cool” to implement a data structure this way, but I was able to provide some important operations in constant time along with other optimizations that were cripplingly slow in the Erlang version. Performance Of course, a major concern with Python is performance. We receive a few thousands of requests per second, and it’s important that the API server can handle spikes in traffic. As a startup, we also need to be careful about our server costs. We could buy several servers to power the API and scale horizontally, but if we can write a fast enough server to begin with, that’s a waste of money. The optimizations I made were relatively minor, and most of my speed came from leveraging the right Python libraries. The community is extremely active, so many of my questions were already answered on Stack Overflow and in eventlet’s documentation. Benchmarking The setup I used to benchmark was pretty simple. I ran the API server (it only used a single core) and kestrel (queue) on the same machine, with 4 GB of RAM and a 2000 MHz AMD Opteron processor. The 4 client processes ran together on a quad-core machine with 1 GB of RAM and the same type of CPU. Each client had 25 eventlet green threads and ran requests randomly from access logs I pre-processed. Everything was running on Rackspace. Here are the results:
As you can see, we maintained roughly 1000-1200 requests per second (on a single core), with a latency of almost always less than 100 milliseconds. We plan to deploy the API on a quad-core machine with 4 GB of RAM, so these numbers are definitely fast enough. A Little Treat If you read our API spec, you’ll notice that we return ‘0’ on failure and ‘1’ on success. The failure could be anything from invalid base-64 encoding to an error on our end. Developers writing their own clients have complained about this, and we listened. With the rewrite you can set “verbose=1” in the query string and we’ll respond with a helpful error message if your request fails. We’ll post again when this feature is fully live. Reflections I’ve written a few servers like this as personal projects, but I’ve never gotten the opportunity to throw one against real scale. With actual traffic, the challenges are much different, and unlike the prototypes I’ve written before, the code has to work. Our customers rely on the API server reliably storing their requests, and we need to recover from every type of possible failure. The biggest challenge for me was pushing the server from working 99.9% of the time to 99.99% of the time, because those last few bugs were especially hard to find. I’ve learned a lot about how to scale a real service in the couple of weeks I’ve been here. I went into Mixpanel thinking Erlang was a really cool and fast language, but after spending a significant amount of time sorting through a real implementation, I understand how important code clarity and maintainability is. Scalability is as much about being able to think through your code as it is about systems-level optimizations. If you’re itching to learn more about real-world scaling problems, then you should work here. Mixpanel offers the unique combination of a fun startup environment with the scaling challenges that companies like Facebook and Twitter face every day. [1] The other popular one is gevent. Avery wrote about some of the tradeoffs earlier: http://code.mixpanel.com/2010/10/29/gevent-the-good-the-bad-the-ugly/ Comments Hey Ankur, I’m disappointed that you guys didn’t stick with Erlang. I think you’d be VERY pleased with its performance if you had stuck with it. I’ve only recently been introduced to Erlang and have been using it over the last year professionally. We have been able to solve so many problems with Erlang that would have been nearly impossible to solve with any other language. Performance was one of the biggest things. Without going into proprietary details, at work, we are doing on one Mac Mini with Erlang what takes our competition 78 Windows servers to do. Before I go any farther, congratulations on inspiring me to write code in my free time. That doesn’t happen often. I read your problem description and was curious how Erlang, written by someone who has some professional experience with it in a performance-oriented world, would do in comparison to the Python results. I saw in your requirements that the intention was to hand the JSON object off to another process through RabbitMQ, but I didn’t understand why when all it seemed like all you wanted to do was log the event. So, I just wrote it to MongoDB instead. From the testing I’ve done at work, it’s faster to offload the message to Rabbit than it is to write it to Mongo. So, if anything, writing it to Mongo made my results worse. I ran the sample data you have in a single “thread” (called a light-weight process in Erlang). I used mochiweb to receive the request, parse out the “data” value from the query string, base64 decode it, JSON decode it, and write the JSON to Mongo DB. (BTW, doing the base64 and JSON decoding together took 63 microseconds.) With a single process, mind you, it was handling 4,863 messages per second for the 1,000,000 messages I ran. In that test, I was sending, receiving, processing, writing to the DB, responding, and receiving the response in the same Erlang VM on my laptop. I then wondered how much of that time was taken up by writing to the DB, so I removed that part of the test and re-ran it. That got me 6,700 messages per second. Needless to say, at over 5 times the performance of Python, it might be worth your company’s money to send you to an Erlang class or for you to invest the time in learning it yourself. I know it’s super weird when you first get into it, but you’ll definitely grow to love it for problems like this. Like I said, I’m writing code in my free time, and that says a lot! Jeremy Hood ------- Few points. #1. Having no expert on staff that major application X is written in, is absolutely silly, and it basically nullifies all your other points about maintainability, speed and downtime. The same would happen in Ruby or C++ or YYY. Erlang is generally considered exceptionally maintainable due to low number of lines of code, functional idioms and tons of real world reuse. #2. The only major (valid?) point you had on performance seemed to be based around a single implementation (mochijson) of a JSON library and more generally strings. I would advise you to look at Jiffy (NIF C based JSON encoder / decoder) and binaries (general Erlang language feature). Most of the time you don’t need to muck with strings too much, just pass ’em around or compare them to something. #3. The other performance issue — as you mentioned, was a haphazard implementation done by non-experts, I think you can hardly hold this against Erlang, anymore than you could hold a horrible C++ implementation of X against C++. … almost seems amazing that you ran a production app in an language you don’t have experts in for two years at high scale… I would give the language bonus points for that, triple-dog-bonus-points. Erlang as a functional language can seem daunting at first, but is exceptionally maintainable on very large scales, but it has to have a proper architecture and have experts on staff. |
How do you support 450 million users with only 32 engineers? For WhatsApp, acquired earlier this week by Facebook, the answer is Erlang, a programming language developed in the '80s that is finally having its moment in the spotlight. But with other languages starting to co-opt its ideas, does Erlang have a future? Erlang was developed by Swedish telecom giant Ericsson over 25 years ago, and now it's finding a home at messaging apps like WhatsApp and TigerText. Even Facebook was singing the language’s praises when it used Erlang to launch Facebook Chat back in 2009—the same year it turned down the job application of WhatsApp cofounder Brian Acton. CONCURRENCY IS THE NEW BLACK "With Erlang you could build a messaging app with many connected users and not worry about how they communicate," says Chad DePue, CTO of Whisper, an app for anonymous posts that has opted to use the language. Instead, he says, "You can worry about creating a great app." Ericsson engineer Joe Armstrong developed Erlang with the logic of telecommunications in mind: millions of parallel conversations happening at the same time, with almost zero tolerance for downtime. Other programming languages can only give the illusion of managing all those conversations—some have gotten very good at keeping up appearances, but they aren't natural "multi-taskers." Erlang, in contrast, loves to multithread or "juggle" in this way—got a another spinning plate? Toss it over! "The language is very expressive," says Igor Clark, a creative technologist. "You can talk at a high level and do quite a lot with its few key concepts." In practical terms, that initially made Erlang very good at efficiently executing commands across processors within a single machine. Fast-forward to 2014, and it’s making Erlang very good at executing an even more bewildering volume of commands across the global networks of servers we have come to know as "the cloud." For use cases in gaming, financial services, and anything that mimics the behavior of a real-time auction, that speed and reliability, at massive scale, is absolutely essential. Equally attractive to engineering teams, and highly unusual, is the way that Erlang allows for bug fixes and updates without downtime. In effect, you can change the lighting on your set or execute a costume change without a pause in the action. This Erlang property is a legacy of the telecom imperative: As DePue says, "When you're on the phone, you can't have someone hang up on you because they're upgrading the system." TURNING THE TABLES ON TELECOM Today the industry that open-sourced Erlang in 1998 is feeling less than generous. As WhatsApp, WeChat, Line, and other startups have blossomed, the messaging fees that telecoms once relied on have been rapidly eroding. KPN, a Dutch telecom, tried to block the upstarts from its network after seeing a drop in SMS revenue, but lost a battle in court over the practice. Some established players, such as Vodafone, have tried to build messaging services of their own, with little success. Nearly all have been tinkering with the mix and pricing of SMS and data in their users’ contracts. While the telecoms were busying themselves with payment plans, WhatsApp was focused on product. Its engineering team has repeatedly wowed the Erlang community with feats of scale and speed, jumping from 10,000 to over 2 million connections per server in just a few short years. Those achievements rest on the shoulders of Armstrong, whose signature project for Ericsson, AXD301, achieved a "nine nines" reliability (99.9999999%). But for now, at least, the young challengers are keeping the aging telecoms on their heels. TO ERLANG, AND BEYOND? With such a strong track record, you might think that Erlang is poised for breakout adoption. The reality is a bit more complicated. Until now, the developer community around Erlang has remained small, and largely concentrated in Europe. The language’s syntax is admittedly "weird." And if you need to analyze complex data or prototype a simple web app, there are better tools. Moreover, newer languages and variations have been borrowing from Erlang principles, leading to an increasingly cluttered landscape, from Google’s Go toDocker. There’s hope for Erlang in Elixir—Armstrong recently bestowed his blessing—which normalizes the Erlang syntax and could help the language take off in the same way that Rails did for Ruby. Projects like Chicago Boss are also aiming to make the language more user-friendly. If a broader community does coalesce around Erlang, they will discover OTP (open telecom platform) libraries that more than make up for the language’s limited presence on GitHub. "They abstracted patterns from the software they were writing and made bulletproof libraries that you can piggyback on," Clark says. "It’s the glue between the small parts and the top level." The tipping point for adoption could be the one ecosystem with enough scale to dwarf mobile as we know it: the internet of things. Smart object hobbyists have been experimenting with Erlang as part of MQTT, a lightweight protocol for transferring messages between sensors. If you can SMS your thermostat, surely you can WhatsApp it, too? |
EARLIER THIS MONTH, in a post to his Facebook page, WhatsApp CEO Jan Koum announced that his company’s instant messaging service is now used by more than 900 million people. And then Facebook CEO Mark Zuckerberg promptly responded with two posts of his own. One said “congrats,” and the other included a cheeky photo Zuckerberg had taken of Koum as the WhatsApp CEO keyed his 900-million-user post into a smartphone. “Here’s an action shot of you writing this update,” Zuckerberg wrote. WhatsApp is owned by Facebook, after Zuckerberg and company paid $19 billion for the startup a little more than a year ago. That means Facebook now runs three of the most popular apps on the internet. Its primary social networking service is used by more than 1.5 billion people worldwide, and Facebook Messenger, an instant messaging service spun off from Facebook proper, spans 700 million. But the 900 million-user milestone announced by Koun is very much a WhatsApp achievement, not a product of the formidable Facebook machine. One of the (many) intriguing parts of the WhatsApp story is that it has achieved such enormous scale with such a tiny team. When the company was acquired by Facebook, it had 35 engineers and reached more than 450 million users. Today, it employs only about 50 engineers, though the number of WhatsApp users has doubled, and this tiny engineering staff continues to run things almost entirely on its own. In a world where so many internet services are rapidly expanding to millions upon millions of users, WhatsApp shows the way forward—at least in part. WhatsApp doesn’t talk much about its engineering work—or any other part of its operation, for that matter—but yesterday, at an event in San Jose, California, WhatsApp software engineer Jamshid Mahdavi took the stage to briefly discuss the company’s rather unusual methods. Part of the trick is that the company builds its service using a programming language called Erlang. Though not all that popular across the wider coding community, Erlang is particularly well suited to juggling communications from a huge number of users, and it lets engineers deploy new code on the fly. But Mahdavi says that the trick is as much about attitude as technology. Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen—in part because it used Erlang and a computer operating system called FreeBSD, but also because it strove to keep its operation so simple. “It was a completely different way of building a high-scale infrastructure,” he said on Monday. “It was an eye-opener to see the minimalistic approach to solving … just the problems that needed to be solved.” Code in Parallel In using Erlang, WhatsApp is part of a larger push towards programming languages that are designed for concurrency, where many processes run at the same time. As internet services reach more people—and juggle more tasks from all those people—such languages become more attractive. Naturally. With its new anti-spam system—a system for identifying malicious and otherwise unwanted messages on its social network—Facebook uses a language called Haskell. Haskell began as a kind of academic experiment in the late ’80s, and it’s still not used all that often. But it’s ideal for Facebook’s spam fighting because it’s so good at juggling parallel tasks—and because it lets coders tackle urgent tasks so quickly. Meanwhile, Google and Mozilla, maker of the Firefox browser, are striving for a similar sweet spot with new languages called Go and Rust. Like Haskell, Erlang is a product of the ’80s. Engineers at Ericsson, the Swedish multinational that builds hardware and software for telecom companies, developed the language for use with high-speed phone networks. “Instead of inventing a language and then figuring out what to do with it, they set out to invent a language which solved a very specific problem,” says Francesco Cesarini, an Erlang guru based in the UK. “The problem was that of massive scalability and reliability. Phone networks were the only systems at the time who had to display those properties.” (NW: Lesson: Keep your eye out for tech being developed by the big guys!) Erlang remains on the fringes of the modern coding world, but at WhatsApp and other internet companies, including WeChat and Whisper, it has found a home with new applications that operate not unlike a massive phone network. In essence, WhatsApp is a replacement for cellphone texting services. It too requires that “scalability and reliability.” What’s more, Erlang lets coders work at high speed—another essential part of modern software development. It offers a way of deploying new code to an application even as the application continues to run. In an age of constant change, this is more useful than ever. Keep It Simple, Smarty The language does have its drawbacks. Relatively few coders know Erlang, and it doesn’t necessarily dovetail with a lot of the code already built by today’s internet companies. Facebook built its original Facebook Chat app in Erlang but eventually rebuilt so that it would better fit with the rest of its infrastructure. “You had this little island that was Erlang, and it was hard to build enough boats back to the island to make everything hook in,” says Facebook vice president of engineering Jay Parikh. Of course, WhatsApp didn’t have to integrate with an existing infrastructure in this way. And Mahdavi believes the relative scarcity of Erlang coders isn’t a problem. “Our strategy around recruiting is to find the best and brightest engineers. We don’t bring them in specifically because the engineer knows Erlang,” Mahdavi said on Monday. “We expect the engineer to come in and spend their first week getting familiar with the language and learning to use the environment. If you hire smart people, they’ll be able to do that.” The company has succeeded by hiring engineers who are adaptable—in more ways than one. Asked to explain the company’s secret, Mahdavi’s response seems far too simple. But that’s the point. “The number-one lesson is just be very focused on what you need to do,” he said. “Don’t spend time getting distracted by other activities, other technologies, even things in the office, like meetings.” At WhatsApp, employees almost never attend a meeting. Yes, there are only a few dozen of them. But that too is the point. |
We love Rails, but we believe that Ruby is the wrong choice of platform on which to build a simple, fast, reliable website. The alternatives aren't much better: the JVM, V8, Python, and Perl all eventually run into the problem of dog-slow server-side templates.Web developers usually "solve" the problem in a number of ways: buying lots of hardware, hiring a big ops team, implementing complex nested hierarchical cache structures, ripping out features, and moving to a "thick-client" architecture that takes forever to load. But there's a much simpler solution: it's called respecting the RAM.
Erlang Respects Your RAM!
Erlang is different from other platforms because when rendering a server-side template, it doesn't create a separate copy of a web page in memory for each connected client. Instead, it constructs pointers to the same pieces of immutable memory across multiple requests.
So if two people request two different profile pages at the same time, they're actually sent the same chunks of memory for the header, footer, and other shared template snippets. The result is a server that can construct complex, uncached web pages for hundreds of users per second without breaking a sweat.
With Erlang, you can run a website on a fraction of the hardware that Ruby and the JVM require, saving you money and operational headaches. You can run a website on "wimpy-core" ARM servers, helping your bottom-line along with the environment. Your test suite will run much faster than you thought possible, shortening your development cycle and letting you deliver features more quickly.
Chicago Boss Makes Erlang Accessible
Erlang has a reputation for being a mystical technology available only to an elite few willing to spend years learning its arcane syntax. It's true that Erlang "looks weird" to developers accustomed to C-like syntax, and the semantics of functional programming takes some getting used to. By adopting familiar Rails conventions, Chicago Boss has finally made Erlang accessible to programmers who don't have years and years of free time.
There is no separate Boolean type in Erlang. The atoms true and false are used instead. The following Boolean operators are available: and, andalso, or, orelse, xor, not
Expressions
