A Practical Introduction to Erlang

Since first hearing about functional programming, I have made many attempts to ‘get it’. I felt it was something cool and worthwhile to learn, but I didn’t make any real progress until…

For some reason my brain, spoiled by a decade of imperative programming, just didn’t work this way. I was able to write a few snippets in Common Lisp and Scheme but I felt I couldn’t write anything real. Racket was much better and I almost ‘got’ it, but this language seems overly complex to me. Haskell is still beyond me. But at some point I got a book called Programming Erlang by Joe Armstrong [Armstrong]. And at last, my journey into the functional world had begun.

To begin with, take a look at a quote from that book:

A few years ago, when I had my research hat on, I was working with PlanetLab. I had access to the PlanetLab network, so I installed ‘empty’ Erlang servers on all the PlanetLab machines (about 450 of them). I didn’t really know what I would do with the machines, so I just set up the server infrastructure to do something later.

That was cool I thought. If I had a cluster of 450 machines to play with, just imagine what could I do with this! Bare bones C or C++ don’t give me much. Messing around with POSIX threads and TCP/IP directly will take ages to implement anything plausible from scratch. Even boostified C++ is lacking in this scenario. But this guy has an infrastructure out of the box!

So, what is Erlang? The language was developed at Ericsson to program telecoms devices. I expected it would be a system language like C, or at least Go if it had been created at that time. But fasten your seat belts – Erlang works in a managed environment (simply, a virtual machine executing a byte code), and secondly: it is a functional language. I was shocked – how can you deal with bits and bytes, packing and unpacking low-level network messages, process them efficiently under massive load and similar stuff, in a functional language?

Guys from Ericsson research were given the task of developing a language to build sophisticated but robust and scalable systems, and they had ended up with functional Erlang. This just got me.

After some time messing around with Erlang, I explored a few parallels with my everyday development in C++. I remember a John Carmack tweet with his sad story about a day spent finding a bug caused by a variable that had been accidentally changed somewhere. To avoid this waste of time, the variable should be simply declared as const. At some point I began to be obsessed by using const in C++. Increasing demand to write complex code safely led to my understanding that immutability is what I really want. In C and C++ that can be just putting const everywhere the logic of the code permits.

But Erlang takes immutability to another level. All variables in this language are immutable. Period. In fact, a variable can be assigned only once, when it gets created. Right after that it turns into a constant. Imagine in C or C++ that you must put const in front of any variable. Any mutation can be achieved only by copying and creating another variable. This is an extreme for C or C++, but this is the world of Erlang.

At first glance this looks like an obviously silly overhead. But let’s slow down and give it a second thought. Yes, there is an overhead in copying, but at the same time the code has fewer side effects, and as a consequence fewer bugs caused by hidden data mutation in the complicated branching. Moreover, data is mutated only via copying, so the compiler and runtime (remember, Erlang is a managed environment) have many more clues on how to optimize and eliminate unnecessary copying. The runtime provides a set of native functions to mutate the data efficiently. For example, in Erlang, adding a new head to a list is quick but appending a tail is slow because it causes a deep copy. When you understand such peculiarities you can organize the mutations of data to be very efficient, yet free of side effects.

Finally, code without an internal state (and the immutability is a good guarantor of it) is much easier to parallelize in a multi-core or multi-machine environment, and Erlang has great support for multi-threading.

Well, I hope I’ve sown a seed of interest in Erlang, and it is time to code something real.

When I discover a new language, and after playing with trivial snippets, I often code a task called TCP/IP proxy. Simply, this application listens on a given TCP/IP port and for every incoming connection it connects to another remote host and passes the traffic back and forth between the caller and the remote host. Also the proxy logs all transmitted packets in the form of hexadecimal dumps. The application has to process multiple connections simultaneously.

This application can be very useful when you need, for example, to reverse engineer or debug an application protocol. From the implementation perspective it also very indicative – it involves string processing, multi-threading, sockets and file I/O.

In the past I’ve implemented it in C, C++, Python, PHP, Ruby and Go, and every time it was fun.

The code is below. Of course I cannot explain every single line if you’re a newbie in Erlang, so you could check out the book Programming Erlang I mentioned above (at least the few first chapters to get basic concepts of Erlang).

I will go through the code and try to stress important elements. I recommend following and try to get a taste of Erlang.

Here we go. A file called tcp_proxy.erl (see Listing 1).

 1 -module(tcp_proxy).
 2 -export([main/1]).
			

In lines 1 and 2 we define our unit of code, a module, and export one function main having one parameter – a list of command line arguments.

The definition of the function main (Listing 2) is similar in many other languages – if three command line arguments are supplied then this version is invoked.

 3 main([ListenPort, RemoteHost, RemotePort]) ->
 4   ListenPortN = list_to_integer(ListenPort),
 5   RemotePortN = list_to_integer(RemotePort),
 6   start(ListenPortN, RemoteHost, RemotePortN);
			

There’s a second definition of the function main (Listing 3) which matches any other use that doesn’t match the first. It simply prints usage information.

 7 main(_) -> usage().
 8 usage() ->
 9   io:format("~n~s local_port remote_port
        remote_host~n~n", [?FILE]),
10   io:format("Example:~n~n"),
11   io:format("tcp_proxy.erl 50000 google.com
     80~n~n").
			

Now the fun begins in the function start (Listing 4). We create a TCP/IP listener (line 16) and then launch an acceptor thread (lines 19–21). A parallel thread (known as a process) is created by the Erlang spawn function.

12 start(ListenPort, CalleeHost, CalleePort) ->
13   io:format("Start listening on port ~p and
        forwarding data to ~s:~p~n",
14     [ListenPort, CalleeHost, CalleePort]),
15   ListenOptions = [binary, {packet, 0},
        {reuseaddr, true}, {active, true}],
16   case gen_tcp:listen(ListenPort,
        ListenOptions) of
17     {ok, ListenSocket} ->
18       io:format("Listener started ~s~n",
            [socket_info(ListenSocket)]),
19       spawn(fun() -> 
20         acceptor(ListenSocket, CalleeHost,
              CalleePort, 0) 
21       end),
22       receive _ -> void end;
23     {error, Reason} ->
24       io:format("Unable to start listener,
            error '~p'~n", [Reason])
25 end.
			

At this point it is worth explaining the concept of processes in Erlang. Normally by ‘a process’ we mean an OS container having threads running within it. In turn ‘a thread’ usually means a single execution flow within a process planned for execution by the OS scheduler.

In Erlang the term ‘process’ means a different thing. It is a lightweight thread scheduled for execution not by the OS scheduler, but by the Erlang runtime. It is possible to launch a few thousand processes in Erlang and the runtime will multiplex them onto native OS threads. Processes in Erlang are very fast to create and have a small memory footprint.

The concept of lightweight processes is quite similar to goroutines in Go (maybe goroutines were even inspired by Erlang). [Go] Moreover, the runtime can launch processes on remote Erlang nodes in exactly the same way as locally. Remember the quote from Joe’s book at the beginning about 450 servers? This is where the magic begins.

From here I will use the term ‘process’ to refer to these lightweight parallel execution flows and not OS processes or threads.

In line 22 the main process starts receiving messages (we’ll take a look at messaging a bit later). In fact nobody will be sending messages to the main process, so it will be blocked indefinitely unless you stop the application from outside.

Now the main process sleeps, but the acceptor process is ready to serve incoming connections.

In lines 26–37 (Listing 5) there are few string formatting routines. Nothing fancy.

26 format_socket_info(Info) ->
27   {ok, {{A, B, C, D}, Port}} = Info,
28   lists:flatten(
        io_lib:format("~p.~p.~p.~p-~p",
        [A, B, C, D, Port])).

29 peer_info(Socket) ->
   format_socket_info(inet:peername(Socket)).
30 socket_info(Socket) -> 
   format_socket_info(inet:sockname(Socket)).

31 format_date_time({{Y, M, D}, {H, MM, S}}) ->
32   lists:flatten(
33     io_lib:format("~4.10.0B.~2.10.0B.~2.10.
       0B-~2.10.0B.~2.10.0B.~2.10.0B", 
34                   [Y, M, D, H, MM, S])).

35 format_duration({Days, {H, M, S}}) ->
36   lists:flatten(
37     io_lib:format(
          "~2.10.0B-~2.10.0B.~2.10.0B.~2.10.0B",
          [Days, H, M, S])).
			

Now the acceptor (Listing 6). In line 39 we accept an incoming connection, then in lines 41–43 launch another acceptor, and finally within the current process we connect to the remote host (line 44) and start forwarding data between the local and remote sockets by calling process_connection function (line 46).

38 acceptor(ListenSocket, RemoteHost,
      RemotePort, ConnN) ->
39   case gen_tcp:accept(ListenSocket) of
40     {ok, LocalSocket} -> 
41       spawn(fun() -> 
42         acceptor(ListenSocket, RemoteHost,
              RemotePort, ConnN + 1) 
43       end),
44       case gen_tcp:connect(RemoteHost,
            RemotePort, [binary, {packet, 0}]) of
45         {ok, RemoteSocket} ->
46           process_connection(ConnN,
             LocalSocket, RemoteSocket);
47         {error, Reason} ->
48           io:format("Unable to connect to ~s:~s
             (error: '~p')", 
49             [RemoteHost, RemotePort, Reason])
50        end;
51    {error, Reason} ->
52      io:format("Socket accept error '~w'~n",
           [Reason])
53   end.
			

I’d like you to pause and think. There is a pattern in many languages like C and C++ of how to implement a TCP/IP server: we have a main listener process; when an incoming connection comes up the listener creates a worker process, passes the accepted socket to the worker for processing and continues to listen.

But our logic here is different: our listener processes the connection data within its own process because it also plays a worker role, but prior to the processing it clones itself to continue listening.

The former listener is now a worker. It processes the connection and then terminates.

This approach is natural for Erlang. It can be partially explained because in Erlang a process accepting a connection becomes its owner, and all messages from that connection will be delivered to the owner. This rule can be abused, but the point here is that processes are cheap and easy to create, and you’re free to create as many as you want.

An Erlang developer usually fires up processes not per task, which tend to perform multiple activities, but per logically concurrent activity. And the activities don’t need to multiplex anything.

Now we begin to process the connection (Listing 7). The process_connection function takes the number of the current connection and a pair of sockets. In line 57 it launches its own logger. Then via sending messages (line 59, 62 and 64) it communicates to it. By having the logger in a separate process we split the data transferring activity and the logger.

54 process_connection(ConnN, LocalSocket,
      RemoteSocket) ->
55   LocalInfo = peer_info(LocalSocket),
56   RemoteInfo = peer_info(RemoteSocket),
57   Logger = start_connection_logger(ConnN,
        LocalInfo, RemoteInfo),
58   StartTime = calendar:local_time(),
59   Logger ! {connected, StartTime},
60   pass_through(LocalSocket, RemoteSocket,
        Logger, 0),
61   EndTime = calendar:local_time(),
62   Logger ! {finished, StartTime, EndTime},
63   % Stop the logger.
64   Logger ! {stop, self(), ack},
65   receive ack -> void end.	
			

In lines 59 and 62 we send asynchronous messages but in the line 64 we send a synchronous one. At line 64, the processing of the connection is finished and we need to stop the logger by sending a stop command. But we must get an ack response back when the logger terminates (line 65).

Now the data transmitter (Listing 8). The pass_through function transmits data between two sockets and backs up the traffic to the logger. In line 67 we receive a next portion of the data from the sockets. Then in lines 68, 73, 78 and 80, using the pattern matching syntax of Erlang, we decide what kind of data has arrived and from where. The two branches starting at lines 69 and 74 mirror each other. Let’s take a look at the first one. At line 69 it sends the received binary packet to the logger. Then it transmits the packet to the peer socket (line 70) and at line 71 it sends a notification message to the logger saying that the packet was delivered.

66 pass_through(LocalSocket, RemoteSocket,
      Logger, PacketN) ->
67   receive
68     {tcp, LocalSocket, Packet} ->
69       Logger ! {received, from_local, Packet,
            PacketN},
70       gen_tcp:send(RemoteSocket, Packet),
71       Logger ! {sent, to_remote, PacketN},
72       pass_through(LocalSocket, RemoteSocket,
            Logger, PacketN + 1);
73     {tcp, RemoteSocket, Packet} ->
74       Logger ! {received, from_remote, Packet,
            PacketN},
75       gen_tcp:send(LocalSocket, Packet),
76       Logger ! {sent, to_local, PacketN},
77       pass_through(LocalSocket, RemoteSocket,
            Logger, PacketN + 1);
78     {tcp_closed, RemoteSocket} -> 
79       Logger ! {disconnected, from_remote};
80     {tcp_closed, LocalSocket} -> 
81       Logger ! {disconnected, from_local}
82   end.
			

Take a look at line 72. This is a very important one. In Erlang there are no reserved words or operators for loops. You can simulate loops using lambdas and list comprehensions, but usually looping is implemented in Erlang by tail recursion. In line 72 the function pass_through calls itself. This doesn’t mean that for every nested call it creates a new frame of the stack. Instead, the call to itself is the last one in the function execution flow, and the compiler optimizes such calls into tail recursion instead of normal recursion.

In short, tail recursion is a jump to the start of the calling function, but without using a proper context saving and restoring approach.

Finally in line 78 and 80 when we have the situation of a disconnected socket, the function doesn’t call itself any more and just exits.

The function start_connection_logger (Listing 9) begins a logging activity. It forms a name for the log and fires up a connection_logger function in a separate process. The function spawn_link differs from spawn by making the main process and a newly created one linked. If one dies or exits, its counterpart will be notified.

83 start_connection_logger(ConnN, From, To) ->
84   {{Y, M, D}, {H, MM, S}} =
        calendar:local_time(),
85   LogName = lists:flatten(
86     % YYYY.MM.DD-hh.hh.ss-ConnN-From-To.log
87     io_lib:format(
         "log-~4.10.0B.~2.10.0B.~2.10.0B-"
88       "~2.10.0B.~2.10.0B.~2.10.0B-~4.10.0B-"
89       "~s-~s.log", 
90       [Y, M, D, H, MM, S, ConnN, From, To])),
91   spawn_link(fun() -> connection_logger(ConnN,
        From, To, LogName) end).
			

In Listing 10 (lines 92–95) we see four functions having the same name. To choose which function to call, Erlang applies the concept of pattern matching on the data, but not argument types like in C++ for instance. The key here is the first argument. In Erlang, identifiers starting with a lowercase letter are atoms. Atoms are implicit constants, enums if you like. When calling the peer_name function (lines 104, 112 and 124) if the first argument is the from_local atom, for example, it calls the first version of the function.

92   peer_name(from_local, LocalInfo,
        _RemoteInfo) -> LocalInfo;
93   peer_name(from_remote, _LocalInfo,
        RemoteInfo) -> RemoteInfo;
94   peer_name(to_local, LocalInfo, _RemoteInfo)
        -> LocalInfo;
95   peer_name(to_remote, _LocalInfo, RemoteInfo)
        -> RemoteInfo.
			

Of course pattern matching is a much wider technique in Erlang, also used for branching. For example, given an expression you can provide a list of possible values expected in it and Erlang will try to match them all against the expression and pick a matching option. We’ll see an example of this in a moment.

Again in lines 96 and 101 (Listings 11 and 12), we see two functions with the same name. The first one is called from start_connection_logger.

 96   connection_logger(ConnN, From, To, LogName)
         when is_list(LogName) ->
 97     Putter = fun(Message) ->
 98       append_message_to_file(Message, LogName)
 99     end,
100     connection_logger(ConnN, From,
           To, Putter);
			

101 connection_logger(ConnN, FromInfo, ToInfo,
       LogWriter) ->
102   receive
103     {received, From, Packet, PacketN} ->
104       PeerName = peer_name(From, FromInfo,
             ToInfo),
105       Message = fun(Printer) -> 
106         Printer("Received (#~p) ~p byte(s)
               from ~s~n", 
107           [PacketN, byte_size(Packet),
              PeerName]),
108         binary_to_hex(Packet, Printer)
109       end,
110       write_message(ConnN, FromInfo, ToInfo,
             LogWriter, Message);
111     {sent, To, PacketN} ->
112       PeerName = peer_name(To, FromInfo,
             ToInfo),
113       Message = fun(Printer) ->
114         Printer("Sent (#~p) to ~s~n",
               [PacketN, PeerName])
115       end,
116       write_message(ConnN, FromInfo, ToInfo,
             LogWriter, Message);
117     {connected, Time} ->
118       When = format_date_time(Time),
119       Message = fun(Printer) ->
120         Printer("Connected to ~s at ~s~n",
               [ToInfo, When])
121       end,
122       write_message(ConnN, FromInfo, ToInfo,
             LogWriter, Message);
123     {disconnected, From} ->
124       PeerName = peer_name(From, FromInfo,
             ToInfo),
125       Message = fun(Printer) ->
126         Printer("Disconnected from ~s~n",
             [PeerName])
127       end,
128       write_message(ConnN, FromInfo, ToInfo,
             LogWriter, Message);
129     {finished, StartTime, EndTime} ->
130       Duration = calendar:time_difference
             (StartTime, EndTime),
131       When = format_date_time(EndTime),
132       Message = fun(Printer) ->
133         Printer("Finished at ~s,
              duration ~s~n",
134           [When, format_duration(Duration)])
135       end,
136       write_message(ConnN, FromInfo, ToInfo,
             LogWriter, Message);
137     {stop, CallerPid, Ack} ->
138       CallerPid ! Ack 
139  end.
			

Take a closer look at the lines 97–99 (Listing 11). We create a lambda function called Putter. This lambda binds the two arguments function append_message_to_file to one argument lambda gluing the LogName variable as the second parameter.

The function connection_logger waits for an incoming message (line 102, Listing 12) and then does different type of logging activities depending on the received message, decided by pattern matching on the received data.

You may spot on that in lines 105, 113, 119, 125 and 132 we also create a lambda named Message and pass it to the write_message function. I will explain in a minute why we create and pass around a function rather than a string, for example.

The function write_message at the line 140 (Listing 13) calls the function passed by value in the LogWriter variable (remember the Putter at lines 97-99) and feeds it a variable called Message which also holds a function (remember the Message lambdas above). It then calls connection_logger again to get the next packet of data.

140 write_message(ConnN, FromInfo, ToInfo,
       LogWriter, Message) ->
141   LogWriter(Message),
142   connection_logger(ConnN, FromInfo, ToInfo,
         LogWriter).
			

And this is a final bit – append_message_to_file function (line 143, Listing 14). This function again creates a lambda assigned to a Printer variable (line 145) and calls a function passed as a value in the Putter variable by passing the Printer as a parameter.

143 append_message_to_file(Putter, LogName) ->
144    {_, File} = file:open(LogName, [write,
          append]),
145    Printer = fun(Format, Args) ->
146      io:format(Format, Args),
147      io:format(File, Format, Args)
148    end,
149    Putter(Printer),
150    file:close(File).
151
152% --------------------------------------------
			

The whole idea of this multi-level cascade of lambdas is to split the putter activity formatting the data, and the printer activity writing the data into a log file and the console. The data formatting putter activity can be an expensive operation and should be done only once. That is why the Putter calls the Printer for every chunk of already formatted data, and the Printer in its turn outputs the data to the log file and the console (lines 146 and 147).

In lines 153 to 174 (Listing 15) we produce a nicely formatted hexadecimal dump. As we saw previously these conversion routines print out the formatted lines by calling a function passed in the Printer variable.

153 -define(WIDTH, 16).
154 binary_to_hex(Bin, Printer) ->
155   binary_to_hex(Bin, Printer, 0).
156 binary_to_hex(<<Bin:?WIDTH/binary,
       Rest/binary>>, Printer, Offset) ->
157   binary_to_dump_line(Bin, Printer, Offset),
158   binary_to_hex(Rest, Printer,
         Offset + ?WIDTH);
159 binary_to_hex(Bin, Printer, Offset) ->
160   Pad = fun() -> Printer("~*c",
         [(?WIDTH - byte_size(Bin)) * 3, 32]) end,
161   binary_to_dump_line(Bin, Printer,
         Pad, Offset).
162 binary_to_dump_line(Bin, Printer, Offset) ->
163   binary_to_dump_line(Bin, Printer, fun() ->
         ok end, Offset).
164 binary_to_dump_line(Bin, Printer,
       Pad, Offset) ->
165   Printer("~4.16.0B: ", [Offset]),
166   Printer("~s", [binary_to_hex_line(Bin)]),
167   Pad(),
168   Printer("| ~s~n",
         [binary_to_char_line(Bin)]).
169 binary_to_hex_line(Bin) ->
       [[(byte_to_hex(<<B>>) ++ " ") ||
       << B >> <= Bin]].
170 byte_to_hex(<< N1:4, N2:4 >>) ->
171   [integer_to_list(N1, 16),
         integer_to_list(N2, 16)].
172 binary_to_char_line(Bin) ->
       [[mask_invisiable_chars(B) ||
       << B >> <= Bin]].
173 mask_invisiable_chars(X) when
       (X >= 32 andalso X < 128) -> X;
174 mask_invisiable_chars(_) -> $..
			

This is the end of the code.

Now it is time to try the application in action. You need to download and install the latest version of Erlang from http://www.erlang.org for your system. For Windows they ship pre-built bundles. I use a Mac and had to build Erlang from the source. It is a very straightforward procedure and worked for me without any problems.

To run our code you can try this:

   escript tcp_proxy.erl 50000 pop.yandex.ru 110

escript is one of the Erlang tools and should be in your path. It combines the compilation and execution phases together.

Once it gets started you type in a different window: telnet localhost 50000 then when it gets connected type QUIT and press ENTER.

In the first window you should see something like Figure 1.

Start listening on port 50000 and forwarding data to pop.yandex.ru:110
Listener started 0.0.0.0-50000
Connected to 93.158.134.37-110 at 2011.12.15-00.59.22
Received (#0) 38 byte(s) from 93.158.134.37-110
0000: 2B 4F 4B 20 50 4F 50 20 59 61 21 20 76 31 2E 30 | +OK POP Ya! v1.0
0010: 2E 30 6E 61 40 31 34 20 4D 78 55 55 33 74 66 48 | .0na@14 MxUU3tfH
0020: 52 57 32 31 0D 0A                               | RW21..
Sent (#0) to 127.0.0.1-51042
Received (#1) 6 byte(s) from 127.0.0.1-51042
0000: 51 55 49 54 0D 0A                               | QUIT..
Sent (#1) to 93.158.134.37-110
Received (#2) 20 byte(s) from 93.158.134.37-110
0000: 2B 4F 4B 20 73 68 75 74 74 69 6E 67 20 64 6F 77 | +OK shutting dow
0010: 6E 2E 0D 0A                                     | n...
Sent (#2) to 127.0.0.1-51042
Disconnected from 93.158.134.37-110
Finished at 2011.12.15-00.59.29, duration 00-00.00.07
			

That’s it! It works.

Analyzing the code, if I implement such an application in C++ for instance, I always think twice before putting anything into a separate thread and so eventually end up with only a few threads – a listener, workers and one logger multiplexing logging from all the workers. Also I would think about a thread pool to limit the number of running native threads. Otherwise, accepting a thousand connections will spawn a thousand native threads which is not a clever approach even on a 32-core machine.

But in Erlang the multithreading is managed by the runtime. You can focus on the business logic of threading rather than on the OS resource management.

To conclude I’d like to underscore that I didn’t want to explain every single character in the code. I touched very briefly on the fundamentals of Erlang such as pattern matching, messaging and list comprehensions. For better understanding I would recommend two books: Programming Erlang: Software for a Concurrent World [Armstrong] and Erlang Programming [Cesarini]. They perfectly complement each other.

I hope I have inspired someone to take a closer look at this wonderful language. There is a lot of stuff in there: passing function by values over the wire, hot swapping of code on live servers without restarting them, developing generic servers which can be turned to do anything (remember again that quote at the beginning), extending Erlang with your native code written in other languages and much more.

Have fun.

Source code

I’ve put the source from the article at [Github]. The version there is more advanced. As well as the text logger, it logs data in a binary form as well. It is not that exciting to study but is useful in real applications.

References

[Armstrong] Programming Erlang: Software for a Concurrent World by Joe Armstrong

[Cesarini] Erlang Programming by Francesco Cesarini and Simon Thompson

[Github] https://github.com/begoon/tcp_proxy/tree/logger_threads

[Go] http://golang.org/doc/effective_go.html#goroutines