Title: Golang programming on AppEngine

Author: Martin Moene

Date: 10 July 2015 21:43:46 +01:00 or Fri, 10 July 2015 21:43:46 +01:00

Summary: Silas S. Brown tries his hand at writing native code for the Cloud.

Body: 

Go [1] is Google’s new C-like programming language which compiles to machine code but with various built-in safety mechanisms. Their ‘AppEngine’ public cloud servers currently support Python, Java, PHP and Go, so Go is the only option if you want the speed of machine code (albeit with some added safety checks) and don’t want to set up a full virtual machine. (Google will let you set up a full virtual machine but at an extra cost, and RedHat’s OpenShift offering allows you to run arbitrary binaries, but of course the rules could change at any time.)

If using AppEngine, you can write code with different modules in different programming languages, which is useful if you already have most of it in Python or Java and just want to optimise a small part of it into Go. But different modules have different .yaml files and handle different URL patterns, so the communication between modules will likely involve URL fetching, which might contribute to your quota use depending on how exactly they calculate it (which could change).

Go’s syntax is very much like C, the most immediately obvious differences being:

1. Type names are put after variables and functions, not before as in C. So the C code: int myFunc(int a, int b) would be written in Go as: func myFunc(a int, b int) int and the C++11 statement: auto c = my_expression(); would be written in Go as simply: c := my_expression()
2. Semicolons are automatically added at the end of lines. This means you have to be careful where you do and don’t put newlines. For example, if writing } else { it must be all on one line, which is contrary to the C styles of some, although I for one tend to write that all on one line anyway.
3. Braces are compulsory, even when there is only one statement in the block. You can’t write if (a) b(); – you have to put braces around b(). On the other hand, the parentheses around the a are optional.
4. There is no while keyword, but writing for with one argument effectively turns it into a while.
5. Unsurprisingly for a ‘safe’ language, there is no pointer arithmetic. Most things you did in C with pointer arithmetic have to be done in Go with arrays, but thankfully there is much added support for array operations. ‘Slices’ in Go can expand as needed, but I/O functions tend to take their current size as a cue for how much data you want to read, so one caveat is not to accidentally pass an empty array to a read function and expect it to read anything! If you need to read a string as an array of bytes, you probably have to copy it like this (unless there’s a better way I don’t know about):

        myBytes := make([]byte, len(myString))
        copy (myBytes, myString)

6. ++ and -- are statements, not expressions, so (unlike in C) you can’t write them in the middle of a longer expression. You can still write myVar++, but this now has to be a statement on its own.

Go’s ‘packaging’ system works as follows: Every .go source file provides a package (the main package is main), and can ‘import’ other packages from other .go files. One package can be split across more than one .go file. The compiler automatically searches the current directory for all .go files (you don’t have to tell it which ones to look at). You’re not allowed to mix files from different packages in the same directory, so non-standard packages you wish to import must be placed in subdirectories. There are also facilities to automatically download packages from various public source-control systems, but versioning can be an issue so you might prefer to make local copies. At any rate the resulting binary will always be statically linked (Go doesn’t ‘do’ shared libraries), which, while making for larger binaries, might at least mean you don’t have to worry so much about installing different versions of shared libraries on other people’s systems.

Go’s standard library is strong on Internet data processing and includes support for concurrency and synchronization. Mark Summerfield wrote a more in-depth article about concurrent programming in Go in Overload 3 years ago [2], but if you’re just writing a simple server for AppEngine then perhaps all you really need to know is that multiple threads of control might be running through your code so it’s best to avoid mutable shared resources (or use locking as a last resort).

If you are writing for AppEngine then I’d suggest you download AppEngine’s version of Go, which has the AppEngine-related packages already included, but you could also install the ‘golang’ packages in your GNU/Linux package manager, and there are versions for the Raspberry Pi. The golang-doc package provides a web server for offline browsing of the documentation; this is also included in AppEngine’s Go download.

Compilation speed is quite fast, but you do need enough RAM: while experimenting with a Go back-end for my Annotator Generator [3], I found the compiler can take between 400 and 500 times the size of its input program in RAM. Because Go is a garbage-collected language, it’s possible to manage on less RAM at the expense of more CPU by running the GC more frequently, but the problem is most operating systems don’t have a standard way of asking ‘are we running out of RAM yet’: if you allocate too much, then typically you will either take the huge speed penalty of being extensively swapped out to virtual memory without being told (unless VM has been disabled), or else the OS would sooner kill your process entirely than tell you about the memory shortage. The standard ‘fix’ for this (which is also used by the GCC compiler) is to query the system’s specification for total physical RAM size, and set your internal limits accordingly. This can go wrong if too much of that physical RAM is in active use by other programs or is otherwise unavailable, and in this case it can be necessary to manually trick the compiler or other program into reading an artificially reduced value, otherwise it will sprawl itself out into swap space without knowing.

References and notes

[1] www.golang.org. Some prefer to call it Golang (Go Language), as I have done in the title of this article because ‘go programming’ seems more open to misunderstanding.

[2] Mark Summerfield: Concurrent Programming with Go. Overload #106 (December 2011)

[3] ‘Web Annotation with Modified-Yarowsky and Other Algorithms’, Overload #112 (December 2012); code is at http://people.ds.cam.ac.uk/ssb22/adjuster/annogen.html

Notes: 

More fields may be available via dynamicdata ..