Go + Python = ❤️

Calling go from python

You can do all sorts of tricks in python to speed up your python and if it’s still not fast enough you can use Cython or C. But if you like me who don’t know/want to learn C then we can write it in Go too. Although I have to emphasize by simply writing in Go doesn’t necessarily speed up python. It has its own data transfer overhead and as well as time spent in debugging, writing, and managing two separate codebases.

Okay after the disclaimer, now it’s time to code in go and call it in python. Although you may not want to do anything with C but like it or not python and go cannot talk with each other they need C to do that. So we have to use Go package Cgo that compiles go to C library which then can be called from python. Let’s get started. Let’s start with a simple example hello world example. Let’s start with go code create file lib.go.

package main

import (
    "C"
    "log"
)

//export helloWorld
func helloWorld(valPtr *C.char) {
    val := C.GoString(valPtr)
    log.Println("Hello go from", val)
}

func main(){
}

So we need to use go C package cgo. The comment //export helloWorld tells go that we cant to to export to C. And we have empty main function because our go code is to export go function to C not to return anything in go. Now all we need to do is compile code to C by invoking this in the command line go build -buildmode=c-shared -o lib.so lib.go then you will see two file lib.h and lib.so.

Now we can create python to access the go function we created and exported to C. Let’s create app.py in the same directory where go file C binary file generated by cgo.

In python first thing we need to do is import [ctypes](https://docs.python.org/3/library/ctypes.html). This will help us to load the binary file we generated from cgo and also provides compatible data types between C and python.

import ctypes

lib = ctypes.cdll.LoadLibrary('./lib.so')

# calling go function
hello =  lib.helloWorld
hello.argtypes = [ctypes.c_char_p]
hello("python".encode("utf-8"))

Output
❯ python3 app.py
Hello go from python

So we first load binary file using ctypes.cdll.LoadLibrary. The we assign exported function to python variable hello hello = lib.helloWorld. Then we need to define the C argument type that exported function accepts at line hello.argtypes = [ctypes.c_char_p]. Finally we call the function hello with proper string encoding hello("python".encode("utf-8")).

Let’s build something substantial and export it to python. This time I am going to write a function in go that takes a text file from URL and prints out the top 5 most used words in the text file.

Here’s how the go function looks like

//export wordCount
func wordCount(filePath *C.char) {
	resp, err := http.Get(C.GoString(filePath))
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	scanner := bufio.NewScanner(resp.Body)
	scanner.Split(bufio.ScanWords)
	counts := make(map[string]int)
	for scanner.Scan() {
		word := strings.ToLower(scanner.Text())
		counts[word]++
	}
	if err := scanner.Err(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	var ordered []Count
	for word, count := range counts {
		ordered = append(ordered, Count{word, count})
	}
	sort.Slice(ordered, func(i, j int) bool {
		return ordered[i].Count > ordered[j].Count
	})
	fmt.Println(ordered[:5])
}

type Count struct {
	Word  string
	Count int
}

So the function wordCount accepts string specifically C char. In our case, this string/char is a URL where the text file exists and then we read the text file and count the word and save the word and its frequency in a dictionary counts. Then we create the list of struct ordered, which has field Word and Count representing the word and its frequency from the dictionary/map counts. Finally, we sort our list according to its frequency. I have adapted this go function from this interesting blog post. I am not returning anything but simply printing the std output. If I was returning any value then I need to make sure that I return proper C type and we need to be careful as there is a limitation on C type supported by go.

Now we just repeat the drill we regenerate the binary file go build -buildmode=c-shared -o lib.so lib.go

Getting back to our python we need to import the new function and define the argument types in C that it accepts. Here is the python code.

# calling countwords
count_words = lib.wordCount
count_words.argtypes = [ctypes.c_char_p]
url ="https://ocw.mit.edu/ans7870/6/6.006/s08/lecturenotes/files/t8.shakespeare.txt"
count_words(url.encode("utf-8"))

❯ python3 app.py
[{the 27549} {and 26037} {i 19540} {to 18700} {of 18010}]

So there you have it we called function written in go from python.