Runes, Bytes, and Graphemes in Go

Text processing in Go operates across three distinct layers of abstraction: bytes, runes (Unicode code points), and extended grapheme clusters (user-perceived characters).

Treating code points or bytes as visible characters corrupts non-Latin scripts (like Tamil, Hindi, or Arabic) and multi-code-point emoji sequences.

graph TD
    G["Grapheme Cluster (User-Perceived Character)"]
    R1["Rune 1 (Base Code Point)"]
    R2["Rune 2 (Zero-Width Joiner / Modifier)"]
    R3["Rune 3 (Combining Mark)"]
    B1["UTF-8 Bytes (1-4 B)"]
    B2["UTF-8 Bytes (1-4 B)"]
    B3["UTF-8 Bytes (1-4 B)"]

    G --> R1
    G --> R2
    G --> R3
    R1 --> B1
    R2 --> B2
    R3 --> B3

    class G diag-ingress;
    class R1,R2,R3 diag-compute;
    class B1,B2,B3 diag-storage;

1. Bytes: UTF-8 Storage Layer

In Go, a string is an immutable, read-only slice of arbitrary bytes. The built-in len(s) function returns the number of bytes, not characters:

s := "வணக்கம்"
fmt.Println(len(s)) // 21 bytes (3 bytes per Tamil code point)

Direct indexing s[0] extracts a raw uint8 byte. If you slice a multi-byte UTF-8 sequence halfway through (s[:2]), you produce invalid, corrupted UTF-8.

2. Runes: Unicode Code Points

A rune in Go is an alias for int32, representing a single Unicode Code Point:

rs := []rune(s)
fmt.Println(len(rs)) // 7 runes

While []rune(s) converts a string to code points, it allocates a new slice on the heap. For zero-allocation iteration, use a range loop or the standard library unicode/utf8 package:

import "unicode/utf8"

// Zero heap allocation
count := utf8.RuneCountInString(s) // 7

3. Grapheme Clusters: User-Perceived Characters

A single visible character often consists of multiple Unicode code points joined together.

Example A: Tamil Syllables with Virama

In the word "வணக்கம்", the syllable "க்" combines two independent code points:

• Base consonant: க (U+0B95)
• Pulli / Virama combining mark: ் (U+0BCD)

To a human reader, "க்" is 1 character, but to Go’s []rune, it is 2 runes.

Example B: Zero-Width Joiner (ZWJ) Emoji

Consider the family emoji sequence: 👨‍👩‍👧‍👦

emoji := "👨‍👩‍👧‍👦"
fmt.Println(len(emoji))                    // 25 bytes
fmt.Println(utf8.RuneCountInString(emoji)) // 7 runes (Man + ZWJ + Woman + ZWJ + Girl + ZWJ + Boy)

If you slice this emoji by rune or reverse it with a naive rune loop, the family breaks apart into four separate individuals separated by invisible joiners.

graph LR
    T["வணக்கம் (Tamil)"] -->|"len"| B1["21 Bytes"]
    T -->|"utf8.RuneCount"| R1["7 Runes"]
    T -->|"uniseg"| G1["5 Graphemes (வ, ண, க், க, ம்)"]

    E["👨‍👩‍👧‍👦 (Family Emoji)"] -->|"len"| B2["25 Bytes"]
    E -->|"utf8.RuneCount"| R2["7 Runes"]
    E -->|"uniseg"| G2["1 Grapheme"]

    class T,E diag-ingress;
    class B1,B2 diag-storage;
    class R1,R2 diag-compute;
    class G1,G2 diag-success;

Proper Grapheme Iteration with uniseg

Go’s standard library stops at code points (rune). For Unicode Standard Annex #29 compliant grapheme clustering, use github.com/rivo/uniseg:

package main

import (
    "fmt"
    "slices"
    "github.com/rivo/uniseg"
)

// ReverseGraphemes reverses a string without breaking combining marks or ZWJ sequences
func ReverseGraphemes(s string) string {
    var clusters []string
    gr := uniseg.NewGraphemes(s)
    for gr.Next() {
        clusters = append(clusters, gr.Str())
    }
    slices.Reverse(clusters)

    var result string
    for _, c := range clusters {
        result += c
    }
    return result
}

func main() {
    fmt.Println(ReverseGraphemes("வணக்கம்")) // ம் க க் ண வ (Valid Tamil graphemes preserved)
}

Text Processing Reference