Hyper-scale OCR in GO.

I was recently tasked with building a service that reads SIM card details from photos. The obvious move is PaddleOCR in Python. Image in, text out, ship the demo.

On my Intel Mac laptop, a clean, well-lit photo took 9 seconds. Bad images (blurry, skewed, low contrast) took 10–30 seconds. The SLA was 2 seconds at p99, including domain-specific post-processing of the extracted text. The demo worked. Production did not.

The fix was not a faster Python library. I deleted Python and rebuilt the pipeline in Go from first principles, keeping only what the service needs at runtime.

---

The core insight

You do not need Python to run classical ML inference in production.

If you can export the model to ONNX, study its input/output shapes, and wire tensors in and out of a pipeline, your serving language is a choice. Train in Python. Run in Go.

That leaves two runtime dependencies:

• ONNX Runtime for inference (github.com/yalue/onnxruntime_go)

• GoCV for image handling (Go bindings to OpenCV)

We use PP-OCRv3 for detection and PP-OCRv5 for recognition in production (hybrid set: small v3 det, accurate v5 rec). Same weights as the Python PaddleOCR stack, exported as .onnx files. The Go server is just an orchestrator around them.

What OCR is - simply.

OCR is detection plus recognition.

1. Detection finds where text is. The model outputs a probability map (heatmap). Post-processing turns that map into boxes.

2. Recognition reads each cropped box. The model outputs a sequence of character probabilities. CTC collapses that sequence into a string.

Two models, two ONNX sessions, one pipeline. The rest of this post walks through that pipeline in Go.

Step 1: Download the PP-OCR ONNX models

You do not need PaddlePaddle installed to run inference. Pre-converted ONNX models are published on Hugging Face. We fetch PP-OCRv5 det + rec + dictionary, plus a lightweight PP-OCRv3 detector for the hybrid setup:

Download the models (and cache them on disk)

Here's a sample download script:

HF="https://huggingface.co/monkt/paddleocr-onnx/resolve/main"

mkdir -p models/ppocrv5 models/ppocrv4

curl -fsSL -L -o models/ppocrv5/det.onnx  "$HF/detection/v5/det.onnx"
curl -fsSL -L -o models/ppocrv5/rec.onnx  "$HF/languages/english/rec.onnx"
curl -fsSL -L -o models/ppocrv5/dict.txt   "$HF/languages/english/dict.txt"

# Optional: lightweight v3 detector (~2 MB vs ~88 MB v5 server det)
curl -fsSL -L -o models/ppocrv4/det.onnx  "$HF/detection/v3/det.onnx"

You also need the ONNX Runtime shared library for your OS. The download script i made pulls libonnxruntime.so (Linux) or libonnxruntime.dylib (macOS) from the official GitHub releases.

Expected layout:

models/
  ppocrv5/det.onnx rec.onnx dict.txt
  ppocrv4/det.onnx          # hybrid: small detector
  libonnxruntime.so

Step 2: Load the models in Go

ONNX models expose named inputs and outputs. Paddle exports vary (x, fetch_name_0, etc.), so we try a small list of known names until one opens:

import ort "github.com/yalue/onnxruntime_go"

func openSession(modelPath string, opts *ort.SessionOptions) (*ort.DynamicAdvancedSession, error) {
    inputs  := []string{"x", "input", "images"}
    outputs := []string{"fetch_name_0", "sigmoid_0.tmp_0", "output"}

    for _, in := range inputs {
        for _, out := range outputs {
            sess, err := ort.NewDynamicAdvancedSession(modelPath, []string{in}, []string{out}, opts)
            if err == nil {
                return sess, nil
            }
        }
    }
    return nil, fmt.Errorf("could not open %s", modelPath)
}

Initialize the runtime once, set thread counts, open det + rec:

ort.SetSharedLibraryPath("models/libonnxruntime.so")
ort.InitializeEnvironment()

opts, _ := ort.NewSessionOptions()

// I find that 2 threads each is the sweet spot for my usecase.
opts.SetIntraOpNumThreads(2)
opts.SetInterOpNumThreads(2)

detSess, _ := openSession("models/ppocrv5/det.onnx", opts)
recSess, _ := openSession("models/ppocrv5/rec.onnx", opts)

Tip: run one probe inference at startup and read the output shape. Detection map scale differs between v3 (1/4 resolution) and v5 (full resolution). We derive that from the actual output tensor instead of guessing from the filename.

Step 3: Prepare input tensors

Neural nets want NCHW float tensors: [batch, channels, height, width]. Images arrive as JPEG bytes in BGR byte order.

Detection preprocess (resize, pad to a multiple of 32, normalize with ImageNet mean/std):

// Resize so the long side <= 960, pad to 32px grid, copy into a black canvas.
gocv.Resize(img, &resized, image.Pt(rw, rh), 0, 0, gocv.InterpolationLinear)
padded := gocv.NewMatWithSize(paddedH, paddedW, gocv.MatTypeCV8UC3)
padded.SetTo(gocv.NewScalar(0, 0, 0, 0))
resized.CopyTo(&padded.Region(image.Rect(0, 0, rw, rh)))

inputData := matToDetInput(padded) // BGR Mat -> []float32 NCHW
inputShape := ort.NewShape(1, 3, int64(paddedH), int64(paddedW))
inputTensor, _ := ort.NewTensor(inputShape, inputData)
defer inputTensor.Destroy()

The tensor packing step is where we spent most of our optimization effort. Read the Mat buffer once, then iterate in pure Go:

func matToDetInput(padded gocv.Mat) []float32 {
    h, w := padded.Rows(), padded.Cols()
    mean := [3]float32{0.485, 0.456, 0.406}
    std  := [3]float32{0.229, 0.224, 0.225}

    data, _ := padded.DataPtrUint8() // one CGO call for the whole image
    out := make([]float32, 3*h*w)

    for c := 0; c < 3; c++ {
        ch := rgbChannelOrder[c] // BGR -> RGB
        plane := c * h * w
        for p := 0; p < h*w; p++ {
            pixel := float32(data[p*3+ch]) / 255.0
            out[plane+p] = (pixel - mean[c]) / std[c]
        }
    }
    return out
}

Recognition uses the same pattern but normalizes to [-1, 1] and fixes height at 48px with variable width.

Do not do this in a hot loop:

// Slow: ~2.76M CGO crossings on a 960x960 image
pixel := float32(padded.GetVecbAt(y, x)[ch]) / 255.0

Each GetVecbAt crosses CGO and allocates a small slice. Profile first. Fix the glue before tuning ONNX thread counts.

Step 4: Run inference and read the output

Detection output: probability map -> boxes

Pass nil as the output slot and let ONNX Runtime allocate. Then read shape + data:

outputs := []ort.Value{nil}
if err := detSess.Run([]ort.Value{inputTensor}, outputs); err != nil {
    return err
}
defer outputs[0].Destroy()

ft := outputs[0].(*ort.Tensor[float32])
oshape := ft.GetShape()           // e.g. [1, 1, 240, 240] for 1/4-scale v3
outH := int(oshape[len(oshape)-2])
outW := int(oshape[len(oshape)-1])
probMap := ft.GetData()           // flat []float32, row-major

// Map pixel -> original image coordinates
sx := float64(paddedW) / float64(outW) / scale
sy := float64(paddedH) / float64(outH) / scale
boxes := dbBoxesFromBitmap(probMap, outW, outH, /* thresholds */, sx, sy, origW, origH)

Recognition output: logits -> text via CTC

For each box, warp the crop upright, pack a rec tensor, run the rec session:

outputs := []ort.Value{nil}
recSess.Run([]ort.Value{recTensor}, outputs)
ft := outputs[0].(*ort.Tensor[float32])

seqLen := int(ft.GetShape()[1])      // time steps
numClasses := int(ft.GetShape()[2])  // dict size + CTC blank
text, conf, _ := ctcGreedyDecode(ft.GetData(), seqLen, numClasses, dict)

Greedy CTC decode in Go is short and easy to test:

func ctcGreedyDecode(data []float32, seqLen, numClasses int, dict []string) (string, float32, error) {
    var b strings.Builder
    prev := -1
    for t := 0; t < seqLen; t++ {
        offset := t * numClasses
        bestIdx, bestProb := 0, data[offset]
        for c := 1; c < numClasses; c++ {
            if data[offset+c] > bestProb {
                bestProb, bestIdx = data[offset+c], c
            }
        }
        if bestIdx != 0 && bestIdx != prev { // 0 = CTC blank
            b.WriteString(dict[bestIdx-1])
        }
        prev = bestIdx
    }
    return b.String(), 0, nil
}

This is the "working with the output" half people skip when they only call paddleocr.ocr(). ONNX gives you raw tensors. You own the post-processing.

Step 5: DB post-processing in Go (abridged)

PaddleOCR's Differentiable Binarization post-process is not in the ONNX file. You reimplement it: threshold the map, find contours, score each candidate box, unclip, map back to image coordinates.

func dbBoxesFromBitmap(prob []float32, w, h int, thresh, boxThresh, unclip float32,
    minSize int, sx, sy float64, ow, oh int) []quad {

    // 1. Threshold probability map -> binary bitmap
    bitmap := gocv.NewMatWithSize(h, w, gocv.MatTypeCV8U)
    defer bitmap.Close()
    for y := 0; y < h; y++ {
        row := y * w
        for x := 0; x < w; x++ {
            if prob[row+x] > thresh {
                bitmap.SetUCharAt(y, x, 255)
            }
        }
    }

    // 2. Find contours, fit min-area rotated rects
    contours := gocv.FindContours(bitmap, gocv.RetrievalExternal, gocv.ChainApproxSimple)
    defer contours.Close()

    var boxes []quad
    for i := 0; i < contours.Size(); i++ {
        cnt := contours.At(i)
        rr := gocv.MinAreaRect2f(cnt)
        if min(rr.Width, rr.Height) < float32(minSize) {
            continue
        }

        // 3. Mean probability inside the oriented box (not axis-aligned bbox)
        score := boxScoreFast(prob, w, h, rr)
        if score < boxThresh {
            continue
        }

        // 4. Unclip, order corners, scale to original image size
        expanded := unclipRect(rr, unclip)
        // ... append quad with pts mapped by sx, sy ...
    }
    return boxes
}

boxScoreFast matters for accuracy: for rotated text lines, scoring the axis-aligned bounding box dilutes the signal with background pixels. We mask to the oriented rectangle instead.

Tests worth writing

Tensor packing: prove the fast path matches the naive path bit-for-bit:

func TestMatToDetInputBitIdenticalToSlow(t *testing.T) {
    m := gocv.NewMatWithSize(123, 77, gocv.MatTypeCV8UC3)
    defer m.Close()
    fillPattern(t, m) // deterministic bytes via DataPtrUint8
    assertIdentical(t, matToDetInput(m), matToDetInputSlow(m))
}

DB post-processing: seed a fake probability map with rectangular blobs, assert you get boxes back:

func TestDBBoxesFindsSeededRegions(t *testing.T) {
    const w, h = 320, 320
    prob := make([]float32, w*h)
    for y := 40; y < 60; y++ {
        for x := 50; x < 200; x++ {
            prob[y*w+x] = 0.95
        }
    }
    boxes := dbBoxesFromBitmap(prob, w, h, 0.3, 0.6, 1.5, 3, 1, 1, w, h)
    if len(boxes) == 0 {
        t.Fatal("expected at least one box")
    }
}

We also benchmark each stage in isolation so the next optimization target is obvious:

go test -run=^$ -bench=MatToDetInput -benchmem ./internal/ocr/
go test -run=^$ -bench=DBBoxesFromBitmap ./internal/ocr/
BENCH_MODELS_DIR=./models go test -run=^$ -bench=DetectAndRecognize ./internal/ocr/

---

Step 6: Performance work (what profiling actually found)

Both GoCV and ONNX Runtime sit behind CGO. Occasional calls are fine. Millions of per-pixel calls are not.

Metric	Before (GetVecbAt)	After (DataPtrUint8)
CGO calls per det tensor	~2.76M	1
Heap allocs (tensor pack)	~2.76M	1
matToDetInput benchmark	baseline	~170× faster
End-to-end happy-path CPU	~520ms	250ms (2×)

The micro-benchmark number is large. The end-to-end number is the one that matters. Tensor packing stopped dominating detection. ONNX Run became the next place worth tuning.

Workflow:

1. go test -bench=... -cpuprofile=cpu.prof ./internal/ocr/

2. go tool pprof -top cpu.prof

3. Fix the hottest non-model stage

4. Keep the slow path + bit-identical tests so you do not regress correctness

Step 7: Concurrency with engine leases

ONNX Run is thread-safe on a session, but you still need a concurrency budget. Each concurrent inference pins CPU and memory. If you allow unbounded goroutines, latency spikes under load.

We use an engine lease: a fixed pool of N engine slots (each slot = one det + rec session pair). Every inference borrows a slot for the duration of one DetectAndRecognize call:

type engineLease interface {
    acquireEngine(ctx context.Context) (*Engine, error)
    releaseEngine(*Engine)
}

func detectAndRecognize(ctx context.Context, lease engineLease, img gocv.Mat) ([]string, error) {
    eng, err := lease.acquireEngine(ctx)
    if err != nil {
        return nil, err
    }
    defer lease.releaseEngine(eng)
    return eng.DetectAndRecognize(img)
}

The pool is a buffered channel of engines:

type Pool struct {
    slots chan *Engine
    size  int
}

func (p *Pool) acquireEngine(ctx context.Context) (*Engine, error) {
    select {
    case eng := <-p.slots:
        return eng, nil
    case <-ctx.Done():
        return nil, ctx.Err()
    }
}

func (p *Pool) releaseEngine(eng *Engine) {
    p.slots <- eng
}

Results

Scenario	Python (PaddleOCR, demo)	Go + ONNX (production)
Good image (clean, well-lit)	~9s	~200ms
Hard image (blur, skew, low contrast)	10–30s	~900ms (fallback strategies)
Tensor packing	n/a	~170× faster (micro-benchmark)

Same model weights. Same accuracy target. Different runtime and different care about what runs in the hot path.

---

Takeaways

1. Train in Python, serve in Go. Export to ONNX. You do not need the Python runtime in production.

2. Own the post-processing. DB box extraction and CTC decode are plain Go. That is where you get control and testability.

3. Profile the glue. Our bottleneck was not the neural net. It was 2.76 million CGO calls reading pixels that were already in memory.

4. Bound concurrency with leases. A channel of engine slots is simpler than layered semaphores, and it keeps fallback strategies from multiplying your thread budget.

The models were trained in Python, exported as ONNX graphs, and the Go server handles the rest: decode, tensor packing, concurrent session management, and a multi-strategy pipeline that spends extra compute on hard images without blowing the latency budget on easy ones.

That is the whole trick. The rest is operations.