Using the Agrona library (Part 2)

"Agrona buffers are the ByteBuffer you wish the JDK gave you — with atomic operations, no position state, and direct Unsafe access."

Agrona's buffer hierarchy is the foundation for everything else in the library. Ring buffers, broadcast buffers, counters, and SBE codecs all build on top of these buffer abstractions. This article explains each layer and when to use it.

---

Buffer Hierarchy

Key insight: The hierarchy separates concerns:

• DirectBuffer — read-only access (for consumers)

• MutableDirectBuffer — read-write access (for producers)

• AtomicBuffer — thread-safe atomic operations (for concurrent access)

• UnsafeBuffer — the primary implementation that does it all

---

DirectBuffer — Read-Only Access

The base interface. All Agrona buffers implement this. It provides offset-based reading without any mutable position state (unlike JDK's ByteBuffer):

import org.agrona.DirectBuffer;

void processMessage(DirectBuffer buffer, int offset, int length) {
    int messageType = buffer.getInt(offset);
    long timestamp  = buffer.getLong(offset + 4);
    double price    = buffer.getDouble(offset + 12);
    String symbol   = buffer.getStringAscii(offset + 20);
    
    // No flip(), no rewind(), no position tracking needed!
}

Type-Safe Accessors

// Every primitive type has a direct accessor
byte   b = buffer.getByte(0);
short  s = buffer.getShort(1);
int    i = buffer.getInt(3);
long   l = buffer.getLong(7);
float  f = buffer.getFloat(15);
double d = buffer.getDouble(19);

// Strings with length prefix
String ascii = buffer.getStringAscii(27);
String utf8  = buffer.getStringUtf8(27);

// Raw bytes
byte[] dst = new byte[10];
buffer.getBytes(27, dst, 0, 10);

---

MutableDirectBuffer — Read-Write Access

Extends DirectBuffer with write operations:

import org.agrona.MutableDirectBuffer;
import org.agrona.concurrent.UnsafeBuffer;

MutableDirectBuffer buffer = new UnsafeBuffer(new byte[256]);

// Write fields at explicit offsets
buffer.putInt(0, 42);
buffer.putLong(4, System.nanoTime());
buffer.putDouble(12, 149.95);
buffer.putStringAscii(20, "AAPL");

// Bulk operations
buffer.setMemory(64, 32, (byte) 0);  // Zero out 32 bytes at offset 64

// Copy from another buffer (zero-copy if same backing)
MutableDirectBuffer dst = new UnsafeBuffer(new byte[128]);
dst.putBytes(0, buffer, 0, 64);  // Copy 64 bytes from buffer to dst

---

UnsafeBuffer — The Workhorse

UnsafeBuffer is the primary implementation. It can wrap anything:

import org.agrona.concurrent.UnsafeBuffer;
import java.nio.ByteBuffer;

// Wrap a byte array
UnsafeBuffer buf1 = new UnsafeBuffer(new byte[1024]);

// Wrap a JDK direct ByteBuffer
UnsafeBuffer buf2 = new UnsafeBuffer(ByteBuffer.allocateDirect(1024));

// Wrap a section of an existing buffer
UnsafeBuffer buf3 = new UnsafeBuffer(buf1, 64, 128); // offset=64, length=128

// Wrap a raw memory address (e.g., from mmap)
UnsafeBuffer buf4 = new UnsafeBuffer();
buf4.wrap(address, length); // address from Unsafe.allocateMemory or mmap

How Does It Work Internally?

UnsafeBuffer uses sun.misc.Unsafe (or modern VarHandle) to perform direct memory reads/writes. For a heap byte array, it computes:

// Conceptually:
address = arrayBaseOffset + index
value = UNSAFE.getInt(byteArray, address)

// For off-heap (direct) buffers:
address = bufferAddress + index
value = UNSAFE.getInt(null, address)

This bypasses all bounds checking in the JDK's ByteBuffer, making it faster but requiring you to manage safety yourself.

---

AtomicBuffer — Thread-Safe Memory Access

AtomicBuffer adds atomic operations for concurrent access. This is what makes ring buffers, counters, and IPC possible:

import org.agrona.concurrent.UnsafeBuffer;
import org.agrona.concurrent.AtomicBuffer;

AtomicBuffer buffer = new UnsafeBuffer(ByteBuffer.allocateDirect(1024));

// Volatile reads/writes — full memory barrier
buffer.putIntVolatile(0, 42);      // Store with release semantics
int val = buffer.getIntVolatile(0); // Load with acquire semantics

// Ordered writes — release semantics only (cheaper than volatile)
buffer.putIntOrdered(0, 42);       // Store-release (lazy set)
buffer.putLongOrdered(8, 100L);

// Compare-And-Swap — the foundation of lock-free algorithms
boolean success = buffer.compareAndSetInt(0, 42, 43);
// If [offset 0] == 42 → set to 43 and return true
// Otherwise → return false (someone else changed it)

// Atomic add — thread-safe increment
int previous = buffer.getAndAddInt(16, 1);   // Atomically add 1, return old
long prevL   = buffer.getAndAddLong(24, 10); // Atomically add 10

---

Memory Ordering: Plain vs Ordered vs Volatile

This is critical for understanding concurrent Agrona code:

In Practice: Ring Buffer Protocol

// Producer (writer):
buffer.putInt(offset, messageType);          // Plain write (field)
buffer.putLong(offset + 4, timestamp);       // Plain write (field)
buffer.putIntOrdered(tailOffset, newTail);   // ORDERED write — publishes the message!
// All writes above are guaranteed visible before the tail update

// Consumer (reader):
int tail = buffer.getIntVolatile(tailOffset); // VOLATILE read — sees latest tail
// If new tail > head, there's a message to read
int msgType   = buffer.getInt(offset);        // Plain read — safe after volatile read
long timestamp = buffer.getLong(offset + 4);   // Plain read — safe after volatile read

---

Buffer Wrapping — Zero-Copy Views

One of Agrona's most powerful features — you can create views into existing memory without copying:

UnsafeBuffer sourceBuffer = new UnsafeBuffer(ByteBuffer.allocateDirect(4096));

// Write a protocol: [HEADER(8 bytes)][BODY(variable)]
sourceBuffer.putInt(0, 1);          // message type
sourceBuffer.putInt(4, 64);         // body length
sourceBuffer.putStringAscii(8, "Hello, Agrona!"); // body starts at offset 8

// Create a view of just the body — NO COPY!
UnsafeBuffer bodyView = new UnsafeBuffer();
bodyView.wrap(sourceBuffer, 8, 64);  // Share the same memory!

String body = bodyView.getStringAscii(0); // "Hello, Agrona!"
// bodyView[0] is actually sourceBuffer[8] — zero copy!

---

Agrona Buffers vs JDK ByteBuffer

Feature	JDK ByteBuffer	Agrona UnsafeBuffer
Position state	✅ Yes (position, limit, mark)	❌ Stateless (explicit offsets)
flip() required	✅ Yes — error-prone	❌ No — nothing to flip
Atomic operations	❌ None	✅ CAS, volatile, ordered
Wrap byte[]	✅ ByteBuffer.wrap()	✅ new UnsafeBuffer(byte[])
Wrap raw address	❌ Not directly	✅ wrap(address, length)
Sub-views	✅ slice() (creates new BB)	✅ wrap(buf, offset, len)
Bounds checking	✅ Always (safe)	⚠️ Optional (fast)
String read/write	❌ Manual encoding	✅ getStringAscii/Utf8()
Allocation	allocateDirect() allocates	Wraps existing memory
Thread safety	❌ Not thread-safe	✅ Via AtomicBuffer methods

---

Common Patterns

Pattern 1: Flyweight Message Decoding (Zero-Alloc)

// A reusable "flyweight" decoder — wraps different buffers without allocation
class TradeDecoder {
    private DirectBuffer buffer;
    private int offset;
    
    // Wrap a new message — NO allocation
    void wrap(DirectBuffer buffer, int offset) {
        this.buffer = buffer;
        this.offset = offset;
    }
    
    int instrumentId()  { return buffer.getInt(offset); }
    long timestamp()    { return buffer.getLong(offset + 4); }
    double price()      { return buffer.getDouble(offset + 12); }
    int quantity()      { return buffer.getInt(offset + 20); }
}

// Usage — decode millions of messages with zero allocation
TradeDecoder decoder = new TradeDecoder(); // Allocate ONCE
ringBuffer.read((msgTypeId, buffer, index, length) -> {
    decoder.wrap(buffer, index);  // Just sets two fields — no allocation!
    processOrder(decoder.instrumentId(), decoder.price(), decoder.quantity());
});

Pattern 2: Building a Binary Protocol

// Define the message layout
static final int MSG_TYPE_OFFSET    = 0;
static final int TIMESTAMP_OFFSET   = 4;
static final int INSTRUMENT_OFFSET  = 12;
static final int PRICE_OFFSET       = 16;
static final int QUANTITY_OFFSET    = 24;
static final int MESSAGE_LENGTH     = 28;

// Encode
void encode(MutableDirectBuffer buffer, int offset, long ts, int instrument,
            double price, int qty) {
    buffer.putInt(offset + MSG_TYPE_OFFSET, 1);
    buffer.putLong(offset + TIMESTAMP_OFFSET, ts);
    buffer.putInt(offset + INSTRUMENT_OFFSET, instrument);
    buffer.putDouble(offset + PRICE_OFFSET, price);
    buffer.putInt(offset + QUANTITY_OFFSET, qty);
}

// Decode
void decode(DirectBuffer buffer, int offset) {
    int msgType    = buffer.getInt(offset + MSG_TYPE_OFFSET);
    long timestamp = buffer.getLong(offset + TIMESTAMP_OFFSET);
    int instrument = buffer.getInt(offset + INSTRUMENT_OFFSET);
    double price   = buffer.getDouble(offset + PRICE_OFFSET);
    int quantity   = buffer.getInt(offset + QUANTITY_OFFSET);
}

Pattern 3: Expandable Buffers

// When you don't know the size upfront
import org.agrona.ExpandableArrayBuffer;

ExpandableArrayBuffer buffer = new ExpandableArrayBuffer(64);
buffer.putStringAscii(0, "Short message");   // Fits in initial 64 bytes
buffer.putStringAscii(100, "This will trigger expansion!"); // Auto-grows!
// Buffer capacity automatically expanded — no manual resize needed

---

Summary

The one-liner: Agrona's buffer hierarchy provides stateless, offset-based memory access with atomic CAS/volatile/ordered operations — making it the foundation for zero-allocation message encoding, lock-free IPC, and off-heap data structures.