JVM Low-Level I/O - Part 1
ByteBuffer Deep Dive
Welcome to the first article in our JVM Low-Level I/O series. We're going to tear apart java.nio.ByteBuffer — the single most important class for high-performance I/O on the JVM. By the end of this article, you'll understand not just how to use ByteBuffer, but why it exists and what's happening underneath.
1. Why ByteBuffer Exists
Before Java NIO (introduced in Java 1.4), all I/O was stream-based. You had InputStream and OutputStream, which processed data one byte at a time. This was simple but painfully slow for high-throughput scenarios.
The problem? Every byte had to cross the JVM boundary. The JVM manages its own heap memory, while the operating system manages native memory. Traditional streams required copying data from native OS buffers → JVM heap → your application. That's at least two copies of memory for every read operation.
ByteBuffer solves this by giving you a block of memory that you can read from and write to in bulk — and in some cases, it can live outside the JVM heap, eliminating one of those copies entirely.
2. The Memory Model Behind ByteBuffer
Let's visualise where ByteBuffers actually live in memory:
Key insight: A HeapByteBuffer is backed by a regular Java byte[] array on the garbage-collected heap. A DirectByteBuffer lives in native memory — the GC never touches the buffer's data, only its Java wrapper object.
3. Creating ByteBuffers
There are three primary ways to create a ByteBuffer:
3.1 Heap Buffer (allocate)
// Allocate 1024 bytes on the JVM heap
ByteBuffer heapBuffer = ByteBuffer.allocate(1024);
System.out.println(heapBuffer.isDirect()); // false
System.out.println(heapBuffer.hasArray()); // true — backed by byte[]
System.out.println(heapBuffer.capacity()); // 1024
When to use: General-purpose work, short-lived buffers, when you need fast allocation, and GC can handle cleanup.
3.2 Direct Buffer (allocateDirect)
// Allocate 1024 bytes in native (off-heap) memory
ByteBuffer directBuffer = ByteBuffer.allocateDirect(1024);
System.out.println(directBuffer.isDirect()); // true
System.out.println(directBuffer.hasArray()); // false — no backing byte[]
System.out.println(directBuffer.capacity()); // 1024
When to use: I/O-heavy operations, large buffers, long-lived buffers that interact with the OS (FileChannel, SocketChannel).
3.3 Wrapping an Existing Array (wrap)
byte[] data = {0x48, 0x65, 0x6C, 0x6C, 0x6F}; // "Hello" in ASCII
ByteBuffer wrapped = ByteBuffer.wrap(data);
System.out.println(wrapped.isDirect()); // false
System.out.println(wrapped.hasArray()); // true
System.out.println(wrapped.capacity()); // 5
// IMPORTANT: Changes to the array affect the buffer and vice versa!
data[0] = 0x4A; // changes buffer too
System.out.println((char) wrapped.get(0)); // 'J' (not 'H')
When to use: When you already have data in a byte[] and want to use ByteBuffer's API without copying.
Allocation Cost Comparison
Rule of thumb: Allocate direct buffers once and reuse them. Allocate heap buffers when you need them temporarily.
4. The Four Core Properties
Every ByteBuffer has four properties that control which part of the buffer you can read from or write to:
| Property | Description | Invariant |
|---|---|---|
| capacity | Total size of the buffer (fixed at creation) | Always constant |
| limit | First index you cannot read/write | limit ≤ capacity |
| position | Next index to read/write | position ≤ limit |
| mark | A saved position (for reset) | mark ≤ position |
The invariant is always: 0 ≤ mark ≤ position ≤ limit ≤ capacity
ByteBuffer buf = ByteBuffer.allocate(10);
// After allocation:
// position=0, limit=10, capacity=10
System.out.println("position=" + buf.position()); // 0
System.out.println("limit=" + buf.limit()); // 10
System.out.println("capacity=" + buf.capacity()); // 10
// remaining() = limit - position
System.out.println("remaining=" + buf.remaining()); // 10
5. Reading and Writing Data
ByteBuffer provides two styles of access: relative (which uses and advances the position) and absolute (which uses an explicit index).
5.1 Relative Put/Get
ByteBuffer buf = ByteBuffer.allocate(16);
// Write various types — position advances automatically
buf.put((byte) 0x41); // 1 byte → position = 1
buf.putInt(42); // 4 bytes → position = 5
buf.putLong(123456789L); // 8 bytes → position = 13
buf.putShort((short) 7); // 2 bytes → position = 15
System.out.println("position after writes: " + buf.position()); // 15
// To read, we must flip first (see next section)
buf.flip();
byte b = buf.get(); // reads 0x41, position → 1
int i = buf.getInt(); // reads 42, position → 5
long l = buf.getLong(); // reads 123456789, position → 13
short s = buf.getShort(); // reads 7, position → 15
5.2 Absolute Put/Get
ByteBuffer buf = ByteBuffer.allocate(16);
// Write at specific indices — position does NOT change
buf.put(0, (byte) 0x41);
buf.putInt(4, 42);
buf.putLong(8, 123456789L);
System.out.println("position after absolute writes: " + buf.position()); // 0
// Read at specific indices — position does NOT change
byte b = buf.get(0); // 0x41
int i = buf.getInt(4); // 42
long l = buf.getLong(8); // 123456789
System.out.println("position after absolute reads: " + buf.position()); // 0
5.3 Bulk Operations
ByteBuffer buf = ByteBuffer.allocate(32);
// Bulk write
byte[] source = "Hello, ByteBuffer!".getBytes(StandardCharsets.UTF_8);
buf.put(source); // copies entire array into buffer
// Bulk read
buf.flip();
byte[] dest = new byte[buf.remaining()];
buf.get(dest); // copies buffer content into array
System.out.println(new String(dest, StandardCharsets.UTF_8));
// "Hello, ByteBuffer!"
Type Sizes Reference
6. Flip, Clear, Compact — The State Machine
This is where most beginners get confused. ByteBuffer operates like a state machine — after writing data, you must prepare it for reading (and vice versa).
The State Transitions
6.1 flip() — Switch from Writing to Reading
flip() sets limit = position and then position = 0. This makes all the data you just wrote available for reading.
ByteBuffer buf = ByteBuffer.allocate(10);
// WRITE MODE: position=0, limit=10
buf.put((byte) 'H'); // position=1
buf.put((byte) 'i'); // position=2
// Before flip: position=2, limit=10
buf.flip();
// After flip: position=0, limit=2
// Now we can read exactly the 2 bytes we wrote
while (buf.hasRemaining()) {
System.out.print((char) buf.get()); // prints "Hi"
}
Visual walkthrough:
6.2 clear() — Reset for Full Rewrite
clear() sets position = 0 and limit = capacity. The data is still there, but will be overwritten.
buf.clear();
// position=0, limit=10, capacity=10
// Ready to write fresh data
6.3 compact() — Keep Unread Data, Make Room for More
compact() copies unread data to the beginning of the buffer, then sets position after the last copied byte.
ByteBuffer buf = ByteBuffer.allocate(10);
buf.put(new byte[]{'A', 'B', 'C', 'D', 'E'});
buf.flip(); // position=0, limit=5
buf.get(); // reads 'A', position=1
buf.get(); // reads 'B', position=2
// We've read A,B but C,D,E remain unread
buf.compact();
// C,D,E are moved to positions 0,1,2
// position=3, limit=10
// We can now write more data starting at position 3
buf.put((byte) 'F'); // position=4
buf.put((byte) 'G'); // position=5
buf.flip(); // position=0, limit=5
// Buffer now contains: C,D,E,F,G
Visual walkthrough of compact():
Decision Flowchart: Which Operation to Use?
7. Byte Order (Endianness)
Multi-byte values (int, long, double, etc.) can be stored in two orders:
| Order | Description | Example (int 0x01020304) |
|---|---|---|
| Big-Endian | Most significant byte first | 01 02 03 04 |
| Little-Endian | Least significant byte first | 04 03 02 01 |
Java's ByteBuffer defaults to Big-Endian. You can change it:
ByteBuffer buf = ByteBuffer.allocate(4);
// Default: Big-Endian
buf.order(ByteOrder.BIG_ENDIAN);
buf.putInt(0x01020304);
buf.flip();
// bytes: 01, 02, 03, 04
buf.clear();
// Switch to Little-Endian
buf.order(ByteOrder.LITTLE_ENDIAN);
buf.putInt(0x01020304);
buf.flip();
// bytes: 04, 03, 02, 01
// Check native platform order
System.out.println("Native: " + ByteOrder.nativeOrder());
// Usually LITTLE_ENDIAN on x86/x64
When does this matter?
Network protocols: almost always Big-Endian
Interoperating with C/C++ code: usually Little-Endian on x86
Reading binary file formats: check the spec!
Shared memory IPC: must agree on byte order
8. View Buffers
You can create a "view" of a ByteBuffer as a buffer of a different type. The view shares the same underlying memory:
ByteBuffer byteBuffer = ByteBuffer.allocate(16);
// Create an IntBuffer view (each int = 4 bytes, so capacity = 4)
IntBuffer intView = byteBuffer.asIntBuffer();
intView.put(100);
intView.put(200);
intView.put(300);
// The ByteBuffer now contains those ints as bytes!
byteBuffer.position(0);
for (int i = 0; i < 12; i++) {
System.out.printf("%02X ", byteBuffer.get(i));
}
// Output (Big-Endian): 00 00 00 64 00 00 00 C8 00 00 01 2C
// (100) (200) (300)
Available view types: asCharBuffer(), asShortBuffer(), asIntBuffer(), asLongBuffer(), asFloatBuffer(), asDoubleBuffer().
9. Slicing and Duplicating
9.1 slice() — A View of a Portion
slice() creates a new ByteBuffer that shares the same memory but starts at the current position with a capacity equal to remaining:
ByteBuffer original = ByteBuffer.allocate(10);
original.putInt(111); // position=4
original.putInt(222); // position=8
original.position(4);
original.limit(8);
ByteBuffer slice = original.slice();
System.out.println(slice.capacity()); // 4
System.out.println(slice.getInt(0)); // 222
// Modifying the slice modifies the original!
slice.putInt(0, 333);
System.out.println(original.getInt(4)); // 333
9.2 duplicate() — Independent Position, Shared Data
ByteBuffer original = ByteBuffer.allocate(10);
original.putInt(42);
ByteBuffer dup = original.duplicate();
// Different position/limit/mark, same data
dup.flip();
System.out.println(dup.getInt()); // 42
// Modifying data through either buffer is visible in both
original.putInt(0, 99);
dup.position(0);
System.out.println(dup.getInt()); // 99
10. Direct vs Heap — When to Use What
Here's a comprehensive comparison:
| Aspect | Heap Buffer | Direct Buffer |
|---|---|---|
| Location | JVM heap (byte[]) |
Native memory |
| Allocation speed | Fast (~50ns) | Slow (~1μs) |
| GC Impact | Subject to GC | Only wrapper object is GC'd |
| I/O performance | Slower (extra copy) | Faster (zero-copy possible) |
hasArray() |
true |
false |
| Max size | Limited by JVM heap | Limited by -XX:MaxDirectMemorySize |
| Deallocation | Automatic (GC) | Requires GC of wrapper or Cleaner |
| Best for | Short-lived, small buffers | Long-lived, I/O-intensive buffers |
How Direct Buffers Avoid Copies
Practical Guidelines
// ❌ BAD: Allocating direct buffer in a hot loop
for (int i = 0; i < 100_000; i++) {
ByteBuffer buf = ByteBuffer.allocateDirect(1024); // SLOW!
// ... use and discard
}
// ✅ GOOD: Allocate once, reuse
ByteBuffer reusable = ByteBuffer.allocateDirect(1024);
for (int i = 0; i < 100_000; i++) {
reusable.clear();
// ... use
}
// ✅ GOOD: ThreadLocal for thread-safe reuse
private static final ThreadLocal<ByteBuffer> BUFFER =
ThreadLocal.withInitial(() -> ByteBuffer.allocateDirect(4096));
public void process() {
ByteBuffer buf = BUFFER.get();
buf.clear();
// ... use
}
11. Common Pitfalls
Pitfall 1: Forgetting to flip()
// ❌ BUG: Reading without flip
ByteBuffer buf = ByteBuffer.allocate(10);
buf.putInt(42);
int value = buf.getInt(); // BufferUnderflowException! position is at 4, limit is 10
// reads garbage bytes 4-7 (all zeros), returns 0
// ✅ FIX: Always flip before reading
buf.flip();
int value = buf.getInt(); // 42 ✓
Pitfall 2: Double flip()
// ❌ BUG: Flipping twice
ByteBuffer buf = ByteBuffer.allocate(10);
buf.putInt(42); // position=4
buf.flip(); // position=0, limit=4 ← correct
buf.flip(); // position=0, limit=0 ← OOPS! limit is now 0
buf.getInt(); // BufferUnderflowException!
Pitfall 3: Direct Buffer Memory Leak
// ❌ RISK: Direct buffers aren't freed promptly
void processFile() {
ByteBuffer buf = ByteBuffer.allocateDirect(100_000_000); // 100MB
// ... use buf
// buf goes out of scope but native memory may NOT be freed
// until the next GC cycle
}
// ✅ FIX: Explicit cleanup (Java 9+, or use reflection pre-9)
import sun.misc.Unsafe; // or use Cleaner API
// Better: use try-with-resources pattern around a wrapper
Pitfall 4: Wrong byte order for interop
// ❌ BUG: C program writes little-endian, Java reads big-endian
ByteBuffer buf = ByteBuffer.wrap(dataFromC);
int value = buf.getInt(); // Wrong value!
// ✅ FIX: Match the byte order
buf.order(ByteOrder.LITTLE_ENDIAN);
int value = buf.getInt(); // Correct!
12. Summary
Key takeaways:
ByteBuffer is the foundation of all NIO I/O on the JVM
Heap buffers live on the GC heap; direct buffers live in native memory
Always flip() before switching from writing to reading
Use compact() when you've partially consumed a buffer
Byte order matters for interoperability with native code and network protocols
Allocate direct buffers once and reuse them — they're expensive to create