Handling large files using JavaNIO MappedByteBuffer
In almost every project we need to deal with files. But what if size of file is too large ?? Heap gets full and JVM rewards us by throwing OutOfMemoryError.
Java provides a class called MappedByteBuffer (part of JavaNIO), which helps us dealing with large size files.
MappedByteBuffer is a class which creates Virtual-memory mapping with JVM memory. File content is loaded in virtual memory instead of heap and JVM doesn’t need to call OS specific read/write system calls to read write data in JVM memory. We can also map specific part of a file instead of entire file.
We can map file with MappedByteBuffer by getting FileChannel. FileChannel is link for reading, writing and manipulating a file. FileChannel can be access by RandomAccessFile, FileInputStream (for read only) and FileOutputStream (for write only).
FileChannel provide map() to map the file. It takes 3 arguments
1. Map mode (READ_ONLY, READ_WRITE and PRIVATE)
2. Position
3. Size
After getting MappedByteBuffer we can call get() and put() methods to read and write data.
Reading Data from mappedByteBuffer into byte array
[java]
RandomAccessFile randomAccessFile = new RandomAccessFile("file.txt", "r");
MappedByteByffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length());
byte[] data = new byte[100]; // Needs to be Byte Array only as MappedBuffer play only with Byte[]
while (mappedByteBuffer.hasRemaining()) {
int remaining = data.length;
if(mappedByteBuffer.remaining() < remaining)
remaining = mappedByteBuffer.remaining();
mappedByteBuffer.get(data, 0, remaining);
// do somthing with data
}
[/java]
In above code we are mapping entire file. We can call clear() on mappedByteBuffer to reset the position to 0, so that we can read buffer again from start.
We can also have mappedByteBuffer in read-write mode. Lets see how it works :-
[java]
RandomAccessFile randomAccessFile = new RandomAccessFile("FileToWrite.txt", "rw");
MappedByteByffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, 512);
String str = "This is testing string";
mappedByteBuffer.put(str.getBytes());
[/java]
Note: There are some limitation with MappedByteBuffer. It has size limit 2GB and it can result in page fault if requested page is not in memory.
The purpose of MappedByteBuffer is to load file into memory or RAM. Java will die with OutofMemory if you load files with size greater than -XmX.