How to unmap a file from memory mapped using FileChannel in java?

I am mapping a file("sample.txt") to memory using FileChannel.map() and then closing the channel using fc.close(). After this when I write to the file using FileOutputStream, I am getting the following error:

java.io.FileNotFoundException: sample.txt (The requested operation cannot be per formed on a file with a user-mapped section open)

开发者_开发知识库File f = new File("sample.txt");
RandomAccessFile raf = new RandomAccessFile(f,"rw");
FileChannel fc = raf.getChannel();
MappedByteBuffer mbf = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
fc.close();
raf.close();

FileOutputStream fos = new FileOutputStream(f);
fos.write(str.getBytes());
fos.close();

I presume this may be due to file being still mapped to the memory even after I close the FileChannel. Am I right?. If so, how can I "unmap" the file from memory?(I can't find any methods for this in the API). Thanks.

Edit: Looks like it(adding an unmap method) was submitted as RFE to sun some time back: http://bugs.sun.com/view_bug.do?bug_id=4724038

Following static method could be used:

public static void unmap(MappedByteBuffer buffer)
{
   sun.misc.Cleaner cleaner = ((DirectBuffer) buffer).cleaner();
   cleaner.clean();
}

But this is unsafe solution because of following:
1) Lead to failures if someone use MappedByteBuffer after unmap
2) It relies on MappedByteBuffer implementation details

[WinXP,SunJDK1.6] I had a mapped ByteBuffer taken from filechannel. After reading SO posts finally managed to call a cleaner through reflection without any sun.* package imports. No longer file lock is lingering.

edit Added JDK9+ code(Luke Hutchison).

private static void closeDirectBuffer(ByteBuffer cb) {
    if (cb==null || !cb.isDirect()) return;
    // we could use this type cast and call functions without reflection code,
    // but static import from sun.* package is risky for non-SUN virtual machine.
    //try { ((sun.nio.ch.DirectBuffer)cb).cleaner().clean(); } catch (Exception ex) { }

    // JavaSpecVer: 1.6, 1.7, 1.8, 9, 10
    boolean isOldJDK = System.getProperty("java.specification.version","99").startsWith("1.");  
    try {
        if (isOldJDK) {
            Method cleaner = cb.getClass().getMethod("cleaner");
            cleaner.setAccessible(true);
            Method clean = Class.forName("sun.misc.Cleaner").getMethod("clean");
            clean.setAccessible(true);
            clean.invoke(cleaner.invoke(cb));
        } else {
            Class unsafeClass;
            try {
                unsafeClass = Class.forName("sun.misc.Unsafe");
            } catch(Exception ex) {
                // jdk.internal.misc.Unsafe doesn't yet have an invokeCleaner() method,
                // but that method should be added if sun.misc.Unsafe is removed.
                unsafeClass = Class.forName("jdk.internal.misc.Unsafe");
            }
            Method clean = unsafeClass.getMethod("invokeCleaner", ByteBuffer.class);
            clean.setAccessible(true);
            Field theUnsafeField = unsafeClass.getDeclaredField("theUnsafe");
            theUnsafeField.setAccessible(true);
            Object theUnsafe = theUnsafeField.get(null);
            clean.invoke(theUnsafe, cb);
        }
    } catch(Exception ex) { }
    cb = null;
}

Ideas were taken from these posts.
* How to unmap a file from memory mapped using FileChannel in java?
* Examples of forcing freeing of native memory direct ByteBuffer has allocated, using sun.misc.Unsafe?
* https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferAllocator.java#L40

From the MappedByteBuffer javadoc:

A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.

Try calling System.gc()? Even that's only a suggestion to the VM.

sun.misc.Cleaner javadoc says:

General-purpose phantom-reference-based cleaners. Cleaners are a lightweight and more robust alternative to finalization. They are lightweight because they are not created by the VM and thus do not require a JNI upcall to be created, and because their cleanup code is invoked directly by the reference-handler thread rather than by the finalizer thread. They are more robust because they use phantom references, the weakest type of reference object, thereby avoiding the nasty ordering problems inherent to finalization. A cleaner tracks a referent object and encapsulates a thunk of arbitrary cleanup code. Some time after the GC detects that a cleaner's referent has become phantom-reachable, the reference-handler thread will run the cleaner. Cleaners may also be invoked directly; they are thread safe and ensure that they run their thunks at most once. Cleaners are not a replacement for finalization. They should be used only when the cleanup code is extremely simple and straightforward. Nontrivial cleaners are inadvisable since they risk blocking the reference-handler thread and delaying further cleanup and finalization.

Running System.gc() is acceptable solution if your buffers total size is small, but if I was mapping gigabytes of files I would try to implement like this:

((DirectBuffer) buffer).cleaner().clean()

But! Make sure you don't access that buffer after cleaning or you will end up with:

A fatal error has been detected by the Java Runtime Environment: EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000002bcf700, pid=7592, tid=10184 JRE version: Java(TM) SE Runtime Environment (8.0_40-b25) (build 1.8.0_40-b25) Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode windows-amd64 compressed oops) Problematic frame: J 85 C2 java.nio.DirectByteBuffer.get(I)B (16 bytes) @ 0x0000000002bcf700 [0x0000000002bcf6c0+0x40] Failed to write core dump. Minidumps are not enabled by default on client versions of Windows An error report file with more information is saved as: C:\Users\?????\Programs\testApp\hs_err_pid7592.log Compiled method (c2) 42392 85 4 java.nio.DirectByteBuffer::get (16 bytes) total in heap [0x0000000002bcf590,0x0000000002bcf828] = 664 relocation [0x0000000002bcf6b0,0x0000000002bcf6c0] = 16 main code [0x0000000002bcf6c0,0x0000000002bcf760] = 160 stub code
[0x0000000002bcf760,0x0000000002bcf778] = 24 oops
[0x0000000002bcf778,0x0000000002bcf780] = 8 metadata
[0x0000000002bcf780,0x0000000002bcf798] = 24 scopes data
[0x0000000002bcf798,0x0000000002bcf7e0] = 72 scopes pcs
[0x0000000002bcf7e0,0x0000000002bcf820] = 64 dependencies
[0x0000000002bcf820,0x0000000002bcf828] = 8

Good luck!

The method covered in other answers that employs ((DirectBuffer) byteBuffer).cleaner().clean() doesn't work on JDK 9+ (even in reflective form) without displaying an An illegal reflective access operation has occurred warning. This will stop working altogether in some future JDK version. Fortunately sun.misc.Unsafe.invokeCleaner(ByteBuffer) can make that exact same call for you without the warning: (from OpenJDK 11 source):

public void invokeCleaner(java.nio.ByteBuffer directBuffer) {
    if (!directBuffer.isDirect())
        throw new IllegalArgumentException("buffer is non-direct");

    DirectBuffer db = (DirectBuffer)directBuffer;
    if (db.attachment() != null)
        throw new IllegalArgumentException("duplicate or slice");

    Cleaner cleaner = db.cleaner();
    if (cleaner != null) {
        cleaner.clean();
    }
}

Being a sun.misc class, it will be removed at some point. Interestingly all calls but this one in sun.misc.Unsafe are proxied directly to jdk.internal.misc.Unsafe. I don't know why invokeCleaner(ByteBuffer) is not proxied in the same way as all the other methods -- it was probably omitted because there will be a new way to directly free memory references (including DirectByteBuffer instances) as of about JDK 15.

I wrote the following code that is able to clean/close/unmap DirectByteBuffer/MappedByteBuffer instances on JDK 7/8, as well as JDK 9+, and this does not give the reflection warning:

private static boolean PRE_JAVA_9 = 
        System.getProperty("java.specification.version","9").startsWith("1.");

private static Method cleanMethod;
private static Method attachmentMethod;
private static Object theUnsafe;

static void getCleanMethodPrivileged() {
    if (PRE_JAVA_9) {
        try {
            cleanMethod = Class.forName("sun.misc.Cleaner").getMethod("clean");
            cleanMethod.setAccessible(true);
            final Class<?> directByteBufferClass =
                    Class.forName("sun.nio.ch.DirectBuffer");
            attachmentMethod = directByteBufferClass.getMethod("attachment");
            attachmentMethod.setAccessible(true);
        } catch (final Exception ex) {
        }
    } else {
        try {
            Class<?> unsafeClass;
            try {
                unsafeClass = Class.forName("sun.misc.Unsafe");
            } catch (Exception e) {
                // jdk.internal.misc.Unsafe doesn't yet have invokeCleaner(),
                // but that method should be added if sun.misc.Unsafe is removed.
                unsafeClass = Class.forName("jdk.internal.misc.Unsafe");
            }
            cleanMethod = unsafeClass.getMethod("invokeCleaner", ByteBuffer.class);
            cleanMethod.setAccessible(true);
            final Field theUnsafeField = unsafeClass.getDeclaredField("theUnsafe");
            theUnsafeField.setAccessible(true);
            theUnsafe = theUnsafeField.get(null);
        } catch (final Exception ex) {
        }
    }
}

static {
    AccessController.doPrivileged(new PrivilegedAction<Object>() {
        @Override
        public Object run() {
            getCleanMethodPrivileged();
            return null;
        }
    });
}

private static boolean closeDirectByteBufferPrivileged(
            final ByteBuffer byteBuffer, final LogNode log) {
    try {
        if (cleanMethod == null) {
            if (log != null) {
                log.log("Could not unmap ByteBuffer, cleanMethod == null");
            }
            return false;
        }
        if (PRE_JAVA_9) {
            if (attachmentMethod == null) {
                if (log != null) {
                    log.log("Could not unmap ByteBuffer, attachmentMethod == null");
                }
                return false;
            }
            // Make sure duplicates and slices are not cleaned, since this can result in
            // duplicate attempts to clean the same buffer, which trigger a crash with:
            // "A fatal error has been detected by the Java Runtime Environment:
            // EXCEPTION_ACCESS_VIOLATION"
            // See: https://stackoverflow.com/a/31592947/3950982
            if (attachmentMethod.invoke(byteBuffer) != null) {
                // Buffer is a duplicate or slice
                return false;
            }
            // Invoke ((DirectBuffer) byteBuffer).cleaner().clean()
            final Method cleaner = byteBuffer.getClass().getMethod("cleaner");
            cleaner.setAccessible(true);
            cleanMethod.invoke(cleaner.invoke(byteBuffer));
            return true;
        } else {
            if (theUnsafe == null) {
                if (log != null) {
                    log.log("Could not unmap ByteBuffer, theUnsafe == null");
                }
                return false;
            }
            // In JDK9+, calling the above code gives a reflection warning on stderr,
            // need to call Unsafe.theUnsafe.invokeCleaner(byteBuffer) , which makes
            // the same call, but does not print the reflection warning.
            try {
                cleanMethod.invoke(theUnsafe, byteBuffer);
                return true;
            } catch (final IllegalArgumentException e) {
                // Buffer is a duplicate or slice
                return false;
            }
        }
    } catch (final Exception e) {
        if (log != null) {
            log.log("Could not unmap ByteBuffer: " + e);
        }
        return false;
    }
}

/**
 * Close a {@code DirectByteBuffer} -- in particular, will unmap a
 * {@link MappedByteBuffer}.
 * 
 * @param byteBuffer
 *            The {@link ByteBuffer} to close/unmap.
 * @param log
 *            The log.
 * @return True if the byteBuffer was closed/unmapped (or if the ByteBuffer
 *            was null or non-direct).
 */
public static boolean closeDirectByteBuffer(final ByteBuffer byteBuffer,
            final Log log) {
    if (byteBuffer != null && byteBuffer.isDirect()) {
        return AccessController.doPrivileged(new PrivilegedAction<Boolean>() {
            @Override
            public Boolean run() {
                return closeDirectByteBufferPrivileged(byteBuffer, log);
            }
        });
    } else {
        // Nothing to unmap
        return false;
    }
}

Note that you will need to add the requires jdk.unsupported to your module descriptor in a modular runtime on JDK 9+ (needed for the use of Unsafe).

Your jar may also need RuntimePermission("accessClassInPackage.sun.misc"), RuntimePermission("accessClassInPackage.jdk.internal.misc"), and ReflectPermission("suppressAccessChecks").

A more complete method for garbage collecting a MappedByteBuffer or a DirectByteBuffer is implemented in ClassGraph (I am the author) -- the entry point is the closeDirectByteBuffer() method at the end of FileUtils:

https://github.com/classgraph/classgraph/blob/latest/src/main/java/nonapi/io/github/classgraph/utils/FileUtils.java#L543

This code is written to use reflection, since the Java APIs that are used (including Unsafe) are going away in the near future.

Note that there's an additional problem in JDK 16+:

this code will not work out of the box in JDK 16+ unless you use the Narcissus or JVM-Driver libraries to circumvent strong encapsulation. This is because MappedByteBuffer.clean() is a private method, and JDK 16 enforces strong encapsulation. ClassGraph abstracts away access to private encapsulated methods by calling Narcissus or JVM-driver through reflection at runtime:

https://github.com/classgraph/classgraph/blob/latest/src/main/java/nonapi/io/github/classgraph/reflection/ReflectionUtils.java

Warning: if you try to access the DirectByteBuffer after it is cleaned (freed), it will crash the VM.

There are other security considerations, discussed in the last comment in this bug report:

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-4724038

To work around this bug in Java, I had to do the following, which will work ok for small to medium-sized files:

    // first open the file for random access
    RandomAccessFile raf = new RandomAccessFile(file, "r");

    // extract a file channel
    FileChannel channel = raf.getChannel();

    // you can memory-map a byte-buffer, but it keeps the file locked
    //ByteBuffer buf =
    //        channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());

    // or, since map locks the file... just read the whole file into memory
    ByteBuffer buf = ByteBuffer.allocate((int)file.length());
    int read = channel.read(buf);

    // .... do something with buf

    channel.force(false);  // doesn't help
    channel.close();       // doesn't help
    channel = null;        // doesn't help
    buf = null;            // doesn't help
    raf.close();           // try to make sure that this thing is closed!!!!!

I found out information about unmap, it is a method of FileChannelImpl and not accessible, so you can invoke it by java reflect like:

public static void unMapBuffer(MappedByteBuffer buffer, Class channelClass) {
    if (buffer == null) {
        return;
    }

    try {
        Method unmap = channelClass.getDeclaredMethod("unmap", MappedByteBuffer.class);
        unmap.setAccessible(true);
        unmap.invoke(channelClass, buffer);
    } catch (NoSuchMethodException e) {
        e.printStackTrace();
    } catch (IllegalAccessException e) {
        e.printStackTrace();
    } catch (InvocationTargetException e) {
        e.printStackTrace();
    }
}

The mapped memory is used until it is freed by the garbage collector.

From FileChannel docs

A mapping, once established, is not dependent upon the file channel that was used to create it. Closing the channel, in particular, has no effect upon the validity of the mapping.

From MappedByteBuffer java doc

A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.

So I would suggest ensuring there are no remaining references to the mapped byte buffer and then requesting a garbage collection.

It is funny to see so many recommendations to do what Item 7 in 'Effective Java' specifically says not to do. A termination method like what @Whome did and no references to the buffer is what is needed. GC cannot be forced. But that doesn't stop developers from trying. Another workaround I found was to use WeakReferences from http://jan.baresovi.cz/dr/en/java#memoryMap

final MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, size);
....
final WeakReference<mappedbytebuffer> bufferWeakRef = new WeakReference<mappedbytebuffer>(bb);
bb = null;

final long startTime = System.currentTimeMillis();
while(null != bufferWeakRef.get()) {
  if(System.currentTimeMillis() - startTime > 10)
// give up
    return;
    System.gc();
    Thread.yield();
}

I would try JNI:

#ifdef _WIN32
UnmapViewOfFile(env->GetDirectBufferAddress(buffer));
#else
munmap(env->GetDirectBufferAddress(buffer), env->GetDirectBufferCapacity(buffer));
#endif

Include files: windows.h for Windows, sys/mmap.h for BSD, Linux, OSX.

Try https://github.com/real-logic/agrona

Its IOUtil class has an unmap(MappedByteBuffer) method that does exactly what you need. It allows to explicitly unmap a MappedByteBuffer.

However, it uses sun.misc.Unsafe internally, but that's not any different from the other answers here.

If the mapped file buffer object can be guaranteed to be eligible for garbage collection, you don't need to GC the whole VM to get the buffer's mapped memory to be released. You can call System.runFinalization() . This will call the finalize() method on the mapped file buffer object (if it there are no references to it in your app threads) which will release the mapped memory.

The correct solution here is to use try-with-resources.

This allows the creation of the Channel & the other resources to be scoped to a block. Once the block exits, the Channel & other resources are gone & subsequently cannot be used (as nothing has a reference to them).

The memory-mapping still won't be undone until the next GC run, but at least there aren't any dangling references to it.