Java Nio Large File Transfer Tutorial

JJJune 8th, 2017Last Updated: March 12th, 2019

1 1,139 9 minutes read

This article is a tutorial on transferring a large file using Java Nio. It will take shape via two examples demonstrating a simple local file transfer from one location on hard disk to another and then via sockets from one remote location to another remote location.

1. Introduction

2. Technologies used

3. FileChannel

4. Background

5. Program

5.1. Local copy

5.2. Remote copy

6. Running the program

7. Summary

8. Download the source code

1. Introduction

This tutorial will make use of the FileChannel abstraction for both remote and local copy. Augmenting the remote copy process will be a simple set of abstractions (ServerSocketChannel & SocketChannel) that facilitate the transfer of bytes over the wire. Finally we wrap things up with an asynchronous implementation of large file transfer. The tutorial will be driven by unit tests that can run from command line using maven or from within your IDE.

2. Technologies used

The example code in this article was built and run using:

Java 1.8.101 (1.8.x will do fine)
Maven 3.3.9 (3.3.x will do fine)
Spring source tool suite 4.6.3 (Any Java IDE would work)
Ubuntu 16.04 (Windows, Mac or Linux will do fine)

2. FileChannel

A FileChannel is a type of Channel used for writing, reading, mapping and manipulating a File. In addition to the familiar Channel (read, write and close) operations, this Channel has a few specific operations:

Has the concept of an absolute position in the File which does not affect the Channels current position.
Parts or regions of a File can be mapped directly into memory and work from memory, very useful when dealing with large files.
Writes can be forced to the underlying storage device, ensuring write persistence.
Bytes can be transferred from one ReadableByteChannel / WritableByteChannel instance to another ReadableByteChannel / WritableByteChannel, which FileChannel implements. This yields tremendous IO performance advantages that some Operating systems are optimized for.
A part or region of a File may be locked by a process to guard against access by other processes.

FileChannels are thread safe. Only one IO operation that involves the FileChannels position can be in flight at any given point in time, blocking others. The view or snapshot of a File via a FileChannel is consistent with other views of the same File within the same process. However, the same cannot be said for other processes. A file channel can be created in the following ways:

… FileChannel.open(...)
… FileInputStream(...).getChannel()
… FileOutputStream(...).getChannel()
… RandomAccessFile(...).getChannel()

Using one of the stream interfaces to obtain a FileChannel will yield a Channel that allows either read, write or append privileges and this is directly attributed to the type of Stream (FileInputStream or FileOutputStream) that was used to get the Channel. Append mode is a configuration artifact of a FileOutputStream constructor.

4. Background

The sample program for this example will demonstrate the following:

Local transfer of a file (same machine)
Remote transfer of a file (potentially remote different processes, although in the unit tests we spin up different threads for client and server)
Remote transfer of a file asynchronously

Particularly with large files the advantages of asynchronous non blocking handling of file transfer cannot be stressed enough. Large files tying up connection handling threads soon starve a server of resources to handle additional requests possibly for more large file transfers.

5. Program

The code sample can be split into local and remote domains and within remote we further specialize an asynchronous implementation of file transfer, at least on the receipt side which is arguably the more interesting part.

5.1. Local copy

FileCopy

final class FileCopy
 
    private FileCop() {
        throw new IllegalStateException(Constants.INSTANTIATION_NOT_ALLOWED);
    }
 
    public static void copy(final String src, final String target) throws IOException {
        if (StringUtils.isEmpty(src) || StringUtils.isEmpty(target)) {
            throw new IllegalArgumentException("src and target required");
        }
 
        final String fileName = getFileName(src);
 
        try (FileChannel from = (FileChannel.open(Paths.get(src), StandardOpenOption.READ));
                FileChannel to = (FileChannel.open(Paths.get(target + "/" + fileName), StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE))) {
            transfer(from, to, 0l, from.size());
        }
    }
 
    private static String getFileName(final String src) {
        assert StringUtils.isNotEmpty(src);
 
        final File file = new File(src);
        if (file.isFile()) {
            return file.getName();
        } else {
            throw new RuntimeException("src is not a valid file");
        }
    }
 
    private static void transfer(final FileChannel from, final FileChannel to, long position, long size) throws IOException {
        assert !Objects.isNull(from) && !Objects.isNull(to);
 
        while (position < size) {
            position += from.transferTo(position, Constants.TRANSFER_MAX_SIZE, to);
        }
    }
}

line 14: we open the from Channel with the StandardOpenOption.READ meaning that this Channel will only be read from. The path is provided.
line 15: the to Channel is opened with the intention to write and create, the path is provided.
line 31-37: the two Channels are provided (from & to) along with the position (initially where to start reading from) and the size indicating the amount of bytes to transfer in total. A loop is started where attempts are made to transfer up to Constants.TRANSFER_MAX_SIZE in bytes from the from Channel to the to Channel. After each iteration the amount of bytes transferred is added to the position which then advances the cursor for the next transfer attempt.

5.2. Remote copy

FileReader

final class FileReader {
 
    private final FileChannel channel;
    private final FileSender sender;
 
    FileReader(final FileSender sender, final String path) throws IOException {
        if (Objects.isNull(sender) || StringUtils.isEmpty(path)) {
            throw new IllegalArgumentException("sender and path required");
        }
 
        this.sender = sender;
        this.channel = FileChannel.open(Paths.get(path), StandardOpenOption.READ);
    }
 
    void read() throws IOException {
        try {
            transfer();
        } finally {
            close();
        }
    }
 
    void close() throws IOException {
        this.sender.close();
        this.channel.close();
    }
 
    private void transfer() throws IOException {
        this.sender.transfer(this.channel, 0l, this.channel.size());
    }
}

line 12: the FileChannel is opened with the intent to read StandardOpenOption.READ, the path is provided to the File.
line 15-21: we ensure we transfer the contents of the FileChannel entirely and the close the Channel.
line 23-26: we close the sender resources and then close the FileChannel
line 29: we call transfer(...) on the sender to transfer all the bytes from the FileChannel

FileSender

final class FileSender {
 
    private final InetSocketAddress hostAddress;
    private SocketChannel client;
 
    FileSender(final int port) throws IOException {
        this.hostAddress = new InetSocketAddress(port);
        this.client = SocketChannel.open(this.hostAddress);
    }
 
    void transfer(final FileChannel channel, long position, long size) throws IOException {
        assert !Objects.isNull(channel);
 
        while (position < size) {
            position += channel.transferTo(position, Constants.TRANSFER_MAX_SIZE, this.client);
        }
    }
     
    SocketChannel getChannel() {
        return this.client;
    }
 
    void close() throws IOException {
        this.client.close();
    }
}

line 11-17: we provide the FileChannel, position and size of the bytes to transfer from the given channel. A loop is started where attempts are made to transfer up to Constants.TRANSFER_MAX_SIZE in bytes from the provided Channel to the SocketChannel client. After each iteration the amount of bytes transferred is added to the position which then advances the cursor for the next transfer attempt.

FileReceiver

final class FileReceiver {
 
    private final int port;
    private final FileWriter fileWriter;
    private final long size;
 
    FileReceiver(final int port, final FileWriter fileWriter, final long size) {
        this.port = port;
        this.fileWriter = fileWriter;
        this.size = size;
    }
 
    void receive() throws IOException {
        SocketChannel channel = null;
 
        try (final ServerSocketChannel serverSocketChannel = ServerSocketChannel.open()) {
            init(serverSocketChannel);
 
            channel = serverSocketChannel.accept();
 
            doTransfer(channel);
        } finally {
            if (!Objects.isNull(channel)) {
                channel.close();
            }
 
            this.fileWriter.close();
        }
    }
 
    private void doTransfer(final SocketChannel channel) throws IOException {
        assert !Objects.isNull(channel);
 
        this.fileWriter.transfer(channel, this.size);
    }
 
    private void init(final ServerSocketChannel serverSocketChannel) throws IOException {
        assert !Objects.isNull(serverSocketChannel);
 
        serverSocketChannel.bind(new InetSocketAddress(this.port));
    }
}

The FileReceiver is a mini server that listens for incoming connections on the localhost and upon connection, accepts it and initiates a transfer of bytes from the accepted Channel via the FileWriter abstraction to the encapsulated FileChannel within the FileWriter. The FileReceiver is only responsible for receiving the bytes via socket and then delegates transferring them to the FileWriter.

FileWriter

final class FileWriter {
 
    private final FileChannel channel;
 
    FileWriter(final String path) throws IOException {
        if (StringUtils.isEmpty(path)) {
            throw new IllegalArgumentException("path required");
        }
 
        this.channel = FileChannel.open(Paths.get(path), StandardOpenOption.WRITE, StandardOpenOption.CREATE_NEW);
    }
 
    void transfer(final SocketChannel channel, final long bytes) throws IOException {
        assert !Objects.isNull(channel);
 
        long position = 0l;
        while (position < bytes) {
            position += this.channel.transferFrom(channel, position, Constants.TRANSFER_MAX_SIZE);
        }
    }
     
    int write(final ByteBuffer buffer, long position) throws IOException {
        assert !Objects.isNull(buffer);
 
        int bytesWritten = 0;        
        while(buffer.hasRemaining()) {
            bytesWritten += this.channel.write(buffer, position + bytesWritten);            
        }
         
        return bytesWritten;
    }
 
    void close() throws IOException {
        this.channel.close();
    }
}

The FileWriter is simply charged with transferring the bytes from a SocketChannel to it’s encapsulated FileChannel. As before, the transfer process is a loop which attempts to transfer up to Constants.TRANSFER_MAX_SIZE bytes with each iteration.

5.2.1. Asynchronous large file transfer

The following code snippets demonstrate transferring a large file from one remote location to another via an asynchronous receiver FileReceiverAsync.

OnComplete

@FunctionalInterface
public interface OnComplete {
 
    void onComplete(FileWriterProxy fileWriter);
}

The OnComplete interface represents a callback abstraction that we pass to our FileReceiverAsync implementation with the purposes of executing this once a file has been successfully and thoroughly transferred. We pass a FileWriterProxy to the onComplete(...) and this can server as context when executing said method.

FileWriterProxy

final class FileWriterProxy {
 
    private final FileWriter fileWriter;
    private final AtomicLong position;
    private final long size;
    private final String fileName;
 
    FileWriterProxy(final String path, final FileMetaData metaData) throws IOException {
        assert !Objects.isNull(metaData) && StringUtils.isNotEmpty(path);
 
        this.fileWriter = new FileWriter(path + "/" + metaData.getFileName());
        this.position = new AtomicLong(0l);
        this.size = metaData.getSize();
        this.fileName = metaData.getFileName();
    }
     
    String getFileName() {
        return this.fileName;
    }
 
    FileWriter getFileWriter() {
        return this.fileWriter;
    }
     
    AtomicLong getPosition() {
        return this.position;
    }
     
    boolean done() {
        return this.position.get() == this.size;
    }
}

The FileWriterProxy represents a proxy abstraction that wraps a FileWriter and encapsulates FileMetaData. All of this is needed when determining what to name the file, where to write the file and what the file size is so that we know when the file transfer is complete. During transfer negotiation this meta information is compiled via a custom protocol we implement before actual file transfer takes place.

FileReceiverAsync

final class FileReceiverAsync {
 
    private final AsynchronousServerSocketChannel server;
    private final AsynchronousChannelGroup group;
    private final String path;
    private final OnComplete onFileComplete;
 
    FileReceiverAsync(final int port, final int poolSize, final String path, final OnComplete onFileComplete) {
        assert !Objects.isNull(path);
 
        this.path = path;
        this.onFileComplete = onFileComplete;
         
        try {
            this.group = AsynchronousChannelGroup.withThreadPool(Executors.newFixedThreadPool(poolSize));
            this.server = AsynchronousServerSocketChannel.open(this.group).bind(new InetSocketAddress(port));
        } catch (IOException e) {
            throw new IllegalStateException("unable to start FileReceiver", e);
        }
    }
 
    void start() {
        accept();
    }
 
    void stop(long wait) {
         
        try {            
            this.group.shutdown();            
            this.group.awaitTermination(wait, TimeUnit.MILLISECONDS);
        } catch (InterruptedException e) {
            throw new RuntimeException("unable to stop FileReceiver", e);
        }
    }
 
    private void read(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {
        assert !Objects.isNull(channel) && !Objects.isNull(proxy);
 
        final ByteBuffer buffer = ByteBuffer.allocate(Constants.BUFFER_SIZE);
        channel.read(buffer, proxy, new CompletionHandler<Integer, FileWriterProxy>() {
 
            @Override
            public void completed(final Integer result, final FileWriterProxy attachment) {
                if (result >= 0) {
                    if (result > 0) {
                        writeToFile(channel, buffer, attachment);
                    }
                     
                    buffer.clear();
                    channel.read(buffer, attachment, this);
                } else if (result < 0 || attachment.done()) {
                    onComplete(attachment);
                    close(channel, attachment);
                }
            }
 
            @Override
            public void failed(final Throwable exc, final FileWriterProxy attachment) {
                throw new RuntimeException("unable to read data", exc);
            }
        });
    }
     
    private void onComplete(final FileWriterProxy proxy) {
        assert !Objects.isNull(proxy);
         
        this.onFileComplete.onComplete(proxy);
    }
 
    private void meta(final AsynchronousSocketChannel channel) {
        assert !Objects.isNull(channel);
 
        final ByteBuffer buffer = ByteBuffer.allocate(Constants.BUFFER_SIZE);
        channel.read(buffer, new StringBuffer(), new CompletionHandler<Integer, StringBuffer>() {
 
            @Override
            public void completed(final Integer result, final StringBuffer attachment) {
                if (result < 0) {
                    close(channel, null);
                } else {
                     
                    if (result > 0) {
                        attachment.append(new String(buffer.array()).trim());
                    }
 
                    if (attachment.toString().contains(Constants.END_MESSAGE_MARKER)) {
                        final FileMetaData metaData = FileMetaData.from(attachment.toString());
                        FileWriterProxy fileWriterProxy;
 
                        try {
                            fileWriterProxy = new FileWriterProxy(FileReceiverAsync.this.path, metaData);
                            confirm(channel, fileWriterProxy);
                        } catch (IOException e) {
                            close(channel, null);
                            throw new RuntimeException("unable to create file writer proxy", e);
                        }
                    } else {
                        buffer.clear();
                        channel.read(buffer, attachment, this);
                    }
                }
            }
 
            @Override
            public void failed(final Throwable exc, final StringBuffer attachment) {
                close(channel, null);
                throw new RuntimeException("unable to read meta data", exc);
            }
        });
    }
 
    private void confirm(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {
        assert !Objects.isNull(channel) && !Objects.isNull(proxy);
 
        final ByteBuffer buffer = ByteBuffer.wrap(Constants.CONFIRMATION.getBytes());
        channel.write(buffer, null, new CompletionHandler<Integer, Void>() {
 
            @Override
            public void completed(final Integer result, final Void attachment) {
                while (buffer.hasRemaining()) {
                    channel.write(buffer, null, this);
                }
 
                read(channel, proxy);
            }
 
            @Override
            public void failed(final Throwable exc, final Void attachment) {
                close(channel, null);
                throw new RuntimeException("unable to confirm", exc);
            }
 
        });
    }
 
    private void accept() {
        this.server.accept(null, new CompletionHandler() {
            public void completed(final AsynchronousSocketChannel channel, final Void attachment) {
 
                // Delegate off to another thread for the next connection.
                accept();
 
                // Delegate off to another thread to handle this connection.
                meta(channel);
            }
 
            public void failed(final Throwable exc, final Void attachment) {
                throw new RuntimeException("unable to accept new connection", exc);
            }
        });
    }
 
    private void writeToFile(final AsynchronousSocketChannel channel, final ByteBuffer buffer, final FileWriterProxy proxy) {
        assert !Objects.isNull(buffer) && !Objects.isNull(proxy) && !Objects.isNull(channel);
 
        try {
            buffer.flip();
 
            final long bytesWritten = proxy.getFileWriter().write(buffer, proxy.getPosition().get());
            proxy.getPosition().addAndGet(bytesWritten);
        } catch (IOException e) {
            close(channel, proxy);
            throw new RuntimeException("unable to write bytes to file", e);
        }
    }
 
    private void close(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {
        assert !Objects.isNull(channel);
 
        try {
 
            if (!Objects.isNull(proxy)) {
                proxy.getFileWriter().close();
            }
            channel.close();
        } catch (IOException e) {
            throw new RuntimeException("unable to close channel and FileWriter", e);
        }
    }

The FileReceiverAsync abstraction builds upon the idiomatic use of AsynchronousChannels demonstrated in this tutorial.

6. Running the program

The program can be run from within the IDE, using the normal JUnit Runner or from the command line using maven. Ensure that the test resources (large source files and target directories exist).
Running tests from command line

1	`mvn clean` `install`

You can edit these in the AbstractTest and FileCopyAsyncTest classes. Fair warning the FileCopyAsyncTest can run for a while as it is designed to copy two large files asynchronously, and the test case waits on a CountDownLatch without a max wait time specified.

I ran the tests using the “spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz” file downloaded from the SpringSource website. This file is approximately 483mb large and below are my test elapsed times. (using a very old laptop).

Test elapsed time

Running com.javacodegeeks.nio.large_file_transfer.remote.FileCopyTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.459 sec - in com.javacodegeeks.nio.large_file_transfer.remote.FileCopyTest
Running com.javacodegeeks.nio.large_file_transfer.remote.FileCopyAsyncTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.423 sec - in com.javacodegeeks.nio.large_file_transfer.remote.FileCopyAsyncTest
Running com.javacodegeeks.nio.large_file_transfer.local.FileCopyTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.562 sec - in com.javacodegeeks.nio.large_file_transfer.local.FileCopyTest

7. Summary

In this tutorial, we demonstrated how to transfer a large file from one point to another. This was showcased via a local copy and a remote transfer via sockets. We went one step further and demonstrated transferring a large file from one remote location to another via an asynchronous receiving node.

8. Download the source code

This was a Java NIO Large File Transfer tutorial

Download
You can download the full source code of this example here: Java Nio Large File Transfer

Java Nio Large File Transfer Tutorial

Table Of Contents

1. Introduction

2. Technologies used

2. FileChannel

4. Background

5. Program

5.1. Local copy

5.2. Remote copy

5.2.1. Asynchronous large file transfer

Thank you!

6. Running the program

7. Summary

8. Download the source code

Thank you!

JJ

Thank you!

Table Of Contents

1. Introduction

2. Technologies used

2. FileChannel

4. Background

5. Program

5.1. Local copy

5.2. Remote copy

5.2.1. Asynchronous large file transfer

Thank you!

6. Running the program

7. Summary

8. Download the source code

Thank you!

Related Articles

Thank you!