nio

Java Nio Iterate Over Files in Directory

This example will demonstrate iterating over the files and directories recursively from a root directory. The example program will utilize some of the Java NIO features. Central to this example are the DirectoryStream, FilterPath and Paths classes.
 
 
 
 
  
 
 
 
 

1. Introduction

Before diving into the example program an understanding of the 4 abstractions listed above:

1.1 DirectoryStream

The DirectoryStream interface enables us to iterate over the files and directories within a specific directory. The interface extends Iterable meaning we can use it in a general purpose for loop . While we cannot obtain more than one Iterator when invoking the iterator()method, the hasNext() method is guaranteed to read ahead by at least one element.

This means if it returns truethen an invocation of next()will not throw an IOException. DirectoryStreams are opened when created and closed when invoking the close()method making it a prime candidate for try ... finally idioms or try with resources.

1.2 Filter

A FunctionalInterface type that specifies the criteria under which a directory or file entry should be accepted or filtered out when iterating over the entries for a directory.

1.3 Path

An immutable and thus thread safe abstraction that represents a way to locate a File in a file system. A Path is hierarchial and could contain a root element with many subsequent nested elements and a leaf element representing the name of the file or directory being referred to. (eg: /home/user/Documents). A Path can be registered with a WatchService and watched for changes.

1.4 Paths

The Paths class provides static utility for converting an URI or String to a Path.

2. Technologies used

The example code in this article was built and run using:

  • Java 1.8.101 (1.8.x will do fine)
  • Maven 3.3.9 (3.3.x will do fine)
  • Spring source tool suite 4.6.3 (Any Java IDE would work)
  • Ubuntu 16.04 (Windows, Mac or Linux will do fine)

3. Overview

The example program is a console application that can be run with the java -jarcommand once built using the maven instruction mvn clean install package. It makes use of the Apache commons commnad line parser to parse command line arguments when running the application.

To run the application simply navigate to the project root and execute the following in order:

  1. mvn clean install package
  2. navigate to the “target” folder within the project root and execute the following: java -jar iterate-directory-0.0.1-SNAPSHOT.jar -r <path to root folder> -v -f <filter> where -ris required and -vand -fare optional. The mere presence of -vwill indicate verbose mode and does not require an additional argument whereas -fdoes and will merely filter our any file names that do not contain the literal you supply as argument. For the purposes of this example that was sufficient for demonstrating iteration with filtering, however the filtering can facilitate far more powerful filtering should the need arise.

Examples running the application:

  1. java -jar iterate-directory-0.0.1-SNAPSHOT.jar -r /home/user/temp -v this will list all the directories and files contained within /home/user/temp in verbose mode. This will mean it will include file creation and modification time stamps and file / directory sizes.
  2. java -jar iterate-directory-0.0.1-SNAPSHOT.jar -r /home/user/temp -v -f Test this will list all the directories contained within /home/user/temp in verbose mode. This will also list the files within each of the nested directories that contain the given pattern in their file names.

4. The code

Listing – Responsible for listing the directory contents

public final class Listing {

    private final CLIConfig config;
    private final ConsoleView view;

    public Listing(final CLIConfig config) {
        this.config = config;
        this.view = new ConsoleView(config);
    }

    public void list() {        
        final Path root = Paths.get(this.config.getRoot().orElseThrow(RuntimeException::new));

        if (Files.isDirectory(root)) {
            final Directory rootDir = new Directory(root);

            list(rootDir, this.config.getFilter());
            this.view.render(rootDir);
        } else {
            throw new RuntimeException("Root needs to be a directory");
        }
    }

    private void list(final Directory directory, final Optional<String> filter) {
        assert !Objects.isNull(directory) && !Objects.isNull(filter);

        DirectoryStream stream = null;
        try {
            if (filter.isPresent()) {
                stream = Files.newDirectoryStream(directory.getPath(), createFilter(filter.get()));
            } else {
                stream = Files.newDirectoryStream(directory.getPath());
            }

            for (Path path : stream) {
                if (Files.isDirectory(path)) {
                    final Directory child = new Directory(path);
                    directory.addDirectory(child);

                    list(child, filter);
                } else {
                    directory.addFile(new File(path));
                }
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        } finally {
            if (!Objects.isNull(stream)) {
                try {
                    stream.close();
                } catch (IOException e) {
                    throw new RuntimeException("unable to close stream while listing directories", e);
                }
            }
        }
    }

    private Filter<Path> createFilter(final String pattern) {

        return new Filter<Path>() {

            @Override
            public boolean accept(final Path entry) throws IOException {
                return Files.isDirectory(entry) || entry.toFile().getName().contains(pattern);
            }
        };
    }
}
  • line 12: we call Paths.get(...)passing the parsed command line argument representing the root directory to begin the listing from.
  • line 14: it is paramount that our starting point is a directory and we guard against this
  • lines 29-33: involve the creation of a DirectoryStream. Depending on if there is a filter pattern provided or not we create a Directory by creating and supplying a Filter or not.
  • lines 35-43: we iterate over the Path entries from the DirectoryStream. We check if the Path is a directory Paths.isDirectory(...)and if it is we create our own Directory proxy to encapsulate the specific Path and recursively iterate again. Should it be a file, we create our own File proxy to encapsulate the file and add it to the current Directory proxy we are currently listing.
  • lines 58-67: we create a Filter that matches all directories and any files that contain the given pattern in their names.

ConsoleView – Responsible for rendering the result of the directory listing

final class ConsoleView {

    private final boolean verbose;

    ConsoleView(final CLIConfig config) {
        this.verbose = config.isVerbose();
    }

    void render(final Directory directory) {
        render(directory, StringUtils.EMPTY);
    }

    private void render(final Directory directory, final String padding) {
        assert !Objects.isNull(directory) && !StringUtils.isNotEmpty(padding);

        System.out.println(padding + " d -- " + enrichContent(directory));

        directory.getFileChildren().stream().forEach(file -> render(file, padding + "\t"));
        directory.getDirectoryChildren().stream().forEach(dir -> render(dir, padding + "\t"));
    }

    private void render(final File file, final String padding) {
        assert !Objects.isNull(file) && !StringUtils.isNotEmpty(padding);

        System.out.println(padding + " f -- " + enrichContent(file));
    }

    private String enrichContent(final Directory directory) {
        assert !Objects.isNull(directory);

        try {
            return this.verbose ? directory.getCreated().toString() + " : " + directory.getModified() + " : " + directory.getSize() + " " + directory.getName()
                    : directory.getName();
        } catch (IOException e) {
            return this.verbose ? "E " + directory.getName() : directory.getName();
        }
    }

    private String enrichContent(final File file) {
        assert !Objects.isNull(file);

        try {
            return this.verbose ? file.getCreated() + " : " + file.getModified() + " : " + file.getSize() + " " + file.getName() : file.getName();
        } catch (IOException e) {
            return this.verbose ? "E " + file.getName() : file.getName();
        }
    }
}
  • Trying to conform to an MVC approach, we only depend on the our domain model, Directoryand Filecustom abstractions and help render them via a tree hierarchical view.
  • Whats important to note is that the Path abstractions are wrapped by our custom domain model and thus ultimately pooled into memory prior to writing them to console. This would mean deep listings might prove challenging on memory requirements, but for this example I suspect the concept is illustrated well enough.

FSNode – FileSystemNode encapsulating shared behaviour between File and Directory domain model

public abstract class FSNode {
    
    private final Path path;
    
    protected FSNode(final Path path) {
        Objects.requireNonNull(path);
        
        this.path = path;
    }

    protected final Path getPath() {
        return this.path;
    }

    protected final String getName() {
        return this.path.toFile().getName();
    }

    protected final long getSize() throws IOException {
        return Files.readAttributes(this.path, BasicFileAttributes.class).size();
    }
    
    protected final FileTime getCreated() throws IOException {
        return Files.readAttributes(this.path, BasicFileAttributes.class).creationTime();
    }
    
    protected final FileTime getModified() throws IOException {
        return Files.readAttributes(this.path, BasicFileAttributes.class).lastModifiedTime();
    }
}
  • line 20,24,28: we read the BasicFileAttributes via the Files.readAttributes(...)method on the Files class. This gives us access to attributes such as date created, date modified, size etc.

5. The output

Sample output from running application in non verbose mode

d -- surefire-reports
                                 f -- TEST-com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferIdentityCrisisTest.xml
                                 f -- TEST-com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferMemoryTest.xml
                                 f -- TEST-com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferReadOnlyTest.xml
                                 f -- com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferReadOnlyTest.txt
                                 f -- com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferMemoryTest.txt
                                 f -- TEST-com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferIdentityTest.xml
                                 f -- com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferIdentityCrisisTest.txt
                                 f -- com.javacodegeeks.nio.heapbytebuffer.HeapByteBufferIdentityTest.txt
                         d -- maven-archiver
                                 f -- pom.properties
                         d -- test-classes
                                 d -- com
                                         d -- javacodegeeks
                                                 d -- nio
                                                         d -- heapbytebuffer
                                                                 f -- HeapByteBufferIdentityCrisisTest.class
                                                                 f -- HeapByteBufferReadOnlyTest.class
                                                                 f -- HeapByteBufferIdentityTest.class
                                                                 f -- HeapByteBufferMemoryTest.class
  • dstands for directory and fstands for file
  • the hierarchical structure is preserved on layout to show the directory hierarchy

Sample output from running application in verbose mode

 d -- 2017-06-29T15:18:03Z : 2017-06-29T15:18:03Z : 4096 main
                         d -- 2017-06-29T15:20:31Z : 2017-06-29T15:20:31Z : 4096 java
                                 d -- 2017-06-29T15:20:31Z : 2017-06-29T15:20:31Z : 4096 com
                                         d -- 2017-06-29T15:20:31Z : 2017-06-29T15:20:31Z : 4096 javacodegeeks
                                                 d -- 2017-06-29T15:20:31Z : 2017-06-29T15:20:31Z : 4096 nio
                                                         d -- 2017-07-02T07:37:01Z : 2017-07-02T07:37:01Z : 4096 iterate_directory
                                                                 f -- 2017-07-02T07:40:32Z : 2017-07-02T07:40:32Z : 170 File.java
                                                                 f -- 2017-07-01T11:57:41Z : 2017-07-01T11:57:41Z : 273 Constants.java
                                                                 f -- 2017-07-02T07:38:45Z : 2017-07-02T07:38:45Z : 1083 FSNode.java
                                                                 f -- 2017-07-02T08:36:14Z : 2017-07-02T08:36:14Z : 2548 Listing.java
                                                                 f -- 2017-07-02T07:39:30Z : 2017-07-02T07:39:30Z : 868 Directory.java
                                                                 f -- 2017-07-01T11:57:09Z : 2017-07-01T11:57:09Z : 48 package-info.java
                                                                 f -- 2017-07-02T08:49:04Z : 2017-07-02T08:49:04Z : 1948 ConsoleView.java
                                                                 d -- 2017-07-01T11:56:10Z : 2017-07-01T11:56:10Z : 4096 runner
  • Verbose sample output showing created (first) and modified date (second) with the size of the Path element (directory or file).

6. Summary

In this example we demonstrated how to iterate over the files and directory entries within a specified directory. We covered some Nio abstractions core to this example and working with files and directories in general. These included the DirectoryStream, FilterPath and Paths abstractions.

7. Download the Source Code

This was a Java Nio Iterate Over Files in Directory example.

Download
You can download the full source code of this example here: Java Nio Iterate Over Files in Directory example

JJ

Jean-Jay Vester graduated from the Cape Peninsula University of Technology, Cape Town, in 2001 and has spent most of his career developing Java backend systems for small to large sized companies both sides of the equator. He has an abundance of experience and knowledge in many varied Java frameworks and has also acquired some systems knowledge along the way. Recently he has started developing his JavaScript skill set specifically targeting Angularjs and also bridged that skill to the backend with Nodejs.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments
Back to top button