RSS Feed for This PostCurrent Article

Develop a Java File Watcher

Download Source Code

These are some utlities classes that I wrote some time back to monitor file creation, deletion, and modification in a specified directory.The DirectoryWatcher implements AbstractResourceWatcher to monitor a specified directory. It accepts a ResourceListener class, which is FileListener in thi case. Whenever DirectoryWatcher detects any changes, it will raise the onAdd, onChange, or onDelete events in the listener class, passing in the necessary File object.It is quite straightforward, but is useful when you are developing Java backend ETL processes.

public class DirectoryWatcher extends AbstractResourceWatcher {
/**
* The current map of files and their timestamps (String fileName => Long
* lastMod)
*/
private Map currentFiles = new HashMap();

/**
* The directory to watch.
*/
private String directory;

/**
* The map of last recorded files and their timestamps (String fileName =>
* Long lastMod)
*/
private Map prevFiles = new HashMap();

/**
* Constructor that takes the directory to watch.
*
* @param directoryPath the directory to watch
* @param intervalSeconds The interval to use when monitoring this
* directory. I.e., ever x seconds, check this directory to see
* what has changed.
* @throws IllegalArgumentException if the argument does not map to a
* valid directory
*/
public DirectoryWatcher(String directoryPath, int intervalSeconds)
throws IllegalArgumentException {

//Get the common thread interval stuff set up.
super(intervalSeconds, directoryPath + ” interval watcher.”);

//Check that it is indeed a directory.
File theDirectory = new File(directoryPath);

if (theDirectory != null && !theDirectory.isDirectory()) {

//This is bad, so let the caller know
String message = “The path ” + directory +
” does not represent a valid directory.”;
throw new IllegalArgumentException(message);

}

//Else all is well so set this directory and the interval
this.directory = directoryPath;

}

/**
* For testing only.
*
* @param args
*/
public static void main(String[] args) {
// Monitor c:/temp every 5 seconds
DirectoryWatcher dw = new DirectoryWatcher(“c:/temp/”, 5);
dw.addListener(new FileListener());
dw.start();
}

/**
* Start the monitoring of this directory.
*/
public void start() {

//Since we’re going to start monitoring, we want to take a snapshot of the
//current directory to we have something to refer to when stuff changes.
takeSnapshot();

//And start the thread on the given interval
super.start();

//And notify the listeners that monitoring has started
File theDirectory = new File(directory);
monitoringStarted(theDirectory);
}

/**
* Stop the monitoring of this directory.
*/
public void stop() {

//And start the thread on the given interval
super.stop();

//And notify the listeners that monitoring has started
File theDirectory = new File(directory);
monitoringStopped(theDirectory);
}

/**
* Store the file names and the last modified timestamps of all the files
* and directories that exist in the directory at this moment.
*/
private void takeSnapshot() {

//Set the last recorded snap shot to be the current list
prevFiles.clear();
prevFiles.putAll(currentFiles);

//And get a new current state with all the files and directories
currentFiles.clear();

File theDirectory = new File(directory);
File[] children = theDirectory.listFiles();

//Store all the current files and their timestamps
for (int i = 0; i < children.length; i++) {

File file = children[i];
currentFiles.put(file.getAbsolutePath(),
new Long(file.lastModified()));

}

}

/**
* Check this directory for any changes and fire the proper events.
*/
protected void doInterval() {

//Take a snapshot of the current state of the dir for comparisons
takeSnapshot();

//Iterate through the map of current files and compare
//them for differences etc…
Iterator currentIt = currentFiles.keySet().iterator();

while (currentIt.hasNext()) {

String fileName = (String) currentIt.next();
Long lastModified = (Long) currentFiles.get(fileName);

//If this file did not exist before, but it does now, then
//it’s been added
if (!prevFiles.containsKey(fileName)) {
//DirectorySnapshot.addFile(fileName);
resourceAdded(new File(fileName));
}
//If this file did exist before
else if (prevFiles.containsKey(fileName)) {

Long prevModified = (Long) prevFiles.get(fileName);

//If this file existed before and has been modified
if (prevModified.compareTo(lastModified) != 0) {
// 27 June 2006
// Need to check if the file are removed and added
// during the interval
/* if (!DirectorySnapshot.containsFile(fileName)) {
resourceAdded(new File(fileName));
} else {*/
resourceChanged(new File(fileName));
//}
}
}
}

//Now we need to iterate through the list of previous files and
//see if any that existed before don’t exist anymore
Iterator prevIt = prevFiles.keySet().iterator();

while (prevIt.hasNext()) {

String fileName = (String) prevIt.next();

//If this file did exist before, but it does not now, then
//it’s been deleted
if (!currentFiles.containsKey(fileName)) {
//DirectorySnapshot.removeFile(fileName);
resourceDeleted(fileName);
}
}
}
}

public class FileListener extends BaseListener implements IFileListener {

/**
* Connstructor
*/
public FileListener() {
super();
}

public void onStart(Object monitoredResource) {
// On startup
if (monitoredResource instanceof File) {
File resource = (File) monitoredResource;
if (resource.isDirectory()) {

System.out.println(“Start to monitor ” + resource.getAbsolutePath());

/*File[] files = resource.listFiles();
for (int i = 0; i < files.length; i++) {
File f = (File) files[i];
onAdd(f);
}*/
}
}
}

public void onStop(Object notMonitoredResource) {

}

public void onAdd(Object newResource) {
if (newResource instanceof File) {
File file = (File) newResource;
if (file.isFile()) {
System.out.println(file.getAbsolutePath() + ” is added”);
}
}
}

public void onChange(Object changedResource) {
if (changedResource instanceof File) {
File file = (File) changedResource;
if (file.isFile()) {
System.out.println(file.getAbsolutePath() + ” is changed”);
}

}
}

public void onDelete(Object deletedResource) {
if (deletedResource instanceof String) {
String deletedFile = (String) deletedResource;
System.out.println(deletedFile + ” is deleted”);
}
}
}


Trackback URL


RSS Feed for This Post34 Comment(s)

  1. Jason Daggs | Oct 2, 2007 | Reply

    This was the initial approach I tried on several occasions. However, one problem I ran into was that even though the file name shows up in the directory list the file itself may not actually exist on the file system. Let me explain. If I copy a large file from a remote network source to the local directory being monitored, that file will still show up in the directory listing, but before the network copy has completed. If I try to do almost anything non trivial to the file at that moment like move it to another directory or open it for writing, an exception will be thrown because really the file is not yet completely there and the OS still has a write lock on it. So, what I did was perform a test on each new file before I raised the “file ready” event. I opened up a FileIputStream on the file if a FileNotFoundException is thrown then sleep a few seconds and then keep trying( bit forever but you get the picture) If no exception is thrown the the file is ready.(tested on windows not yet on linux) This approach will not solve all issues of course, but did allow us to implement a very robust approach to file system monitoring. Of course, IMHO avoiding directory monitoring whenever possible would be my preferred approach.

  2. Rob Di Marco | Oct 2, 2007 | Reply

    Another implementation is available using the Java Native Access API. You can see an example FileMonitor at:

    https://jna.dev.java.net/source/browse/
    jna/trunk/jnalib/src/com/sun/jna/examples/
    FileMonitor.java?rev=HEAD&view=markup

  3. Tim Wall | Oct 2, 2007 | Reply

    The polling approach has a number of drawbacks that rapidly become apparent as the number of ‘watched’ files increases. It’s better to rely on an OS-provided service for this sort of thing.

    You’ll find that the underlying ‘stat’ triggered by many of java.io.File methods can quickly become a performance bottleneck.

    So polling is not a bad solution if you’re looking at one or two files, but I wouldn’t recommend it for an IDE tracking external changes to a project’s files.

  4. thoughtworks | Oct 2, 2007 | Reply

    Jason,

    I faced the some problem as what you encountered also. I resolved it using 2 ways.

    1. 1st way is similar to what you did, I monitor the file size/modified date and if the file size/modified date are not the same during an interval, then I will keep on waiting.

    2. Another way to resolve the large file issue is to monitor on dummy file with empty content. E.g. copying a large file to directory A and once it is done, create a dummy file in directory B, and monitoring is only only directory B. Of course this depends on the program pushing the file to create another dummy file once the upload/copying is completed.

    Rob,
    I need it on Unix/Linux platform. Let me know if you have a similiar solution on Unix/Linux platform.

    Thanks

  5. Sander | Oct 2, 2007 | Reply

    Why are you calling yourself ThoughtWorks? Do you have anything to do with http://www.thoughtworks.com?

  6. timber | Oct 3, 2007 | Reply

    There is always this project on sourceforge. Not used it though.

    http://jpoller.sourceforge.net/

  7. GuidoLx | Oct 3, 2007 | Reply

    There is just one thing I don’t like in your design.
    Whenever you are doing file-related operations, use the java.io.File object and do not use a java.lang.String!
    My 2c,

    Guido

  8. Alex Miller | Oct 3, 2007 | Reply

    BTW, support for file (or directory) watching is included in JSR 203 (the new new Java I/O API) which should be part of Java 7. More info on my Java 7 page here.

  9. sudhakar | Apr 22, 2008 | Reply

    this DirectoryWatcher applicaton is almost similar to our requirements.with this code it shows only modified or added or deleted filenames when ever you updated the directory. but in our requirement, basically directory contains number of log files and not only monitering the directory i need to parse the content of a modified or new log file and pick the required information from that log file.

    can anybody help me out please? this is urgent requirement…

  10. kyle | May 2, 2008 | Reply

    Instead of a directory, I want to watch a table inside a database for entries, and the call the appropriate methods to handle the records. Basically the requirement is:
    if the timestamp on a particular row is greater then 4 minutes, meaning its not been processed, then it should send a message to group list notifying them of this condition.

  11. vamsi krishna | Jul 16, 2008 | Reply

    hi thanks for this code i am looking for the same functionality in java.

  12. Oljas | Oct 7, 2008 | Reply

    Hi everyone.
    I try to use this program.
    I run it and open TaskManager on XP too.
    So, the resource size this program is using is growing up, each second(or 2-3 seconds) its growing for some bytes … thats because i can’t use it(after some time my computer won’t have any resource to use it). Is there any way to solve this program?

  13. Nitin Gupta | Nov 24, 2008 | Reply

    Hi, Can you please explain me how can i use this code?

    Thanks
    Nitin

  14. Suman K | Dec 9, 2008 | Reply

    Oljas,

    Add the following in the end of the takeSnapshot() function:

    Runtime runtime = Runtime.getRuntime();
    runtime.gc();

    This should solve the memory leak problem.

  15. sam | Jun 5, 2009 | Reply

    Hi,

    nice code, thx!

    However, I was wondering if the java.io.File declaration defining the watched directory could actually watch a “remote” directory (on a different server) or not? If yes, what would then be the syntax of the directory pathname (\\x.x.x.x\…)?

    Thx!

  16. debol | Nov 23, 2009 | Reply

    http://www2.hawaii.edu/~qzhang/FileSystemWatcher.html

  17. Uwe Pachler | Mar 20, 2010 | Reply

    jpatchwatch is a Java library for monitoring directories for changes. It uses the host platform’s native OS functions to achieve this to avoid polling.

    Currently the following platforms are supported natively:

    * Windows (Windows 2000, XP, Vista, 7)
    * Linux
    * Mac OS X (x86, 10.6)
    * FreeBSD (x86)

    Have a look at http://jpathwatch.wordpress.com for details.

    Cheers,

    Uwe

  18. Seph | Apr 9, 2010 | Reply

    @Uwe

    Will there be support for Solaris also?

  19. novuss | May 12, 2010 | Reply

    sounds very interesting…i try to use this code!

  20. Bob | May 13, 2010 | Reply

    Interesting post!
    I like it!

  21. Lederpuschen | Jun 3, 2010 | Reply

    Thanks for your post and the helpfull informations. I will see if I can work with your code. Maybe I will come back to you and ask some stupid questions… ;o)

  22. Uwe Pachler | Jun 9, 2010 | Reply

    @Seph
    Yes, I’m currently working on a port for OpenSolaris. However, I have no Sparc machine that I can test on, so it’ll be x86 only until someone volunteers *hint*

  23. Diesel | Jun 30, 2010 | Reply

    12 minute 50 question iq test
    alvin glenn detention center columbia south carolina
    1999 taurus problems firing order 24 valve
    all time nfl rushers
    allen roth closets
    25 tons is ow many pounds
    17 caliber ballistics
    18 foot mobil home manufacture
    22 lr ammo reviews
    actrices pornograficas mexicanas
    285 75 16 on 16×8 tundra
    380 caliber handguns for sale
    annasophia robb in a bathing suit
    30 milliliters is how many ounces
    2002 nickel
    anyone shop lazata
    308 at 1000 yards
    50 bmg ballistic chart
    ac97 audio c device driver media
    1980 hair style feathered

  24. david | Aug 10, 2010 | Reply

    Hi,

    Due the lack of OS polling mechanism on Solaris. I’ve try to use this DirectoryWatcher app. At least for me it didn’t worked very well, despite some changes I made in the code, the Thread mechanism is doesn’t adapt when you use thousands of directories to watch.

    Maybe a meachanism to watch subdirectories using the same thread would prevent the “DoS”.

    Cheers ,

    David

  25. vignesh | Aug 11, 2010 | Reply

    the code u written was very easy to understand,Thanks for ur code, with explanation for every line

  26. Secure Wireless Internet | Oct 4, 2010 | Reply

    This website is great — the codes are excellent. Thanks for sharing this information. However, I was reading everyones comments relating to this code in particular — it seems that its not the best mechanism to adapt to thousands of directories.

  27. JinwoodKrabbelpuschen | Nov 11, 2010 | Reply

    Thanks for this post. I will have to try this out, glad to see that you`ll help here, so I can ask if something goes wrong. Nice!
    Andre J. Krabbelpuschen

  28. ataaw | Nov 14, 2010 | Reply

    great blog, can i make friend link with you please!Thx

  29. dean | Feb 14, 2011 | Reply

    hi,I am trying to implement this program in servlet can you suggest me how it is possible because i am trying to call the methods and the problem is everytime it throws exception …………waiting for ur reply.

  30. sravan | Mar 10, 2011 | Reply

    Hi ur code is fully match my requirement…thx first of all..my question is if i add a new folder then why it is not detecting?? but if i delete that new folder it is detecing fine…wait for ur reply..

  31. sravan | Mar 10, 2011 | Reply

    k i got the solution after adding below code in FileListener…
    if (file.isDirectory()) { System.out.println(file.getAbsolutePath() + ” is added”);
    }
    Now can detect newly added directories also…

  32. Alessandro | Jan 4, 2012 | Reply

    Hi,
    what about a large file? there is a way to handle the onComplete event??

  33. Satan | Jan 9, 2012 | Reply

    Why waste time on this old school crap ! Look into Spring Batch it’s kicks butt. Im surprised this code is still getting traffic. !!!! Suckah born every minute.

  34. เรียนหนังสือ | Sep 7, 2013 | Reply

    Have you ever thought about publishing an e-book or guest authoring
    on other blogs? I have a blog centered on the same information you
    discuss and would love to have you share some
    stories/information. I know my visitors would enjoy
    your work. If you are even remotely interested, feel free to shoot me an e-mail.

2 Trackback(s)

  1. From iLenceel» Blog 存档 » links for 2007-10-09 | Nov 28, 2007
  2. From Technical Related Notes » Blog Archive » links for 2010-09-27 | Apr 18, 2011

RSS Feed for This PostPost a Comment

*