(Java) Changed directory (via bash script) not reflected in folder.listfiles

Question

I run a bash script from my Java program which takes a chunk of data, manipulates it, and splits it up.

It's not a question of whether the bash script works -- I can see the split files in the directory.

Say the original file was "bigFile" in data/

Then

    try
    {
        Process proc = Runtime.getRuntime().exec("bash " + SCRIPT_DIR + "/" + SPLIT_SCRIPT_NAME + " " + args[_MESSAGES_PER_UPLOAD_] + " " + args[_MAXIMUM_MESSAGES_PER_FEED_] + " " + (60000*Integer.parseInt(args[_DURATION_BEFORE_EACH_UPLOAD_IN_MINUTES_])/Integer.parseInt(args[_DURATION_OF_EACH_FEED_IN_MILLISECONDS_])));
        proc.waitFor();
    }
    catch(IOException e) { error(e); }

    String fileNames;
    File folder = new File(DATA_DIR);

    File[] filesToUpload = folder.listFiles();

    for (int i = 0; i < filesToUpload.length; ++i)
        if (filesToUpload[i].isFile()) 
        {
            fileNames = filesToUpload[i].getName();
            System.out.println(fileNames);
        }

Will print bigFile, not...

$ ls data/

dataChunk_00000 dataChunk_00001 dataChunk_00002 dataChunk_00003 dataChunk_00004 dataChunk_00005 dataChunk_00006 dataChunk_00007 dataChunk_00008 dataChunk_00009 dataChunk_00010 dataChunk_00011 dataChunk_00012 dataChunk_00013 dataChunk_00014 dataChunk_00015 dataChunk_00016 dataChunk_00017 dataChunk_00018 dataChunk_00019 dataChunk_00020 dataChunk_00021 dataChunk_00022 dataChunk_00023 dataChunk_00024 dataChunk_00025 dataChunk_00026 dataChunk_00027

as it should. I'm guessing this is a compiler optimization or something.

Edit: If somebody could explain to me why proc.waitFor() isn't working and/or a better way to solve this, I'd much appreciate it.

Could you share the content of your bash script? Or at least explain what it is doing step-by-step? — Jesse Webb
– Jesse Webb, Commented Jul 18, 2011 at 21:17
As an aside, you should use Runtime.exec(String[]) instead of .exec(String) like you're doing here, for all but the absolute simplest things. Runtime.exec() is not a shell or parser and doesn't honor quoting or escaping. Providing all your arguments in a single string can often lead to unexpected treatment of arguments because exec(String) blindly splits on spaces. — Stephen P
– Stephen P, Commented Jul 18, 2011 at 21:43

Kal · Accepted Answer · 2011-07-18 21:13:53Z

3

The problem with this is not compiler optimization or anything like that.

Its because you are invoking your script with a "bash" in front of it . This causes the process to fork -- so your bash command returns successfully immediately , but your script continues to run in the background and terminate.

The proc.waitFor() has nothing to wait for, the rest of the java program executes before your file has been "split".

answered Jul 18, 2011 at 21:13

Kal

24.9k8 gold badges67 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Ryan Over a year ago

I see. Is there a better workaround to this other than using Thread.sleep()?

Kal Over a year ago

You could get rid of the "bash" and let the script execute directly ( so it doesn't spawn a new process ).

Ryan Over a year ago

It gives me !-- Cannot run program "scripts/splitAndEnumerate.sh": java.io.IOException: error=13, Permission denied

Kal Over a year ago

chmod +x on scripts/splitAndEnumerate.sh

Ryan Over a year ago

Erm, simple fix... chmod. Thanks, Kal.

|

krico · Accepted Answer · 2011-07-18 21:31:02Z

2

You cannot change the directory with java. If you want to "simulate" it, all you need to do is set the property "user.dir".

answered Jul 18, 2011 at 21:31

krico

5,7282 gold badges27 silver badges28 bronze badges

Comments

Jesse Webb · Accepted Answer · 2011-07-18 21:19:31Z

1

I am guessing that your bash script is performing actions asynchronously from its own process/thread. This means that the script finishes executing before the work is complete. This would still pass the waitFor() check and continue executing the code.

EDIT: Kal's answer explains this more clearly, and it was posted first. The problem is the fact that you use the bash command to execute the script.

answered Jul 18, 2011 at 21:19

Jesse Webb

45.6k32 gold badges110 silver badges146 bronze badges

Comments

opyate · Accepted Answer · 2011-07-18 21:28:29Z

I suspect your arguments aren't all passed to your script.

Put all your arguments in an ArrayList instance, pass the instance to the ProcessBuilder, then call the start method on the builder instance, which returns the proc on which you call waitFor.

Here's sample Scala code to show what I mean (I can port it to Java if you're really interested ;-):

import java.lang.{ Process => JProcess, ProcessBuilder => JProcessBuilder }
import java.util.{ArrayList => JArrayList, List => JList, Map => JMap}
import java.io.{InputStreamReader, BufferedReader}

def call(args: String*) = {
    val command: JList[String] = new JArrayList[String]()

    args.foreach {arg =>
      command.add(arg)
    }

    //log.debug("argument list: %s", command.toString)

    val builder = new JProcessBuilder(command)

    val proc: JProcess = builder.start()
    proc.waitFor()

    val read = new BufferedReader(new InputStreamReader(proc.getInputStream()));

    val sb: StringBuffer = new StringBuffer()
    while(read.ready()) {
      sb.append(read.readLine)
    }

    // sb now has the output of the called process...

    val exitValue: Int = proc.exitValue

    // http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/
    read.close
    proc.destroy

    (exitValue, sb.toString) // return it
}

Example call in REPL:

scala> call("date")
res156: (Int, java.lang.String) = (0,Mon 18 Jul 2011 22:29:58 BST)

ddimitrov · Accepted Answer · 2011-07-19 16:10:02Z

There are a number of wrong assumptions with this program:

Every time you do 'exec' you fork a new process, with its own environment, current directory, etc. Any change of the current directory would have been local to that process and will not affect the parent (your Java process). In other words, there is no way to change the current path of an application using a command in a sub-process, there is no Java API for that either - if you really need this, you have to use native call.
The 'cd' command on Unix is a real command, you do not need the shell in order to run it (unlike Windows).
When you fork a process, you need to make sure that you drain the stdout and stderr, or it is going to block when the OS buffer gets full (see next)
Process.waitFor() works. Always.

A better way to approach the problem is to read carefully the File API and as much as possible work with absolute paths. The 'current directory' is something very usefull when you are in shell, but for applications it ends up being more confusing, so the sooner you resolve it to absolute path - the better.

Collectives™ on Stack Overflow

(Java) Changed directory (via bash script) not reflected in folder.listfiles

5 Answers 5

6 Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

6 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related