5

I have a FileInputStream which has 200MB of data. I have to retrieve the bytes from the input stream.

I'm using the below code to convert InputStream into byte array.

private byte[] convertStreamToByteArray(InputStream inputStream) {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    try {
        int i;
        while ((i = inputStream.read()) > 0) {
            bos.write(i);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
    return bos.toByteArray();
}

I'm getting OutOfMemory exception while coverting such a large data to a byte array.

Kindly let me know any possible solutions to convert InputStream to byte array.

Improve this question
10
  • 4
    Why would you need to load 200 MBs in RAM? Any android cell using this app would collapse. Commented Apr 5, 2013 at 4:22
  • 3
    The out of memory error is because you are trying to hold everything in memory at once. Changing the way you are trying to convert it to a byte array isn't going to help; the problem is the idea of converting to a byte array. Commented Apr 5, 2013 at 4:24
  • Actually there is a scenario where I need to send large data to the server. I'm having a solution to handle that part. Kindly let me know if you have any solutions for the above query. Thanks. Commented Apr 5, 2013 at 4:26
  • 1
    You can read a chunk of the file, send it to server and repeat the process until there is nothing more to read. Also, when you read an InputStream, you should use a byte[1024*X] buffer, where X should have a value of 1, 2, 4 or 8. AFAIK 1024*4 is one of the fastest. Commented Apr 5, 2013 at 4:26
  • If you need to send a large file to the server (and your customers are just going to love the data charges for transmitting 200MB!), just write it to the server connection on the fly as you are reading it. Commented Apr 5, 2013 at 4:27

5 Answers 5

Reset to default
10

Why do you want to hold the 200MB file in memory? What are you going to to with the byte array?

If you are going to write it to an OutputStream, get the OutputStream ready first, then read the InputStream a chunk at a time, writing the chunk to the OutputStream as you go. You'll never store more than the chunk in memory.

eg:

     public static void pipe(InputStream is, OutputStream os) throws IOException {

        int read = -1;
        byte[] buf = new byte[1024];

        try {
            while( (read = is.read(buf)) != -1) {
                os.write(buf, 0, read);
            }
        }
        finally {
            is.close();
            os.close();
        }
    }

This code will take two streams and pipe one to the other.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

nice code, but you are still saving all the 200MB in the memory. you should add os.flush() call inside the while loop.
"you are still saving all the 200MB in the memory". Not strictly true - that depends on the underlying implementation of OutputStream. ByteArrayOutputStream will certainly buffer it all in memory (and calling flush() will do nothing), but FileOutputStream will manage its own internal buffering and should be trusted to do as it sees fit. Unnecessarily calling flush() would be second-guessing the implementation and might break any performance gains from internal buffering.
2

Android application has limited Heap Memory and which depend on devices. Currently most of the new devices has 64 but it could be more or less depend on Manufacturer. I have seen device come come with 128 MB heap Memory.

So what this really mean?

Its simply means that regardless of available physical memory your application is not allowed to grow more then allocated heap size.

From Android API level 11 you can request for additional memory by using manifest tag android:largeHeap="true" which will be double your heap size. That simply means if your devices has 64 you will get 128 and in case of 128 you will get 256. But this will not work for lower API version.

I am not exactly sure what is your requirement, but if you planning to send over HTTP then read file send data and read again. You can follow the same procedure for file IO also. Just to make sure not to use memory more then available heap size. Just to be extra cautious make sure you leave some room for application execution.

Improve this answer

Comments

1

Your problem is not about how to convert InputStream to byte array but that the array is to big to fit in memory. You don't have much choice but to find a way to process bytes from InputStream in smaller blocks.

Improve this answer

Comments

0

You'll probably need to massively increase the heap size. Try running your Java virtual machine with the -Xms384m -Xmx384m flag (which specifies a starting and maximum heap size of 384 megabytes, unless I'm wrong). See this for an old version of the available options: depending on the specific virtual machine and platform you may need to do some digging around, but -Xms and -Xmx should get you over that hump.

Now, you probably really SHOULDN'T read it into a byte array, but if that's your application, then...

Improve this answer

3 Comments

Do you have an android cell? Do you know how much available RAM it has after turning it on?
This is Android, which runs the Dalvik VM, not a JVM. The user doesn't get to set startup options like heap size.
Ah..missed the tag: my mistake. Yeah, that's never going to work for him: Android apps generally have heap sizes limited to less than 64 MB, even at the high end. android:largeHeap="true" might get him partway there on very very recent devices (as covered stackoverflow.com/questions/5350465/…) but in general that's just really not possible.
-2

try this code 

private byte[] convertStreamToByteArray(InputStream inputStream) {
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
                    int readByte = 0;
        byte[] buffer = new byte[2024];

                    while(true)
                    {
                        readByte = inputStream.read(buffer);
                        if(readByte == -1)
                        {
                            break;
                        }
                        byteOutStream.write(buffer);
                    }
                    inputStream.close();
                    byteOutStream.flush();
                    byteOutStream.close();
                    byte[] byteArray= byteOutStream.toByteArray();
                    return byteArray;
}

try to read chunk of data from InputStream .

Improve this answer

3 Comments

This is going to be absolutely no help with the out-of-memory error. If anything, it makes the problem a tiny bit worse by using a 2K buffer.
-1 for OuyOfMemory exception in your code. your code should include 'byteOutStream.flush();' inside the while loop or you want to load 200MB to the heap
There is absolutely no need to call flush(). As [user_half] says, this will actually be slightly worse. On top of that, you are also writing the complete buffer to the output stream, even if it's only partially filled. So you are going to corrupt the stream. You should call byteOutStream.write(buffer, 0, readByte); Lastly, readByte isn't actually a byte, it is an int that represents the number of bytes read, so most people call it 'read'. Your name suggests that it is the byte that was read, rather than the count.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.