Load files bigger than 1M from assets folder

I'm going crazy, I created a file object, so it can be read with ObjectInputStream, and I placed the assets folder. The method works with a file smaller than 1M, and give error with larger files. I read that is a limit of Android platform, but I also know that can be "easily" avoided. Those who have downloaded the game Reging Thunder, for example, can easily see that 开发者_C百科in their assets folder is a file 18.9M large. This is my code for read 1 object from a ObjecInputStream

File f = File.createTempFile("mytempfile", "dat");
FileOutputStream fos = new FileOutputStream(f);

InputStream is = mc.getAssets().open(path,3);

ObjectInputStream ois=new ObjectInputStream(is);
byte[] data = (byte[]) ois.readObject();
fos.write(data);

fos.flush();
fos.close();
ois.close();
is.close();

now I have an uncompressed file and I can use it without worrying about the error "This file can not be opened as a file descriptor; it is probably compressed"

This function works well with files smaller than 1M, with bigger files return an java.io.IOException on line "ObjectInputStream ois=new ObjectInputStream(is);"

why??

Faced the same issue. I've cut up my 4MB file into 1 MB chunks, and on the first run I join the chunks into a data folder on the phone. As an added bonus, the APK is properly compressed. The chunk files are called 1.db, 2.db, etc. The code goes like this:

File Path = Ctxt.getDir("Data", 0);
File DBFile = new File(Path, "database.db");

if(!DBFile.exists() || DatabaseNeedsUpgrade)  //Need to copy...
    CopyDatabase(Ctxt, DBFile);


static private void CopyDatabase(Context Ctxt, File DBFile) throws IOException
{
    AssetManager assets = Ctxt.getAssets();
    OutputStream outstream = new FileOutputStream(DBFile);
    DBFile.createNewFile();
    byte []b = new byte[1024];
    int i, r;
    String []assetfiles = assets.list("");
    Arrays.sort(assetfiles);
    for(i=1;i<10;i++) //I have definitely less than 10 files; you might have more
    {
        String partname = String.format("%d.db", i);
        if(Arrays.binarySearch(assetfiles, partname) < 0) //No such file in assets - time to quit the loop
            break;
        InputStream instream = assets.open(partname);
        while((r = instream.read(b)) != -1)
            outstream.write(b, 0, r);
        instream.close();
    }
    outstream.close();
}

The limitation is on compressed assets. If the asset is uncompressed, the system can memory-map the file data and use the Linux virtual memory paging system to pull in or discard 4K chunks as appropriate. (The "zipalign" tool ensures that uncompressed assets are word-aligned in the file, which means they'll also be aligned in memory when directly mapped.)

If the asset is compressed, the system has to uncompress the entire thing to memory. If you have a 20MB asset, that means 20MB of physical memory is tied up by your application.

Ideally the system would employ some sort of windowed compression, so that only parts need to be present, but that requires some fanciness in the asset API and a compression scheme that works well with random access. Right now APK == Zip with "deflate" compression, so that's not practical.

You can keep your assets uncompressed by giving them a suffix of a file type that doesn't get compressed (e.g. ".png" or ".mp3"). You may also be able to add them manually during the build process with "zip -0" instead of having them bundled up by aapt. This will likely increase the size of your APK.

Like Seva suggested you can split up your file in chunks. I used this to split up my 4MB file

public static void main(String[] args) throws Exception {  
    String base = "tracks";  
    String ext = ".dat";  
    int split = 1024 * 1024;  
    byte[] buf = new byte[1024];  
    int chunkNo = 1;  
    File inFile = new File(base + ext);  
    FileInputStream fis = new FileInputStream(inFile);  
    while (true) {  
      FileOutputStream fos = new FileOutputStream(new File(base + chunkNo + ext));  
      for (int i = 0; i < split / buf.length; i++) {  
        int read = fis.read(buf);  
        fos.write(buf, 0, read);  
        if (read < buf.length) {  
          fis.close();  
          fos.close();  
          return;  
        }  
      }  
      fos.close();  
      chunkNo++;  
    }  
  }  

If you do not need to combine the files into a single file on the device again, just use this InputStream, which combines them into one on the fly.

import java.io.IOException;  
import java.io.InputStream;  

import android.content.res.AssetManager;  

public class SplitFileInputStream extends InputStream {  

  private String baseName;  
  private String ext;  
  private AssetManager am;  
  private int numberOfChunks;  
  private int currentChunk = 1;  
  private InputStream currentIs = null;  

  public SplitFileInputStream(String baseName, String ext, int numberOfChunks, AssetManager am) throws IOException {  
    this.baseName = baseName;  
    this.am = am;  
    this.numberOfChunks = numberOfChunks;  
    this.ext = ext;  
    currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);  
  }  

  @Override  
  public int read() throws IOException {  
    int read = currentIs.read();  
    if (read == -1 && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      return read();  
    }  
    return read;  
  }  

  @Override  
  public int available() throws IOException {  
    return currentIs.available();  
  }  

  @Override  
  public void close() throws IOException {  
    currentIs.close();  
  }  

  @Override  
  public void mark(int readlimit) {  
    throw new UnsupportedOperationException();  
  }  

  @Override  
  public boolean markSupported() {  
    return false;  
  }  

  @Override  
  public int read(byte[] b, int offset, int length) throws IOException {  
    int read = currentIs.read(b, offset, length);  
    if (read < length && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      read += read(b, offset + read, length - read);  
    }  
    return read;  
  }  

  @Override  
  public int read(byte[] b) throws IOException {  
    return read(b, 0, b.length);  
  }  

  @Override  
  public synchronized void reset() throws IOException {  
    if (currentChunk == 1) {  
      currentIs.reset();  
    } else {  
      currentIs.close();  
      currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      currentChunk = 1;  
    }  
  }  

  @Override  
  public long skip(long n) throws IOException {  
    long skipped = currentIs.skip(n);  
    if (skipped < n && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      skipped += skip(n - skipped);  
    }  
    return skipped;  
  }  
}

Usage:
ObjectInputStream ois = new ObjectInputStream(new SplitFileInputStream("mytempfile", ".dat", 4, getAssets()));

Change the file extension to .mp3

I know this is an old question but I thought of a good solution. Why not store the file already pre-zipped in the asset folder. Then since it is already a zip file and therefore compressed it won't need to be compressed again. So if you wanted the file to be compressed to decrease the size of your apk but you don't want to deal with splitting up files I think this is easier.

When you need to read that file off the device just wrap the inputstream in a zipinputstream http://developer.android.com/reference/java/util/zip/ZipInputStream.html

I found another solution, maybe you're interested in it.

In the root of your sources, where you have the build.xml file, you can overwrite the -package-resources target in the custom_rules.xml file, which is used for adding/modifying targets in ant without breaking anything in the standard android app build system.

Just create a file with this content:

<?xml version="1.0" encoding="UTF-8"?>
<project name="yourAppHere" default="help">

    <target name="-package-resources" depends="-crunch">
        <!-- only package resources if *not* a library project -->
        <do-only-if-not-library elseText="Library project: do not package resources..." >
            <aapt executable="${aapt}"
                    command="package"
                    versioncode="${version.code}"
                    versionname="${version.name}"
                    debug="${build.is.packaging.debug}"
                    manifest="${out.manifest.abs.file}"
                    assets="${asset.absolute.dir}"
                    androidjar="${project.target.android.jar}"
                    apkfolder="${out.absolute.dir}"
                    nocrunch="${build.packaging.nocrunch}"
                    resourcefilename="${resource.package.file.name}"
                    resourcefilter="${aapt.resource.filter}"
                    libraryResFolderPathRefid="project.library.res.folder.path"
                    libraryPackagesRefid="project.library.packages"
                    libraryRFileRefid="project.library.bin.r.file.path"
                    previousBuildType="${build.last.target}"
                    buildType="${build.target}"
                    ignoreAssets="${aapt.ignore.assets}">
                <res path="${out.res.absolute.dir}" />
                <res path="${resource.absolute.dir}" />
                <nocompress /> <!-- forces no compression on any files in assets or res/raw -->
                <!-- <nocompress extension="xml" /> forces no compression on specific file extensions in assets and res/raw -->
            </aapt>
        </do-only-if-not-library>
    </target>
</project>

add file extension is mp3.I use mydb.mp3in assets folder and copy .this run without error.show check it.

Using GZIP would be another method. you only need to wrap InputStream inside GZIPInputStream.

I used this for a database which size about 3.0 MB and output compress file was about 600KB.

• For copying DB in firs run, I gzipped my source .db file using GZIP tool.
• Then renamed it to .jpg in order to avoid more compression (these processes are done before compile APK FILE).
• Then for reading compressed GZIP file from assetss

and copying it:

private void copydatabase() throws IOException {
        // Open your local db as the input stream
        InputStream myinput = mContext.getAssets().open(DB_NAME_ASSET);
        BufferedInputStream buffStream = new BufferedInputStream(myinput);
        GZIPInputStream zis = new GZIPInputStream(buffStream);

        // Path to the just created empty db
        String outfilename = DB_PATH + DB_NAME;

        // Open the empty db as the output stream
        OutputStream myoutput = new FileOutputStream(outfilename);


        // transfer byte to inputfile to outputfile
        byte[] buffer = new byte[1024];
        int length;
        while ((length = zis.read(buffer)) > 0) {
            myoutput.write(buffer, 0, length);
        }

        // Close the streams
        myoutput.flush();
        myoutput.close();
        zis.close();
        buffStream.close();
        myinput.close();
    }

set database to multiple part with a program e.g "Win Hex", you can download from Link

and continue Load files bigger than 1M from assets folder

Instead of assets folder, I have put my large files in the raw folder. It works for me.

I use NetBeans to build the package and I not found how to change the settings of AAPT. I have not tried the png, but the mp3 are compressed. I can compile the package and then enter the assets folder with the parameter -0? what would be the correct command to use?