[fixed]Loading an archive that is embedded in the executable

[cheshirekow]

I have a program that is based on a plugin-like architecture. The base program doesn't require a lot of resources, so I've been thinking about embedding them all into the executable (about 1MB worth).

In order to keep the space down, I thought I would archive my meshes together with their textures into a tar.gz, embed that into the program, and then just point the archive reader to that spot in memory.

Creating a file pointer from the memory is easy:

Code: Select all

IFileSystem* fs = m_device->getFileSystem();
IReadFile* file = fs->createMemoryReadFile(&resources::axes_tar_gz, resources::axes_tar_gz_len, "axes.tar.gz", false);


But there doesn't seem to be an interface to actually read the archive file after that. There only seems to be one function to do that:

Code: Select all

IFileSystem::addFileArchive (const path &filename, bool ignoreCase=true, bool ignorePaths=true, E_FILE_ARCHIVE_TYPE archiveType=EFAT_UNKNOWN, const core::stringc &password="")


I gave figured I would give it a try and started by checking to see if the file system could find the file I added.

Code: Select all

if(fs->existFile("axes.tar.gz"))
    std::cout << "Irrlicht file system seems to see axes.tar.gz" << std::endl;
else
    std::cout << "Irrlicht file system can't find axes.tar.gz" << std::endl;


existFile returns false (i.e "Irrlicht file system can't find axes.tar.gz"). I also tried calling

Code: Select all

fs->setFileListSystem(irr::io::FILESYSTEM_VIRTUAL);


But the result was unchanged.

Is there any way to do what I want, or is this a feature that has not been implemented, and will require a lot of work to fix. I don't mind patching the source if I have to, but I'd prefer going the easy way. So here are the questions:

1. When a memory file is loaded, is it added to the virtual file system?
2. If not, is there a way to manually add it to the file system?
3. If not, how difficult will it be do add that functionality (and where will the modifications need to be? CFileSystem.cpp?
4. If it's too difficult, is there a way to retrieve an IArchiveLoader by EArchiveType? The IArchiveLoader interface seems to be able to read from an IReadFile* so if I can get a pointer to the right IArchiveLoader, then that should provide me with what I need, because I can use the IArchiveLoader directly.

[hybrid]

The .tar.gz file is not added to the file system. Instead, you can ask the archive for a list of files that it contains, or simply one of the files you know to be in the tar archive.

[cheshirekow]

but how do I get the IArchive object. All I have is an IReadFile object.

[cheshirekow]

With no better alternative, I altered the source, adding the following method to the IFileSystem interface and CFileSystem class.

Code: Select all

//  patch by cheshirekow 30 Aug 2010
bool CFileSystem::addFileArchive( IReadFile* file, bool takeOwnership,
                                    bool ignoreCase, bool ignorePaths,
                                    const core::stringc& password )
{
    IFileArchive* archive = 0;

    for(u32 i=0; i < ArchiveLoader.size(); i++)
    {
        if( ArchiveLoader[i]->isALoadableFileFormat(file) )
        {
            file->seek(0);
            archive = ArchiveLoader[i]->createArchive(file, ignoreCase, ignorePaths);
            break;
        }
    }

    if (archive)
    {
        FileArchives.push_back(archive);
        if (password.size())
            archive->Password=password;
        if(takeOwnership)
            file->drop();
        return true;
    }
    else
        return false;
}


It seems to work. If anyone has any feedback it would be greatly appreciated (i.e. "hey dummy, there's a method just like that over here").

[Sudi]

nice addition. i really like it!

[hybrid]

Indeed I thought that we have such a method already, but maybe not. I'll check, meanwhile moving this to bug forum to keep track of it.

[pc0de]

+1 from me. I'm using this patch in a new irrb feature that allows scene, mesh, & texture data to be embedded in iwalktest.

I didn't see it in the patch tracker, so I added it (3110231) as a diff against the trunk revision.

[hybrid]

Yeah, at least this is a thing I'm still owrking on. Problem is that we need to have a way to release this file again, and ressource management becomes pretty difficult here. Many things will have to change for this to properly work.

[pc0de]

Bear with me here - When I mentioned I was embedding scene, mesh, & texture data I meant I was zipping those files into a single .zip file and embedding the generated .zip file into the executable.

When the executable starts up and detects it has an embedded .zip, it then reads the .zip data into memory and creates an IReadFile via IFileSystem->createMemoryReadFile(). The .zip archive is then added via the patched IFileSystem->addFileArchive(IReadFile*, ...).

Without the the patched addFileArchive(), I would have to write the embedded data to a temporary file ("temp.zip") and then add the archive with the existing IFileSystem->addFileArchive("temp.zip", ...). This method eventually creates an IReadFile on "temp.zip" in CArchiveLoaderZIP::createArchive.

Both methods end up with an IReadFile* in CZipReader so I'm not seeing why resource management would be different than the existing method. What am i missing?

[hybrid]

Yes, you're right. This method would work without the additional overhead. However, memory read files and other virtual file system elements would still not be found by many of the file system methods. I started to add all those files to internal tables and change the access scheme for them. This developed into a very large change of the file system after all.

[pc0de]

Ah, okay got it - you're looking at memory readfile usage on a grander scale. Thanks for the clarification.