Irrlicht XML-Writer very slow with long strings?

[Squarefox]

Hello,

I've a problem with exporting several very long std::string with Irrlichts XML-Writer. The strings can go up to 1,000,000 chars in size.
Exporting several of them takes 12 minutes for 1.9 MB resulting XML, which seems extremly slow.

The code looks like this:

cpp Code: Select all

// create File
irr::io::IWriteFile* file = FileSystem->createAndWriteFile(_filename);
         
if (file)
{
        // create XML-Writer
        irr::io::IXMLWriter* xmlWriter = FileSystem->createXMLWriter(file);
        xmlWriter->writeXMLHeader();
        xmlWriter->writeElement(L"FraktalIterations");
 
        for (unsigned int i = 0; i<Fraktal->FraktalIteration.size() ; i++)
                xmlWriter->writeElement(L"Iteration",true,
                        L"Number",irr::core::stringw(i).c_str(),
                        L"String",irr::core::stringw(Fraktal->FraktalIteration.at(i).c_str()).c_str());
               
        xmlWriter->writeClosingTag(L"FraktalIterations");
        // close File
        xmlWriter->drop();
        file->drop();
}

As mentioned, the datatype for Fraktal->FraktalIteration.at(i) is std::string.
Maybe someone has an idea why it's so slow - is this the fault of the XML-Writer or something other?
How could I speed this up?

As a note: I don't think that the datatype conversion from std::string to (const wchar_t* ), which is done by irr::core::stringw(Fraktal->FraktalIteration.at(i).c_str()).c_str(), is the problem.
I tried the same thing without writing it to XML, and it was really fast.

Greetings,
Squarefox

[REDDemon]

tried to profile wich is the function taking most time?

[Squarefox]

Ok,

I got the answer: function void CXMLWriter::writeText(const wchar_t* text) checks each input for special characters and replaces them.
The problem is that a new string is created and each char, whether it's replaced or not, is appended with .append() to that string.
That means that you get O(n) memory allocations for each string you want to write (n ... number of characters in string) in worst case.

I bypassed the check and replaced:

cpp Code: Select all

xmlWriter->writeElement(L"Iteration",true,
        L"Number",irr::core::stringw(i).c_str(),
        L"String",irr::core::stringw(Fraktal->FraktalIteration.at(i).c_str()).c_str());

with:

cpp Code: Select all

file->write(L"\t", sizeof(wchar_t)); // Tabulator
file->write(L"<Iteration Number=\"", 19*sizeof(wchar_t)); // Begin
tempStrW = irr::core::stringw(i);
file->write(tempStrW.c_str(), tempStrW.size()*sizeof(wchar_t)); // Number
                       
file->write(L"\" String=\"", 10*sizeof(wchar_t));
tempStrW = irr::core::stringw(Fraktal->FraktalIteration.at(i).c_str());
file->write(tempStrW.c_str(), tempStrW.size()*sizeof(wchar_t)); // String
 
file->write(L"\" />", 4*sizeof(wchar_t)); // End

The export takes one second now, down from 12 minutes.

Greetings,
Squarefox

[hybrid]

I guess we should pre-alloc the string then, shouldn't hurt in the standard case and give much better performance for huge strings

[CuteAlien]

Hm, seems I had forgotten about this. I just checked in a slight change. Will still not be as fast as writing directly, but it's at least 3-4 times faster.

edit: Grr - faster but still slow. We have to get rid of the new/delete in this function completely.