irrBenchmark (Maybe!)

[DeM0nFiRe]

Hey guys, just making this post here about something CuteAlien and I discussed. Currently, I am in a class which is baqsically about how to do software development on a team. For this class, we're going to work on an open source project. I wanted us to work on irrlicht, but most of the class picked a different project. However, there's still a chance the professor will let me and a few other students work on irrlicht anyway.

If we do get he chance to work on irrlicht, it looks like we won't be working on irrlicht directly. CuteAlien mentioned to me that some of the irrlicht developers were looking for a benchmark suite to help them test the performance of irrlicht, and I think that would be a great project to work on.

This thread is here so, hopefully, CuteAlien and hybrid can leave comments abouut what exactly they would like such a tool to do, and of course anyone else can leave comments on it as well. I am not sure yet if my class will be able to work on this, but even if we're not this thread will be here if anyone decides to pick it up in the future.

Whoo! wall of text! Anyway, here are my thoughts so far:

It will have to test rendering performance obviously:
--meshes
----A lot of meshes with a (relatively) small amount of vertices
----A few meshes with a lot of vertices
----A lot of meshes with the same materials
----A lot of meshes with different materials
--Paricles
----A lot of emitters with a few particles
----A few emitters with a lot of particles
--collision

If anyone else has any comments, especially any irrlicht developers, please leave them here!

[christianclavet]

There are also other things you might consider to check for performance:

- Mipmap generation
- Occtree generation
- Raycast and collision detection
- Skinning (render performance with a skinned vs non-skinned object)
- Node tree performance (Irrlicht node tree system) (finding nodes, changing node placement in the hierachym etc) (render and GUI)

[hendu]

Since rendering something is half testing irr and half testing the driver, you should decide if you want that or to focus on testing irr only.

[REDDemon]

A big point in wich performance also matter is HD access time. Would be nice to see a relly complete framework for testing wich archives are faster etc. etc. File throughput, bytes from HD throghput etc.

Maybe we can finally see a cross platform timer in irrlicht with precision and resolution to microseconds.

[DeM0nFiRe]

Unfortunately, the professor has now said that we won't be able to work on this project. I am incredibly disappointed :/

Well, perhaps a team of irrlicht community members can work on this instead. I, unfortunately, don't have the time to work on this unless it was for the class

[REDDemon]

would be a nice idea for now in my projects I just needed to profile cullingtimeVS rendering time for parameters tuning. nothing more.

[CuteAlien]

Ok, too bad it didn't happen. But it's still something that would be cool to have, so some ideas in case this ever is getting on someone's project list.

I suppose one of challenges isn't about which tests to write or even the writing of the tests, but writing an infrastructure that makes it easy to add new tests.
I guess it's not so much infrastructure - each test probably has a name, a description, some parameters and a run function. And it needs probably access to the Irrlicht-device. And maybe some functions for output (for example function like OnStart, OnUpdate, OnFinish which could be overloaded). Not sure about results - maybe a common result-format is needed. At least when the tests have run we should afterward have all the information in a format that allows comparing the results.

I suspect tests can have parameters. I would propose to use the Irrlicht attribute system for passing those, as it will make it easy to read the parameters from xml and maybe also write them out together with the results (and it's also flexible enough to allow any kind of parameters).

Probably we also need a bunch of common tool-functions, mostly certainly those to measure time. I've already done some speed-tests in the past, maybe that can give some basic ideas: http://www.michaelzeilfelder.de/irrlich ... IrrStl.zip (the strange defines in there are mostly for comparing 2 types of code while reducing effects like caching, but the Profiler itself might be useful). Also it could be that our "tests" folder in Irrlicht already contains some useful code for this.

And then some application main-screen with buttons for run-tests, watch-last-results, maybe compare-to-other-tests, quit .... anything missing?

Also important is collecting as much information as possible about the environment in which tests are run. Fortunately Irrlicht already has most of that information (in IrrlichtDevice and IOSOperator):
- version of the engine (would be interesting to have even the svn-revision if available).
- operating system
- memory
- graphic-card
- processor.

All test-results should be written in some file. Maybe together with the parameters used for tests as those might change over time.
I would propose again to use the Irrlicht attribute system for that - and write the results out as xml's. That allows easy further processing with other tools.
The result-file should probably have the date as part of the filename so it will produce a new one in each run.

Some functions to compare test-results would be cool (but can be done in a second step as just comparing xml's by hand can be done for a start).

[hybrid]

Now that there are only minor chances for this to become a project soon, I'll move it to open discussion to re-use this thread for planning phase.

[REDDemon]

be sure also to get drivers version for each rendering API so that is possible to compare improvements to performance that comes from updating video drivers.