irrBenchmark (Maybe!)

Discuss about anything related to the Irrlicht Engine, or read announcements about any significant features or usage changes.
Post Reply
DeM0nFiRe
Posts: 117
Joined: Thu Oct 16, 2008 11:59 pm

irrBenchmark (Maybe!)

Post by DeM0nFiRe »

Hey guys, just making this post here about something CuteAlien and I discussed. Currently, I am in a class which is baqsically about how to do software development on a team. For this class, we're going to work on an open source project. I wanted us to work on irrlicht, but most of the class picked a different project. However, there's still a chance the professor will let me and a few other students work on irrlicht anyway.

If we do get he chance to work on irrlicht, it looks like we won't be working on irrlicht directly. CuteAlien mentioned to me that some of the irrlicht developers were looking for a benchmark suite to help them test the performance of irrlicht, and I think that would be a great project to work on.

This thread is here so, hopefully, CuteAlien and hybrid can leave comments abouut what exactly they would like such a tool to do, and of course anyone else can leave comments on it as well. I am not sure yet if my class will be able to work on this, but even if we're not this thread will be here if anyone decides to pick it up in the future.

Whoo! wall of text! Anyway, here are my thoughts so far:

It will have to test rendering performance obviously:
--meshes
----A lot of meshes with a (relatively) small amount of vertices
----A few meshes with a lot of vertices
----A lot of meshes with the same materials
----A lot of meshes with different materials
--Paricles
----A lot of emitters with a few particles
----A few emitters with a lot of particles
--collision

If anyone else has any comments, especially any irrlicht developers, please leave them here!
christianclavet
Posts: 1638
Joined: Mon Apr 30, 2007 3:24 am
Location: Montreal, CANADA
Contact:

Re: irrBenchmark (Maybe!)

Post by christianclavet »

There are also other things you might consider to check for performance:

- Mipmap generation
- Occtree generation
- Raycast and collision detection
- Skinning (render performance with a skinned vs non-skinned object)
- Node tree performance (Irrlicht node tree system) (finding nodes, changing node placement in the hierachym etc) (render and GUI)
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: irrBenchmark (Maybe!)

Post by hendu »

Since rendering something is half testing irr and half testing the driver, you should decide if you want that or to focus on testing irr only.
REDDemon
Developer
Posts: 1044
Joined: Tue Aug 31, 2010 8:06 pm
Location: Genova (Italy)

Re: irrBenchmark (Maybe!)

Post by REDDemon »

A big point in wich performance also matter is HD access time. Would be nice to see a relly complete framework for testing wich archives are faster etc. etc. File throughput, bytes from HD throghput etc.

Maybe we can finally see a cross platform timer in irrlicht with precision and resolution to microseconds.
Junior Irrlicht Developer.
Real value in social networks is not about "increasing" number of followers, but about getting in touch with Amazing people.
- by Me
DeM0nFiRe
Posts: 117
Joined: Thu Oct 16, 2008 11:59 pm

Re: irrBenchmark (Maybe!)

Post by DeM0nFiRe »

Unfortunately, the professor has now said that we won't be able to work on this project. I am incredibly disappointed :/

Well, perhaps a team of irrlicht community members can work on this instead. I, unfortunately, don't have the time to work on this unless it was for the class :(
REDDemon
Developer
Posts: 1044
Joined: Tue Aug 31, 2010 8:06 pm
Location: Genova (Italy)

Re: irrBenchmark (Maybe!)

Post by REDDemon »

would be a nice idea :) for now in my projects I just needed to profile cullingtimeVS rendering time for parameters tuning. nothing more.
Junior Irrlicht Developer.
Real value in social networks is not about "increasing" number of followers, but about getting in touch with Amazing people.
- by Me
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: irrBenchmark (Maybe!)

Post by CuteAlien »

Ok, too bad it didn't happen. But it's still something that would be cool to have, so some ideas in case this ever is getting on someone's project list.

I suppose one of challenges isn't about which tests to write or even the writing of the tests, but writing an infrastructure that makes it easy to add new tests.
I guess it's not so much infrastructure - each test probably has a name, a description, some parameters and a run function. And it needs probably access to the Irrlicht-device. And maybe some functions for output (for example function like OnStart, OnUpdate, OnFinish which could be overloaded). Not sure about results - maybe a common result-format is needed. At least when the tests have run we should afterward have all the information in a format that allows comparing the results.

I suspect tests can have parameters. I would propose to use the Irrlicht attribute system for passing those, as it will make it easy to read the parameters from xml and maybe also write them out together with the results (and it's also flexible enough to allow any kind of parameters).

Probably we also need a bunch of common tool-functions, mostly certainly those to measure time. I've already done some speed-tests in the past, maybe that can give some basic ideas: http://www.michaelzeilfelder.de/irrlich ... IrrStl.zip (the strange defines in there are mostly for comparing 2 types of code while reducing effects like caching, but the Profiler itself might be useful). Also it could be that our "tests" folder in Irrlicht already contains some useful code for this.

And then some application main-screen with buttons for run-tests, watch-last-results, maybe compare-to-other-tests, quit .... anything missing?

Also important is collecting as much information as possible about the environment in which tests are run. Fortunately Irrlicht already has most of that information (in IrrlichtDevice and IOSOperator):
- version of the engine (would be interesting to have even the svn-revision if available).
- operating system
- memory
- graphic-card
- processor.

All test-results should be written in some file. Maybe together with the parameters used for tests as those might change over time.
I would propose again to use the Irrlicht attribute system for that - and write the results out as xml's. That allows easy further processing with other tools.
The result-file should probably have the date as part of the filename so it will produce a new one in each run.

Some functions to compare test-results would be cool (but can be done in a second step as just comparing xml's by hand can be done for a start).
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: irrBenchmark (Maybe!)

Post by hybrid »

Now that there are only minor chances for this to become a project soon, I'll move it to open discussion to re-use this thread for planning phase.
REDDemon
Developer
Posts: 1044
Joined: Tue Aug 31, 2010 8:06 pm
Location: Genova (Italy)

Re: irrBenchmark (Maybe!)

Post by REDDemon »

be sure also to get drivers version for each rendering API so that is possible to compare improvements to performance that comes from updating video drivers.
Junior Irrlicht Developer.
Real value in social networks is not about "increasing" number of followers, but about getting in touch with Amazing people.
- by Me
Post Reply