Irrlicht unit tests

You discovered a bug in the engine, and you are sure that it is not a problem of your code? Just post it in here. Please read the bug posting guidelines first.
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Irrlicht unit tests

Post by CuteAlien »

This thread is for problems with the Irrlicht unit tests. The tests can be found in the tests folder and are generally there to prevent accidentally breaking working functionality. They should in generally work on all systems and this thread has the target of getting us there.

Anyone can help by running the tests once in a while on his own system and giving feedback about the results. Tests must be run from within the tests folder, which means that on Linux you have to copy the tests binary after running the Makefile to the tests folder before starting it. The results are then written to tests.log. When there had been failing tests it helps us if the corresponding tests.log is posted as well, as it contains additional information. Post it for example to a pastebin side or put it on some server.
- When you post test-results please also give your configuration: OS, Graphic-Card, Compiler, 32-bit or 64-bit (system as well as target), Irrlicht version (svn revision), debug/release
- Please update your graphic card drivers to the newest version.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Irrlicht unit tests

Post by CuteAlien »

Currently many tests are failing (15 today on Linux gcc 32 bit, more or less on Windows and/or 64-bit platforms).
I could fix a few over the last months, but spend today a few hours on figuring out _when_ stuff got broken.
Unfortunately results were somewhat ambigious as it seems the tests broke on different revision on Windows and Linux.
Additionally there is the problem that on my Linux system the majority of tests broke between some revisions which fails completely to work. Still I hope it brings us again a little closer to finding those problems, so here my results.

I couldn't really test revisions between 3729 and 3755 as those failed on my Linux system completely (meaning I could only outcomment them to finish at all).
The tests in this table are those failing in the current revision (as of this morning) r4137.
ok = worked.
f = failed
- = test didn't exist yet
crash = test crashed
// = had to be outcommented to run tests through (usually because of asserts)

Some revisions which seem to be obviously missing here couldn't be tested because Irrlicht didn't compile (3400, 3650, 3700).
Also mosty some more tests needed to be outcommented to run through on Linux at all

Code: Select all

 
SVN Revision                3089     3200     3500      3600      3675      3725    3726/VS32      3728    3729/VS32     3756     3762     3800     4137    4137/VS32
'drawPixel'                  ok        ok       ok        ok        ok        ok        f            ok        f            f        f        f        f        f
'material'                    -         -        -         -        ok        ok        f            ok        f            f        f        f        f        f
'renderTargetTexture'        ok        ok        f         f         f         f        f            f         f            f        f        f        f        f
'textureFeatures'            ok        ok       ok        ok        ok        ok        ok           ok        ok          ok        f        f        f        f
'textureRenderStates'        ok        ok        f        ok        ok        ok        ok           ok        f            f        f        f        f        f
'transparentMaterials'                                                                  ok                     f                                                f
'antiAliasing'                -        ok        f         f         f         f        f            f         f            f        f        f        f        f
'draw2DImage'                 -        ok       ok        ok        ok        ok        f            ok        f            f        f        f        f        f
'lights'                      -         -       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'twodmaterial'                -         -        f        ok        ok        ok        f            ok        f            f        f        f        f        f
'viewPort'                    -         -       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'mrt'                         -         -    crash        //     crash        ok        ok           ok        ok           f        f        f        f        ok
'projectionMatrix'           //        //       //        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'lightMaps'                  ok        ok       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'triangleSelector'            -         -        -         -        -          -        ok           -         -            -        -        -        f        ok
 
Certainly more tests are needed to find where exactly it breaks - this is just how far I got today (I had hoped they would break on same revisions on Windows-Linux, only found out in the evening that this wasn't the case).

I'm not sure yet if I continue with that table - bascially I'll just use this table as startingpoint when I work on figuring out the next failing tests. And I hope others can also use this info maybe to get faster an idea around when something started going wrong (although results might be different on other systems obviously).
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

Yeah, especially RTTs have many problems on my systems. Sometimes they are not even run due to missing support, sometimes they give strange results. But at least on one system they gave the correct results :-)
I've also fixed drawPixel - at least I hope so. Also, from time to time, I add test cases without actually fixing the cause of the failure. Test first so to say. This is usually not possible with rendered results, though.
On one of my Windows systems I get grey GUI element background instead of white - not really knowing how this can happen. Also make sure that no driver settings override the mipmap or anti-aliasing settings, as this results in severe drifts in the comparisons.

Could someone tell me how to run the tests under recent Linux systems with compiz? Seems that the colors are all messed up then.
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

For me, all tests above transparentMaterials do work under VS32 now with rev 4142. Would be also good to know, which subtest fails, and if the problem is merely a threshold one (i.e. result is largely correct) or is a major one with completely failing rendering.
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Irrlicht unit tests

Post by CuteAlien »

Ok, I just run a quick-test, drawPixel is now fixed on VS32 with r4142.
My other results there: http://pastebin.com/7XiyJLBS

edit: Hm.. failing tests only write "failure" to the log. Guess we have to add more info first.
edit2: Also for Info: I compiled with Debug on VS2010 and XP and graphic card is Nvidia GeForce 6200.
edit3: I just run it once more with updated drivers, but same results: http://pastebin.com/fTfh6vgN
edit4: My testresults for VS/Win32 can now be found in http://www.michaelzeilfelder.de/pub/IrrTestR4142VS32/
The failing screenshots are always mentioned in the tests.log, so for example:
http://www.michaelzeilfelder.de/pub/Irr ... onBack.png
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Yoran
Site Admin
Posts: 96
Joined: Fri Oct 07, 2005 8:55 am
Location: The Netherlands
Contact:

Re: Irrlicht unit tests

Post by Yoran »

I have run a test with SVN rev 4142 on my system on 2 graphic cards:

Geforce GT330M:
50 tests of 65 passed http://pastebin.com/qSPfWUvL

and now the surprise, Intel HD graphics
62 test of 65 passed http://pastebin.com/Fd7JxKZW

- debug dll
- system: windows 7 pro 64 bit 6.1.7601 Service Pack 1, build 7601
- core i5, 4gb ram
Yoran
Site Admin
Posts: 96
Joined: Fri Oct 07, 2005 8:55 am
Location: The Netherlands
Contact:

Re: Irrlicht unit tests

Post by Yoran »

D3D fails to capture the correct screenshots, explaining some failed tests
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

Ok, for the GForce it looks indeed as if the d3d driver had severe problems with capturing, while the Intel graphics seems to have basically just one major bug. Because the draw2DImage failure is basically because the file size differs, the difference is negelctable, though. It seems that this is again caused by the 1pixel offset in 2d rendering under d3d. The reflection2textures in transparent materials is a weird bug under OpenGL, which seems to produce different results on each system I've ever run this material. And the Burnings bug seems to vanish from time to time. Just run the test two or three times and it may disappear. So basically the textureMatrixInMixedScenes is a bug, which could be caused by wrongly created reference outputs, though. We should make sure that the colors used in the GUI are as seen in the outputs, because the reference images of the different drivers are also not equal.
@CuteAlien: GForce 6200, whoa ;-) Anyway, looks like the OpenGL driver has some severe problems, as even mipmap locking does not work. I'll check if I can see some more things in the screenshots.
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Irrlicht unit tests

Post by CuteAlien »

hybrid wrote: @CuteAlien: GForce 6200, whoa ;-) Anyway, looks like the OpenGL driver has some severe problems, as even mipmap locking does not work. I'll check if I can see some more things in the screenshots.
Haha, will hopefully get a new system the next days - but for my 32-bit system I'll have to live with that card, there's not many betters cards for AGP ;-)

In case screenshots just had permission errors please try again, I tried allowing to view the whole folder, but seems my provider doesn't like the .htaccess file. Will put screenshots on another server next time where I have more control over permissions.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

Ok, draw2DImage test is now also fixed. Well, actually it's only relaxed with comparison. The problem is that in case of down scaling (dest rect is smaller than source rect), d3d driver seems to move the texture by one pixel to the top left. In all other cases, the exact same call does place the texture exactly aligned with OpenGL and the other drivers. I couldn't see any obvious reason for this, and one pixel shift is for now acceptable.
RdR
Competition winner
Posts: 273
Joined: Tue Mar 29, 2011 2:58 pm
Contact:

Re: Irrlicht unit tests

Post by RdR »

OS: Ubuntu 10.10 (32-bit)
Compiler: GCC 4.7.0
Revision: 4143

Tests finished. 49 tests of 65 passed.

Log: http://pastebin.com/Wx4ebRSd
Results: http://ferket.net/results.zip
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

Yeah, just as I already wrote. Without special care taking it does not seem to make sense to run the tests under Linux. Every render test case fails under OpenGL. Red and blue seem to be exchanged, which of course leads to very large differences. Not sure if this is just a problem with screenshots, or general rendering.
Nadro
Posts: 1648
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: Irrlicht unit tests

Post by Nadro »

It's my log:
http://pastebin.com/tz43Abvc

Hardware: AMD Phenom X3 8650 + AMD Radeon HD2400 PRO
Software: Windows 7 x64 + Catalyst 12.1

I will show a results from a test under MacOSX soon (on GF9 series card)

I see a problem with a OpenGL floating point texture support on my machine. I'll fix it ;)
Library helping with network requests, tasks management, logger etc in desktop and mobile apps: https://github.com/GrupaPracuj/hermes
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Irrlicht unit tests

Post by CuteAlien »

@Nadro: Which svn version? (svn already received some fixes today)
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Re: Irrlicht unit tests

Post by hybrid »

Latest, because texture pointer test is running
Post Reply