MSAA on Render Targets - BAW Irrlicht (GIT repo, v 0.2.5)

Discuss about anything related to the Irrlicht Engine, or read announcements about any significant features or usage changes.

MSAA on Render Targets - BAW Irrlicht (GIT repo, v 0.2.5)

Postby devsh » Fri Mar 07, 2014 10:05 pm

We are finally giving back to the community and here is our irrlicht

Current git HEAD:
https://github.com/buildaworldnet/IrrlichtBAW
New changes to the texture loading interface to enable cubmaps in the near future.
15 Dec 2017

Release 0.2.5 is out
https://github.com/buildaworldnet/IrrlichtBAW/releases/tag/v0.2.5
7 Sep 2017

Repo URL has migrated:
https://github.com/buildaworldnet/IrrlichtBAW
Dont complain you can't build if you try to use the Makefile
Only working build systems are:
1) the Makefile with target serverlib (no graphics)
2) Codeblocks project under Linux Release Fast Math /Debug static library
3) Windows Visual Studio solution static library

We are working on a CMake build and documentation.

OVERVIEW:
- OpenGL&(Linux|Windows) only
- Reverse 32bit Floating Point Depth Buffers so you have enough Z-Buffer precision to look in someone's nose and view a mountain 10km away
- Direct State Access, Texture Storage and Buffer Storage extensions used by default!
- COpenGLState class to capture OpenGL global state of a context and restore any changes or default state before and after using 3rd party middlewares like SilverLining or CEGUI
- Reworked Shader Constant Callback and setting uniforms with a more performance friendly mechanism (looking up shader uniforms by name every time you want to set them is REALLY bad)
- Tessellation Shaders (DX11 feature)
- Transform Feedback
- IInstancedMeshSceneNode with GPU culling and LoD determination
- Hardware (GPU) Skinning
- MetaBuffer classes allowing to allocate from a GPU buffer in various ways
- 3D Texture Support
- 2D Texture Arrays Support
- All types of Texture-friendly API cleanup
- Texture Buffer Objects
- DDS Texture Support with compression and explicit mip-chains
- Manual Mip Map specification and loading (useful when you generate your own with the superior sinc filter or load DDS DXT compressed files with explicit mip maps))
- FBO Blitting (super fast rendertarget copy)
- Explicit FBO setups for render targets (with render to 2DArray and 3D texture support)
- OpenGL 4.3 core compliance
- No fixed function materials
- All shader materials
- IGPUBuffer and ICPUBuffer architecture making it possible to circulate data around GPU-side (mesh to texture, texture to mesh and more!)
- Flexible Vertex Format in Meshes (up to 16 attributes of all formats like float, signed/unsigned and normalized/unnormalized ints!) which can share the same underlying GPU-Buffer!
- MeshBuffers which can have 16,32bit or NO INDICES AT ALL!
- Support for All Primitive Types in meshes (patches, lines, strips, fans etc.)
- IGPUTransientBuffer for threaded streaming of data into OpenGL (yes, direct DMA pointer dereference into GPU VRAM in non-OpenGL thread!!!)
- Granular GPU/CPU Buffer (basically a std::vector<>/core::array<> for your GPU data)
- NEW FLEXIBLE VERTEX FORMAT MESHES
- Automatic 10bit normal quantization for static meshes (.obj and .x) saving 8 bytes-per-vertex
- Quantization of OBJ mesh Texture Coordinates into uint8_t and uint16_t when range is low and can be used instead of float saving 4 to 6bpv
- 14 Small Code examples
- SSE3 Accelerated Quaternion class, resulting in performance increase of Skinning
- Tested on NVIDIA, Intel and AMD
- Visual Studio 2010-2016 Project
- SSE3 SIMD Vector{3/4}D class
- 4x3 Matrix class with faster inverses and multiplications
- Cached Absolute Transformation updates (only recompute the whole ancestor chain of transforms for nodes that actually need it and only as far as needed)
- Cached Model View Projection states (combinations of world, view and proj matrices available at all times and corresponding normal matrices)
- RasterizerDiscard flag which prevents the shader from progressing onto the fragment stage (feedback transform loops with no display)
- Full GPU Query Support, not only occlusion, but TIMESTAMP, TIME_ELAPSED, and primitves-generated
- ARB_query_buffer_object support so you can store your query results in IGPUBuffer{s} avoiding a GPU->CPU transfer
- *Precise* functions for multiplying matrix4x3 together and matrix4 with matrix4x3 using double precision arithmetic
- Fixed Irrlicht's insane implementation of glPolygonOffset so now you can render your decals without killing your Z-Buffer data or Early-Z
- OpenCL device stub, finds associated device to the rendering GPU and can report number of GPU cores
- CPU culling workarounds for small instance counts to avoid GPU idling
- Normal quantization to 30bit in x meshes, even ones which use Skinning
- MultiDrawIndirect and DrawIndirect handles to GL functions (can use this OpenGL feature if desired)
- Stack Unwinding object tracker which helps to find memory leaks/lost resources
- ConvertUTF library for conversions between UTF8 and WString
- GLSync Fences for asynchronous GPU<=>RAM data transfers and omission of mesh rendering when data not ready
- Fast Hash algorithm for high throughtput non-secure non-cryptographic hashing (xxHash256)
- Multithreaded OpenGL context creation functions
- Splines for 3D path interpolation
- Some other stuff I cant remember but is soo epic

Known issues:
1) Using RasterizerDiscard=true affects clearing and blitting frame buffers but setMaterial doesn't!!!
In other words you either have to draw something to change the RasterizerDiscard flag in OpenGLDriver or force reset all render states.
Last edited by devsh on Fri Dec 15, 2017 5:06 pm, edited 21 times in total.
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby devsh » Sun Mar 09, 2014 2:10 am

oh yeah, forgot to mention:
A) direct access to the depth buffer as a texture (useful for your deferred renderer GBuffers so you have 24bits free)
B) you can share depth buffers between same size multiple render targets (no need for obscure copies)
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby CuteAlien » Sun Mar 09, 2014 9:54 am

Thanks for sharing. It would be nice if you would consider using some easy accessible source control system instead of a single zip file. Something like sourceforge, googlecode or github. I suppose you use sourcecontrol internally already anyway (otherwise every time is a good time to start doing that). So you might even be able to push it including history. Then we can figure out stuff like which patches belong together as it's otherwise near impossible for other coders to see that.
IRC: #irrlicht on irc.freenode.net
Code snippets, patches&stuff: http://www.michaelzeilfelder.de/irrlicht.htm
Free racer created with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
User avatar
CuteAlien
Admin
 
Posts: 8363
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby devsh » Sun Mar 09, 2014 2:48 pm

diff will tell you everything

plus we'll number zips by release and keep the old ones
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby CuteAlien » Sun Mar 09, 2014 3:43 pm

Unfortunately diff only gives one big patch for everything. Figuring out which changed line belongs to which feature is not even trivial when you wrote the stuff yourself. For code from other people it's close to impossible.

And in case you are not so familiar yet with source control systems please give them a try. You need at most half a dozend commands and they will make your life as coder _a lot_ easier. Doing versions manually is a constant pain and you are missing out on all the automatic feature of a version control system (branching for tests, reverting problems, checking specific versions, automatic patches, automatic documentation for changes if you use good commit messages, 1-line commands to push your updates on the web, trivial diffs to any old version, trival checks of what has changed, etc... you miss out on _all_ that with zip's). Version control systems are not just there for team coordination - even as a single coder you will never go back once you learned to use them in your projects.
IRC: #irrlicht on irc.freenode.net
Code snippets, patches&stuff: http://www.michaelzeilfelder.de/irrlicht.htm
Free racer created with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
User avatar
CuteAlien
Admin
 
Posts: 8363
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby devsh » Sun Mar 09, 2014 4:10 pm

WE do use VCS

but we dont have a separate repo for irrlicht, so I can only give diffs between versions
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby CuteAlien » Sun Mar 09, 2014 4:42 pm

Ah, I understand ... admittably I'm doing the same in my projects usually *sigh*. My workaround is creating patch-files for Irrlicht-changes and keeping those around in an additional directory. Probably there is a better solution somehow... but that way I find at least the patches I have to apply on Irrlicht updates relatively fast (I also got a txt file describing each patch some more).
IRC: #irrlicht on irc.freenode.net
Code snippets, patches&stuff: http://www.michaelzeilfelder.de/irrlicht.htm
Free racer created with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
User avatar
CuteAlien
Admin
 
Posts: 8363
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby devsh » Sun Mar 09, 2014 5:28 pm

what I'd like to see is our stuff getting merged into irrlicht

no offense but the OcclusionQuery support is pathetic
just rewrote the irrlicht occlusion query mechanisms and now support GL_SAMPLES_PASSED and GL_ANY_SAMPLES_PASSED

you NEED a IVideoDriver::beginQuery and endQuery mechanism, as you cant just hard-lock the user the trying in a mesh with and occlusion query

ALSO YOU CANT SEARCH THROUGH AN ARRAY OF ALL THE occlusion queries each time you want to do something with one

So now IOcclusionQuery object is created by driver, you attach it to multiple ISceneNodes and then drop()

You manually run the query and render things inbetween to get pixel count
You manually retrieve query results and update pixel count
If ISceneNode has occlusion query attached and AutomaticCulling set to EAC_OCCLUSION_QUERY, it will get pixel count and cull

Now I added ConditionalRendering

So you dont retrieve query result and update pixel count
you set ISceenNode culling type to EAC_CONDITIONAL_RENDER and it passes IOccludionQuery to driver draw functions, which then pass all the way to drawPrimitivesIndexedVertex (or whatever all these things get called) and that does the OGL connditional rendering
you can also pass IOcclusionQuery object to all 3D functions that draw in IVideoDriver
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby CuteAlien » Sun Mar 09, 2014 7:22 pm

No offence taken. There are many areas in the engine where we'd like improvements ourself. But I don't think anyone in the team is able to spend days on splitting up a huge patch to understand which change belongs to which feature.

There is a difference in coding for one application and coding for an engine. You can for example break interfaces as you don't have to care about which code no longer runs or will have trouble (so for example you don't want fixed function pipeline you can just kick it out). You also don't have to think about if an interface will be supportable for several years as long as it does what you need in your application. I have lot's of patches like that myself - and haven't figured out how to apply many of them even after half a decade. I mean I could just check them in - and then go on a long holiday and hope the next guy supports them somehow. We already got a bunch of code like that (and yes, very fun to support stuff that breaks on every real test - or in some cases never really worked at all).

I'm sure your code is good and useful and full of featues we'd all like to have. But we can' just merge such a huge blob and hope everthing will be fine. And we have no-one who will be able to go over it line by line and figure out what to do with it.

I appreciate that you open up your changes and want to help - I just don't know what we can do with it like this.
IRC: #irrlicht on irc.freenode.net
Code snippets, patches&stuff: http://www.michaelzeilfelder.de/irrlicht.htm
Free racer created with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
User avatar
CuteAlien
Admin
 
Posts: 8363
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby Nadro » Sun Mar 09, 2014 8:26 pm

Thanks for share it to community, however I agree with CuteAlien that it's impossible to merge thats big patch.

Anyway Irrlicht will get OpenGL 3.x + ES3.0 drivers with similar features in coming days. Patch is mostly done, but w8 for review.

Maybe we'll be able to merge some stuff from your fork (if it will be compatible with Irrlicht Interfaces).
Nadro
 
Posts: 1647
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby hendu » Sun Mar 09, 2014 9:37 pm

While it's all well and good to request a public repo, that's no guarantee of anything. Not only did I do so, I also posted important patches separately to the tracker as SVN diffs against the then-trunk.

You can view the tracker and see that the majority of them are years old now and have had zero comments.

So forgive me for calling out the hypocrisy.
hendu
 
Posts: 2587
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby Sudi » Sun Mar 09, 2014 11:02 pm

what i do is checkout the libs i want to change in a subfolder of my project. fortunatly most of them use git these days so i can easily use my own branch locally. This allows me to patch stuff and do proper versioning a different repo than the main project and still keep everything neatly together.
We're programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We're not excited by renovation:tinkering,improving,planting flower beds.
User avatar
Sudi
 
Posts: 1685
Joined: Fri Aug 26, 2005 8:38 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby CuteAlien » Sun Mar 09, 2014 11:31 pm

I know that Hendu. But we did apply several of your patches over the years. While in this case we won't be able to. I think you are an experienced enough developer to realize that it's impossible that way. Also as long as they are on the patch-tracker they still can be applied by someone - some patches are applied years later. And even more important they can be used by other people which are customizing their Irrlicht versions. For example I've applied several of the patches on Hybrid's homepage which didn't make it in the engine when I needed them in a game to my local Irrlicht. And there was also the thread where you mentioned one of your old patches this week in the forum - sure would be nicer to have that in the engine, but that way it was still useful to someone.

The reason so many patches are uncommented and stay for years is simply that we always have more to do than we can do. And that's not something that's ever going to change. So each of us sets his priorities and ignores the rest - what else could we do? My priority is for example nearly always hunting bugs or cleaning up existing code before doing anything else and that pretty much takes all the time. And as you can see by the fact that even some bugs I reported there are years old (and I got more on my todo-list) I can't even catch up with that.

Smaller patches or a repository are better - I guess that's pretty obvious. Certainly I'm also fine if devsh doesn't do that - it's really up to each developer what he wants to do. Just reading through the feature-list which he needed in a game was already rather interesting in itself. It tells us a little bit about what someone working on one of the serious projects needs currently from the engine.
IRC: #irrlicht on irc.freenode.net
Code snippets, patches&stuff: http://www.michaelzeilfelder.de/irrlicht.htm
Free racer created with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
User avatar
CuteAlien
Admin
 
Posts: 8363
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby devsh » Mon Mar 10, 2014 1:19 am

well there is a principal reason why we're releasing the changes:
Everytime irrlicht gets an update, we have to merge LESS if you integrate some of the amazing features of mine

Again, irrlicht has the potential to evolve to be the "OpenSource 3D Engine of Choice" over lets say Horde3D or OGRE3D

There are big speed issues with stock irrlicht, its mostly the way the interfaces are designed... some notable examples are:
a) setting shader uniforms by name (lookup with a binary search in an array best case, but implemented as a call to OpenGL glGetUniformLocation) - possibly 10% FPS penalty
b) cant access depth buffer as a texture, no fine tuned control over this process
c) no render target blitting
d) REALLY AWFUL IMPLEMENTATION OF OCCLUSION QUERIES

This is not the difference between coding for an engine or application, this engine is the way it is because no one tried to develop an AAA or just modern render technology title with it... so the interfaces and inner workings of it havent had to be changed to deal with the performance concerns adequately. After all this engine needs to be geared towards creating modern day desktop and mobile render technology applications!
We chose to stream mesh data from Multiple OpenGL Contexts in many threads and do the other things, not because they are easy, but because they are hard! - JFK
User avatar
devsh
Competition winner
 
Posts: 1768
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Postby hendu » Mon Mar 10, 2014 8:57 am

CuteAlien wrote:The reason so many patches are uncommented and stay for years is simply that we always have more to do than we can do. And that's not something that's ever going to change. So each of us sets his priorities and ignores the rest - what else could we do? My priority is for example nearly always hunting bugs or cleaning up existing code before doing anything else and that pretty much takes all the time. And as you can see by the fact that even some bugs I reported there are years old (and I got more on my todo-list) I can't even catch up with that.


The issue with that is that nobody is responsible for the whole when everybody goes "not me". Irr lacks a BDFL.

I guess Niko took that role previously, but now that he's gone, nobody stepped in to fill his shoes. In this situation the project is as good as dead, because what good is fixing a rare bug here and there if users and needs go unanswered?

For a project leader, responding even within two weeks to anything should be of the highest priority. There is no project leader ATM. Nobody's in charge.
hendu
 
Posts: 2587
Joined: Sat Dec 18, 2010 12:53 pm

Next

Return to Open Discussion and Dev Announcements

Who is online

Users browsing this forum: Exabot [Bot] and 1 guest