3000th commit - IrrlichtBAW (GIT repo, v 0.3.0-gamma1)

Discuss about anything related to the Irrlicht Engine, or read announcements about any significant features or usage changes.
Post Reply
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

3000th commit - IrrlichtBAW (GIT repo, v 0.3.0-gamma1)

Post by devsh »

We are finally giving back to the community and here is our Irrlicht fork,
but we are also looking for users, contributors and general feedback.

Documentation coming soon on a static github page.

Here is our completely non-exhaustive list of issues for potential contributors.
https://github.com/buildaworldnet/IrrlichtBAW/issues


Official Support Discord
https://discord.gg/4MTCVaN

There's also a skype support group, ask me for a private invite.

Our Project Goal:
Our goal with Irrlicht BAW is to have a GPU programming sandbox, so it
would be a pure data-oriented HPC and Hardware Rasterization library
with optional access to raytracing kernels, encompassing OpenGL and Vulkan.

We want to use it on headless GPU servers (Vulkan Compute), Nvidia GRID servers
(OpenGL/Vulkan) down to Mobile (but only very recent Android Vulkan
capable devices).

We want to keep very close to the metal, so you should be able to use
all OpenGL/Vulkan functionality you want without ripping it out of layer
upon layer of abstraction.

The idea is to have something that you'd want to use to develop a High
Performance "Application Specific" Graphics Program, instead of building
everything from scratch with pure GL or Vulkan.

If you just want to load a data-set (i.e. textures, spatial data),
process it with a compute shader, visualize your results, and export
them; you wouldn't want to spend hours writing PNG loaders from libPNG
and fussing around with the basics of uploading textures to OpenGL (or worse in Vulkan), or
worse the basics of creating an OpenGL core profile context and window (or worse in Vulkan)
as well as loading all the function pointers, you'd want to jump
straight to the Compute Shader or OpenCL kernel.

So essentially we want to make a skeleton of a 3D Engine without
actually forcing rendering solutions down people's throats.
This is why the following are being implemented and are viewed as important:
1) Compute Shaders DONE
2) Access to OpenGL from multiple threads simultaneously DONE
3) Our own extensible native binary format with optional aes128_gcm DONE
encryption followed by lz4 or lzma compression DONE
4) An OpenGL global state tracker for interfacing with 3rd party OpenGL
middlewares (Silverlining, Triton, CEGUI) and intra-thread material system DONE (but will get phased out)
5) Access to Vulkan with OpenGL as a fallback
6) Generic library of useful functions for GLSL with some PBR-BRDFs, GPU
sort kernels and other functions in-progress mostly as irr::ext
7) Sparse Textures, Sparse Buffers
8) GPU Buffer memory allocators/pools DONE
9) Examples showing the power of unobstructed OpenGL usage DONE
10) Multithreaded Model Loading DONE
11) GLI integration AVAILABLE
12) AVX/SSE3 2d and 3d math classes in-progress
13) GPU-side skinned mesh boning and scenegraph in-progress

For the same reason following things would not ever be in the
core-engine, would not be priority and if-done would have to live in an
extensions repository https://github.com/buildaworldnet/Irrli ... master/ext:
1) A Deferred Renderer
2) Lightmapper
3) COLLADA Loader
4) Shadows
5) Postprocessing of any kind Already Have a Tonemapper, but in irr::ext::
6) Transparency but we have a demo/benchmark of tenchniques
7) Automated Occlusion Culling
This is simply because there are many many ways of doing the above, and
none of them is "the right one" for even 80% of the use cases.
So if we were to support one in the core engine, we'd have to support
the cornucopia of others (even as alternatives).
Secondly the really good solutions to Deferred Rendering and Order
Independent Transparency require the end user integrating the engine to
'make their assets fit into the pipeline' such as modifying shaders so
that they write the correct data in the correct order to a G-Buffer or
even change the entire order and the way their engine sets up the draw loop.
So without someone's clear use case, already existing work and
onobtrusive implementation which would not tie down the API, such things
would have no right to exist in engine core.
A brilliant example of an item which would have been on that list is the
IGPUTransientBuffer, which is a useful stand-alone side-addition.
We developed it for streaming very small data (<4kb) to and from the GPU
to avoid VAO and Buffer object explosion (especially for the GUI).


There are however some things in the grey area, such as a Bullet
Physics (for which there is irr::ext::Bullet), Occulus, etc.
integration of which could see the light of day
being implemented in the engine core.

Release 0.3.0-gamma1 is out (RELEASES WILL NOT WORK IF NOT GIT CLONED)
https://github.com/buildaworldnet/Irrli ... 3.0-gamma1
08 Jun 2019

USE CMAKE-GUI TO BUILD
Check out the github wiki page for more info.

I guess you only care about pics, so selected examples from the examples_tests folder:

07.HardwareSkinning - 100 dwarves, pure GPU skinning with a configurable 120Hz limiter, no instancing, no batching - 1800 FPS
Image

08.HardwareInstancing - your favorite cow mesh and a hundred poly flower mesh demonstrating GPU instance cloud culling and LoD selection via Transform Feedback
Image

13. BRDF Explorer - See how GGX PBR and various diffuse models work with a single light
Image

14.MSAA - 32 sample MSAA, 100 dwarves, 1000FPS
Image

15.LayeredCubeMapShadows - 100 to 625 dwarves, GPU skinned, moving point light, omni shadows drawn in one pass, 56FPS with 625 dwarves (1MIl+ polygons)
Image

16.OITZoo - over 10 planned transparency methods compared on the same scene (screenshot shows DevSH's modified stencil-routed k-buffer).
Image

18.RaytracedAO - showcasing Radeon Rays integration to compute shadows from area and environment lights.
Image

23.Autoexposure - A quick test of compute shaders for something useful (a must for HDR pipelines - uses ext::CToneMapper).
Image

25. Blur -Compute Shader Radius-Independent Gaussian Blur Approximation (uses ext::CBlurPerformer)
Image


Known issues:
1) Using RasterizerDiscard=true affects clearing and blitting frame buffers but setMaterial doesn't!!!
In other words you either have to draw something to change the RasterizerDiscard flag in OpenGLDriver or force reset all render states.
Last edited by devsh on Thu May 28, 2020 12:41 pm, edited 50 times in total.
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

oh yeah, forgot to mention:
A) direct access to the depth buffer as a texture (useful for your deferred renderer GBuffers so you have 24bits free)
B) you can share depth buffers between same size multiple render targets (no need for obscure copies)
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

Thanks for sharing. It would be nice if you would consider using some easy accessible source control system instead of a single zip file. Something like sourceforge, googlecode or github. I suppose you use sourcecontrol internally already anyway (otherwise every time is a good time to start doing that). So you might even be able to push it including history. Then we can figure out stuff like which patches belong together as it's otherwise near impossible for other coders to see that.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

diff will tell you everything

plus we'll number zips by release and keep the old ones
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

Unfortunately diff only gives one big patch for everything. Figuring out which changed line belongs to which feature is not even trivial when you wrote the stuff yourself. For code from other people it's close to impossible.

And in case you are not so familiar yet with source control systems please give them a try. You need at most half a dozend commands and they will make your life as coder _a lot_ easier. Doing versions manually is a constant pain and you are missing out on all the automatic feature of a version control system (branching for tests, reverting problems, checking specific versions, automatic patches, automatic documentation for changes if you use good commit messages, 1-line commands to push your updates on the web, trivial diffs to any old version, trival checks of what has changed, etc... you miss out on _all_ that with zip's). Version control systems are not just there for team coordination - even as a single coder you will never go back once you learned to use them in your projects.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

WE do use VCS

but we dont have a separate repo for irrlicht, so I can only give diffs between versions
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

Ah, I understand ... admittably I'm doing the same in my projects usually *sigh*. My workaround is creating patch-files for Irrlicht-changes and keeping those around in an additional directory. Probably there is a better solution somehow... but that way I find at least the patches I have to apply on Irrlicht updates relatively fast (I also got a txt file describing each patch some more).
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

what I'd like to see is our stuff getting merged into irrlicht

no offense but the OcclusionQuery support is pathetic
just rewrote the irrlicht occlusion query mechanisms and now support GL_SAMPLES_PASSED and GL_ANY_SAMPLES_PASSED

you NEED a IVideoDriver::beginQuery and endQuery mechanism, as you cant just hard-lock the user the trying in a mesh with and occlusion query

ALSO YOU CANT SEARCH THROUGH AN ARRAY OF ALL THE occlusion queries each time you want to do something with one

So now IOcclusionQuery object is created by driver, you attach it to multiple ISceneNodes and then drop()

You manually run the query and render things inbetween to get pixel count
You manually retrieve query results and update pixel count
If ISceneNode has occlusion query attached and AutomaticCulling set to EAC_OCCLUSION_QUERY, it will get pixel count and cull

Now I added ConditionalRendering

So you dont retrieve query result and update pixel count
you set ISceenNode culling type to EAC_CONDITIONAL_RENDER and it passes IOccludionQuery to driver draw functions, which then pass all the way to drawPrimitivesIndexedVertex (or whatever all these things get called) and that does the OGL connditional rendering
you can also pass IOcclusionQuery object to all 3D functions that draw in IVideoDriver
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

No offence taken. There are many areas in the engine where we'd like improvements ourself. But I don't think anyone in the team is able to spend days on splitting up a huge patch to understand which change belongs to which feature.

There is a difference in coding for one application and coding for an engine. You can for example break interfaces as you don't have to care about which code no longer runs or will have trouble (so for example you don't want fixed function pipeline you can just kick it out). You also don't have to think about if an interface will be supportable for several years as long as it does what you need in your application. I have lot's of patches like that myself - and haven't figured out how to apply many of them even after half a decade. I mean I could just check them in - and then go on a long holiday and hope the next guy supports them somehow. We already got a bunch of code like that (and yes, very fun to support stuff that breaks on every real test - or in some cases never really worked at all).

I'm sure your code is good and useful and full of featues we'd all like to have. But we can' just merge such a huge blob and hope everthing will be fine. And we have no-one who will be able to go over it line by line and figure out what to do with it.

I appreciate that you open up your changes and want to help - I just don't know what we can do with it like this.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Nadro
Posts: 1648
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by Nadro »

Thanks for share it to community, however I agree with CuteAlien that it's impossible to merge thats big patch.

Anyway Irrlicht will get OpenGL 3.x + ES3.0 drivers with similar features in coming days. Patch is mostly done, but w8 for review.

Maybe we'll be able to merge some stuff from your fork (if it will be compatible with Irrlicht Interfaces).
Library helping with network requests, tasks management, logger etc in desktop and mobile apps: https://github.com/GrupaPracuj/hermes
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by hendu »

While it's all well and good to request a public repo, that's no guarantee of anything. Not only did I do so, I also posted important patches separately to the tracker as SVN diffs against the then-trunk.

You can view the tracker and see that the majority of them are years old now and have had zero comments.

So forgive me for calling out the hypocrisy.
sudi
Posts: 1686
Joined: Fri Aug 26, 2005 8:38 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by sudi »

what i do is checkout the libs i want to change in a subfolder of my project. fortunatly most of them use git these days so i can easily use my own branch locally. This allows me to patch stuff and do proper versioning a different repo than the main project and still keep everything neatly together.
We're programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We're not excited by renovation:tinkering,improving,planting flower beds.
CuteAlien
Admin
Posts: 9628
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

I know that Hendu. But we did apply several of your patches over the years. While in this case we won't be able to. I think you are an experienced enough developer to realize that it's impossible that way. Also as long as they are on the patch-tracker they still can be applied by someone - some patches are applied years later. And even more important they can be used by other people which are customizing their Irrlicht versions. For example I've applied several of the patches on Hybrid's homepage which didn't make it in the engine when I needed them in a game to my local Irrlicht. And there was also the thread where you mentioned one of your old patches this week in the forum - sure would be nicer to have that in the engine, but that way it was still useful to someone.

The reason so many patches are uncommented and stay for years is simply that we always have more to do than we can do. And that's not something that's ever going to change. So each of us sets his priorities and ignores the rest - what else could we do? My priority is for example nearly always hunting bugs or cleaning up existing code before doing anything else and that pretty much takes all the time. And as you can see by the fact that even some bugs I reported there are years old (and I got more on my todo-list) I can't even catch up with that.

Smaller patches or a repository are better - I guess that's pretty obvious. Certainly I'm also fine if devsh doesn't do that - it's really up to each developer what he wants to do. Just reading through the feature-list which he needed in a game was already rather interesting in itself. It tells us a little bit about what someone working on one of the serious projects needs currently from the engine.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

well there is a principal reason why we're releasing the changes:
Everytime irrlicht gets an update, we have to merge LESS if you integrate some of the amazing features of mine

Again, irrlicht has the potential to evolve to be the "OpenSource 3D Engine of Choice" over lets say Horde3D or OGRE3D

There are big speed issues with stock irrlicht, its mostly the way the interfaces are designed... some notable examples are:
a) setting shader uniforms by name (lookup with a binary search in an array best case, but implemented as a call to OpenGL glGetUniformLocation) - possibly 10% FPS penalty
b) cant access depth buffer as a texture, no fine tuned control over this process
c) no render target blitting
d) REALLY AWFUL IMPLEMENTATION OF OCCLUSION QUERIES

This is not the difference between coding for an engine or application, this engine is the way it is because no one tried to develop an AAA or just modern render technology title with it... so the interfaces and inner workings of it havent had to be changed to deal with the performance concerns adequately. After all this engine needs to be geared towards creating modern day desktop and mobile render technology applications!
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by hendu »

CuteAlien wrote:The reason so many patches are uncommented and stay for years is simply that we always have more to do than we can do. And that's not something that's ever going to change. So each of us sets his priorities and ignores the rest - what else could we do? My priority is for example nearly always hunting bugs or cleaning up existing code before doing anything else and that pretty much takes all the time. And as you can see by the fact that even some bugs I reported there are years old (and I got more on my todo-list) I can't even catch up with that.
The issue with that is that nobody is responsible for the whole when everybody goes "not me". Irr lacks a BDFL.

I guess Niko took that role previously, but now that he's gone, nobody stepped in to fill his shoes. In this situation the project is as good as dead, because what good is fixing a rare bug here and there if users and needs go unanswered?

For a project leader, responding even within two weeks to anything should be of the highest priority. There is no project leader ATM. Nobody's in charge.
Post Reply