Textures compression tool (BINARY RELEASED)

[Nadro]

Yep it's an option, but it would be nice to have a separate method to read a raw data from a file without graphic context access (like in current createImageFromFile methods), why? Because in this way we can read a data in a separate thread and in the main thread, we only have to create a texture from a loaded data (maybe we should create a texture for this specific problem in another ogl context, because after an image process creation we will drop it).

[REDDemon]

That would require cross-platform MT environment for Irrlicht. a big work O_O.

Seems that ATI has a utility "ATI_compress" wich supports only VS 2008 and 2010. Seems that can works without a graphics context but don't know if can work even without a GPU (I don't know if internally creates a graphic context.. probably not). Maybe is possible with that library to have texture compression also for BURNINGS_VIDEO and SOFTWARE_DRIVER. the problem is that will be available only for windows since they dropped support for other compilers. anyway that's interesting.

Any bug to report for the KTX exporter ? It works fine or not?

[hendu]

nv texture tools work fine in cli, it's portable code.

[Nadro]

Next topic which looks like a freezed. I'm still waiting for Hybrid's decision, how compressed texture support in Irrlicht should looks (mainly IImage part) and after 3-5 days we will have an official support for this feature.

[CuteAlien]

Coding is usually the easy&fast part of programming ;-) But right now adding more features when already 1/3 of the Irrlicht-testsuite is broken? Please don't - I've already spend too many evenings in the last weeks trying to figure out the current mess. Don't pile up new features without careing how much code gets broken by it. And yes, I would also prefer working on new features instead of cleaning up existing problems as well, that's always more fun, but if anyone adds another maybe working/maybe broken feature while we're in this state I'll go berserk!

[Nadro]

Hmmm I thinked that compressed texture support is one of the common feature of today's graphic engines and thats why I talking about it in last time very often. Yep, I know that implementation in many cases isn't the hardest part, but in this situation I rly don't see better way than locked manipulator methods in IImage (IImageManipulator) for compressed images and thats why I'm w8 for Hybrid, because as I know he already have a better solution for it than me.

I think that we can't use hardware downloading an image from a texture (like in REDDemon solution) in createImageFromFile method, because many users call these methods in another thread (I think that we don't want create an OpenGL context and destroy them in each createImageFromFile call). We also can't use software images decoding because of software patents and loading image process slowdown. If somebody want uncompressed dds data, he can call ITexture::lock(HERE_EG_COLOR_FORMAT_PARAM) method and create uncompressed image (on uncompressed images all manipulation methods are available) next he can create compressed texture from this image calling addTexture method with compressed flag enabled (we will use compression on the fly by the hardware for this). In this solution we have only one minor cons -> "user can't manipulate compressed images without manual decoding them, which will require call 2 methods".

[CuteAlien]

Really - I love to have compressed texture support :-) You just scared me with "after 3-5 days we will have an official support for this feature". Because given the current state of our testsuite it looks to me like it will take a while just cleaning up behind the last features that got added. Imho what we need right now are not more features but an engine that get's back in a state that is releasable. As soon as we got rid of the broken test-suite I'm glad thinking about new features again (I have a few myself I still wanted to add to 1.8....).

[hybrid]

Not really intended for this thread, but seems to need it that way: Irrlicht's trunk is now feature freezed. We won't change any API or other feature-related things, besides bug fixes. Irrlicht 1.8 won't have any other feature besides those which are already in trunk. Mail on the ML will follow.
Back to topic: We cannot mix up compressed uploading and IImage, that's correct. All the gfx hw related stuff is in ITexture and below. So compressed images will contain comressed data. When turned into ITextures, they can directly upload the compressed data to the card. Compress and decompress are manipulator methods, which may require a gfx card to handle compression and decompression. That's not a problem, we can place the manipulators in video driver. Whether these manipulators will work on IImage or ITexture is yet open. The major problem is that we need to clean up all interfaces to handle the cases properly where pixels or color informations are accessed. These cases may require API changes, code fixes in the places where those functions are called, etc. The way from file to compressed data on the GPU is the simplest one, hard part is *all the rest*
I know about this quite good. I added the generic primitives rendering (lines, quads, etc) once on a higher level than it is today. Was really simple. The hard stuff (which I did not recognize at that time) was to make mesh manipulator etc. working again. Niko immediately recognized this and reverted the patch. And that was the only correct way to deal with it. You cannot provide implementations which do not adhere to the defined API.

[REDDemon]

So my patch is ok for you?

It respect all the points listed by you
1) don't mix hw uploading with image stuff.
2) adhere to current api (no api changes infact!)
3) already works for pixel access (what was working for old textures will also work for compressed textures with my patch so users don't have to mind if that texture is compressed or not).

Sounds good. I can continue working on that so?

(you already looked that post?
viewtopic.php?f=2&t=46041&p=265891#p265891
)

[Nadro]

So my patch is ok for you?

No, we can't use video driver call like a ITexture::lock() in a createImageFromFIle method (threading problem). Now I understand Hybrid's solution and it's similar to my idea, it's only extended. We have to also remove extra class from Your patch - CKhronosTexture and prepare KTX support in standard C[DriverName]Texture and CImageLoaderKTX (it will include many lines of code from Your CKhronosTexture file) a files.

BTW. We'll firstly release v1.8, before adding a compression texture support.

[REDDemon]

So my patch is ok for you?

No, we can't use video driver call like a ITexture::lock() in a createImageFromFIle method (threading problem). Now I understand Hybrid's solution and it's similar to my idea, it's only extended. We have to also remove extra class from Your patch - CKhronosTexture and prepare KTX support in standard C[DriverName]Texture and CImageLoaderKTX (it will include many lines of code from Your CKhronosTexture file) a files.

BTW. We'll firstly release v1.8, before adding a compression texture support.

When Hybrid will tell his solution I'll decide to agree or less to your statement. For now he has just told some general guidelines ^^. I think that what he sad is mostly related to the patch I 'm working on than your solution.

Also If you say so I think that you just didn't looked at my patch on the tracker but only at the snippet on my project page. The snippet is made for Irrlicht 1.7 and 1.8 users to have texture compression already available. The real patch is currently in testing phase (and is directly builded above the latest revision). The pre-patch gives an Idea of what will be changed but does not have any CKhronosTexture. Please before criticize my patch at least read it ^^ (and read that there: viewtopic.php?f=2&t=46041 ). I'm taking you seriously, but there are slight pros with your solution.

I think that Hybrid will consider also my solution.

You have solved a thread problem... (not just really solved, you moved the thread problem from "createImageFromFile" to "Itexture->lock" because at some point the user will need to compress/decompress data in some way and that can be done only with a GL context. So there will be exactly the same problem of creating the Image, with the added difficulty that there's the need to the users to handle decompression. What if the user didn't checked if compression is supported at this point? with my solution the user simply was not able to load unsupported "images", this make the application even easier to debug. if something works, it works immediatly. If something don't works it don't works immediatly.)

Seriously. If the only deal is solving a specific thread problem with huge API changes.

Why it should be added instead of a general solution wich is simpler/more maintenable/without API changes? You should be masocist for doing do that^^.

Multythreading Is really specific on how you implement it. So if you want to make that method thread safe why you want to do that only for that method? at this point you should make whole Irrlicht thread safe. Even some times ago (don't remember who) someone sad that texture compression will not be added to Irrlicht. And now everyone have its own solution ^^. Making all that changes for a single method and then other methods not thread safe? If your solution will be added I hope that also lots of other methods will be made threadsafe^^.


Multithreading is specific on how you implement it. For example I'm using texture compression on my terrain manager and texture streamin^^ And works perfectly with my solution..
1) Thread 1: Irrlicht start (any driver)
2) Thread 2: Irrlicht start (OpenGL) // the second thread don't share any data with first thread. Except for sporadic read-only locks.

3) Thread1 (main thread) -> request to Thread2 of loading Image
4) Thread2 ->receive the message-> createImageFromFile (download from hardware the texture already uncompressed).
5) Thread2-> loading finished-> signal to Thread1
5) Thread1-> receive the message-> lock the data of the Image and copy (readonly) it in its memory.

(are 2 devices expensive? probably yes. but only at startup. At run time there is huge improvement without pauses for loading and anyway that's just a specific usage. No one will do again the same. )

I also now that you are already involved in irrlicht development so your opinion just matters for that^^. But just give hybrid time to see/consider my solution at least. AFAIK if a feature can be added without API changes that is great. What if everyone added API changes for every improvement?

[Nadro]

We cannot mix up compressed uploading and IImage, that's correct. All the gfx hw related stuff is in ITexture and below. So compressed images will contain comressed data. When turned into ITextures, they can directly upload the compressed data to the card.

That why I said that this patch isn't ok in current form, but I based in my statement only on Hybrid words. I don't criticize Your whole patch (When I said "patch" I mean about it: https://sourceforge.net/projects/fixeen ... 0Irrlicht/", now I'm looking on Your patch from a tracker and it seems that it provides partition a CKhronoxTexture on C[DriverName]Texture and CImageLoaderKTX, so second part of my previous post isn't actually). As I said before this is very good stuff, but require some modification.

What about threading? The mostly important part is read a file from a hard drive (mainly textures, so createImageFromFile method) and these operations should be ready for call them from another thread without problems (in Your approach this isn't possible cause of OpenGL context, share lists aren't good solution, because it will require redesign of many engine parts and many context switch calls). Uploading a texture to a GPU is really, really fast compared to reading a data from a file, so You can't say that I moved a problem from a "createImageFromFile" to "ITexture::lock". This approach is enough eg. for a loading screen splash with a progress bar. I'm looking on Your example and You just moved a device with an OpenGL driver to a second thread, but how it solve a OpenGL context problem? In Your description it looks like a whole rendering is done in a second thread, so when You will call createImageFromFile from this thread You will notice temporary rendering freeze, cause of reading from a hard drive operations, if You will move these operations to another thread and left only uploading an image to a GPU in a OpenGL thread freeze will be noticeable shorter. But remember that we can eliminate freeze from my approach thanks to a PBO (it's require for an efficient texture streaming), in Your approach freeze still be noticeable because of file reading. You should left all related to a GPU and OpenGL stuff in a main thread.

Please don't get my post as aggressive towards You. We both want to have the best possible solution for a compressed texture problem, so I think that this disscussion is important. I think that I showed cons and pros of my solution.

I also now that you are already involved in irrlicht development so your opinion just matters for that^^.

I don't think that this makes that my opinion is more important than Yours. I only looking at the advantages and disadvantages of the solution and no matter which of us is the author, because this is an open source project, so we should work together

[REDDemon]

This approach is enough eg. for a loading screen splash with a progress bar. I'm looking on Your example and You just moved a device with an OpenGL driver to a second thread, but how it solve a OpenGL context problem?

Are speaking seriously? that solve the context problem. Image is created in the second thread with the OpenGL context. Then is decompressed (always in the 2nd thread). The rendering is done only in the first thread. (wich can be OpenGL or not). This is not only enough for a loading screen, but works also for data streaming. My solution clearly separe 2 different contexts wich should be kept independed as much as possible. (because 1 is rendering thread, the other one is loading thread).

The frist thread have always green light and run always at maximum possible framerate.. There are no deadlocks. There are no freezes. There is just background loading. And there are no particular problems with my patch. (and I'm already using that example for streaming ^^ in my terrain manager )

You sad I'm freezing cause of hard disk access. That's not true, harddisk access is done in the second thread. Rendering thread will not be freezed by this.


@Hybrid: just take a look to that patch:

viewtopic.php?f=2&t=46041

[hendu]

Hm, this patch would force two GL contexts? Bad idea, it should be left up to the user. Some drivers are not happy with multiple contexts.

[Nadro]

Yes I said 100% seriously, because You didn't say before that You have already created 2 OpenGL contexts in Your example (You wrote that first device doesn't have an OpenGL video driver...), please be more precise in Your description next time. But even if You have 2 contexts, Your solution doesn't work better than my (without share lists You have to create a texture in a main thread, so without PBO freeze required for an uploading texture will be noticeable and Your main thread doesn't have green light every time like in my approach, PBO will solve this problem, only image uploading and decompression is done in second thread, but remember that for texture streaming etc. users mostly doesn't need an access to an uncompressed image data, so automatic call of ITexture::lock() is waste of GPU power for them). I also don't like an idea where You use more than one OpenGL context. For OpenGL ver <= 4.2 the best idea is have one thread (main) for all related GPU things, in future it should change.

Short summary of both approaches:

REDDemon:
- Require 2 OpenGL contexts for operations like texture streaming, dynamic loading splash etc
- We have an access to an uncompressed image after image loading

Nadro:
- We don't have an access to an uncompressed image after image loading (most users doesn't need it)
- createImageFromFile method is little faster than in REDDemon solution
- require some internal API changes related to an IImage class.

For me really a second solution is better, remember that reupload of modified uncompressed image to a GPU as compressed texture isn't cheap operation.

@Hendu
This patch wouldn't force 2 contexts, but it will require from an user to create a second OpenGL context if he'll want to loading an images data from a separate thread (in my solution second OpenGL context isn't required).