Skinning Bone matrix with 4x3 instead of 4x4

You are an experienced programmer and have a problem with the engine, shaders, or advanced effects? Here you'll get answers.
No questions about C++ programming or topics which are answered in the tutorials!
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Mel »

Just this:

Code: Select all

 
this->mesh->setHardwareMappingHint(EHM_DYNAMIC, EBT_VERTEX_AND_INDEX); // Didn't really help..
 
You shouldn't upload your mesh every frame, or else, you lose quite an important advantage of the shader skinning, which is the ability to skin very large meshes (30.000 tris for instance, isn't that crazy to skin on the GPU) in fact, a large deal of the skinning process is to send the mesh to the GPU each frame, and it is even worse the larger the mesh. You want this:

this->mesh->setHardwareMappingHint(EHM_STATIC, EBT_VERTEX_AND_INDEX);

and this grants you that you store the mesh on the GPU permanently, until you delete it. The other, the setHardwareSkinning() setting just grants that Irrlicht won't try to animate any vertex on its own. it will only performs the joints calculations, which are vital to this example.

Also, there are some comparisons you really don't need (so far), as, if it isn't Direct3D9, it will be OpenGL
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
Vectrotek
Competition winner
Posts: 1087
Joined: Sat May 02, 2015 5:05 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Vectrotek »

Yes! I forgot to change it back to that what it was! Did now..
It still doesn't really do anything for the frame rate.

I think Irrlicht H/W Skinning needs a LOT of attention with Culling, Normals, Buffer Boxes, etc, etc, etc.

You could download my project and pick it apart as you know more of this than me!
I'd love some help! Try running my H/W Skinning project it to see if you also get minimal frame rate difference..

Thanks for all your valuable input Mel!

By the way, I might, later, want to ask you about an HLSL shader I'm working on..
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

I've been able to get the skinning shader to accept up to 60 bones which is not bad, it doesn't have advance lighting yet but it's an improvement. I was never able to get it to accept a float4x3 bone matrix only float4x4. I'm not sure what on irrlicht side does not like or agree with the float4x3 matrix. "Nadros area of expertise"

Image


I'm going to try to implement the dual Quaternion Skinning shader next as I've read it's faster and better visual results and less expensive as matrix skinning.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Mel »

https://youtu.be/jA5tWZg_7Kk

I reached 78 bones on that animation, lighting and all, using the 4x3 matrices approach. I want to be able to use the dqs and push more than 100 bones on a single model. It sacrifices the scaling of the bones but on the other hand, some stuff gets simplified for instance, the limb twists ave the shoulders deformation
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

Can you help us fix what's wrong with ours?
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Mel »

From what i see, the problem is that some vertices don't get a correct bone, so they get dragged to the origin of coordinates, instead of, for instance, getting a base bone, and remaining in place. The bone set up seems correct, I do the same with the vertex colors. What is the vertex shader code?
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

Mel heres vertex shader. I'd like to use 4x3 If I can.
Thanks Glitch

Code: Select all

 
 
#define MaxBones 60
 
// Input parameters.
float4x4 wvp;
float4x4 Bones[MaxBones];
 
float3 Light1Direction = normalize(float3(1, 1, -2));
float3 Light1Color = float3(0.9, 0.8, 0.7);
 
float3 Light2Direction = normalize(float3(-1, -1, 1));
float3 Light2Color = float3(0.1, 0.3, 0.8);
 
float3 AmbientColor = 0.2;
 
texture Texture;
 
sampler Sampler = sampler_state
{
    Texture = (Texture);
 
    MinFilter = Linear;
    MagFilter = Linear;
    MipFilter = Linear;
};
 
 
// Vertex shader input structure.
struct VS_INPUT
{
    float4 Position : POSITION0;
    float3 Normal : NORMAL0;
    float2 TexCoord : TEXCOORD0;
    float4 BlendWeight : BLENDWEIGHT;
    float4 BlendIndex : BLENDINDICES;
};
 
 
// Vertex shader output structure.
struct VS_OUTPUT
{
    float4 Position : POSITION0;
    float3 Lighting : COLOR0;
    float2 TexCoord : TEXCOORD0;
};
 
 
// Vertex shader program.
VS_OUTPUT main(VS_INPUT input)
{
    VS_OUTPUT output;
 
 
    
    // Blend between the weighted bone matrices.
    float4x4 skinTransform = 0;
    
    skinTransform += Bones[input.BlendIndex.x] * input.BlendWeight.x;
    skinTransform += Bones[input.BlendIndex.y] * input.BlendWeight.y;
    skinTransform += Bones[input.BlendIndex.z] * input.BlendWeight.z;
    skinTransform += Bones[input.BlendIndex.w] * input.BlendWeight.w;
    
    // Skin the vertex position.
   float4 position =  mul(input.Position,skinTransform);  
    
    output.Position = mul(position , wvp);
    //output.Position = mul(wvp , position);
 
    // Skin the vertex normal, then compute lighting.
    float3 normal = normalize(mul(input.Normal, skinTransform));
    
    float3 light1 = max(dot(normal, Light1Direction), 0) * Light1Color;
    float3 light2 = max(dot(normal, Light2Direction), 0) * Light2Color;
 
    output.Lighting = light1 + light2 + AmbientColor;
 
    output.TexCoord = input.TexCoord;
    
    return output;
}
 
Vectrotek
Competition winner
Posts: 1087
Joined: Sat May 02, 2015 5:05 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Vectrotek »

Very nice "Dangerous, Demented Asylum Dweller" The Glitch! Lovely! I just know that's yours!
Also absolutely beautiful looking "Winged Sonja" Mel!

O.K.
I'm looking at The Glitch's HWS-HLSL shader.. Looks cool, but I cant learn from it as I cant see his "Calling Function",
so here is my "less professional" version..
It comes from the old example (24. Hardware Skinning) but I altered it somewhat..
My normals don't get sent to the shader in an animated way
so Lighting isn't even attempted.
(you'll see in the shader that I ignored them completely)

I know nothing of the 4 X 3 Mat issue yet, but here is my altered HLSL shader.
(maybe you guys can give some pointers)

So, here goes.. Your wise comments would be appreciated!

"HWSkin.hlsl":

Code: Select all

 
 //  HLSL VERTEX PROGRAM (Ja, we know we have more bones in GLSL, but this concerns HLSL)
 
 #define MAX_JOINT_NUM 57   // 57 is MAX before compilation fails..  Got it up to 62!!
 #define MAX_LIGHT_NUM 8
 
 float4x4 JointTransform[MAX_JOINT_NUM];  // WE HAD 16 X 62 = 992.. So how does this impact on Hardware "Registers"??
 float4x4 mWorldViewProj;
 float4x4 mWorld;
 
 // THESE DONT WORK IN HERE SO I DROPPED THEM TOTALLY..
 // Anyway, I don't think Vertex Programs is the right place for lighting.
 // I'll worry about lighting in the Fragment Shader when the Normals are sent
 // to the Vertex Program "properly animated".
 // I can do lighting pefectly in GLSL and HLSL but NOT under H/W Skinning activated.
 // (is hardware skinning really all that hot? I don't seem to see much of a frame rate improvement?)
 
 // Getting rid of lights to see if we can get more bones in.. We could (hey hey!), but NOT ENOUGH!!
 // This I tried after Mel's comment on "Max Registers" being exceeded.
 // float3 lightPosArray[MAX_LIGHT_NUM];   (In a perfect world this happens in the Fragment Shader..)     
 // float4 lightColorArray[MAX_LIGHT_NUM];
 // ----------------------------------- ---- --- -- - 
 struct VS_OUTPUT 
  {float4 Position   : POSITION0;
   float2 TexCoords  : TEXCOORD0; 
   float2 TexCoords2 : TEXCOORD1;
   float4 Color      : COLOR0;     // In this implementation Vertex Colour, it seems, is used for Weights and Indices..
                                   // How it is this structured ?..(i ain't got a clue)
  };
 // ----------------------------------- ---- --- -- - 
 VS_OUTPUT vertexMain (float3 Position     : POSITION0,
                       float2 TexCoords    : TEXCOORD0,
                       float2 TexCoords2   : TEXCOORD1, // Is this used?
                       float4 Color        : COLOR0,
                       float3 VertexNormal : NORMAL
                      )
  {VS_OUTPUT  OUT;
   float4     MeshPos = float4(Position.x,Position.y,Position.z,1.0);
   float4x4   ModelTransform = mWorldViewProj;
   int        verCol = int(Color.r  * 255.9);       // What is the "255.9" all about?
   float4x4   matTran = JointTransform[verCol - 1]; // O.K. this facilitates an array of SIXTEEN Values!
                                                    // DO WE REALLY NEED ALL 16 VALUES..
                                                    // OR.. Could we have a longer array of simple floats..
                                                    // BUT! Indices are INTEGERS and Weights are FLOATS!! (clue?)
                                                    // How do we RESTRUCTURE this there..
   // I have no idea what's going on here!
   // Is the following still relevant in this HWS discussion?
   verCol = int(Color.g  * 255.9);
   if(verCol != 0) {matTran += JointTransform[verCol - 1];}
   verCol = int(Color.b  * 255.9);
   if(verCol != 0) {matTran += JointTransform[verCol - 1];}
   verCol = int(Color.a  * 255.9);
   if(verCol != 0) {matTran += JointTransform[verCol - 1]; }
 
   // Apply final transformation.
   ModelTransform  = mul(matTran, ModelTransform);
   OUT.Position    = mul(MeshPos, ModelTransform);
   OUT.TexCoords   = TexCoords;
   OUT.TexCoords2  = TexCoords2;
   float4 finalCol = float4(1,1,1,1);
 
   // All Light stuff disabled as they just complicate things unecessarily..
 
   OUT.Color = clamp(finalCol,0.3,1.0);
   // This "should" just render the normals so we have something to look at as I have no real lighting in HWS yet..
   OUT.Color.x = (VertexNormal.x / 2.0) + 0.5;
   OUT.Color.y = (VertexNormal.y / 2.0) + 0.5;
   OUT.Color.z = (VertexNormal.z / 2.0) + 0.5;
   
   // OFF-TOPIC..
   // I consider the fact that Irrlicht insists on rendering textures when in shader mode
   // (even if none have been specified, see GLSL) a serious Irrlicht bug because this makes it
   // so I can't set a DIFFERENT "Diffuse" image for the model if I should want to..
   return(OUT);
  }
 
Last edited by Vectrotek on Fri Jan 15, 2016 4:47 pm, edited 1 time in total.
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

Vectrotek the float4x3 issue is this :
If you use a float4x3 instead of float4x4 you can increase your bone count by a pretty significant count since it uses less space, remember d3d9 hlsl shader model 3.0 is limited to a certain amount.

The Irrlicht problem is that the shader I posted should be working fine but the mesh is loaded it is destroyed similar to you picture. The problem is either us or maybe Irrlicht specific which is what I aim to find.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Mel »

My bet is that probably, some vertices get excluded from the vertex shader as their driving bones aren't reached by the constants (i.e. the bone index is larger than 62) Try getting a model that has less bones than the maximum, then, debug the skinning until you have a working shader, and increase the bone count to see what happens with the bones that get

As for the usage of the 4x3 matrices is straight forward in the shader, not so in the shader callback. In the shader, change into float4x3 the joint matrices, and when you have your 4x3 skinTransform matrix built, extend it to a 4x4 matrix, and you're done. In your shader callback, now you have to remove the last row (or column... check it by yourself so you can do what is best for you, it is the row/column that is always 0,0,0,1) of every joint matrix, as it is now a redundant data, and send it to the shader, and here comes the catch. Irrlicht doesn't have 4x3 matrices, so you will have to build your own array and fill it using pointer maths so you can pass it to the shader.

The good part is that you pass this whole data block in a single call.

Also, if you're familiar with the transformation matrices, you should if you want to achieve something skinning in hardware, you know a 4x4 matrix is actually a 3x3 rotation matrix with a 4th row/column added which stores the translations. Thus, to calculate properly the normals, you have to get the dominant 3x3 matrix out of the skin transform matrix (you get it removing the last column of the 4x3 matrix) and multiply the normal/bi-vector/tangent to orient it correctly
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

Yeah Mel your right I'm positive my issue is that I'm not building a float4x3 matrix in my shader callback for the shader I'm not quite sure how to do that though.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by Mel »

What i do is to allocate 4x3xMAXJOINTS floats

Code: Select all

#define MAXJOINTMATRIXCOUNT 79
float jointMatrices[MAXJOINTMATRIXCOUNT*12]; //This is a variable of the shader callback, it grants that the memory is consecutive
Actually, it is a static allocation... and then fill the whole structure with the joints matrix data.

Code: Select all

            for(i = 0;i<SMesh->getAllJoints().size() && i<MAXJOINTMATRIXCOUNT ;i++)
            {
                mat = SMesh->getAllJoints()[i]->GlobalAnimatedMatrix * SMesh->getAllJoints()[i]->GlobalInversedMatrix;
                //jointMatrices[i] = mat;
 
                jointMatrices[i*12+0] = mat[0];
                jointMatrices[i*12+1] = mat[1];
                jointMatrices[i*12+2] = mat[2];
 
                jointMatrices[i*12+3] = mat[4];
                jointMatrices[i*12+4] = mat[5];
                jointMatrices[i*12+5] = mat[6];
 
                jointMatrices[i*12+6] = mat[8];
                jointMatrices[i*12+7] = mat[9];
                jointMatrices[i*12+8] = mat[10];
 
                jointMatrices[i*12+9] = mat[12];
                jointMatrices[i*12+10] = mat[13];
                jointMatrices[i*12+11] = mat[14];
            }
It is pretty straight forward
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by devsh »

I shall be making my own Hardware Skinning Dual Quaternion system soon... with proper normal calculation :D
The_Glitch
Competition winner
Posts: 523
Joined: Tue Jan 15, 2013 6:36 pm

Re: Skinning Bone matrix with 4x3 instead of 4x4

Post by The_Glitch »

I hate you lol
Post Reply