Skinning Bone matrix with 4x3 instead of 4x4

[The_Glitch]

I'm trying to increase my bone count by using float4x3 instead of float4x4. My mesh becomes horribly destroyed I'm posting my vertex shader perhaps someone can spot something I've missed.

Code: Select all

 
float4x4 uMVP;
float4x3 uBone[59];
 
sampler2D uTexture : register(s0);
 
struct VS_INPUT
{
    float4 Position : POSITION;
    float3 Normal : NORMAL;
    float2 TexCoord : TEXCOORD0;
    float4 BlendWeight : BLENDWEIGHT;
        float4 BlendIndex : BLENDINDICES;
};
 
struct VS_PS
{
    float4 Position : POSITION;
    float2 TexCoord : TEXCOORD0;
};
 
struct PS_OUTPUT
{
    float4 Color : COLOR0;
};
 
VS_PS VertexShaderFunction(VS_INPUT Input)
{
    VS_PS Output;
    
    float4 Position = float4(0.0, 0.0, 0.0, 0.0);
    
    for (int i = 0; i < 4; ++i)
            //Position += float4(mul(Input.Position, uBone[int(Input.BlendIndex[i])]) * Input.BlendWeight[i]); // commented out for float4x3 below
          Position += float4((mul(Input.Position.xyz, uBone[int(Input.BlendIndex[i])]) * Input.BlendWeight[i]),1.0);
        
 
    Output.Position = mul(float4(Position.xyz ,1.0),uMVP);  
    Output.TexCoord = Input.TexCoord;
 
    return Output;
}
 

[Mel]

Irrlicht won't fill by default the BLENDWEIGHTS and BLENDINDICES semantics, you have to use the shader pipeline version of irrlicht which implements the flexible vertex formats. If you're not using them, you can store the vertex indices on the vertex colors and the vertex weights on some other places. For instance, I use the vertex tangents, I pack the weights and the tangents together, so I can have animated meshes and normal mapped models at the same time. Keep also in mind that these data are passed on the TEXCOORD1 and TEXCOORD2 semantics, at least, again, on the standard Irrlicht

[Mel]

Also, if you don't use bone scalings, you can have even more bones available by using dual quaternion skinning, 2x4 floats instead of 3x4 floats

[The_Glitch]

I'm using the shader pipeline hardware skinning example and the mesh is destroyed horribly. I believe theres something in the example that expects a float4x4 but I've changed things and nothing works the mesh is destroyed still. Heres the shader I'm using from nvidia compiles fine but the mesh is not rendered :/

Code: Select all

 
 
// Matrix Palette Skinning - DirectX 9 Demo
 
float4x4 WorldViewProj : WorldViewProj;
float4x3 Bones[29];
float3   LightVec;
 
struct VS_INPUT {
    float3 Position   : POSITION;
    float3 Normal     : NORMAL;
    float3 Color      : COLOR0;
    float2 TexCoord0  : TEXCOORD0;
    float3 S          : TEXCOORD1;
    float3 T          : TEXCOORD2;
    float3 SxT        : TEXCOORD3;
    float4 Weights    : TEXCOORD4;
    float4 Indices    : TEXCOORD5;
};
 
struct VS_OUTPUT {
    float4 HPosition : POSITION;
    float2 TexCoord0 : TEXCOORD0;
    float2 TexCoord1 : TEXCOORD1;
    float3 Color0    : COLOR0;
};
 
VS_OUTPUT main(VS_INPUT IN)
{
    VS_OUTPUT OUT;
 
    float i;        // Index into matrix palette
 
    float4 inPos;
    inPos.xyz = IN.Position;
    inPos.w = 1.0;
 
    // Pass through texcoords
    OUT.TexCoord0.xy = IN.TexCoord0.xy;
    OUT.TexCoord1.xy = IN.TexCoord0.xy;
 
    float3 tempPos, tempNormal;
 
    /////////////////////////////////////////////////////////////////////
    // FIRST BONE
    // We don't worry about the ELSE condition because we defined the 
    // initial conditions.
 
    // grab first bone matrix
    i = IN.Indices.x;
 
    // First transformed position and normal
    tempPos = mul(inPos, Bones[i]) * IN.Weights.x;
    tempNormal = mul(IN.Normal, (float3x3)Bones[i]) * IN.Weights.x;
 
    /////////////////////////////////////////////////////////////////////
    // SECOND BONE
    // Next bone.
 
    if(IN.Weights.y > 0.0f)
    {
        i = IN.Indices.y;
 
        // Add second transformed position and normal
        tempPos += mul(inPos, Bones[i]) * IN.Weights.y;
        tempNormal += mul(IN.Normal, (float3x3)Bones[i]) * IN.Weights.y;
 
        /////////////////////////////////////////////////////////////////////
        // THIRD BONE
        // Note we only skin the normal by the first two bones, these are by 
        // far the most significant.
 
        if(IN.Weights.z > 0.0f)
        {
            i = IN.Indices.z;
 
            // Add third transformed position only
            tempPos += mul(inPos, Bones[i]) * IN.Weights.z;
 
            /////////////////////////////////////////////////////////////////////
            // FOURTH BONE
 
            if(IN.Weights.w > 0.0f)
            {
                i = IN.Indices.w;
                
                // Add fourth transformed position only
                tempPos += mul(inPos, Bones[i]) * IN.Weights.w;
            }
        }
    }
 
    // Normalize the final skinned normal
    float3 finalNormal = normalize(tempNormal);
    OUT.Color0.xyz = max(dot(finalNormal, LightVec), 0).xxx;
 
    float4 finalPos;
    finalPos.xyz = tempPos;
    finalPos.w = 1.0;
 
    // Transform the final skinned position
    OUT.HPosition = mul(finalPos, WorldViewProj);
    return OUT;
}
 
 

[Vectrotek]

Hi! The_Glitch!
I started looking at Example 10 etc like you suggested.. I also looked at your shader..

[this was edited.. Ignore the last stuff]

With HLSL I Get..


With GLSL I get.. (needed a trick to get it working with GLSL)


It looks like with HLSL there is "even less bones" available??

This immediately bothers me, but I'll admit I need to take a much closer look at it..
I'm in the deep end here with the way the Callback sends things like Weights and Indices to the shaders!
(I wish I understood the "SVN / Trunk / Pipeline" thing as that would probably help me understand your code)

Would it be correct to say: "Your H/W Skin shader started of as the HLSL Vertex Program in Example 10, then evolved to what it is now, with extra goodies offered by "Shader Pipeline" advantages" ?

I am trying to keep my own GLSL and HLSL progress as parrallel as possible because the one always helps me to better understand the other..

I'm beginning to fee like a true Irrlicht noob!

[Vectrotek]

Oh! I was also wondering.. When a good Idea in the "SVN & Trunk" thing turns out to be useful, then how does it find it's way into the next "Release" of Irrlicht?
I ask because I've noticed a few cool things in older examples and projects that didn't seem to make its way into later releases. (for instance proper shader cube-mapping and the MAX IMAGES thing for shaders)
Is there some good material to read on this (excuse my ignorance) "Svn & Trunk" thing and how to recompile the engine with the "+ code" and "- code"?

Last question.. Why, when I change things in the engine code and compile, my HLSL code suddenly stops working but GLSL is O.K.?
I do have a version of the Direct X SDK installed for engine recompiles to work.. Could it be this version that DX SDK that causes problems?

[The_Glitch]

I felt like my shader turned out like yours because maybe there was something irrlicht was doing that I could not find I asked around but no one replied. Use my shader I posted above the second one I got from NVidia's site and see what you get. It doesn't surprise me about the opengl things as all the development and corrections seem to be poured into opengl in this engine instead of direct3d sadly to say.

[The_Glitch]

Also I should not I used a model that has less than 40 bones and it gave the same results.

[Vectrotek]

Mmm.. I'm in head first!
The_Glitch, Tell me.. That super cool evil looking "Gas Masked, Cloacked Hammerman" with the cool cloth movement.. Is that your work in MAX?

[Vectrotek]

I know you are using HLSL, but for what its worth, this is how I got GLSL working for Example 10..
(this comes straight from Example 10)

Code: Select all

 
 // Part of "void HWSkinCB::OnSetConstants(video::IMaterialRendererServices* services, s32 userData)" in Example 10..
 
 // ...
 // HLSL..
 if(services->getVideoDriver()->getDriverType() == EDT_DIRECT3D9)
  {bool success = services->setVertexShaderConstant("JointTransform", JointArray, mesh->getAllJoints().size() * 16);
  }
 // GLSL..  
 // Using this secret "[0]" in Appcode works for GLSL..
 if(services->getVideoDriver()->getDriverType() == EDT_OPENGL)
  {bool success = services->setVertexShaderConstant("JointTransform[0]", JointArray, mesh->getAllJoints().size() * 16);
   // see the [0] for glsl..
  }
 // see "JointTransform" vs "JointTransform[0]"..
 

This simply helps me to always see the differences between HLSl and GLSL thus increasing my understanding of it.

You're right! We "demand more hard bones" and while were complaining, we also demand more image textures for HLSL as 4 is a bit lame! C'mon devs!

Later!

[The_Glitch]

The masked character is from a SilentHill game he has a high bone count I think 89 bones. Does what you posted work with hlsl as if it's glsl that won't do me any good.

[Mel]

DX9 can only set 1024 constants (namely floats) so the amount of 4x4 matrices you can set is exactly 64, including world transformations, and other stuff. In the most practical case, you can get 58-62 bones like that, or else, the shader won't compile. Using 4x3 matrices you can use up to 78-80 bones, more or less, depending on the effects you want to include, and using dual quaternions, you can set around 115-120 bones. This is a D3D9 limitation, so there is nothing you can do to get over that. While some might find this limiting, it also enforces the eficiency as it is granted that only a small amount of information will be passed to the shader.

In GL seems there is not this limit. and over D3D10 the limitations are higher, and you can transfer memory blocks as shader constants, so you can add even more information.

And even being things like that, you can use vertex textures to store the joint data, resulting in an almost limitless amount of joints.

[Vectrotek]

@The Glitch..
This is a ballsup allright!
I'm posting this because it may be something you
have considered already with your shader and lighting.
Also, just to keep this H/W Skinning question alive.

You see, with H/W Skinning the Skeleton moves like you have it animated,
but when looking at it in "debug mode" I noticed that the "Normals", like the
"Wireframe" representations do not!

I don't know if your normals are animated, but if they are not (like mine)
then the problem gets even bigger!

(I'm still trying to figure how to use Shader Pipeline) Aaargh!

Just for interests sake:

The Normals are animated in HLSL when H/W Skinning is OFF..



The Normals are NOT animated in HLSL when H/W Skinning is ON..



Now I know this does not help you directly, but I believe, that many times,
when looking at a problem from a different perspectives by different people
clues for a solution sometimes comes out of hiding!

So I have the same question as you:
"Why do All the Bones NOT displace ALL the Vertices in HLSL like they do in GLSL?"

The 4 x 4 versus 4 x 3 matrix issue is still a bit above me but I'm trying.

Final note: My Shaders depends heavily on Normals to be animated along with the Bones and Vertices
and if I, or anyone else, dont have them moving with the animation, then It's really a useless
exersize!

Do your Normals move along with the bones in debug view?

[Vectrotek]

I could very well make peace with the fact that HLSL offers a MAX of 57 bones, but seeing that GLSL can handle up to 255 proves that the
hardware can handle it thus begging the question.. "Why?"
Mel seems to have a good grasp of the problem though and what he says makes sense..
Anyway, If the frame-rate is much improved then its probably also due to the fact that normals are not recalculated
at each frame which in turn means that Tangents and Binormals also wont be correct..
Like I said.. It looks like a giant ballsup!

[Vectrotek]

If what Mel stated is correct, which it certainly seems, then I have a filthy suspicion that we've just witnessed the death of DX9 in terms of hardware skinning.. Sorry chaps!
I cannot see the all important normals surviving this either.
Cheers!