Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!
pixel alias has a constant size too... therefore more edges are present far away and AA makes far away pictures blurry (even the real one) also read third paragraph down
Make sure you use Gaussian Weights+Blur and calculate them for a 3x3 kernel with std var of 1.f
You are looking 2 pixels away (1.5 actually)... this wont aid finding an edge I can guaranteee that 100% cause I tried... it just makes the lines on edge detect thicker which makes the whole thing blurry cause the thing isnt an actual edge... getting a sample from 1.5 away will give you an interpolated value between pixel 1 and 2 which can cause haloes etc so the edges could appear to be NOT touching the objects by just 0.5 of a pixel or make the edge lines up to 50% darker
Also please attenuate the normal comparison edges by the depth difference so if there isnt much/any the app assumes its the same surface == not blurring bump maps... this approach however WILL improve parallax occlusion (raycasted) height maps as they modify the fragment depth in the zbuffer so they WILL be detected
Make sure you use Gaussian Weights+Blur and calculate them for a 3x3 kernel with std var of 1.f
You are looking 2 pixels away (1.5 actually)... this wont aid finding an edge I can guaranteee that 100% cause I tried... it just makes the lines on edge detect thicker which makes the whole thing blurry cause the thing isnt an actual edge... getting a sample from 1.5 away will give you an interpolated value between pixel 1 and 2 which can cause haloes etc so the edges could appear to be NOT touching the objects by just 0.5 of a pixel or make the edge lines up to 50% darker
Also please attenuate the normal comparison edges by the depth difference so if there isnt much/any the app assumes its the same surface == not blurring bump maps... this approach however WILL improve parallax occlusion (raycasted) height maps as they modify the fragment depth in the zbuffer so they WILL be detected
right here is my shader... I could have made it better by attenuating the normals depending on depth like I said and weighing the diagonal samples by 1/sqrt(2) to compensate for the length
In the image, first row up and from left: edge detection using color buffer only, final outcome attenuated from all 3 methods, depth, color buffer
Bottom row: edge detection with depth only, normals only, normals+depth and no color edge attenuated, normal buffer
as we can see the normal based edge detection is the crucial one and can be used on its own... depth on its own wont do.. you can add/mix methods in order to get a higher quality render
Link to a 1024 render picture 4k by 2k (the edge detection were screenshotted from a 512by512 app so dont be picky)
http://img202.imageshack.us/img202/4696/edgeaa.jpg
In the image, first row up and from left: edge detection using color buffer only, final outcome attenuated from all 3 methods, depth, color buffer
Bottom row: edge detection with depth only, normals only, normals+depth and no color edge attenuated, normal buffer
as we can see the normal based edge detection is the crucial one and can be used on its own... depth on its own wont do.. you can add/mix methods in order to get a higher quality render
Link to a 1024 render picture 4k by 2k (the edge detection were screenshotted from a 512by512 app so dont be picky)
http://img202.imageshack.us/img202/4696/edgeaa.jpg
Really nice output.
I'm checking 1.5 px away because in glsl the next px is 1.5f far from the current pixel, while it's 1.0f in hlsl ?
I was checking with a cross kernel instead of a 3x3 kernel as it's described in gpugems for Stalker game, however if I were using a 3x3 kernel how should I check either if the pixel is on an edge or not? I were just checking if at least 1 pixel depth difference or normal global difference is more than a critical value, but as you can see, lines produced are too big, not adapted to the distance camera-pixel.
Once again, you shader looks great
I'm checking 1.5 px away because in glsl the next px is 1.5f far from the current pixel, while it's 1.0f in hlsl ?
I was checking with a cross kernel instead of a 3x3 kernel as it's described in gpugems for Stalker game, however if I were using a 3x3 kernel how should I check either if the pixel is on an edge or not? I were just checking if at least 1 pixel depth difference or normal global difference is more than a critical value, but as you can see, lines produced are too big, not adapted to the distance camera-pixel.
Once again, you shader looks great
as I said lines are too big because of the 1.5... if you used 1.f instead then the lines would be smaller
a 3x3 kernel gives AntiAliased edge detection (smooth-ish lines) because of the diagonal checks, mine are not weighted by the length so it would be possible to gain better result unrolling the for functions and attenuating the ones which are diagonal by 1/sqrt(2)
here is my shader code... process your scene with it to see a comparison (especially the grass part).. remember to play around with the settings
a 3x3 kernel gives AntiAliased edge detection (smooth-ish lines) because of the diagonal checks, mine are not weighted by the length so it would be possible to gain better result unrolling the for functions and attenuating the ones which are diagonal by 1/sqrt(2)
here is my shader code... process your scene with it to see a comparison (especially the grass part).. remember to play around with the settings
Code: Select all
// simple fragment shader
// 'time' contains seconds since the program was linked.
uniform float imgX,imgY,exposure;
uniform vec3 atten; // (1.0,0.47,0.71)
uniform sampler2D depth;
uniform sampler2D normal;
uniform sampler2D col;
void main()
{
float orig = texture2D(depth,gl_TexCoord[0].xy).r;
float diff = 0.0;
float normaldiff = 0.0;
float coldiff = 0.0;
vec3 orig_nml = normalize(texture2D(normal,gl_TexCoord[0].xy).rgb*2.0-1.0);
vec2 newTC;
vec3 orig_col = texture2D(col,gl_TexCoord[0].xy).rgb*vec3(0.3,0.59,0.11);
for (int i=-1; i<2; i++) {
for (int j=-1; j<2; j++) {
newTC = clamp(gl_TexCoord[0].xy-vec2(i,j)/vec2(imgX,imgY),0.0001,0.9999);
diff += abs(orig-texture2D(depth,newTC).r);
normaldiff += sin(acos(dot(orig_nml,normalize(texture2D(normal,newTC).rgb*2.0-1.0))));
coldiff += length(orig_col-texture2D(col,newTC).rgb*vec3(0.3,0.59,0.11));
}
}
gl_FragColor = pow(pow(coldiff*0.75,2.5)*atten.x+pow(normaldiff*0.2,1.5)*atten.y+pow(diff*0.3,0.85)*atten.z,exposure);
}
Hi.
I'd like to use your shader in my project, but my project is in DX9 so I need the HLSL version.
I have no idea of shader language but I intend to convert by my self.
It compiles without errors, but for sure it does not work at all.
Could you, please, help me?
I apologize for my poor English, but it doesn't my mother language.
Regards.
I'd like to use your shader in my project, but my project is in DX9 so I need the HLSL version.
I have no idea of shader language but I intend to convert by my self.
Code: Select all
// antialiasing pixel shader
// HLSL version
// Original GLSL version Author:Stefbuet
//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)
sampler2D depthMap: register(s0) ;
sampler2D renderMap: register(s1) ;
float4 textureCoord;
float2 offsetArray[8];
float4 pixelMain(): COLOR0
{
float2 textCoord=textureCoord.xy;
textCoord.x*=-1.0;
float currentDepth=tex2D(depthMap, -textCoord).w;
float3 normal=tex2D(depthMap, -textCoord).xyz*2.0-float3(1.0,1.0,1.0);
offsetArray[0] = float2(1.0/SCREENX, 1.0/SCREENY);
offsetArray[1] = float2(1.0/SCREENX, 0.0);
offsetArray[2] = float2(1.0/SCREENX, -1.0/SCREENY);
offsetArray[3] = float2(0.0, 1.0/SCREENY);
offsetArray[4] = float2(0.0, -1.0/SCREENY);
offsetArray[5] = float2(-1.0/SCREENX, 1.0/SCREENY);
offsetArray[6] = float2(-1.0/SCREENX, 0.0);
offsetArray[7] = float2(-1.0/SCREENX, -1.0/SCREENY);
//depth
float depthDif=0.0;
float minD=0.005;
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));
depthDif/=(1.0+5.0*currentDepth);
if(depthDif<0.01) depthDif=0;
depthDif*=25.0;
depthDif=min(1.0, depthDif);
depthDif=sin(depthDif*2.0);
//normals
float t=1-abs(dot(normal, tex2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-float3(1.0,1.0,1.0)));
t/=15.0;
t/=(1.0+2.5*currentDepth);
if(t<0.16) t=0.0;
t=sin(t*3.74);
depthDif+=t;
if(depthDif>0.2) {
float4 finalColor=tex2D(renderMap, -textCoord.xy);
for(int i=0; i<8; i++)
finalColor+=tex2D(renderMap, -textCoord.xy+offsetArray[i]);
finalColor/=8.0;
return finalColor;
}
else {
return tex2D(renderMap, -textCoord);
}
}
Could you, please, help me?
I apologize for my poor English, but it doesn't my mother language.
Regards.
Hi.
I managed it to work.
This is the HLSL code.
Or almost I think that works.
Regards.
I managed it to work.
This is the HLSL code.
Code: Select all
// antialiasing pixel shader
// HLSL version
// GLSL version Author:Stefbuet
//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)
sampler2D depthMap: register(s0) ;
sampler2D renderMap: register(s1) ;
float2 offsetArray[8];
float4 pixelMain(float4 textureCoord : TEXCOORD0): COLOR0
{
float2 textCoord=textureCoord.xy;
float currentDepth=tex2D(depthMap, textCoord).w;
float3 normal=tex2D(depthMap, textCoord).xyz*2.0-float3(1.0,1.0,1.0);
offsetArray[0] = float2(1.0/SCREENX, 1.0/SCREENY);
offsetArray[1] = float2(1.0/SCREENX, 0.0);
offsetArray[2] = float2(1.0/SCREENX, -1.0/SCREENY);
offsetArray[3] = float2(0.0, 1.0/SCREENY);
offsetArray[4] = float2(0.0, -1.0/SCREENY);
offsetArray[5] = float2(-1.0/SCREENX, 1.0/SCREENY);
offsetArray[6] = float2(-1.0/SCREENX, 0.0);
offsetArray[7] = float2(-1.0/SCREENX, -1.0/SCREENY);
//depth
float depthDif=0.0;
float minD=0.005;
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[0]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[1]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[2]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[3]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[4]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[5]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[6]).w-currentDepth));
depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[7]).w-currentDepth));
depthDif/=(1.0+5.0*currentDepth);
if(depthDif<0.01) depthDif=0;
depthDif*=25.0;
depthDif=min(1.0, depthDif);
depthDif=sin(depthDif*2.0);
//normals
float t=1-abs(dot(normal, tex2D(depthMap, textureCoord.xy+offsetArray[0]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[1]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[2]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[3]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[4]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[5]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[6]).xyz*2.0-float3(1.0,1.0,1.0)));
t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[7]).xyz*2.0-float3(1.0,1.0,1.0)));
t/=15.0;
t/=(1.0+2.5*currentDepth);
if(t<0.16) t=0.0;
t=sin(t*3.74);
depthDif+=t;
if(depthDif>0.2) {
float4 finalColor=tex2D(renderMap,textCoord.xy);
for(int i=0; i<8; i++)
finalColor+=tex2D(renderMap, textCoord.xy+offsetArray[i]);
finalColor/=8.0;
return finalColor;
}
else {
return tex2D(renderMap, textCoord);
}
}
Regards.