Page 1 of 2

Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!

Posted: Tue Jul 27, 2010 7:06 pm
by stefbuet
[Updated 7/03/2011]
-new edge detection filter also based on normal & depth but now constant edge size even if you zoom out, more exact detection thanks to a 9x9 kernel instead of old 2*2 (cross) and noise removal with sin function.
-----


Here is a shader I've done which is using depth and normals to find edges.
Depth can't be used alone because edges which are inside of an object have a continuous depth, that's why I'm also using normals.

Please, if you find a faster/better may to do it, post it :)
Note : on the screenshot, the floor have strange white stuff, it's because it contain grass ;)

Result :

Image

VS:

Code: Select all

// simple vertex shader
// Author:Stefbuet
// Date:20/04/2010

//This shader can be used by a wide range of pixel shader.
//It is just giving to the PS the texture coords, vertex pos & color.

varying vec4 textureCoord;

void main() {
    
    //texture coordinate'll be interpolated to the PS
    textureCoord=gl_MultiTexCoord0;

    //don't change any vertex position & color
    gl_FrontColor=gl_Color;
    gl_Position=ftransform();
    

}

PS:

Code: Select all


// edge detection pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map


uniform sampler2D depthMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       gl_FragColor=vec4(1.0,1.0,1.0,1.0);

   }
   else {  
     
       gl_FragColor=vec4(0.0,0.0,0.0,0.0);
     
   }


}

You have to give to the pixel shader the following data :
depthMap : Rendertarget : XYZ = normal, W = depth (view space)
SCREENX, SCREENY : depthMap size, must be floats.

This is a postprocess effect, so you might apply this shader to the quad you're using to do this post processing and put the depthMap texture on it.

Now, here is the antialiasing stuff :
We blur detected edge with a simple 3x3 blur kernel.

Result of the effect :

Image

new pixel shader :

PS :

Code: Select all

// antialiasing pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)

uniform sampler2D depthMap;
uniform sampler2D renderMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       vec4 finalColor=texture2D(renderMap, -textCoord.xy);
       for(int i=0; i<8; i++)
           finalColor+=texture2D(renderMap, -textCoord.xy+offsetArray[i]);
       
       finalColor/=8.0;
       gl_FragColor=finalColor;

   }
   else {  
     
       gl_FragColor=texture2D(renderMap, -textCoord);
     
   }


}

You have only to give a new data to the shader:
renderMap : render target which contain your scene.

This method is used by Stalker to do antialiasing.
See http://http.developer.nvidia.com/GPUGem ... ter09.html after the deferred lighting part.

I saw that bluring far away things may let them too blurred. Maybe modulate the blur in function of the depth of pixels is a good idea, but I haven't tested it yet!

Hope you will enjoy it :P

Posted: Tue Jul 27, 2010 9:11 pm
by slavik262
Cool. I was thinking of writing a similar effect - I'll definitely use this as a basis for mine.

Posted: Tue Jul 27, 2010 9:14 pm
by andres
Very good. Thanks!

Posted: Wed Jul 28, 2010 8:42 am
by wing64
Good job. :D

Posted: Wed Jul 28, 2010 10:12 am
by greenya
Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON --- incredible indeed :roll:

Posted: Wed Jul 28, 2010 11:59 am
by slavik262
greenya wrote:Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON
:shock: How is that possible?

Posted: Wed Jul 28, 2010 1:18 pm
by ACE247
Whats going on here? 48 without and 230 with??? Some hardware fps boost.
And i see you didn't mix them up, since the blurred vs unblurred is clearly visible.

Posted: Wed Jul 28, 2010 4:18 pm
by DtD
lol on the FPS. Looks great!

Posted: Thu Jul 29, 2010 3:40 pm
by ACE247
Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?

Posted: Thu Jul 29, 2010 4:58 pm
by booe
Hum, stefbuet, you must have unwittingly switched the "on"/"off" labels.
The "on" label should belong to screenie with 48 fps, and "off" label to one with 230.

Posted: Thu Jul 29, 2010 5:43 pm
by ACE247
No he hasn't, you can clearly see the smoothed out edges in the different screenshots.

Posted: Thu Jul 29, 2010 6:16 pm
by slavik262
ACE247 wrote:Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?
It's funny you should say that; I was looking at some soft particle code the other day from the DirectX SDK. Soft particles work by rendering the particles to a seperate RTT. You also render the depth of the particles, and then when you add the particles back into the final frame, you compare the depth of the frame against the depth of the particles. You only draw the particles if they are in front of the rest of the frame, so that way they don't clip into geometry.

Posted: Thu Jul 29, 2010 6:24 pm
by stefbuet
Ok, here is the FPS answer :
3dsmax was opening when I took the screenshot. :P
The shader don't change the FPS, however as I'm on a geforce 260GTX, it might change a little for lower spec graphic card.

Posted: Thu Jul 29, 2010 7:26 pm
by slavik262
Since it's completely dependant on the pixel shader pipeline, performance is mostly a hardware question. On a decent graphics card it won't slow anything down, but it will choke on crap hardware (which is why games usually make post-processing optional)

Posted: Tue Nov 30, 2010 2:39 pm
by devsh
you should be careful with normal map edge detection cause otherwise you will end up blurring bumpmapped surfaces so be sure to attenuate your normal found edges by the depth edge detection.. I could think of a very complicated algo but it would be slightly inefficient... I made my own that also takes the color buffer into account so I might share it in some time