Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!

Post those lines of code you feel like sharing or find what you require for your project here; or simply use them as tutorials.
stefbuet
Competition winner
Posts: 495
Joined: Sun Dec 09, 2007 4:13 pm
Location: france

Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!

Post by stefbuet »

[Updated 7/03/2011]
-new edge detection filter also based on normal & depth but now constant edge size even if you zoom out, more exact detection thanks to a 9x9 kernel instead of old 2*2 (cross) and noise removal with sin function.
-----


Here is a shader I've done which is using depth and normals to find edges.
Depth can't be used alone because edges which are inside of an object have a continuous depth, that's why I'm also using normals.

Please, if you find a faster/better may to do it, post it :)
Note : on the screenshot, the floor have strange white stuff, it's because it contain grass ;)

Result :

Image

VS:

Code: Select all

// simple vertex shader
// Author:Stefbuet
// Date:20/04/2010

//This shader can be used by a wide range of pixel shader.
//It is just giving to the PS the texture coords, vertex pos & color.

varying vec4 textureCoord;

void main() {
    
    //texture coordinate'll be interpolated to the PS
    textureCoord=gl_MultiTexCoord0;

    //don't change any vertex position & color
    gl_FrontColor=gl_Color;
    gl_Position=ftransform();
    

}

PS:

Code: Select all


// edge detection pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map


uniform sampler2D depthMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       gl_FragColor=vec4(1.0,1.0,1.0,1.0);

   }
   else {  
     
       gl_FragColor=vec4(0.0,0.0,0.0,0.0);
     
   }


}

You have to give to the pixel shader the following data :
depthMap : Rendertarget : XYZ = normal, W = depth (view space)
SCREENX, SCREENY : depthMap size, must be floats.

This is a postprocess effect, so you might apply this shader to the quad you're using to do this post processing and put the depthMap texture on it.

Now, here is the antialiasing stuff :
We blur detected edge with a simple 3x3 blur kernel.

Result of the effect :

Image

new pixel shader :

PS :

Code: Select all

// antialiasing pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)

uniform sampler2D depthMap;
uniform sampler2D renderMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       vec4 finalColor=texture2D(renderMap, -textCoord.xy);
       for(int i=0; i<8; i++)
           finalColor+=texture2D(renderMap, -textCoord.xy+offsetArray[i]);
       
       finalColor/=8.0;
       gl_FragColor=finalColor;

   }
   else {  
     
       gl_FragColor=texture2D(renderMap, -textCoord);
     
   }


}

You have only to give a new data to the shader:
renderMap : render target which contain your scene.

This method is used by Stalker to do antialiasing.
See http://http.developer.nvidia.com/GPUGem ... ter09.html after the deferred lighting part.

I saw that bluring far away things may let them too blurred. Maybe modulate the blur in function of the depth of pixels is a good idea, but I haven't tested it yet!

Hope you will enjoy it :P
Last edited by stefbuet on Mon Mar 07, 2011 7:03 am, edited 2 times in total.
slavik262
Posts: 753
Joined: Sun Nov 22, 2009 9:25 pm
Location: Wisconsin, USA

Post by slavik262 »

Cool. I was thinking of writing a similar effect - I'll definitely use this as a basis for mine.
Last edited by slavik262 on Tue Jul 27, 2010 11:14 pm, edited 1 time in total.
andres
Competition winner
Posts: 78
Joined: Tue Jul 08, 2008 5:18 pm
Location: Guarapuava/Brazil
Contact:

Post by andres »

Very good. Thanks!
Prof. Andres Jessé Porfirio
Federal Technological University of Parana (UTFPR)
www.andresjesse.com
http://irrrpgbuilder.sourceforge.net

Image
wing64
Competition winner
Posts: 242
Joined: Wed Jul 23, 2008 2:35 am
Location: Thailand
Contact:

Post by wing64 »

Good job. :D
greenya
Posts: 1012
Joined: Sun Jan 21, 2007 1:46 pm
Location: Ukraine
Contact:

Post by greenya »

Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON --- incredible indeed :roll:
slavik262
Posts: 753
Joined: Sun Nov 22, 2009 9:25 pm
Location: Wisconsin, USA

Post by slavik262 »

greenya wrote:Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON
:shock: How is that possible?
ACE247
Posts: 704
Joined: Tue Mar 16, 2010 12:31 am

Post by ACE247 »

Whats going on here? 48 without and 230 with??? Some hardware fps boost.
And i see you didn't mix them up, since the blurred vs unblurred is clearly visible.
DtD
Posts: 264
Joined: Mon Aug 11, 2008 7:05 am
Location: Kansas
Contact:

Post by DtD »

lol on the FPS. Looks great!
ACE247
Posts: 704
Joined: Tue Mar 16, 2010 12:31 am

Post by ACE247 »

Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?
booe
Posts: 76
Joined: Thu Jul 29, 2010 2:12 pm

Post by booe »

Hum, stefbuet, you must have unwittingly switched the "on"/"off" labels.
The "on" label should belong to screenie with 48 fps, and "off" label to one with 230.
ACE247
Posts: 704
Joined: Tue Mar 16, 2010 12:31 am

Post by ACE247 »

No he hasn't, you can clearly see the smoothed out edges in the different screenshots.
slavik262
Posts: 753
Joined: Sun Nov 22, 2009 9:25 pm
Location: Wisconsin, USA

Post by slavik262 »

ACE247 wrote:Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?
It's funny you should say that; I was looking at some soft particle code the other day from the DirectX SDK. Soft particles work by rendering the particles to a seperate RTT. You also render the depth of the particles, and then when you add the particles back into the final frame, you compare the depth of the frame against the depth of the particles. You only draw the particles if they are in front of the rest of the frame, so that way they don't clip into geometry.
stefbuet
Competition winner
Posts: 495
Joined: Sun Dec 09, 2007 4:13 pm
Location: france

Post by stefbuet »

Ok, here is the FPS answer :
3dsmax was opening when I took the screenshot. :P
The shader don't change the FPS, however as I'm on a geforce 260GTX, it might change a little for lower spec graphic card.
slavik262
Posts: 753
Joined: Sun Nov 22, 2009 9:25 pm
Location: Wisconsin, USA

Post by slavik262 »

Since it's completely dependant on the pixel shader pipeline, performance is mostly a hardware question. On a decent graphics card it won't slow anything down, but it will choke on crap hardware (which is why games usually make post-processing optional)
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Post by devsh »

you should be careful with normal map edge detection cause otherwise you will end up blurring bumpmapped surfaces so be sure to attenuate your normal found edges by the depth edge detection.. I could think of a very complicated algo but it would be slightly inefficient... I made my own that also takes the color buffer into account so I might share it in some time
Post Reply