Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!

Post those lines of code you feel like sharing or find what you require for your project here; or simply use them as tutorials.
stefbuet
Competition winner
Posts: 495
Joined: Sun Dec 09, 2007 4:13 pm
Location: france

Post by stefbuet »

Yeah, it should use a basic normal render without normal maps.
The main problem is that lines have a constant size, so when you zoom out it's too much blured. I haven't figured out how to solve this problem for now, I'm opened to any idea :roll:
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Post by devsh »

pixel alias has a constant size too... therefore more edges are present far away and AA makes far away pictures blurry (even the real one) also read third paragraph down

Make sure you use Gaussian Weights+Blur and calculate them for a 3x3 kernel with std var of 1.f

You are looking 2 pixels away (1.5 actually)... this wont aid finding an edge I can guaranteee that 100% cause I tried... it just makes the lines on edge detect thicker which makes the whole thing blurry cause the thing isnt an actual edge... getting a sample from 1.5 away will give you an interpolated value between pixel 1 and 2 which can cause haloes etc so the edges could appear to be NOT touching the objects by just 0.5 of a pixel or make the edge lines up to 50% darker

Also please attenuate the normal comparison edges by the depth difference so if there isnt much/any the app assumes its the same surface == not blurring bump maps... this approach however WILL improve parallax occlusion (raycasted) height maps as they modify the fragment depth in the zbuffer so they WILL be detected
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Post by devsh »

right here is my shader... I could have made it better by attenuating the normals depending on depth like I said and weighing the diagonal samples by 1/sqrt(2) to compensate for the length

Image

In the image, first row up and from left: edge detection using color buffer only, final outcome attenuated from all 3 methods, depth, color buffer

Bottom row: edge detection with depth only, normals only, normals+depth and no color edge attenuated, normal buffer

as we can see the normal based edge detection is the crucial one and can be used on its own... depth on its own wont do.. you can add/mix methods in order to get a higher quality render

Link to a 1024 render picture 4k by 2k (the edge detection were screenshotted from a 512by512 app so dont be picky)
http://img202.imageshack.us/img202/4696/edgeaa.jpg
stefbuet
Competition winner
Posts: 495
Joined: Sun Dec 09, 2007 4:13 pm
Location: france

Post by stefbuet »

Really nice output.
I'm checking 1.5 px away because in glsl the next px is 1.5f far from the current pixel, while it's 1.0f in hlsl ?
I was checking with a cross kernel instead of a 3x3 kernel as it's described in gpugems for Stalker game, however if I were using a 3x3 kernel how should I check either if the pixel is on an edge or not? I were just checking if at least 1 pixel depth difference or normal global difference is more than a critical value, but as you can see, lines produced are too big, not adapted to the distance camera-pixel.
Once again, you shader looks great :D
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Post by devsh »

as I said lines are too big because of the 1.5... if you used 1.f instead then the lines would be smaller

a 3x3 kernel gives AntiAliased edge detection (smooth-ish lines) because of the diagonal checks, mine are not weighted by the length so it would be possible to gain better result unrolling the for functions and attenuating the ones which are diagonal by 1/sqrt(2)

here is my shader code... process your scene with it to see a comparison (especially the grass part).. remember to play around with the settings

Code: Select all

// simple fragment shader

// 'time' contains seconds since the program was linked.
uniform float imgX,imgY,exposure;
uniform vec3 atten; // (1.0,0.47,0.71)
uniform sampler2D depth;
uniform sampler2D normal;
uniform sampler2D col;

void main()
{
	float orig = texture2D(depth,gl_TexCoord[0].xy).r;
	float diff = 0.0;
	float normaldiff = 0.0;
	float coldiff = 0.0;
	vec3 orig_nml = normalize(texture2D(normal,gl_TexCoord[0].xy).rgb*2.0-1.0);
	vec2 newTC;
	vec3 orig_col = texture2D(col,gl_TexCoord[0].xy).rgb*vec3(0.3,0.59,0.11);
	for (int i=-1; i<2; i++) {
		for (int j=-1; j<2; j++) {
			newTC = clamp(gl_TexCoord[0].xy-vec2(i,j)/vec2(imgX,imgY),0.0001,0.9999);
			diff += abs(orig-texture2D(depth,newTC).r);
			normaldiff += sin(acos(dot(orig_nml,normalize(texture2D(normal,newTC).rgb*2.0-1.0))));
			coldiff += length(orig_col-texture2D(col,newTC).rgb*vec3(0.3,0.59,0.11));
		}
	}
	gl_FragColor = pow(pow(coldiff*0.75,2.5)*atten.x+pow(normaldiff*0.2,1.5)*atten.y+pow(diff*0.3,0.85)*atten.z,exposure);
}
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Post by devsh »

ofc drop the color edge detection if your performance is dying or summink
stefbuet
Competition winner
Posts: 495
Joined: Sun Dec 09, 2007 4:13 pm
Location: france

Post by stefbuet »

I've updated the code using another method between devsh one and mine. The main advantage of the new version is that the edge detected size will stay constant even if you zoom out.
osgoz
Posts: 21
Joined: Mon Apr 23, 2007 10:59 am

Post by osgoz »

Hi.

I'd like to use your shader in my project, but my project is in DX9 so I need the HLSL version.
I have no idea of shader language but I intend to convert by my self.


Code: Select all

// antialiasing pixel shader 
// HLSL version
// Original GLSL version Author:Stefbuet

//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)

sampler2D depthMap: register(s0) ;
sampler2D renderMap: register(s1) ;


float4 textureCoord;

float2 offsetArray[8];

float4 pixelMain(): COLOR0 
{

   float2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=tex2D(depthMap, -textCoord).w;
   float3 normal=tex2D(depthMap, -textCoord).xyz*2.0-float3(1.0,1.0,1.0);

   offsetArray[0] = float2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = float2(1.0/SCREENX, 0.0);
   offsetArray[2] = float2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = float2(0.0, 1.0/SCREENY);
   offsetArray[4] = float2(0.0, -1.0/SCREENY);
   offsetArray[5] = float2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = float2(-1.0/SCREENX, 0.0);
   offsetArray[7] = float2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, tex2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-float3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       float4 finalColor=tex2D(renderMap, -textCoord.xy);
       for(int i=0; i<8; i++)
           finalColor+=tex2D(renderMap, -textCoord.xy+offsetArray[i]);
       
       finalColor/=8.0;
       return finalColor;

   }
   else { 
     
       return tex2D(renderMap, -textCoord);
     
   }


} 

It compiles without errors, but for sure it does not work at all.

Could you, please, help me?

I apologize for my poor English, but it doesn't my mother language.

Regards.
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Post by hendu »

@OP

Just curious, how does your shader compare to the released MLAA ones (I know of at least two)?

Or the proprietary one in AMD's current drivers?
osgoz
Posts: 21
Joined: Mon Apr 23, 2007 10:59 am

Post by osgoz »

Hi.

I managed it to work.
This is the HLSL code.

Code: Select all

// antialiasing pixel shader
// HLSL version
// GLSL version Author:Stefbuet

//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)

sampler2D depthMap: register(s0) ;
sampler2D renderMap: register(s1) ;


float2 offsetArray[8];

float4 pixelMain(float4 textureCoord : TEXCOORD0): COLOR0 
{

   float2 textCoord=textureCoord.xy;

   float currentDepth=tex2D(depthMap, textCoord).w;
   float3 normal=tex2D(depthMap, textCoord).xyz*2.0-float3(1.0,1.0,1.0);

   offsetArray[0] = float2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = float2(1.0/SCREENX, 0.0);
   offsetArray[2] = float2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = float2(0.0, 1.0/SCREENY);
   offsetArray[4] = float2(0.0, -1.0/SCREENY);
   offsetArray[5] = float2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = float2(-1.0/SCREENX, 0.0);
   offsetArray[7] = float2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(tex2D(depthMap, textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, tex2D(depthMap, textureCoord.xy+offsetArray[0]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[1]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[2]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[3]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[4]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[5]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[6]).xyz*2.0-float3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, tex2D(depthMap, textCoord+offsetArray[7]).xyz*2.0-float3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       float4 finalColor=tex2D(renderMap,textCoord.xy);
       for(int i=0; i<8; i++)
           finalColor+=tex2D(renderMap, textCoord.xy+offsetArray[i]);
       
       finalColor/=8.0;
       return finalColor;

   }
   else { 
     
       return tex2D(renderMap, textCoord);
     
   }
} 
Or almost I think that works.

Regards.
Post Reply