forums.ps2dev.org Forum Index forums.ps2dev.org
Homebrew PS2, PSP & PS3 Development Discussions
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Help with filling stencil buffer

 
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PSP Development
View previous topic :: View next topic  
Author Message
arpaagent



Joined: 06 Sep 2007
Posts: 10
PostPosted: Mon Sep 17, 2007 1:50 pm    Post subject: Help with filling stencil buffer Reply with quote
ok, I am trying to draw to only a certain portion of the screen by using stencilling, but I think i am having trouble filling in the stencil data. From snooping around the forums, it seems like the framebuffer alpha channel and the stencil buffer are one in the same. Is this correct? So, for example, if i want to fill the stencil buffer with 0xff and then draw a polygon with stencil value 0x00 in the stencil buffer, could I do it like this?

Code:


sceGuClearStencil( 0xff );
sceGuClear( GU_STENCIL_BUFFER_BIT );
// this lets us draw to the alpha channel without affecting color channels
sceGuPixelMask( 0x00ffffff );

// this will give us 0 in the alpha channel
sceGuColor( 0x00000000 );
// draw polygons in the shape of desired stencil
sceGumDrawArray( ... );



and then later, I could use the stencil values i created with blending...like this?? So that anything with a stencil value of 0xff would pass but anything with stencil value of 0x00 would not get drawn.

Code:


sceGuEnable( GU_BLEND );
sceGuBlendFunc( GU_ADD, GU_DST_ALPHA, GU_ONE_MINUS_DST_ALPHA, 0, 0 );
// draw polygons that will get stencilled
sceGumDrawArray( ... );



I've been messing around with this for a while now and I just can't get it to work like I want it to.

Any insight would be greatly appreciated! thanks!!
_________________
woo

Last edited by arpaagent on Tue Sep 18, 2007 12:57 am; edited 1 time in total
Back to top
View user's profile Send private message
Smong



Joined: 04 Sep 2007
Posts: 82
PostPosted: Mon Sep 17, 2007 9:35 pm    Post subject: Reply with quote
I can't seem to find where sceGuStencilClear is defined, I have r2309 of the psptoolchain from about 2 weeks ago.
_________________
(+[__]%)
Back to top
View user's profile Send private message
arpaagent



Joined: 06 Sep 2007
Posts: 10
PostPosted: Tue Sep 18, 2007 12:56 am    Post subject: Reply with quote
sorry, i meant sceGuClearStencil( );
edited my post
_________________
woo
Back to top
View user's profile Send private message
Smong



Joined: 04 Sep 2007
Posts: 82
PostPosted: Tue Sep 18, 2007 5:05 am    Post subject: Reply with quote
Edit: post completely rewritten with (partial?) solution.

After looking at the gu/reflection and playing around with some code I have come to these conclusions:
- You need to enclose the sceGumDrawArray that you want to be affected by stencilling with sceGuEnable(GU_STENCIL_TEST).
- You need to call sceGuStencilFunc and sceGuStencilOp twice, once before drawing your stencil, then again before you draw the things you want stencilled.
- sceGuPixelMask isn't necessary, but you must set it back to 0 if you do call it.
- The blend function should be done on SRC, not DST.
- The relevant code in the reflection sample is between lines 188 and 205

Here's my test function:
Code:
struct Vertex
{
   float x, y, z;
};

#define DO_VERT(vert, xx, yy, zz) \
   (vert)->x = xx; \
   (vert)->y = yy; \
   (vert)->z = zz; \
   vert++;

/*
when setting up gu:
sceGuClearStencil(0x00);
sceGuEnable(GU_STENCIL_TEST); // Stencil test (might want to move this to only enclose the things you want stencilled)
sceGuEnable(GU_BLEND);
sceGuBlendFunc(GU_ADD, GU_SRC_ALPHA, GU_ONE_MINUS_SRC_ALPHA, 0, 0);

at the start of your rendering loop (note last flag):
sceGuClear(GU_COLOR_BUFFER_BIT | GU_DEPTH_BUFFER_BIT | GU_STENCIL_BUFFER_BIT);

before drawing stuff you want stencilled:
Gfx_StencilRectangle(...)

some parts copied from reflection sample:
Copyright (c) 2005 Jesper Svennevid
Copyright (c) 2005 McZonk <mczonk@gmx.net>
*/

void Gfx_StencilRectangle(int x, int y, int w, int h)
{
   struct Vertex *vdata = sceGuGetMemory(sizeof(struct Vertex) * 4);
   struct Vertex *v = vdata;

   sceGuStencilFunc(GU_ALWAYS, 1, 1); // always set 1 bit in 1 bit mask
   sceGuStencilOp(GU_KEEP, GU_KEEP, GU_REPLACE); // keep value on failed test (fail and zfail) and replace
   sceGuDepthMask(GU_TRUE);

   // turn off alpha (0) so drawing doesn't actually effect the color buffer only the stencil buffer
   // if you turn alpha on (>0) then it will be give the stencilled area a background color
   sceGuColor(0x00000000);

   // draw polygons in the shape of desired stencil
   DO_VERT(v, x, y, 0);
   DO_VERT(v, x + w, y, 0);
   DO_VERT(v, x, y + h, 0);
   DO_VERT(v, x + w, y + h, 0);

   // disable GU_TEXTURE_2D since we aren't using GU_TEXTURE_* flag
   sceGuDisable(GU_TEXTURE_2D);
   sceGumDrawArray(GU_TRIANGLE_STRIP,
      GU_VERTEX_32BITF | GU_TRANSFORM_3D,
      4, 0, vdata);
   sceGuEnable(GU_TEXTURE_2D);

   sceGuStencilFunc(GU_EQUAL, 1, 1); // allow drawing where stencil is 1
   sceGuStencilOp(GU_KEEP, GU_KEEP, GU_KEEP); // keep the stencil buffer unchanged
   sceGuDepthMask(GU_FALSE);
}

It doesn't work 100% I can see some artifacts (text from psplink showing through), probably because I'm also drawing textures with alpha which mess up the stencil buffer.
_________________
(+[__]%)

Last edited by Smong on Wed Sep 19, 2007 4:36 am; edited 1 time in total
Back to top
View user's profile Send private message
arpaagent



Joined: 06 Sep 2007
Posts: 10
PostPosted: Tue Sep 18, 2007 11:07 am    Post subject: Makes Sense Reply with quote
Smong, thanks for the insight

I think it makes a little more sense to me now. The stencilFunc is not only used for testing against the stencil to mask where objects are drawn, but it is also used to put values into the stencil buffer, using the GU_ALWAYS flag. Also, would it be correct to say that in the first StencilOp, the the fail conditions will never occur since the previous call to the StencilFunc is GU_ALWAYS, implying that any drawing done will result in a pass value for stencil purposes? Just trying to get a better grasp on what exactly is going on.

I also did notice that some text from pspDebugScreenPrint was showing up when I was messing with it earlier, although I thought that would be resolved by making a call to sceGuClear( ...GU_STENCIL_BUFFER_BIT... ). I guess I will have to play with it some more.

Later this evening I will try to recode up what i was doing and see if that solves my problems. I will let you know how it goes.

Thanks again!
_________________
woo
Back to top
View user's profile Send private message
Smong



Joined: 04 Sep 2007
Posts: 82
PostPosted: Wed Sep 19, 2007 4:39 am    Post subject: Re: Makes Sense Reply with quote
arpaagent wrote:
Also, would it be correct to say that in the first StencilOp, the the fail conditions will never occur since the previous call to the StencilFunc is GU_ALWAYS, implying that any drawing done will result in a pass value for stencil purposes?
After some quick testing it seems like that is the case. I tried changing the GU_KEEP's to GU_REPLACE, GU_INVERT and GU_ZERO in the first sceGuStencilOp call and it didn't seem to make any difference.
_________________
(+[__]%)
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PSP Development All times are GMT + 10 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group