Re: [MiNT] libgem16: vs_color

[Eero Tamminen <oak@helsinkinet.fi>]

Hi,

On Monday 18 January 2010, Helmut Karlowski wrote:
> I've spend some time to find out why vs_color isn't working in XaAES.
>
> I think it's not XaAES but it is libgem.
>
> I found that in vs_color.c the variables in the stack overwrite
> themselves, though I don't know why this happens.
>
> I have a working version now by making the arrays static:

Assuming they then go to DATA section at same alignment and in same order,
and still don't get messed, then something called by this code is
underwriting it's stack.


> void
> vs_color (short handle, short index, short rgb[])
> {
>    static short vdi_intin[4] = {0,0,0,0};
>    static short vdi_control[VDI_CNTRLMAX] = {0};
>    short *ptr;
>
>    VDI_PARAMS(vdi_control, vdi_intin, 0L, vdi_dummy, vdi_dummy );

You can check it by putting here some unrelated stack array instead of these
arrays and checking whether values in it got overwritten.  If not, then
something else might have gotten overwritten in the DATA section where
the above arrays were moved with "static" keyword.  If nothing there got
overwritten in DATA either, then this may be about stack alignment...


>    ptr = vdi_intin;
>
>    *(ptr ++) = index;           /* vdi_intin[0] = index */
>    if( (long)vdi_params.control & (long)1 )	<- this is true if arrays not
> static!
>
>
> Now also pixel-detection works in the original way using vs_color, as
> well as restore-palette.
>
> Someone will have to find out why the arrays are overwritten (I have
> placed Cconws-output in vs_color to detect these), I don't understand it.
>
> BTW: The CLOBBERED_REGS-patch by Vincent has no effect, because the
> inlined code in gem_vdiP.h isn't used.

If you can run this code in Hatari, I added to Hatari v1.3.1 debugger
conditional breakpoints which can break when data value in memory gets
changed (not just at code addresses).

I'd recommend doing this with the latest Hatari version control version
though because I added to it last week symbol support which may make
debugging a bit more pleasent. :-)


	- Eero