by szo » Sat Mar 03, 2012 7:52 pm
Have been using the dib sections code in my hexen2 project, uhexen2: first tested as is with the bottom-up dib setup, it worked, by magical means, in 32 bits only, whereas in 64 bits it failed miserably. Tried to understand the reason, got bored, and switched to using a top-down setup by feeding -DIBWidth to VID_CreateDIB() as the lazy solution albeit the result being somehow slower.
Noticed today that one big difference between q1 and q2 code is that the width, height, and most importantly the rowbytes member of the viddef_t structure is unsigned int in q1 but signed int in q2: the bottom-up dib section code might as well be the reason for the change. For the bottom-up setup, we are assigning -width to vid.rowbytes, which is unsigned in q1, and it works in 32 bits because the address space is 4 gb and you get a wraparound in pointer atrithmetics. However in win64 that doesn't happen with 64 bit pointers and you simply get a segfault.
In short, the dib code here is really meant for signed int vid.rowbytes. In case anyone ever runs into such failures...
Just applying patches to uhexen2 now, changing the width, height, conwidth, conheight and rowbytes members to signed int. The conrowbytes member is already signed so that's ok already. Shame that I nailed the issue this late, but better late than never.