Color Translation/Fullbright Inconsistencies

[T-Will133]

Playing through Knee-Deep in Knee-Deep in ZDoom has revealed some inconsistencies in PrBoom+/DSDA color translation and fullbrights compared to other ports including Chocolate Doom. Two examples in PrBoom+um 2.6.2 are the imp's teeth and the Nightmare Demon.

The automap colors are also translated wrong.

Oddly enough though the recent builds of PrBoom+um have fixed the Nightmare Demon.

[XaserAcheron]

A couple of anecdotal details from various Discord convos that may help sniff out the specifics:

• I recall some discussion semi-recently (like, a month or two ago) about some inconsistencies with DEHACKED color translations in pr+/dsda-doom, which would explain the Nightmare Demon -- looks like those got all fixed up, so that's one down.
• For the imp teeth, the wad's palette has a set of fullbright colors, including a second pure-white shade. There's presumably some bit of code somewhere that's "optimizing" it by treating it as a duplicate color, and the non-fullbright white in the imp sprites are getting replaced with the fullbright one.
• The automap bug is caused by an old change in Boom's source -- the automap background color got swapped from 0 to 247. This is technically fixable user-side since the colors are customizable, but it'd be good to have the defaults fixed up.

[fabiangreffrath]

• The automap bug is caused by an old change in Boom's source -- the automap background color got swapped from 0 to 247. This is technically fixable user-side since the colors are customizable, but it'd be good to have the defaults fixed up.

#546

[fabiangreffrath]

• For the imp teeth, the wad's palette has a set of fullbright colors, including a second pure-white shade. There's presumably some bit of code somewhere that's "optimizing" it by treating it as a duplicate color, and the non-fullbright white in the imp sprites are getting replaced with the fullbright one.

That probably here:

prboom-plus/prboom2/src/r_patch.c

Lines 127 to 159 in bf872a0

	if (!playpal_duplicate)
	{
	int lump = W_GetNumForName("PLAYPAL");
	const byte *playpal = W_CacheLumpNum(lump);
	// find two duplicate palette entries. use one for transparency.
	// rewrite source pixels in patches to the other on composition.
	int i, j, found = 0;
	for (i = 0; i < 256; i++)
	{
	for (j = i+1; j < 256; j++)
	{
	if (playpal[3*i+0] == playpal[3*j+0] &&
	playpal[3*i+1] == playpal[3*j+1] &&
	playpal[3*i+2] == playpal[3*j+2])
	{
	found = 1;
	break;
	}
	}
	if (found)
	break;
	}
	if (found) { // found duplicate
	playpal_transparent = i;
	playpal_duplicate = j;
	} else { // no duplicate: use 255 for transparency, as done previously
	playpal_transparent = 255;
	playpal_duplicate = -1;
	}

[esselfortium]

I believe GZDoom's recent fix for this bug was to also check the colormap to see if both seemingly-identical colors are also treated the same in that, so that fullbrights are still differentiated as they should be.

[kraflab]

Duplicate detection: kraflab/dsda-doom@708db98
I added mapcolor_* OPTIONS support for dsda-doom but that's probably out of scope for pr+.