vp20compiler: Implement shader format that bundles shader-defined constants

[NZJenkins]

Currently, when a shader is generated using constant uploads, the user has to examine the shader output and manually set constants generated by the compilation.
A workflow to use the shader constants automatically when loading a shader will ease development

[JayFoxRox]

Not sure if this should be changed in vp20compiler itself (which is a pretty bare bones assembler + it implements a specific bit of the hardware, and uses known syntax from nvidia / GL spec).
I'd prefer to not extend the language, unless there's existing tools / conventions.

Most of the time, I ran my shaders through a C preprocessor anyway (to have macros in the shaders); others skip preprocessing because they use Cg (which I hate with a passion, because it's a binary blob).

I don't see an issue with parsing the input shader (with vp20compiler #-comments, or an even earlier stage) or the output shader (with vp20compiler comments turned into C++ //-comments) in a separate step.
It's why #131 exists, which re-uses existing features of vertex-programs instead of extending the language.
To keep vp20compiler lightweight, we could even consider undoing #131, once a new build-system splits shader compilation into separate steps (= add action inbetween steps without modifying nxdk). #131 became necessary because vp20compiler would strip the comments and nxdk would strip the intermediate files between Cg and vp20compiler.

So, historically, lack of variables and constant locations is a shortcoming of the nxdk Makefile or proprietary Cg; not of vp20compiler (which is mostly an assembler, so you define the environment).
Overall, constant and program upload are also performance critical steps, so I'd want to avoid generating pushbuffers (as already done by fp20compiler, which I consider problematic as well).

There are related tasks like relocation of constants to fit multiple vertex programs into the GPU at once, which also deserve some tooling. I don't think vp20compiler has to handle all of this itself, either.

So, to me, extracting constants feels more like a pre- or post-processing step (unless we plan to turn vp20compiler into a full compiler with constant folding etc.). Extracting metadata is similar to pre-processing with C / Cg which are also separate tools.

Putting a wrapper or separate tool in nxdk-bin/ sounds best to me to handle this optional extraction step.
Cg conventions (#const and #var) should be followed and allow extracting to some easily parsable format (C and / or JSON?).
Personally, I just wrote something for each project and I'll likely stick to that approach. Extracting the #const lines is a single unix command / trivial regex / sscanf after all:

• Example to match global variables: ^#var\ ([a-z0-9]*) ([a-zA-Z\.\_]*) : : c\[([0-9]+)\]
• Example to match global constants: ^#const\ c\[([0-9]+)\] = ([0-9\.]+) ([0-9\.]+) ([0-9\.]+) ([0-9\.]+)