Substitution of special attributes of special variables like $SOURCE does not work in builder calls

[mwichmann]

This topic was previously raised in #2905, which was closed in 2018 without apparently having been really resolved. This issue is created because of a fresh mention.

The User Guide claims that you can do substitutions on SOURCE and TARGET in sources and targets, in the chapter about the Command builder:

https://scons.org/doc/production/HTML/scons-user.html#chap-builders-commands

Specifically:

Note that $SOURCE and $TARGET are expanded in the source and target as well, so you can write:

env.Command('${SOURCE.basename}.out', 'foo.in', build)

Besides that SCons variables don't have a special attribute basename, this fails even if filebase (which is documented) is used. You get deep into a chain of calls, and eventually StringSubber.expand fails because it tries to eval the string to be substituted and because of the apparent attribute access in there, you get an AttributeError:

AttributeError: 'str' object has no attribute 'filebase'

At this point, the call trace looks like this:

(Pdb) bt
  /home/mats/.pyenv/versions/venv-system312/bin/scons(8)<module>()
-> sys.exit(main())
  /home/mats/github/scons/SCons/Script/Main.py(1515)main()
-> _exec_main(parser, values)
  /home/mats/github/scons/SCons/Script/Main.py(1469)_exec_main()
-> _main(parser)
  /home/mats/github/scons/SCons/Script/Main.py(1111)_main()
-> SCons.Script._SConscript._SConscript(fs, script)
  /home/mats/github/scons/SCons/Script/SConscript.py(281)_SConscript()
-> exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  /tmp/dot/SConstruct(7)<module>()
-> env.Command('${SOURCE.filebase}.out', 'foo.in', build)
  /home/mats/github/scons/SCons/Environment.py(2320)Command()
-> return bld(self, target, source, **kw)
  /home/mats/github/scons/SCons/Builder.py(673)__call__()
-> return self._execute(env, target, source, OverrideWarner(kw), ekw)
  /home/mats/github/scons/SCons/Builder.py(579)_execute()
-> tlist, slist = self._create_nodes(env, target, source)
  /home/mats/github/scons/SCons/Builder.py(524)_create_nodes()
-> tlist = env.arg2nodes(target, target_factory, target=target, source=source)
  /home/mats/github/scons/SCons/Environment.py(691)arg2nodes()
-> v = node_factory(self.subst(v, **kw))
  /home/mats/github/scons/SCons/Environment.py(722)subst()
-> return SCons.Subst.scons_subst(string, self, raw, target, source, gvars, lvars, conv, overrides=overrides)
  /home/mats/github/scons/SCons/Subst.py(854)scons_subst()
-> result = ss.substitute(strSubst, lvars)
  /home/mats/github/scons/SCons/Subst.py(459)substitute()
-> result = _dollar_exps.sub(sub_match, args)
  /home/mats/github/scons/SCons/Subst.py(454)sub_match()
-> return self.conv(self.expand(match.group(1), lvars))
> /home/mats/github/scons/SCons/Subst.py(393)expand()
-> raise_exception(e, lvars['TARGETS'], old_s)
(Pdb)

Everything at this point looks as one might expect. The string to sub is '${SOURCE.filebase}', the code has extracted the relevant part so the key variable is 'SOURCE.filebase', lvars has the right values for the special variables in it:

{'__env__': <SCons.Script.SConscript.SConsEnvironment object at 0x7f105c0e2780>,
'TARGETS': ['${SOURCE.filebase}.out'],
'TARGET': '${SOURCE.filebase}.out',
'CHANGED_TARGETS': '$TARGETS',
'UNCHANGED_TARGETS': '$TARGETS',
'SOURCES': ['foo.in'],
'SOURCE': 'foo.in',
'CHANGED_SOURCES': '$SOURCES',
'UNCHANGED_SOURCES': '$SOURCES',
'__return__': None}

But the attempt to evaluate it fails:

s = eval(key, self.gvars, lvars)

A few lines prior to that line, there was an attempt to detect a period in the string, but then nothing is done with that information until later. That initial check is here:

scons/SCons/Subst.py

Line 373 in 01f77b1

if key[0] == '{' or '.' in key:

[mwichmann]

Just to wrap up, at this point, the value of $SOURCE is not expected to be a string... you can simulate this a lot more simply:

>>> eval('SOURCE.filebase', {}, {'SOURCE': 'foo'})
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    eval('SOURCE.filebase', {}, {'SOURCE': 'foo'})
  File "<string>", line 1, in <module>
AttributeError: 'str' object has no attribute 'filebase'

[bdbaddog]

A workaround is to use File(), like this:

env.Command('${SOURCE.filebase}.out', File('foo.in') , "echo $TARGET $SOURCE")

It should work.
This should get you unstuck if you were stuck. Until such time as we can get it working as you'd expect.

[mwichmann]

Just to wrap up, at this point, the value of $SOURCE is not expected to be a string.

To update the details a bit: it's not actually a string. It's a Target_or_Source, which wraps an NLWrapper, which wraps a NodeList (the flavor defined in SCons.Util - there are two NodeList classes in SCons, to add to general confusion). NLWrapper defers the conversion of sources and targets to nodes - you get a NodeList whose entries are strings rather than nodes. Target_or_Source and Targets_or_Sources have __getattr__ methods which should be able to retrieve the underlying node attributes, and it's this that fails on the deferred-conversion strings, which of course don't have attributes other than those defined for string objects in general. (anybody got a headache yet?). Apparently there's a path through that doesn't trigger the conversion to nodes that is what trips up this example.

The current proposal is to update the User Guide to give the guidance above, and leave this issue open for a while longer, in case anyone is motivated to dig a bit further.