No module named doit error when building

[jameskim0987]

WhenI run ./doit.sh for building, I get the following error

root@ubuntu1804:~/workspace/metalk8s-2.11.2# ./doit.sh
/root/workspace/metalk8s-2.11.2/buildchain/.venv/bin/python: No module named doit

Initially, I thought the problem was from python not pointing to python3 but the error persists. How can I fix this issue?

[TeddyAndrieux]

You will need to install doit manually, you should be able to install everything you need by running

pip install -r buildchainrequirements.txt

I just checked and yes apparently the requirements section for build did not get updated with the latest changes.

[gdemonet]

Installing doit should be done by the ./doit.sh script:

# Install/reinstall the virtual environment only if it either doesn't exist or
# the requirements have changed since its creation.
if [ "${REQ_TSTAMP}" -gt "${WIT_TSTAMP}" ]
then
    "${PYTHON_SYS}" -m venv --clear "${BUILDENV}"
    "${BUILDENV}/bin/pip" install -r "${REQUIREMENTS}"
    touch "${WITNESS_FILE}"
fi

I'd suggest recreating the venv from scratch (rm -rf buildchain/.venv), and if the issue still occurs, please share the installation logs @jameskim0987.

[jameskim0987]

@TeddyAndrieux It seems like the pip install -r is called from ./doit.sh

@gdemonet I previously did not have pip3 not installed so I installed pip3 and aliased to pip.

After rm -rf buildchain/.venv and running ./doit, I get the following log:

Ignoring macfsevents: markers 'sys_platform == "darwin"' don't match your environment
Collecting certifi==2021.10.8 (from -r buildchain/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl
Collecting charset-normalizer==2.0.10 (from -r buildchain/requirements.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/84/3e/1037abe6498e65d645ce7a22d3402605d49a3b2c7f20c3abb027760da4f0/charset_normalizer-2.0.10-py3-none-any.whl
Collecting cloudpickle==2.0.0 (from -r buildchain/requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/07/3c/bf72ebd3e78eb1ef773f4f0650ecdc29c6454aeafe9c08f6da3f227dd2bc/cloudpickle-2.0.0-py3-none-any.whl
Collecting docker==4.1.0 (from -r buildchain/requirements.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/cc/ca/699d4754a932787ef353a157ada74efd1ceb6d1fc0bfb7989ae1e7b33111/docker-4.1.0-py2.py3-none-any.whl
Collecting doit==0.34.0 (from -r buildchain/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/98/af/569733b2f7ef5e4c865a47685f8ee7a168d00e53dd0b96f8bd027d4cdb4c/doit-0.34.0-py3-none-any.whl
Collecting idna==3.3 (from -r buildchain/requirements.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-none-any.whl
Collecting pyyaml==5.3.1 (from -r buildchain/requirements.txt (line 13))
  Using cached https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Collecting requests==2.27.1 (from -r buildchain/requirements.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl
Collecting six==1.16.0 (from -r buildchain/requirements.txt (line 15))
  Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Collecting urllib3==1.26.8 (from -r buildchain/requirements.txt (line 16))
  Using cached https://files.pythonhosted.org/packages/4e/b8/f5a25b22e803f0578e668daa33ba3701bb37858ec80e08a150bd7d2cf1b1/urllib3-1.26.8-py2.py3-none-any.whl
Collecting websocket-client==1.2.3 (from -r buildchain/requirements.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/88/03/cd5aa2c97356b7eee484a2b63fe850076ecb82fdeda82979a234b62a9a37/websocket_client-1.2.3-py3-none-any.whl
Collecting pyinotify==0.9.6 (from -r buildchain/requirements.txt (line 20))
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... error
  Complete output from command /root/workspace/metalk8s-2.11.2/buildchain/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-7q_m9v5j/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpuwwwe8bapip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for pyyaml
  Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: certifi, charset-normalizer, cloudpickle, idna, urllib3, requests, websocket-client, six, docker, pyinotify, doit, pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed certifi-2021.10.8 charset-normalizer-2.0.10 cloudpickle-2.0.0 docker-4.1.0 doit-0.34.0 idna-3.3 pyinotify-0.9.6 pyyaml-5.3.1 requests-2.27.1 six-1.16.0 urllib3-1.26.8 websocket-client-1.2.3
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/doit_cmd.py", line 299, in run
    return command.parse_execute(args)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/cmd_base.py", line 151, in parse_execute
    return self.execute(params, args)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/cmd_base.py", line 591, in execute
    self.task_list = self.loader.load_tasks(cmd=self, pos_args=args)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/cmd_base.py", line 391, in load_tasks
    args=pos_args, config=self.config)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/loader.py", line 213, in load_tasks
    _process_gen(ref, creator_kwargs)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/loader.py", line 166, in _process_gen
    gen_tasks = generate_tasks(name, ref(**creator_kwargs), ref.__doc__)
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/loader.py", line 384, in generate_tasks
    for task_dict, x_doc in flat_generator(gen_result, gen_doc):
  File "/root/workspace/metalk8s-2.11.2/buildchain/.venv/lib/python3.6/site-packages/doit/loader.py", line 27, in flat_generator
    for item in gen:
  File "/root/workspace/metalk8s-2.11.2/buildchain/buildchain/format.py", line 21, in task_format
    yield create_format_task()
  File "/root/workspace/metalk8s-2.11.2/buildchain/buildchain/format.py", line 51, in format_python
    filepath for filepath in utils.git_ls() if ".py" in filepath.suffixes
  File "/root/workspace/metalk8s-2.11.2/buildchain/buildchain/utils.py", line 131, in git_ls
    ["git", "ls-files", "-z", root], encoding="utf-8"
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'ls-files', '-z', PosixPath('/root/workspace/metalk8s-2.11.2')]' returned non-zero exit status 128.

The errors that stand out to me are

• error: invalid command 'bdist_wheel',
• Failed building wheel for pyyaml
• fatal: not a git repository (or any of the parent directories): .git

The git one and the python callback error log is something that concerns me the most. How can I resolve those errors?

[gdemonet]

@jameskim0987 thanks for reporting back.

The first errors mentioned are not problematic, and quite common when using pip without having the wheel package installed. As can be seen, there's most often a fallback to installation using setuptools (the Running setup.py install for pyyaml ... done line confirms this), which succeeded in your case.

This means the buildchain environment was correctly setup, and the rest is about using the buildchain itself.

The last error explains it: your directory /root/workspace/metalk8s-2.11.2 is not a Git repository, which it needs to be (the buildchain uses Git metadata to know which files are tracked and updated). This directory should be obtained from git clone [email protected]:scality/metalk8s or similar.

Hope this helps!