Generating an SBOM from requirements.txt without including CycloneDX's components?

[moshekaplan]

Currently, I am trying to generate a SBOM for a Python project and following the direct instructions in the documentation (installing CycloneDX via pip, running pip freeze, and then running cyclonedx_py on my requirements.txt leads to my SBOM being filled with unwanted CycloneDX dependencies.

What is the recommended way of generating an SBOM from requirements.txt without including CycloneDX's components?

[jkowalleck]

see also: #427

[jkowalleck]

Could you run pip freeze one step before installing this tool cyclonedx_py and all other non-essential dependencies?
this way, they would not appear in the requirements list you provided.

[moshekaplan]

This seemed like the simplest solution and it seems to have worked without issue. Thank you!

In case it's useful for others, I placed the following sequence of commands in my GitHub Action:

python -m pip install --upgrade pip
python -m pip install .
pip freeze > requirements_frozen.txt
python -m pip install cyclonedx-bom
cyclonedx-py -r -i requirements_frozen.txt -o sbom.xml

[gruebel]

I typically use pipx to install Python based CLI tools. It creates a dedicated venv for each tool, so you don't taint it in any way and you have no package version collision and other weird issues.

So, basically install pipx and then install cyclonedx-bom via pipx and run it in the folder or against the absolute path of the requirements.txt file

pipx install cyclonedx-bom
cyclonedx-py -r

[jkowalleck]

see also: #435 (comment) ff

[behnazh-w]

If you want to take this one step further and only include the package dependencies with no extras (i.e., dev packages), after installing cyclonedx-bom using pipx as suggested by @gruebel, you can prune your environment like in the Makefile of this python-package-template

mkdir -p build/
python -m pip freeze --local --disable-pip-version-check --exclude-editable > build/prune-requirements.txt
python -m pip wheel --wheel-dir build/wheelhouse/ --requirement build/prune-requirements.txt
python -m pip wheel --wheel-dir build/wheelhouse/ .
python -m pip uninstall --yes --requirement build/prune-requirements.txt
python -m pip install --no-index --find-links=build/wheelhouse/ --editable .
rm -fr build/

and then generate the requirements.txt and SBOM as usual or using the Makefile.

make prune requirements sbom

[pombredanne]

Hey, FWIW, I maintain packageurl and pip_requirements_parser used in this tool!

There are two cases:
a. when you run the tool installed in the project under consideration.
B. when you analyze from the outside, not installed in the project.

A. is going to be always be difficult because you cannot wholesale subtract this library requirements as they may also satisfy requirements of the project being analyzed. But if this the approach you want subtracting is the only way ... and even with some prudent checks it is hard to get right. You're hit by the observer effect https://en.wikipedia.org/wiki/Observer_effect

B. is a better approach IMHO... meaning that you use the tool installed elsewhere and point it to a project and you DO NOT modify this project installed packages. This would mean one or more of these:

• B1. parsing manifests/lockfiles (setup.py/setup.cfg/pyproject/requirements/pipenv/etc)
• B2. parsing installed packages (site-packages)
• B3. resolving dependencies (from one or more of the above)

For B., you could include as a library https://github.com/nexB/python-inspector that would do quite a bit of B1., B2. and B3. and it can also avoid biases from the current Python/OS/arch of the tool runtime vs. the project runtime.

[DavidLambertCyber]

The question is, are you generating an sbom from 3rd party source code or are you generating an sbom for a project that is being developed on the local system or in a python virtual environment on the local system?

Either way you need to be aware that when you do a pip freeze it will include your environment packages on the system, or you can try the -l flag to tell pip to ignore globally install packages. Most other tools like pigar or pipreqs will have a similar option. edit So if you are creating an sbom from 3rd party source code you would NOT do a pip freeze because it will include packages installed on your environment, instead try testing pigar and or pipreqs to generate a decent requirements.txt file.

Also with the cyclonedx-py by default, or with adding the -e flag, it will include packages in the local pip environment. If you don't want the localy install packages you would use flag -r and -i together to point to a requirements.txt and omit the -e.

[a1lu]

AFAIK does the SBOM from requirements.txt does not contain license information. So it's not always possible to replace -e by -r -i.

[a1lu]

I found a work around for me.

1. I install my package via pip together with cyclonedx-bom and pipdeptree

pip install . cyclonedis:open x-bom pipdeptree

2. Create the SBOM

cyclonedx-py -e -pb --format json

3. Run pipdeptree on my package, in a CI envrionment you can fetch the name using pip

pipdeptree -p <my package name> --json > dependencies.json

4. Filter SBOM
   Pseudocode, since I hae no access to my code atm

sbom = read(sbom.json)
deps = read(deps.json)
dep_componens = [dep["name"] for dep in deps]

sbom_deps = sbom["components"]

for dep in sbom_deps:
   if dep in dep_componens:
     keep.append(dep)
  else:
   filtered.append(dep)

sbom["components"] = keep

sbom["dependencies"] = [d for d in sbom["dependencies"] if d not in filtered]

With a bit more work in this it would also be possible to generate the missing dependencies per component (#518)

[a1lu]

There you go: https://github.com/a1lu/cyclonedx-sbom-filter

It can also append the dependsOn list of the dependencies. (untested, but the format looks good)

[jkowalleck]

similar case: CycloneDX/cyclonedx-python-lib#568 (comment)