[python] update lxml, PyYAML, libcst, flake8

The current version that we pinned does not build on or work with Python
3.12. lxml, PyYAML, flake8 are upgraded to latest, while libcst is
upgraded to a non-latest version because the latest version dropped
Python 3.8 support.

Signed-off-by: Gary Guo <[email protected]>

doc/contributing/sw/adding_python_depedencies.md

@@ -4,5 +4,5 @@ To comply with software supply chain security requirements of various project pa
 We accomplish this with the help of the `uv pip compile` command, which is part of the `uv` package.
 
 If you need to add another Python package to the project, do so by:
-1. adding the package and version number to the `pyproject.toml` file, in the form of `<package>==<version>`, and
+1. adding the package and version number to the `pyproject.toml` file, in the form of `<package> ~= <version>` (or `<package> == <version>` in case a specific version is needed), and
 1. run the script `util/sh/scripts/gen-python-requirements.sh`, which will auto-generate the updated `python-requirements.txt` file.

pyproject.toml

@@ -22,15 +22,15 @@ dependencies = [
   "beautifulsoup4==4.12.2",
   "hjson==3.1.0",
   "libclang==16.0.0",
-  "libcst==0.4.1",
-  "lxml==4.9.2",
+  "libcst == 1.1.0", # 1.2.0+ needs Python 3.9+
+  "lxml ~= 5.0",
   "mako==1.1.6",
   "pluralizer==1.2.0",
   "pycryptodome==3.15.0",
   "pyelftools==0.29",
   "pytest-timeout==2.1.0",
   "pytest==7.0.1",
-  "pyyaml==6.0",
+  "pyyaml ~= 6.0.2",
   "rich==12.6.0", # maximum version compatible with Python 3.6.8 (used on CentOS7 nightly regression machines)
   "semantic_version==2.10.0",
   "tabulate==0.8.10",
@@ -59,7 +59,7 @@ dependencies = [
   "termcolor==1.1.0",
 
   # Linters
-  "flake8==5.0.4",
+  "flake8 ~= 7.1",
   "isort==5.10.1",
   "mypy==0.971",
   "yapf==0.32.0",