Merge pull request #10 from ussserrr/dev

v1.10

CHANGELOG.md

@@ -110,7 +110,7 @@
  - New: STM32CubeMX is now started more silently (without a splash screen)
  - New: add integration and CLI tests (sort of)
  - New: testing with different Python versions using pyenv (3.6+ target)
- - New: `test_run_editor` is now preliminary automatically checks whether an editor is installed on the machine
+ - New: `test_start_editor` is now preliminary automatically checks whether an editor is installed on the machine
  - New: more typing annotations
  - Fixed: the app has been failed to start as `python app.py` (modify `sys.path` to fix)
  - Changed: `main()` function is now fully modular: can be run from anywhere with given CLI arguments (will be piped forward to be parsed via `argparse`)
@@ -155,7 +155,7 @@
  - Changed: more logging output
  - Changed: change some methods signatures to return result value
 
-## ver. 1.0 (XX.03.20)
+## ver. 1.0 (06.03.20)
  - New: introduce GUI version of the app (beta)
  - New: redesigned stage-state machinery - integrates seamlessly into both CLI and GUI worlds. Python `Enum` represents a single stage of the project (e.g. "code generated" or "project built") while the special dictionary unfolds the full information about the project i.e. combination of all stages (True/False). Implemented in 2 classes - `ProjectStage` and `ProjectState`, though the `Stm32pio.state` property is intended to be a user's getter. Both classes have human-readable string representations
  - New: related to previous - `status` CLI command
@@ -171,5 +171,27 @@
  - Changed: renamed `_load_config_file()` -> `_load_config()` (hide implementation details)
  - Changed: use `logger.isEnabledFor()` instead of manually comparing logging levels
  - Changed: slightly tuned exceptions (more specific ones where it make sense)
-- Changed: rename `project_path` -> `path`
-- Changed: actualized tests, more broad usage of the `app.main()` function versus `subprocess.run()`
+ - Changed: rename `project_path` -> `path`
+ - Changed: actualized tests, more broad usage of the `app.main()` function versus `subprocess.run()`
+
+## ver. 1.10 (15.03.20)
+ - New: table of contents for the README
+ - New: GitHub project wiki
+ - New: `-q/--quiet` option for the `clean` CLI command. The command now by default warns the user about the content deletion
+ - New: embedding example (minimal reproducible code snippet, easier than the full CLI or GUI versions)
+ - New: show the exception name too when the global error has been caught (`stm32pio/app.py`)
+ - New: sanitize `--start-editor` option value using `shlex.quote()`
+ - New: parse `platformio.ini` to establish its correctness when checking for project states (`ProjectStage.PIO_INITIALIZED`, `ProjectStage.PATCHED`)
+ - New: projects are now portable. The user specifying paths relatively to the project folder and using variables (we still don't use `configparser` interpolation but there is no need in it). The backwards compatibility with the old-style config format has been preserved though those projects still will be non-portable unless you manually edit a config
+ - New: analyze STM32CubeMX output to detect errors on execution. This utility does not necessarily returns non-zero code when some error was happened (e.g. `.ioc` and app versions mismatch and so on), and just shows a dialog
+ - New: `platformio_ini_config` `Stm32pio` instance property returning current `platformio.ini` parsed `ConfigParser` value. Used in some internal routines such as correctness determination and doesn't have to be used by the library user
+ - New: `LogPipe` now returns "remote control" `LogPipeRC` - small utility class holding the writable stream and the reference to the string accumulating all incoming messages. It can be accessed later, in the end of the context manager, to store and analyze all the output
+ - New: some new tests, I think, but I do not remember as all the tests are now moved to the new files :)
+ - Fixed: warnings appearing during the `pio_build()` execution were suppressed
+ - Changed: tests are moved out to the root of the repo and excluded from the distribution bundle
+ - Changed: went back to the PlatformIO CLI as a single point to interact with PlatformIO (remove `platformio` package imports and dependencies) (the reason is crushes when the pio is not isolated in a separated subprocess). Use PlatformIO JSON format output to get and filter boards
+ - Changed: remove `required=False` from `argparse` commands as it is a default (and even recommended) value anyway
+ - Changed: remove the unnecessary logging setup when no arguments were given to the program (CLI version)
+ - Changed: separate `Stm32pio` arguments onto 2 categories: project parameters and instance options and use dictionaries for them. First one has now the same form as the project config `configparser.ConfigParser` and merging into the default and file settings on the project creation. Instance options are more related to the programmatic instance itself and contains currently 2 options - `logger` and `save_on_destruction`
+ - Changed: use `append()` instead of `insert()` to modify `sys.path`
+ - Changed: when raising the exceptions use more elegant expressions (e.g. `raise FileNotFoundError(file)` instead of `raise FileNotFoundError("file FILE was not found")`). Use `pathlib.Path().resolve(strict=True)` where appropriate to shorten the code

README.md

@@ -6,10 +6,22 @@ It uses STM32CubeMX to generate a HAL-framework-based code and alongside creates
 ![Logo](/screenshots/logo.png)
 
 
+## Table of contents
+> - [Features](#features)
+> - [Requirements](#requirements)
+> - [Installation](#installation)
+> - [Usage](#usage)
+>   - [Project patching](#project-patching)
+>   - [Embedding](#embedding)
+> - [Example](#example)
+> - [Testing](#testing)
+> - [Restrictions](#restrictions)
+
+
 ## Features
   - Start the new complete project in a single directory using only an `.ioc` file
   - Update an existing project after changing hardware options in CubeMX
-  - Clean-up the project (WARNING: it deletes ALL content of project path except the `.ioc` file!)
+  - Clean-up the project
   - Get the status information
   - *[optional]* Automatically run your favorite editor in the end
   - *[optional]* Automatically make an initial build of the project
@@ -19,7 +31,6 @@ It uses STM32CubeMX to generate a HAL-framework-based code and alongside creates
 ## Requirements:
   - For this app:
     - Python 3.6 and above
-    - `platformio`
   - For usage:
     - macOS, Linux, Windows
     - STM32CubeMX with desired downloaded frameworks (F0, F1, etc.)
@@ -32,18 +43,17 @@ A general recommendation there would be to test both CubeMX (code generation) an
 ## Installation
 You can run the app in a portable way by downloading or cloning the snapshot of the repository and invoking the main script or Python module:
 ```shell script
-$ python3 stm32pio/app.py
-$ # or
-$ python3 -m stm32pio
+stm32pio-repo/ $   python3 stm32pio/app.py  # or
+stm32pio-repo/ $   python3 -m stm32pio  # or
+any-path/ $   python3 path/to/stm32pio-repo/stm32pio/app.py
 ```
-
 (we assume python3 and pip3 hereinafter). It is possible to run the app like this from anywhere.
 
 However, it's handier to install the utility to be able to run stm32pio from anywhere. Use
 ```shell script
-stm32pio-repo/ $   pip3 install .
+stm32pio-repo/ $   pip install .
 ```
-command to launch the setup process. Now you can simply type 'stm32pio' in the terminal to run the utility in any directory.
+command to launch the setup process. Now you can simply type `stm32pio` in the terminal to run the utility in any directory.
 
 Finally, the PyPI distribution (starting from v0.95) is available:
 ```shell script
@@ -76,9 +86,9 @@ It may be useful to tweak some parameters before proceeding. The structure of th
 
 You can always run
 ```shell script
-$ python3 app.py --help
+$ python app.py --help
 ```
-to see help on available commands.
+to see help on available commands. Find the copy of its output on the [project wiki](https://github.com/ussserrr/stm32pio/wiki/stm32pio-help) page, also.
 
 ### Project patching
 
@@ -88,7 +98,7 @@ For those who want to modify the patch (default one is at [`settings.py`](/stm32
 
 ### Embedding
 
-You can also use stm32pio as an ordinary Python package and embed it in your own application. Take a look at the CLI ([`app.py`](/stm32pio/app.py)) or GUI versions to see some possible ways of implementing this. Basically, you need to import `stm32pio.lib` module (where the main `Stm32pio` class resides), (optionally) set up a logger and you are good to go. If you prefer higher-level API similar to the CLI version, use `main()` function in `app.py` passing the same CLI arguments to it (except the actual script name).
+You can also use stm32pio as an ordinary Python package and embed it in your own application. Find the minimal example at the [project wiki](https://github.com/ussserrr/stm32pio/wiki/Embedding-example) page to see some possible ways of implementing this. Basically, you need to import `stm32pio.lib` module (where the main `Stm32pio` class resides), (optionally) set up a logger and you are good to go. If you prefer higher-level API similar to the CLI version, use `main()` function in `app.py` passing the same CLI arguments to it (except the actual script name). Also, take a look at the CLI ([`app.py`](/stm32pio/app.py)) or GUI versions.
 
 
 ## Example
@@ -122,18 +132,18 @@ You can also use stm32pio as an ordinary Python package and embed it in your own
 ## Testing
 There are some tests in file [`test.py`](/stm32pio/tests/test.py) (based on the unittest module). Run
 ```shell script
-stm32pio-repo/ $   python3 -m unittest -b -v
+stm32pio-repo/ $   python -m unittest -b -v
 ```
 or
 ```shell script
-stm32pio-repo/ $   python3 -m stm32pio.tests.test -b -v
+stm32pio-repo/ $   python -m stm32pio.tests.test -b -v
 ```
-to test the app. It uses STM32F0 framework to generate and build a code from the test [`stm32pio-test-project.ioc`](/stm32pio-test-project/stm32pio-test-project.ioc) project file. Please make sure that the test project folder is clean (i.e. contains only an .ioc file) before running the test otherwise it can lead to some cases failing.
+to test the app. It uses STM32F0 framework to generate and build a code from the test [`stm32pio-test-project.ioc`](/stm32pio-test-project/stm32pio-test-project.ioc) project file. Please make sure that the test project folder is clean (i.e. contains only an .ioc file) before running the test otherwise it can lead to some cases failing. Tests automatically create temporary directory (using `tempfile` Python standard module) where all actions are performed.
 
 For the specific test suite or case you can use
 ```shell script
-stm32pio-repo/ $   python3 -m unittest stm32pio.tests.test.TestIntegration -b -v
-stm32pio-repo/ $   python3 -m unittest stm32pio.tests.test.TestCLI.test_verbose -b -v
+stm32pio-repo/ $   python -m unittest stm32pio.tests.test.TestIntegration -b -v
+stm32pio-repo/ $   python -m unittest stm32pio.tests.test.TestCLI.test_verbose -b -v
 ```
 
 
@@ -144,4 +154,3 @@ stm32pio-repo/ $   python3 -m unittest stm32pio.tests.test.TestCLI.test_verbose
     lib_extra_dirs = Middlewares/Third_Party/FreeRTOS
     ```
     You also need to move all `.c`/`.h` files to the appropriate folders respectively. See PlatformIO documentation for more information.
-  - The project folder, once instantiated, is not portable i.e. if you move it at some other place and invoke stm32pio it will report you an error. This because `stm32pio.ini` config is currently stores absolute paths instead of relative.

TODO.md

@@ -6,31 +6,27 @@
  - [ ] GUI. Tests (research approaches and patterns)
  - [ ] GUI. Reduce number of calls to 'state' (many IO operations)
  - [ ] GUI. Drag and drop the new folder into the app window
- - [ ] VSCode plugin
+ - [ ] GUI. Implement some other methods for Qt abstract models
+ - [ ] GUI. Warning on 'Clean' action
+ - [ ] GUI. On 'Clean' clean the log too
+ - [ ] GUI. Stop the chain of commands if someone drops -1 or an exception
+ - [ ] Create VSCode plugin
  - [x] Remove casts to string where we can use path-like objects (related to Python version as new ones receive path-like objects arguments)
- - [ ] We look for some snippets of strings in logs and output for the testing code but we hard-code them and this is not good, probably
- - [ ] Store an initial folder content in .ini config and ignore it on clean-up process. Allow the user to modify such list. Ask the confirmation of a user by-defualt and add additional option for quiet performance
- - [ ] check for all tools (CubeMX, ...) to be present in the system (both CLI and GUI)
- - [ ] exclude tests from the bundle (see `setup.py` options)
+ - [ ] We look for some snippets of strings in logs and output for the testing code but we hard-code them and this is not good, probably (e.g. 'DEBUG')
+ - [ ] Store a folder initial content in .ini config and ignore it on clean-up process. Allow the user to modify such list (i.e. list of exclusion)
+ - [ ] at some point check for all tools (CubeMX, ...) to be present in the system (both CLI and GUI) (global `--check` command (as `--version`), also before execution of the full cycle (no sense to start if some tool doesn't exist))
  - [ ] generate code docs (help user to understand an internal mechanics, e.g. for embedding). Can be uploaded to the GitHub Wiki
  - [ ] colored logs, maybe...
- - [ ] if we require `platformio` package as a dependency we probably can rely on its dependencies too
- - [ ] check logging work when embed stm32pio lib in third-party stuff (no logging setup at all)
+ - [ ] check logging work when embed stm32pio lib in a third-party stuff (no logging setup at all)
  - [ ] merge subprocess pipes to one where suitable (i.e. `stdout` and `stderr`)
- - [ ] redirect subprocess pipes to `DEVNULL` where suitable to suppress output
- - [ ] some `stm32pio.ini` config file validation
- - [ ] CHANGELOG markdown markup
+ - [ ] redirect subprocess pipes to `DEVNULL` where suitable to suppress output (tests)
  - [ ] Two words about a synchronous nature of the lib and user's responsibility of async wrapping (if needed). Also, maybe migrate to async/await approach in the future
- - [ ] `shlex` for `build` command option sanitizing
- - [ ] `__init__`' `parameters` dict argument schema (Python 3.8 feature). Also, maybe move `save_on_desctruction` parameter there. Maybe separate on `project_params` and `instance_opts`
- - [ ] General algo of merging a given dict of parameters with the saved one on project initialization
- - [ ] parse `platformio.ini` to check its correctness in state getter
- - [ ] CubeMX 0 return code doesn't necessarily means the correct generation (e.g. migration dialog has appeared and 'Cancel' was chosen, or CubeMX_version < ioc_file_version), probably should somehow analyze the output (logs can be parsed. i.e. 2020-03-05 12:08:40,765 \[ERROR\] MainProjectManager:806 - Program Manager : The version of the current IOC is too high.)
- - [ ] Dispatch tests on several files (too many code actually)
- - [ ] Do not store absolute paths in config file and make a project portable (use configparser parameters interpolation). Handle renaming
- - [ ] See https://docs.python.org/3/howto/logging-cookbook.html#context-info to maybe replace current scheme
+ - [ ] `__init__`' `parameters` dict argument schema (Python 3.8 feature).
+ - [ ] See https://docs.python.org/3/howto/logging-cookbook.html#context-info to maybe remade current logging schema (current is, perhaps, a cause of the strange error while testing (in the logging thread))
  - [ ] UML diagrams (core, GUI back- and front-ends)
- - [ ] CI is possible
+ - [ ] CI is possible (Arch's AUR has the STM32CubeMX package, also there is a direct link). Deploy Docker in Azure Pipelines, basic at Travis CI
  - [ ] Test preserving user files and folders on regeneration and mb other operations
  - [ ] Move special formatters inside the library. It is an implementation detail actually that we use subprocesses and so on
- - [ ] Mb clean the test project tree before running the tests
+ - [ ] Mb store the last occurred exception traceback in .ini file and show on some CLI command (so we don't necessarily need to turn on the verbose mode). And, in general, we should show the error reason right off
+ - [ ] 'verbose' and 'non-verbose' tests as `subTest` (also `should_log_error_...`)
+ - [ ] the lib sometimes raising, sometimes returning the code and it is not consistent. While the reasons behind such behaviour are clear, would be great to always return a result code and raise the exceptions in the outer scope, if there is need to

setup.py

@@ -11,9 +11,11 @@
 
 import stm32pio.app
 
+
 with open('README.md', 'r') as readme:
     long_description = readme.read()
 
+
 setuptools.setup(
     name='stm32pio',
     version=stm32pio.app.__version__,
@@ -25,7 +27,11 @@
     long_description=long_description,
     long_description_content_type='text/markdown',
     url="https://github.com/ussserrr/stm32pio",
-    packages=setuptools.find_packages(),
+    packages=setuptools.find_packages(
+        exclude=[
+            'tests'
+        ]
+    ),
     classifiers=[
         "Programming Language :: Python :: 3 :: Only",
         "License :: OSI Approved :: MIT License",
@@ -44,9 +50,6 @@
     setup_requires=[
         'wheel'
     ],
-    install_requires=[
-        'platformio'
-    ],
     include_package_data=True,
     entry_points={
         'console_scripts': [