Merge pull request #17 from ussserrr/dev

v1.30

CHANGELOG.md

@@ -230,5 +230,45 @@
  - Changed: remove `from __future__ import annotations` statements
 
 ## ver. 1.21 (19.04.20)
- - Fixed: GUI. All paths are now reliably treated both for QML and Python
- - Changed: README installation process actualized
+ - Fixed: GUI. All resorce paths are now reliably treated both for QML and Python
+ - Changed: README installation process actualized
+
+## ver. 1.30 (05.05.20)
+ - New: `examples` folder (currently, only an embedding one (updated and moved from the wiki page))
+ - New: `docs` folder with some useful internal descriptions (currently, only a logging schematic (with sources))
+ - New: issues guide for GitHub (OS, content of the config, project tree, enable verbose, etc.)
+ - New: GUI. Show the app version in "About" dialog
+ - New: GUI. Handle a theoretical app loading error
+ - New: GUI. Notify a user that the "board" parameter is empty
+ - New: GUI. The app can be started from CLI
+ - New: GUI. `ProjectListItem.fromStartup` property
+ - New: GUI. Expose projects' `config` to QML
+ - New: Tests. Preserving user files and folders on regeneration
+ - New: Tests. 'verbose' and 'non-verbose' tests as `subTest` (also `should_log_error_...`)
+ - New: Inform a user that given parameters have overridden the config ones
+ - Fixed: GUI. `TypeError: Cannot read property 'actionRunning' of null` (deconstruction order) (on project deletion only)
+ - Fixed: GUI. The app now can handle as many projects as needed (use QML `DelegateModel` to store state in the `ListView` delegate)
+ - Fixed: #13 (new parsing algo to analyze the CubeMX output)
+ - Changed: improved `typing` annotations
+ - Changed: wrap imports into `try...except`
+ - Changed: new README logo, add sources (draw.io)
+ - Changed: GUI. Icons instead of a text for "Clean", "Open editor"
+ - Changed: GUI. Gray out "stage" line in all projects except current
+ - Changed: GUI. 2 types of logging formatters for 2 verbosity levels
+ - Changed: GUI. More general `goToProject` signal instead of `duplicateFound`
+ - Changed: GUI. Projects list is now saves to `Settings` in a separate thread using `QThreadPool` and `saveInSettings()` method
+ - Changed: GUI. `ProjectsList.each_project_is_duplicate_of` generator
+ - Changed: GUI. Optimized project' `state` handling
+ - Changed: GUI. Insert board ID from config, if there is one, focus on that input field by default
+ - Changed: logging mechanics is remade from scratch:
+    - add `stm32pio.app.setup_logging()`, `should_setup_logging` argument fo `stm32pio.app.setup_main()`. This also fixes annoying logging errors on testing because the loggers interfere with each other
+    - `stm32pio.util.ProjectLoggerAdapter()` subclass as an individual logger for every project
+    - add `stm32pio.util.log_current_exception()`
+    - get rid of `log_record_factory` substitution
+    - add `stm32pio.util.Verbosity` entity (enum). Acts like an additional degree of freedom for the logging setup
+    - rewritten `stm32pio.util.DispatchingFormatter`
+    - GUI. New `BuffersDispatchingHandler()` class
+    - and some others (see block schema)
+ - Changed: better parameters and configs merging
+ - Changed: make `platformio_ini_is_patched` a property instead of function
+ - Changed: improved in-code docs

README.md

@@ -13,6 +13,7 @@ The [GUI version](/stm32pio_gui) is available, too.
 > - [Requirements](#requirements)
 > - [Installation](#installation)
 > - [Usage](#usage)
+>   - [GUI from CLI](#gui-from-cli)
 >   - [Project patching](#project-patching)
 >   - [Embedding](#embedding)
 > - [Example](#example)
@@ -49,7 +50,7 @@ stm32pio-repo/ $   python3 stm32pio/app.py  # or
 stm32pio-repo/ $   python3 -m stm32pio  # or
 any-path/ $   python3 path/to/stm32pio-repo/stm32pio/app.py
 ```
-(we assume python3 and pip3 hereinafter). It is possible to run the app like this from anywhere.
+(we assume `python3` and `pip3` hereinafter). It is possible to run the app like this from anywhere.
 
 However, it's handier to install the utility to be able to run stm32pio from anywhere. Use
 ```shell script
@@ -94,6 +95,12 @@ $ python app.py --help
 ```
 to see help on available commands. Find the copy of its output on the [project wiki](https://github.com/ussserrr/stm32pio/wiki/stm32pio-help) page, also.
 
+### GUI from CLI
+You can start the [GUI version](/stm32pio_gui) using `gui` subcommand and pass some of the arguments to it:
+```shell script
+$ stm32pio gui -d ./sample-project -b discovery_f4
+```
+
 ### Project patching
 
 Note, that the patch operation (which takes the CubeMX code and PlatformIO project to the compliance) erases all the comments (lines starting with `;`) inside the `platformio.ini` file. They are not required anyway, in general, but if you need them for some reason please consider to save the information somewhere else.
@@ -102,7 +109,7 @@ For those who want to modify the patch (default one is at [`settings.py`](/stm32
 
 ### Embedding
 
-You can also use stm32pio as an ordinary Python package and embed it in your own application. Find the minimal example at the [project wiki](https://github.com/ussserrr/stm32pio/wiki/Embedding-example) page to see some possible ways of implementing this. Basically, you need to import `stm32pio.lib` module (where the main `Stm32pio` class resides), (optionally) set up a logger and you are good to go. If you prefer higher-level API similar to the CLI version, use `main()` function in `app.py` passing the same CLI arguments to it (except the actual script name). Also, take a look at the CLI ([`app.py`](/stm32pio/app.py)) or GUI versions.
+You can also use stm32pio as an ordinary Python package and embed it in your own application. Find the minimal example at the [examples](/examples) to see some possible ways of implementing this. Basically, you need to import `stm32pio.lib` module (where the main `Stm32pio` class resides), (optionally) set up a logger and you are good to go. If you prefer higher-level API similar to the CLI version, use `main()` function in `app.py` passing the same CLI arguments to it (except the actual script name). Also, take a look at the CLI ([`app.py`](/stm32pio/app.py)) or GUI versions.
 
 
 ## Example
@@ -138,16 +145,12 @@ There are some tests in file [`test.py`](/stm32pio/tests/test.py) (based on the
 ```shell script
 stm32pio-repo/ $   python -m unittest -b -v
 ```
-or
-```shell script
-stm32pio-repo/ $   python -m stm32pio.tests.test -b -v
-```
 to test the app. It uses STM32F0 framework to generate and build a code from the test [`stm32pio-test-project.ioc`](/stm32pio-test-project/stm32pio-test-project.ioc) project file. Please make sure that the test project folder is clean (i.e. contains only an .ioc file) before running the test otherwise it can lead to some cases failing. Tests automatically create temporary directory (using `tempfile` Python standard module) where all actions are performed.
 
 For the specific test suite or case you can use
 ```shell script
-stm32pio-repo/ $   python -m unittest stm32pio.tests.test.TestIntegration -b -v
-stm32pio-repo/ $   python -m unittest stm32pio.tests.test.TestCLI.test_verbose -b -v
+stm32pio-repo/ $   python -m unittest tests.test_integration.TestIntegration -b -v
+stm32pio-repo/ $   python -m unittest tests.test_cli.TestCLI.test_verbosity -b -v
 ```
 
 

TODO.md

@@ -1,64 +1,59 @@
 # TODOs
 
-## Business logic, business features
- - [ ] Issues guide for the GitHub (OS, content of the config, project tree, enable verbose)
+## Business logic, general features
  - [ ] GitHub CHANGELOG - separate New, Fixed, Changed into paragraphs
  - [ ] Middleware support (FreeRTOS, etc.)
  - [ ] Arduino framework support (needs research to check if it is possible)
  - [ ] Create VSCode plugin
+ - [ ] UML diagrams (core, GUI back- and front-ends, thread flows, events, etc.)
+ - [ ] CI is possible (Arch's AUR has the STM32CubeMX package, also there is a direct link). Deploy Docker one in Azure Pipelines, basic at Travis CI
 
 ## GUI version
- - [ ] Handle the initialization error (when boards are receiving)
- - [ ] Maybe `data()` `QAbstractListModel` method can be used instead of custom `get()`
+ - [ ] Obtain boards on demand (not at the startup)
  - [ ] Can probably detect Ctrl and Shift clicks without moving the mouse first
- - [ ] Notify the user that the 'board' parameter is empty
  - [ ] Mac: sometimes auto turned off shift highlighting after action (hide-restore helps)
  - [ ] Some visual flaws when the window have got resized (e.g. 'Add' button position doesn't change until the list gets focus, 'Log' area crawls onto the status bar)
- - [ ] Gray out "stage" line in all projects except current
  - [ ] Tests (research approaches and patterns)
- - [ ] Test performance with a large number of projects in the model. First test was made:
-      1. Some projects occasionally change `initLoading` by itself (probably Loader unloads the content) (hence cannot click on them, busy indicator appearing)
+ - [ ] Remade the list item to use States, too. Probably, such properties need to be implemented:
+       ```
+       state: {
+           loaded,
 
-         Note: Delegates are instantiated as needed and may be destroyed at any time. They are parented to ListView's contentItem, not to the view itself. State should never be stored in a delegate.
+           visitedAfterInstantiating,
 
-         Use `id()` in `setInitInfo()`. Or do not use ListView at all (replace by Repeater, for example) as it can reset our "notifications"
-      2. Some projects show OK even after its deletion (only the app restart helps)
+           actionRunning,
+           lastActionStatus,
+           visitedAfterAction,
+           ...
+       }
+       ```
  - [ ] Test with different timings
  - [ ] Divide on multiple modules (both Python and QML)
  - [ ] Implement other methods for Qt abstract models
  - [ ] Warning on 'Clean' action (maybe the window with a checkbox "Do not ask in the future" (QSettings parameter))
- - [ ] 2 types of logging formatters for 2 verbosity levels
- - [ ] `TypeError: Cannot read property 'actionRunning' of null` (deconstruction order) (on project deletion only)
  - [ ] QML logging - pass to Python' `logging` and establish a similar format. Distinguish between `console.log()`, `console.error()` and so on
  - [ ] Lost log box autoscroll when manually scrolling between the actions
  - [ ] Crash on shutdown in Win and Linux (errors such as `[QML] CRITICAL QThread: Destroyed while thread is still running Process finished with exit code 1073741845`)
- - [ ] Start with a folder opened if it was provided on CLI (for example, `stm32pio_gui .`)
- - [ ] Linux:
-      - Not a monospace font in the log area
+ - [ ] Linux: Not a monospaced font in the log area
+ - [ ] Temporarily pin projects with currently running actions to the top (and stay there on scrolling). See QML Package type
 
 ## Core library
- - [ ] Add more checks, for example when updating the project (`generate` command), check for boards matching and so on...
- - [x] Remove casts to string where we can use path-like objects (related to Python version as new ones receive path-like objects arguments while old ones aren't)
+ - [ ] when updating the project (`generate` command), check for boards match
+ - [ ] Remove casts to string where we can use path-like objects (related to a Python version as new ones receives path-like objects arguments while old ones aren't)
  - [ ] We look for some snippets of strings in logs and output for the testing code but we hard-code them and this is not good, probably (e.g. 'DEBUG')
- - [ ] Store a folder initial content in .ini config and ignore it on clean-up process. Allow the user to modify such list (i.e. list of exclusion) in the config file. Mb some integration with `.gitignore`
+ - [ ] Store an initial content of the folder in .ini config and ignore it on clean-up process. Allow the user to modify such list (i.e. list of exclusion) in the config file. Mb some integration with `.gitignore`
  - [ ] at some point check for all tools (CubeMX, ...) to be present in the system (both CLI and GUI) (global `--check` command (as `--version`), also before execution of the full cycle (no sense to start if some tool doesn't exist))
  - [ ] generate code docs (help user to understand an internal mechanics, e.g. for embedding). Can be uploaded to the GitHub Wiki
- - [ ] colored logs, maybe (brakes zero-dependency principle)
- - [ ] check logging work when embed stm32pio lib in a third-party stuff (no logging setup at all)
- - [ ] merge subprocess pipes to one where suitable (i.e. `stdout` and `stderr`)
- - [ ] redirect subprocess pipes to `DEVNULL` where suitable to suppress output (tests)
- - [ ] Two words about a synchronous nature of the lib and user's responsibility of async wrapping (if needed). Also, maybe migrate to async/await approach in the future
+ - [ ] colored logs, maybe (breaks zero-dependency principle)
+ - [ ] maybe migrate to async/await approach in the future (return some kind of a "remote controller" to control the running action)
  - [ ] `__init__`' `parameters` dict argument schema (Python 3.8 feature).
- - [ ] See https://docs.python.org/3/howto/logging-cookbook.html#context-info to maybe remade current logging schema (current is, perhaps, a cause of the strange error while testing (in the logging thread), also modifies global settings (log message factory))
- - [ ] UML diagrams (core, GUI back- and front-ends, thread flows, events, etc.)
- - [ ] CI is possible (Arch's AUR has the STM32CubeMX package, also there is a direct link). Deploy Docker one in Azure Pipelines, basic at Travis CI
- - [ ] Test preserving user files and folders on regeneration and mb other operations
- - [ ] Move special formatters inside the library. It is an implementation detail actually that we use subprocesses and so on
- - [ ] Mb store the last occurred exception traceback in .ini file and show on some CLI command (so we don't necessarily need to turn on the verbose mode). And, in general, we should show the error reason right off
- - [ ] 'verbose' and 'non-verbose' tests as `subTest` (also `should_log_error_...`)
+ - [ ] Mb store the last occurred exception traceback in .ini file and show on some CLI command (so we don't necessarily need to turn on the verbose mode and repeat this action). And, in general, we should show the error reason right off
  - [ ] the lib sometimes raising, sometimes returning the code and it is not consistent. While the reasons behind such behavior are clear, would be great to always return a result code and raise the exceptions in the outer scope, if there is need to
  - [ ] check board (no sense to go further on 'new' if the board in config.ini is not correct)
- - [ ] check if `platformio.ini` config will be successfully parsed when there are interpolation and/or empty parameters
- - [x] check if `.ioc` file is a text file on project initialization. Let `_find_ioc_file()` method to use explicitly provided file (useful for GUI). Maybe let user specify it via CLI
- - [ ] mb add CLI command for starting the GUI version (for example, `stm32pio --gui`)
  - [ ] test using virtualenv
+ - [ ] test for different `.ioc` files (i.e. F0, F1, F4 and so on) as it is not the same actually
+ - [ ] mb allow to use an arbitrary strings (arrays of str) to specify tools commands in stm32pio.ini (shell=True or a list of args (split a string))
+ - [ ] cache boards for a small interval of time
+ - [ ] count another '-v' as '-v' for PlatformIO calls (slider in GUI settings window)
+ - [ ] Project' name (path) can be reused so cannot be used as a unique identifier but so is id(self)? Probably it is better to use a path (human-readable)
+ - [ ] Analyze `.ioc` file for the wrong framework/parameters

docs/logging/logging.drawio

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2020-05-04T16:20:00.248Z" agent="5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15" etag="wxT5b3SQhpxaQQsfQtWn" version="13.0.1" type="device"><diagram id="lKRTrCdrkXKAPuY9JuVQ" name="Page-1">7V1bl9o2EP41nLYPcGzLNx4XNiQ5Jc02254kTxwBwrhrW44sskt/fSVfwLYEOAVf2JA8rDWWbFkz82k0MxI9MPZf3hIYrj/gJfJ6mrJ86YH7nqYBdaixP5yyTSiaZVoJxSHuMqGpe8Kj+y9KiUpK3bhLFBUqUow96oZF4gIHAVrQAg0Sgp+L1VbYK741hA4SCI8L6InUz+6SrhOqrVl7+jvkOuvszao5TO74MKucfkm0hkv8nCOBNz0wJhjT5Mp/GSOPj142Lkm7yYG7u44RFNAqDTbr6fDTw8PceTv85v/+bXtH/GF/1zm6zb4YLdkApEVM6Bo7OIDemz11RPAmWCL+WIWV9nWmGIeMqDLiP4jSbcpNuKGYkdbU99K76MWlX3jzgZGWvubu3L+kT44L27SwwgGdQN/1OGGMN8RFhHX8D8SGdJR8BO/5wcFJSRFruUhrDZ+ev9w/WZOvH/8e9eH97DNxJ7sRoZA4iB6rqO6YyMQfYR9RsmUNCfIgdb8XewJTMXR29facYhcps+SMO/J2TfkOvU36KoqI7zI2CBzFG+q5ARrvVCQbzzH2MInrAPZ/wl8/cghcumh/L8AB53lECX5CuQar+B9/kOt5sgctYbTeCcl3RKjLtGoK58h7wJFLXRywe767XMYylVW481yH35hjSrHPbsCUsGB9YhwvyNHz2qXoMYQxQ58ZAqXflQqeqmXldCx4m4hVdwOHlQAvrWHI7/gvDgewAXyO9EFI8HKzoO8XvIujkCQXxToL7PswWM74wM5c3rVV3I3RQcnjX4heciRRdDLYtFMZ3GagkpafcwiUktZ58FGU86VNChOqeS0wwUadbHONePFr/t6+WVxqHF5AVXjRzoSXuOkdIXCbqxBiJqtR7skPnJATPcMoiF5fNUqzSqmBrilHG7CLpA974dt9zBnoBwT0i6gPtNDFgw11vcEDwf8wrJtix0HkbglDpqC//tbTTI+N+WjOWGs6NFbuuN4sfpsqCPlehA/BTU5cK4jQWbCglodaU0Rc0C0JMOi14QLoDi4oA1s/Dg2s8ICIyz6dTyRluNDaxgv5CFeFC9AGXKhmWSTBcbhgc9vRBmfDxbFBzKGFHznxhfltw+3v0cff99eCSJ/Q+7INMYeRuxhg9rYZM2g8Zn2dkpey8bOzig5AhESUfgA1bBE1hhLQAEZdoKG3Chp5yMjjhWrYAwMYedTQQY5yBDo6iRyVDY1WkEMwNLShWT8QiGZDGQge3n/kD4m4z4KvfHKoULYdVgT7s2gzZzbEAkX8u/8i7LEXQY9Y3D8xy2S8Q5DYqQK4BMXOFJULXZRIO7+cJwoynf+AOXIWsOysj4yDQBeBZTdB5JHFrM0cMbqJLPb/Rhark9Cii9AirWe2gSyGWTIxgHLcJjE042iDeqBIPwlF480cffjCnxNwPqUocwOkqm6TPjAlgCQzdeoDJHFJ2X2/STcxx2zKnDmL4aag1UW/xBQ7D26IuCuiZlfDeao0rLBoUGU+yNpcDdqRoYVhKIxnBmOLDfG2IwIXT1xsTg1skQtHh9krubI9tKK9Cp7uzKFNkkGrdfFXRkRL5KImcyTXBohZ3EzGRc+dDx7Two2dVdgpcwBK+alegJ9StFVtgZ8edhw3cAYMphP366+/ZCz+pX7Yu6yHVaIvplIT6snHd3hwfB8pQdB/BwMmXqSBCeWyIysJadU2snLTzL5C06xty+woBpw0zRJhbi1irpywze7dKIR0sWbKNcHEh5Ren14ZravV8KZWF1KrYUW1ymUTtbDiUa0bwy+UGqBVxVG7VRzVfsTkG6QR9ujqbD8VtI2loEPR9Sv3HoGqupVId9M+a9MCJeE7kXZjWGVpvXDazdFRzKn+YDDoxc7pOI3mD0FgmfbRkn+5kEmYphfmkwhTkpD4d3BlLUOOohrU7S8z1VLMYedmzvvLVAl4gLrAw2g3Fnad87Jc6quGu5KMibYMMSBGlMR5ebWbmGfOxh30rNFsFkAfzWY9676BKfo891cFd+YuCN3IDG3c8mIvpWRGVSU7N6h8npIZB5WsaX/cZVVJ4o9rWJWuZh3ZmEpIQp7yilarKiEG5jKVaNKNdlF1kLjRmtUGIMbJbtt7urO957yMXEUtSlv7+3uMWzDkUqAtCYbIKzYVDDnazWpOvHit8M33ur5AUHeAfCzhodG0FeMWD7mUZkniIfLMylbjIVk3r32FIOqSZJZqVpeyrKTu61JjidiSzeryik1tVj/azWteIgj6IFkjNKsO+u0IgNe7RgB619YI9tWgb9ctGV2yNU9eUW8VtcXdcqfWCA0G+89TL6PCQkGX7V2rDc3ta9wq0k31qhqv01uN1+livK6gSzAMB6PNaoVIlMtMvJbVg6BgkvmrYQXT2lUw7RVpWNVgnd7KDlDVKm1NHjaQG2O3e6pBmnf1c4mX3c6hJ0PdaFy8dDHw3Ins9cvOEpI1dcOTxC2h41JKXDV6rbcavdbF6PU0WeR8xuTpClVI5igAsk2B9fmlxNj1nx+mjPD27/fCYGbeF9ePD2LND13ZH0S5Nh10I+28RfFGzhFcPDkxo2T+qvhld1G4d4XBrLByXzhrR2l/7teU8pNm7/iQaJPFMlAG7gIHK5eJAOEuIEZdQgrZH05n08Yk8qHn9YP+iskvK2rMMJiYlq0qZv+ZNcPPfQ9u8Yb2Vc0ehNwPVZSSd8j7jvg3SlxXJZedoqhDAC4kSWbpMC9VsiaWuZzsugTJusaodDePkdOrhqX1ls6RKx8HNTQasKnEIPhVnCJ56ZV5ltTY2oxhdcjqep3HSOqVg+fn7i37n/ov7L4yGzgOLhuVn/NgSBEIhiIQNHoypHWNe1a6ueHN0JpS+PPSozRBA69k3SVYLPLDWIwGp1Gz3RT6/CSaVx61cPKhWWEO7aZGVY3aJjLdvNM720GxE0itAb+kcfpI1ds5hj8KLeXNvdLT1xo9yDD7VZsuzMxnWugFbMmZ623DS9WodUvwYoLyfKcXLPQKLdQmAiXG6YNVX7NNb5a5ZLVs05vaq0GObtolploROECr2WRZN39OvVSUMhZKTpdpVi9bzld4/XpZdb0AWl2Bm6cN+Fesl0CwUdrWS6tDlvaV+8Csyomg7QS9bLsc9GrAPrZE+/h2hlPCjvKh56om2s7NHuJki+B8Y5aUWaraOq/EtJfx9H3yoRHPUTnAqarsKTm3UyjX9J6YJXKQcQKLL8AKUPZW6ZIpVG304ERbDOzHuUevnRWq8GucdfKCFfe/Xp1MT/sfAQdv/gM=</diagram></mxfile>

examples/README.md

@@ -0,0 +1,8 @@
+# Examples
+This directory contains some useful examples related to the project.
+
+## Table of contents
+> - [embedding](#embedding)
+
+## Embedding
+Refer to this script to see a minimal basic setup enough to perform the most of tasks with stm32pio projects in your code (i.e. how to establish a core library (`stm32pio` folder) in a 3rd party tools).