Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: stdlib models qa #16843

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

Python: stdlib models qa #16843

wants to merge 17 commits into from

Commits on Jun 25, 2024

  1. python: Start modelling using MaD

    - empty models for now
    - `summaryModel` of `codeql/python-all` will be added to shortly.
    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    df406b4 View commit details
    Browse the repository at this point in the history
  2. python: add modelling for urlib.parse

    - `quote` together with `re.compile` recover regex injection alerts on haiwen/seahub
    - `quote_plus` recovers the URL redirection alert on DemocracyClub/EveryElection
    - `unquote` recovers path injection alerts on `cloudera/hue`
    - it was tedious finding justifications for the rest..
    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    281ac05 View commit details
    Browse the repository at this point in the history
  3. python: move model to Stdlib.yml

    There is already a model there so we add to that one.
    
    We did observe that this existing model was blocked by the external MaD model.
    This is concerning and needs to be cleared up.
    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c004ffa View commit details
    Browse the repository at this point in the history
  4. python: compress models

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d410136 View commit details
    Browse the repository at this point in the history
  5. Python: move models

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    1e97600 View commit details
    Browse the repository at this point in the history
  6. python: undo changes to qlpack

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b80a711 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2118f23 View commit details
    Browse the repository at this point in the history
  8. Python: model fnmatch.filter

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    501cda4 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    bc55117 View commit details
    Browse the repository at this point in the history
  10. Python: MaD summary models

    Two of the generated summaries have been excluded:
     - ["re", "Member[split]", "Argument[0,pattern:]", "ReturnValue", "taint"]
       From the documentation, it is not clear why pattern should figure in the return value, as that is the part denoting split point and thus all those instances are filtered out.
       From the implementation
         Spit function: https://github.com/python/cpython/blob/3.12/Lib/re/__init__.py#L199
         _compile function being called by split: https://github.com/python/cpython/blob/3.12/Lib/re/__init__.py#L280
       We see that in case the pattern is already a compiled `Pattern`, it is returned directly from _compile and could thus be part of the return value from split. This is probably not possible to arrange for an attacker, and so an FP in practice.
    
     - ["urllib2", "Member[unquote]", "Argument[0,string:]", "ReturnValue", "taint"]
       urllib2 seems to be only in Python2 (e.g. https://docs.python.org/2.7/library/urllib2.html) and I cannot locate the function unquote.
    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    bdc4808 View commit details
    Browse the repository at this point in the history
  11. Python: codecs.open

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    eb32cbe View commit details
    Browse the repository at this point in the history
  12. Python: model more loggers

    yoff committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    571be8b View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Python: fix compilation

    yoff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    b261145 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a3076f4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    25e6898 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Configuration menu
    Copy the full SHA
    8fabcc6 View commit details
    Browse the repository at this point in the history
  2. Python: adjust expected files

    yoff committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    5667e83 View commit details
    Browse the repository at this point in the history