Generate dependency information / SBOM as part of the MAT build #40
Comments
|
By Andrew Johnson on Sep 27, 2023 09:23 One example is the dash-license-tool project itself. That has a ci build: https://ci.eclipse.org/dash/job/dash-license-tool/\ [INFO] --- cyclonedx:2.7.9:makeAggregateBom (default) @ org.eclipse.dash.licenses --- and archives them |
|
By Krum Tsvetkov on Sep 28, 2023 05:53 Thanks for the hint to look at Dash. It was straightforward to add both dash-license and cyclonedx to the build and let them produce "some" results as part of the local snapshot build :) I added the parameter to make dash-license fail the build in case of an unapproved dependency. Testet it locally with an arbitrary lib and the build failed lile: [INFO] ------------------------------------------------------------------------ I think this is understandable enough, and will keep the flag in. Next step would be for both dash-license and cyclonedx to go through the results in detail and see if we are reporting too many results and try to tune the build to exclude some parts. Both tools support that. What still puzzles me a bit is what to do with the results. We offer for download an update side which contains only MAT bundles + birt. And we do offer also the standalone MAT which includes everything an eclipse RCP needs to run. Having the full bom for the standalone MAT is reasonable I thing, but if people are using only the update site, then the bom will contain many more entries than provided software. I can't say how much of an issue this is. I also see that the Dash project has the DEPENDENCIES file as part of their source repository. Same questions arise here too, plus we do support building against multiple targets/Eclipse versions. So, if I were to put a single file as part of the repository - 1) shall it contain the full bom (needed for the standalone MAT) or only what we need to build the plugins and 2) if we put the full list, shall it be for the target selected by default? Any thoughts on these would be helpful. |
|
By Andrew Johnson on Sep 28, 2023 06:34 Ideally we would have a separate BOM for the update site. We could have a vulnerability in the standalone version due to an older level of Eclipse, and the work-around would be to install MAT into a newer Eclipse. However, do build/test artifacts also appear in the BOM? E.g. SWTBot. |
|
By Krum Tsvetkov on Sep 28, 2023 06:58 All examples I was were excluding test code, therefore tried it also for out build (still need to go through the results to see if this had effect). I also felt like having separate BOMs for the two use cases is the most reasonable approach, but I'll have to see if I can get setup this with the provided tools. I guess I'll need to spend some more time reading and understanding what are typical requirements for BOMs and what are Eclipse specific ones. If in doubt, I would probably reach out the the foundation staff to ask for some guidance. But I'll see first what I can find myself before bothering them. |
|
By Andrew Johnson on Oct 11, 2023 04:41 See also https://gitlab.eclipse.org/eclipsefdn/emo-team/sbom/-/blob/main/docs/sbom.adoc The short version is that generating SBOMs directly as part of your build, and (at least in the case of Maven) sharing them to the software repository is relatively straightforward. To really leverage the the tools to generate SBOMs, however, we need your help to tighten up the metadata captured in your build scripts (e.g., capture license information as SPDX expressions in your pom.xml file) and update your builds to generate the SBOMs. We've started capturing information here. Merge requests, issues, and comments welcome. Date: October 12, 2023 https://eclipse.zoom.us/j/84638945339 See you then! Our office hours calendar, along with links to recordings is available here https://www.eclipse.org/projects/calendar/. Wayne |
|
By Krum Tsvetkov on Oct 11, 2023 16:12 Thanks Andrew! I won't be able to attend tomorrow, but I got the links to the sbom repo already and promised Wayne to give some feedback... then got caught in various other activities :( There was one more useful info - https://gitlab.eclipse.org/eclipsefdn/emo-team/sbom/-/issues/3 . In this issue it is basically discussed, that having sboms for the plugins is not useful and that sboms would be rather suitable for the Eclipse packages (i.e. I read it that we need one for the standalone MAT application, but not for the update site). I still have a few questions, which I need to summarise. |
|
By Krum Tsvetkov on Oct 26, 2023 05:14 I merged https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/204643.\
I adapted the jenkins build config, so that these two files are archived and available as "latest results" in jenkins, as the dash project does. I believe we would need some fine-tuning as we learn more about what is required. Therefore I'll keep this bug open. |
|
By Andrew Johnson on Oct 26, 2023 08:11 Bug 438844 added GZip processing of HPROF files, including a pure Jaza GZip reader via CQ 21570. plugin: org.eclipse.mat.hprof The original code in this package was MIT licensed, I made updates for MAT under the same license. There is some copyright / license text in the package-info.java https://wiki.eclipse.org/About_files\ so I think I need to update the about.html to include this information. If the org.eclipse.mat.hprof plug-in binary is now under both licenses, then |
|
By Andrew Johnson on Oct 26, 2023 10:36 https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/205139\ I made those changes, including to the o.e.mat.hprof pom.xml, but that license report is still the same. I don't think this is very important though.
The Help > About Eclipse Memory Analyzer > Installation Details > Plug-ins > org.eclipse.mat.hprof > Legal Info does show both licenses. |
|
By Krum Tsvetkov on Oct 27, 2023 01:57 I've overlooked that, thanks for the updates! |
|
By Krum Tsvetkov on Oct 27, 2023 02:26 One more thing we should check is versioning. So far we always keep the -SNAPSHOT version for maven and use the plugins with auto-generated versions with timestamps. I see the tools now all report a 1.15.0-SNAPSHOT version for MAT, which doesn't feel right. I don't know hot to deal with this. I found some recent changes for jgit/egit which seem related: AFAIU they try to use the timestamp of the last commit ... I am not sure we need this, but it might be possible to use some parts of the change to influence the sbom results. |
|
By Krum Tsvetkov on Oct 27, 2023 04:55 I was a bit confused about the licenses in the bom.json. It is the aggregated bom (the one linked from jenkins) the license info is missing for the referenced MAT plugins, but is present for "foreign" libraries. No idea why this is so, and if this is fine (I suppose this is not fine as long as the only thing we publish is the aggregated bom). |
|
By Krum Tsvetkov on Oct 27, 2023 06:54 I added some metadata as recommended to the parent pom |
|
By Krum Tsvetkov on Nov 02, 2023 05:31 The recording of the Open hours call about SBOMs on 12 October is now available: |
|
By Krum Tsvetkov on Nov 02, 2023 06:50 @andrew, some questions about the o.e.m.hprof license - I can't open the CQ 21570, (don't know why) and I have no experience with dual licenses, so I was hoping to have some more details in the CQ.
|
|
By Andrew Johnson on Nov 02, 2023 07:43 I don't know properly how it works but the top level Eclipse Foundation Software User Agreement has some lines: The terms and conditions governing Plug-ins and Fragments should be contained in files named "about.html" ("Abouts"). The terms and conditions governing Features and Included Features should be contained in files named "license.html" ("Feature Licenses"). Abouts and Feature Licenses may be located in any directory of a Download or Module including, but not limited to the following locations: then if you look in org.eclipse.mat.hprof there is an about.html and about_files/DEFLATE-mit.html with the extra information. The user can view the o.e.mat.hprof about.html with I don't think there is any need to change the top level Software User Agreement (and that might not be allowed anyway), and the user has agreed to follow all those sub level about.html files. If you go to some of the other plug-ins then they can have different terms. E.g. Apache Commons JXPath |
|
By Andrew Johnson on Nov 02, 2023 08:10 Perhaps for o.e.mat.hprof you could use: Bundle-License: EPL-2.0;link="https://www.eclipse.org/org/documents/epl-2.0/EPL-2.0.txt",MIT;link="https://opensource.org/license/mit/" or possibly https://spdx.org/licenses/MIT.html for the link. |
|
By Krum Tsvetkov on Nov 03, 2023 05:05 Thanks! BTW, the last changes I made (add Bundle-License headers to the MANIFEST.MF files) might be unnecessary - I saw that in the manifest generated during the build the license headers were added. Nevertheless, I think I'll keep the explicit ones, so that we see these also in the source. |
|
By Andrew Johnson on Nov 08, 2023 06:57 The latest build just has the o.e.mat bundles in bom.json and summary, whereas previous had a lot more. Is this due to the Tycho change? https://ci.eclipse.org/mat/job/tycho-mat-nightly/1508/artifact/parent/target/bom.json Build Artifacts https://ci.eclipse.org/mat/job/tycho-mat-nightly/1508/artifact/parent/target/dash/summary/*view*/ maven/mavencentral/org.eclipse.mat/org.eclipse.mat.api/1.15.0-SNAPSHOT, EPL-2.0, approved, tools.mat but a previous build had a lot more: https://ci.eclipse.org/mat/job/tycho-mat-nightly/1508/artifact/parent/target/dash/summary/*view*/ maven/mavencentral/com.ibm.icu/icu4j/73.2, GPL-2.0-or-later AND GPL-3.0-or-later AND LicenseRef-scancode-autoconf-simple-exception AND LicenseRef-scancode-autoconf-simple-exception-2.0 AND LicenseRef-scancode-unicode AND LicenseRef-scancode-unicode-icu-58 AND NTP AND LicenseRef-scancode-unicode, approved, #9093 |
|
By Krum Tsvetkov on Nov 09, 2023 04:40 I added requireEagerResolve to the target platform configuration as I verified that the setting doesn't lead to having more plugins in the standalone version. |
| --- | --- |
| Bugzilla Link | 582480 |
| Status | ASSIGNED |
| Importance | P3 normal |
| Reported | Sep 27, 2023 08:00 EDT |
| Modified | Nov 09, 2023 04:40 EDT |
| Version | 1.14 |
| Reporter | Krum Tsvetkov |
Description
The topic of SBOMs was mentioned on the dev mailing list. I see various aspects and have at present no good idea about the solution, but would use the bug to keep the discussions / ideas.
Removal of IP Logs - the changes introduced last year https://www.eclipse.org/projects/handbook/#ip-history projects are not required to submit IP Logs, but to provide SBOMs. AFAIU there is still no standard way for this. For our last release I used the https://github.com/eclipse/dash-licenses tool and manually checked the report that all listed dependencies are approved. Generating this information can be done as part of the build, and for me this would be the first step.
I still wonder what to do with the information once it is generated. There is an option to fail the build. I haven't tried this one (we use only approved libs), but I intend to add a dependency to something not-approved and see what the effect is.
So far I only did some checks that the plugins I find in the standalone MAT package are all part of the generated dependency list (they are indeed a subset of it, the generated list contains more).
The second aspect is providing an SBOM as part of ...
I guess the best would be to find another Eclipse project which does this and mimic their setup. Any idea of a concrete project to look at?
The text was updated successfully, but these errors were encountered: