Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(spdx): use the hasExtractedLicensingInfos field for licenses that are not listed in the SPDX #8077

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

fix(spdx): use the hasExtractedLicensingInfos field for licenses that are not listed in the SPDX #8077

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
6272fdd
feat: add spdx licenses and exceptions + validation
DmitriyLewen Dec 10, 2024
eb30b49
feat: use otherLicense for non-spdx licenses
DmitriyLewen Dec 10, 2024
b1081e2
test: add/update unit tests
DmitriyLewen Dec 10, 2024
420c15e
refactor: remove duplicates of otherLicenses
DmitriyLewen Dec 10, 2024
b6c1d28
refactor: always fill ExtractedText and LicenseName fields
DmitriyLewen Dec 10, 2024
1bac2ca
fix: tests
DmitriyLewen Dec 11, 2024
f851f9b
add comment
DmitriyLewen Dec 11, 2024
041ab21
feat: add `LicenseComment` field
DmitriyLewen Dec 12, 2024
659f992
refactor: ExtractedText field for license with name
DmitriyLewen Dec 13, 2024
c25d840
refactor: use exception list from spdx.org site
DmitriyLewen Dec 13, 2024
20275c3
test: update tests
DmitriyLewen Dec 13, 2024
c89c4e3
refactor: use `exceptions.json` file
DmitriyLewen Dec 17, 2024
784db9e
feat(mage): add command to create exceptions.json file
DmitriyLewen Dec 17, 2024
ffb5067
feat(licensing): add exceptions.json file
DmitriyLewen Dec 17, 2024
9f0f7bf
fix(mage): fix typo
DmitriyLewen Dec 17, 2024
c5563f5
ci: add spdx-cron
DmitriyLewen Dec 17, 2024
0d13108
ci: add aqua-installer
DmitriyLewen Dec 17, 2024
a2ebc32
fix: linter error
DmitriyLewen Dec 17, 2024
f9ea255
Merge branch 'main' of github.com:DmitriyLewen/trivy into fix/use-oth…
DmitriyLewen Jan 21, 2025
48a46b8
refactor: rename ValidSpdxLicense to ValidateSPDXLicense
DmitriyLewen Jan 21, 2025
ab86fd6
fix: remove duplicate step
DmitriyLewen Jan 21, 2025
8a96a8b
refactor: doesn't check `spdxLicenses` and `spdxExceptions` before init
DmitriyLewen Jan 21, 2025
82a52f1
refactor: use set.Set instead of map[string]struct{}
DmitriyLewen Jan 21, 2025
b38b6eb
refactor: use `spdx` lowercase prefix + use const
DmitriyLewen Jan 21, 2025
d8c4b83
refactor: logic for ValidateSPDXLicense
DmitriyLewen Jan 21, 2025
a8a85ad
refactor: add replaceOtherLicenses function
DmitriyLewen Jan 21, 2025
d4e67dc
refactor: use original spdx exception-id
DmitriyLewen Jan 22, 2025
5d0f7e1
refactor: normalize exceptions in NormalizeForSPDX
DmitriyLewen Jan 22, 2025
08ea0c8
fix: linter error
DmitriyLewen Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 134 additions & 0 deletions pkg/licensing/expression/category.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
package expression

import (
"slices"
"strings"

"github.com/samber/lo"
)

// Canonical names of the licenses.
// ported from https://github.com/google/licenseclassifier/blob/7c62d6fe8d3aa2f39c4affb58c9781d9dc951a2d/license_type.go#L24-L177
const (
Expand Down Expand Up @@ -358,4 +365,131 @@ var (
Unlicense,
ZeroBSD,
}

// SpdxLicenseExceptions contains all supported SPDX Exceptions
// cf. https://github.com/spdx/license-list-data/blob/592c2dcb8497c6fe829eea604045f77d3bce770b/json/exceptions.json
// used `awk -F'"' '/"licenseExceptionId":/ {print toupper("\"" $4 "\"," )}' exceptions.json ` command
spdxLicenseExceptions = []string{
"389-EXCEPTION",
"ASTERISK-EXCEPTION",
"ASTERISK-LINKING-PROTOCOLS-EXCEPTION",
"AUTOCONF-EXCEPTION-2.0",
"AUTOCONF-EXCEPTION-3.0",
"AUTOCONF-EXCEPTION-GENERIC",
"AUTOCONF-EXCEPTION-GENERIC-3.0",
"AUTOCONF-EXCEPTION-MACRO",
"BISON-EXCEPTION-1.24",
"BISON-EXCEPTION-2.2",
"BOOTLOADER-EXCEPTION",
"CGAL-LINKING-EXCEPTION",
"CLASSPATH-EXCEPTION-2.0",
"CLISP-EXCEPTION-2.0",
"CRYPTSETUP-OPENSSL-EXCEPTION",
"DIGIRULE-FOSS-EXCEPTION",
"ECOS-EXCEPTION-2.0",
"ERLANG-OTP-LINKING-EXCEPTION",
"FAWKES-RUNTIME-EXCEPTION",
"FLTK-EXCEPTION",
"FMT-EXCEPTION",
"FONT-EXCEPTION-2.0",
"FREERTOS-EXCEPTION-2.0",
"GCC-EXCEPTION-2.0",
"GCC-EXCEPTION-2.0-NOTE",
"GCC-EXCEPTION-3.1",
"GMSH-EXCEPTION",
"GNAT-EXCEPTION",
"GNOME-EXAMPLES-EXCEPTION",
"GNU-COMPILER-EXCEPTION",
"GNU-JAVAMAIL-EXCEPTION",
"GPL-3.0-389-DS-BASE-EXCEPTION",
"GPL-3.0-INTERFACE-EXCEPTION",
"GPL-3.0-LINKING-EXCEPTION",
"GPL-3.0-LINKING-SOURCE-EXCEPTION",
"GPL-CC-1.0",
"GSTREAMER-EXCEPTION-2005",
"GSTREAMER-EXCEPTION-2008",
"HARBOUR-EXCEPTION", //nolint:misspell
"I2P-GPL-JAVA-EXCEPTION",
"INDEPENDENT-MODULES-EXCEPTION",
"KICAD-LIBRARIES-EXCEPTION",
"LGPL-3.0-LINKING-EXCEPTION",
"LIBPRI-OPENH323-EXCEPTION",
"LIBTOOL-EXCEPTION",
"LINUX-SYSCALL-NOTE",
"LLGPL",
"LLVM-EXCEPTION",
"LZMA-EXCEPTION",
"MIF-EXCEPTION",
"MXML-EXCEPTION",
"NOKIA-QT-EXCEPTION-1.1",
"OCAML-LGPL-LINKING-EXCEPTION",
"OCCT-EXCEPTION-1.0",
"OPENJDK-ASSEMBLY-EXCEPTION-1.0",
"OPENVPN-OPENSSL-EXCEPTION",
"PCRE2-EXCEPTION",
"PS-OR-PDF-FONT-EXCEPTION-20170817",
"QPL-1.0-INRIA-2004-EXCEPTION",
"QT-GPL-EXCEPTION-1.0",
"QT-LGPL-EXCEPTION-1.1",
"QWT-EXCEPTION-1.0",
"ROMIC-EXCEPTION",
"RRDTOOL-FLOSS-EXCEPTION-2.0",
"SANE-EXCEPTION",
"SHL-2.0",
"SHL-2.1",
"STUNNEL-EXCEPTION",
"SWI-EXCEPTION",
"SWIFT-EXCEPTION",
"TEXINFO-EXCEPTION",
"U-BOOT-EXCEPTION-2.0",
"UBDL-EXCEPTION",
"UNIVERSAL-FOSS-EXCEPTION-1.0",
"VSFTPD-OPENSSL-EXCEPTION",
"WXWINDOWS-EXCEPTION-3.1",
"X11VNC-OPENSSL-EXCEPTION",
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment may deserve to be a separate suggestion, but in reading the code I would recommend building the license and exception IDs from the JSON files maintained by the SPDX legal team. The license list is updated every 3 months with new IDs and maintaining these in code can be a challenge to keep up and maintain. What I do in the code I maintain is attempt to access the current JSON files on the website https://spdx.org/licenses/licenses.json and https://spdx.org/licenses/exceptions.json. If I can not access the website or if the user specified not to use the online version, I'll use a cached version of the file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are cases when users run multiple times.
Downloading these files for each run is not good.
But we can save licenses.json and exceptions.json files in the cache dir and use them.
The files contain releaseDate field, so we can update this file only when releaseDate + 3 months has expired.

The license list is updated every 3 months

How strictly is this rule followed?

Anyway let's move this discussion into another issue/pr.

What I do in the code I maintain is attempt to access the current JSON files on the website https://spdx.org/licenses/licenses.json and https://spdx.org/licenses/exceptions.json

I found that https://spdx.org/licenses/exceptions.json and https://github.com/spdx/license-list-data/blob/592c2dcb8497c6fe829eea604045f77d3bce770b/json/exceptions.json are different (see harbour-exception).
Which file would be more correct to use?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How strictly is this rule followed?

Not very strictly. There is, however, a license list version field which is reliably incremented on release.

I found that https://spdx.org/licenses/exceptions.json and https://github.com/spdx/license-list-data/blob/592c2dcb8497c6fe829eea604045f77d3bce770b/json/exceptions.json are different (see harbour-exception).
Which file would be more correct to use?

The lists at https://spdx.org/licenses - these will always be the latest released version. The github repo master will have the latest in development version which may not be stable. The github repo is tagged with release versions, so if you go to the tag for the latest release in github, it will match what is on the website.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, update exception list in 659f992

)

var spdxLicenses map[string]struct{}
knqyf263 marked this conversation as resolved.
Show resolved Hide resolved

func initSpdxLicenses() {
licenseSlices := [][]string{
ForbiddenLicenses,
RestrictedLicenses,
ReciprocalLicenses,
NoticeLicenses,
PermissiveLicenses,
UnencumberedLicenses,
}

for _, licenseSlice := range licenseSlices {
spdxLicenses = lo.Assign(spdxLicenses, lo.SliceToMap(licenseSlice, func(l string) (string, struct{}) {
return l, struct{}{}
}))
}

// Save GNU licenses with "-or-later" and `"-only" suffixes
for _, l := range GnuLicenses {
license := SimpleExpr{
License: l,
}
spdxLicenses[license.String()] = struct{}{}

license.HasPlus = true
spdxLicenses[license.String()] = struct{}{}
}

}

// ValidSpdxLicense returns true if SPDX license lists contain licenseID and license exception (if exists)
func ValidSpdxLicense(license string) bool {
knqyf263 marked this conversation as resolved.
Show resolved Hide resolved
if spdxLicenses == nil {
initSpdxLicenses()
}
knqyf263 marked this conversation as resolved.
Show resolved Hide resolved

id, exception, ok := strings.Cut(license, " WITH ")
if _, licenseIdFound := spdxLicenses[id]; licenseIdFound && (!ok || slices.Contains(spdxLicenseExceptions, strings.ToUpper(exception))) {
return true
}
return false
}
Loading
Loading