Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lower the time tests take to run #1567

Open
another-rex opened this issue Feb 4, 2025 · 2 comments · May be fixed by #1603
Open

Lower the time tests take to run #1567

another-rex opened this issue Feb 4, 2025 · 2 comments · May be fixed by #1603
Labels
performance

Comments

@another-rex
Copy link
Collaborator

Our go tests are starting to take quite a long time now, we should look into performance improvements or gating tests behind flags so they are not always ran. (though caching should already be doing this)

Here are the tests that are currently taking more than 2 seconds, sorted in ascending order:

TestRun_MavenTransitive/scans_transitive_dependencies_by_specifying_pom.xml,2.04
TestRun_MavenTransitive/scans_transitive_dependencies_for_pom.xml_by_default,2.35
TestRun/config_file_can_be_broad,2.38
TestRun_Licenses/No_vulnerabilities_with_license_summary_in_markdown,2.42
TestRun_MavenTransitive/scans_dependencies_from_multiple_registries,2.52
TestRun/PURL_SBOM_case_sensitivity_(local),2.52
TestRun_MavenTransitive/resolve_transitive_dependencies_with_native_data_source,2.6
TestRun_Licenses/Some_packages_with_ignored_licenses,2.7
Test_runGovulncheck,2.9
TestRun/requirements.txt_can_have_all_kinds_of_names,2.97
TestRun_LocalDatabases/only_the_files_in_the_given_directories_are_checked_by_default_(no_recursion),3.39
TestRun_LocalDatabases/.gitignored_files,3.91
TestRun_LocalDatabases/all_supported_lockfiles_in_the_directory_should_be_checked#01,3.97
TestRun_LocalDatabases/nested_directories_are_checked_when_`--recursive`_is_passed,4.53
TestRun_Fix/fix_non-interactive_json_override_pom.xml,4.54
TestRun_Fix/fix_non-interactive_override_pom.xml,4.67
TestPrintCycloneDX14Results_WithVulnerabilities,4.87
TestPrintCycloneDX15Results_WithLicenseViolations,4.95
TestPrintOutputResults_WithVulnerabilities,4.96
TestFromCSVFile,5.05
TestRun_LocalDatabases/ignoring_.gitignore,5.08
TestRunCallAnalysis/Run_with_govulncheck,5.42
TestPrintTableResults_NoTerminalWidth_WithVulnerabilities,6.05
TestCVE202230633,6.12
TestRun/folder_of_supported_sbom_with_vulns,7.27
TestRun_Fix/fix_non-interactive_relax_package.json,8.87
TestRun_LocalDatabases/all_supported_lockfiles_in_the_directory_should_be_checked,9.46
TestRun_Fix/fix_non-interactive_json_relax_package.json,9.61
TestRun_Fix/fix_non-interactive_json_in-place_package-lock.json,10.18
TestRun_Fix/fix_non-interactive_in-place_package-lock.json,11.06
TestRun_LocalDatabases/one_specific_supported_sbom_with_vulns,22.01
@G-Rath
Copy link
Collaborator

G-Rath commented Feb 5, 2025

(though caching should already be doing this)

Sadly caching doesn't tend to help because our top-most packages by definition depend on everything meaning 1. they're generally the slowest of tests as they're the most e2e-y, and 2. pretty much all changes to the codebase invalid their test cache

@G-Rath
Copy link
Collaborator

G-Rath commented Feb 6, 2025

So I knew since we introduced them that the offline/local database based tests are a large source of time because they have to download the all.zip files multiple times and they can be quite large - I've had an idea or two on how we could possibly improve this which I'm exploring now and it looks like it's working (taking us down to 10 seconds!).

There's some trade-offs to discuss and I need to actually cleanup the code to confirm everything properly which is why I'm not sharing more details just yet, but mentioning it now in case anyone else looks at this to avoid possible double work

This was referenced Feb 9, 2025
Merged
Draft
another-rex pushed a commit that referenced this issue Feb 27, 2025
@G-Rath
fix(output): ensure that vulnerabilities are sorted by ID across grou…
ab8123c 
…ps in table output (#1598)

While prototyping some changes to help with #1567, I discovered that the
table output format is not explicitly sorting vulnerabilities across
their groups, meaning we're assumingly using the order that the API
gives us, and means our current output does not actually follow a
predictable order.

To address this, I've modified the table outputter to sort vulnerability
groups by the first ID in each group as a group by definition will
always have at least one ID and the first ID should be the one primary
one since we already sort ids within each group as part of building the
general results
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
2 participants
@G-Rath @another-rex