fix(ci_visibility): handle pytest ATR retry failures in unittest classes #12030

vitor-de-araujo · 2025-01-22T16:30:38Z

Auto Test Retries had a bug where retries of tests defined inside unittest classes would always succeed, even if the test failed. This was because for unittest classes, pytest saves the exception status in the pytest_runtest_makereport hook, which was not called during retries. This PR fixes it so that the hook is called.

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-01-22T16:31:08Z

CODEOWNERS have been resolved as:

releasenotes/notes/ci_visibility-fix-pytest-classes-b54582b09727fdc1.yaml  @DataDog/apm-python
ddtrace/contrib/internal/pytest/_retry_utils.py                         @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/pytest/test_pytest_atr.py                                 @DataDog/ci-app-libraries

datadog-dd-trace-py-rkomorn · 2025-01-22T16:44:52Z

Datadog Report

Branch report: vitor-de-araujo/SDTEST-1469/retry-unittest-classes
Commit report: 1bba8ac
Test service: dd-trace-py

✅ 0 Failed, 130 Passed, 1378 Skipped, 4m 39.99s Total duration (35m 6.44s time saved)

pr-commenter · 2025-01-22T17:09:27Z

Benchmarks

Benchmark execution time: 2025-01-23 09:54:20

Comparing candidate commit 1bba8ac in PR branch vitor-de-araujo/SDTEST-1469/retry-unittest-classes with baseline commit 2f52abb in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 394 metrics, 2 unstable metrics.

…ses (#12030) Auto Test Retries had a bug where retries of tests defined inside unittest classes would always succeed, even if the test failed. This was because for unittest classes, pytest saves the exception status in the [`pytest_runtest_makereport` hook](https://github.com/pytest-dev/pytest/blob/8.3.x/src/_pytest/unittest.py#L368), which was not called during retries. This PR fixes it so that the hook is called. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) (cherry picked from commit dc7037e)

…est classes (#12030)" This reverts commit dc7037e.

…ses [backport 2.20] (#12048) Backport dc7037e from #12030 to 2.20. Auto Test Retries had a bug where retries of tests defined inside unittest classes would always succeed, even if the test failed. This was because for unittest classes, pytest saves the exception status in the [`pytest_runtest_makereport` hook](https://github.com/pytest-dev/pytest/blob/8.3.x/src/_pytest/unittest.py#L368), which was not called during retries. This PR fixes it so that the hook is called. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Vítor De Araújo <[email protected]>

…ses [backport 2.18] (#12046) Backport dc7037e from #12030 to 2.18. Auto Test Retries had a bug where retries of tests defined inside unittest classes would always succeed, even if the test failed. This was because for unittest classes, pytest saves the exception status in the [`pytest_runtest_makereport` hook](https://github.com/pytest-dev/pytest/blob/8.3.x/src/_pytest/unittest.py#L368), which was not called during retries. This PR fixes it so that the hook is called. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Vítor De Araújo <[email protected]> Co-authored-by: Brett Langdon <[email protected]>

…ses (#12030) Auto Test Retries had a bug where retries of tests defined inside unittest classes would always succeed, even if the test failed. This was because for unittest classes, pytest saves the exception status in the [`pytest_runtest_makereport` hook](https://github.com/pytest-dev/pytest/blob/8.3.x/src/_pytest/unittest.py#L368), which was not called during retries. This PR fixes it so that the hook is called. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

fix(ci_visibility): retry tests defined in unittest classes with pytest

f3ce291

vitor-de-araujo added 4 commits January 23, 2025 08:56

tests

6f45a10

tests for skipping

e6421ee

lint

9a35f29

release note

1bba8ac

vitor-de-araujo changed the title ~~fix(ci_visibility): retry tests defined in unittest classes with pytest~~ fix(ci_visibility): handle ATR retry failures in unittest classes with pytest Jan 23, 2025

vitor-de-araujo changed the title ~~fix(ci_visibility): handle ATR retry failures in unittest classes with pytest~~ fix(ci_visibility): handle pytest ATR retry failures in unittest classes Jan 23, 2025

vitor-de-araujo marked this pull request as ready for review January 23, 2025 09:20

vitor-de-araujo requested review from a team as code owners January 23, 2025 09:20

vitor-de-araujo requested review from anmarchenko, gnufede, tabgok and emmettbutler January 23, 2025 09:20

vitor-de-araujo added backport 2.18 backport 2.19 backport 2.20 labels Jan 23, 2025

juan-fernandez approved these changes Jan 23, 2025

View reviewed changes

anmarchenko approved these changes Jan 23, 2025

View reviewed changes

vitor-de-araujo enabled auto-merge (squash) January 23, 2025 16:13

emmettbutler approved these changes Jan 23, 2025

View reviewed changes

vitor-de-araujo merged commit dc7037e into main Jan 23, 2025
520 of 524 checks passed

vitor-de-araujo deleted the vitor-de-araujo/SDTEST-1469/retry-unittest-classes branch January 23, 2025 16:54

github-actions bot mentioned this pull request Jan 23, 2025

fix(ci_visibility): handle pytest ATR retry failures in unittest classes [backport 2.18] #12046

Merged

2 tasks

github-actions bot mentioned this pull request Jan 23, 2025

fix(ci_visibility): handle pytest ATR retry failures in unittest classes [backport 2.19] #12047

Merged

2 tasks

github-actions bot mentioned this pull request Jan 23, 2025

fix(ci_visibility): handle pytest ATR retry failures in unittest classes [backport 2.20] #12048

Merged

2 tasks

wantsui added a commit that referenced this pull request Jan 23, 2025

Revert "fix(ci_visibility): handle pytest ATR retry failures in unitt…

ecf1663

…est classes (#12030)" This reverts commit dc7037e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ci_visibility): handle pytest ATR retry failures in unittest classes #12030

fix(ci_visibility): handle pytest ATR retry failures in unittest classes #12030

vitor-de-araujo commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 22, 2025 •

edited

Loading

pr-commenter bot commented Jan 22, 2025 •

edited

Loading

fix(ci_visibility): handle pytest ATR retry failures in unittest classes #12030

fix(ci_visibility): handle pytest ATR retry failures in unittest classes #12030

Conversation

vitor-de-araujo commented Jan 22, 2025 • edited Loading

Checklist

Reviewer Checklist

github-actions bot commented Jan 22, 2025 • edited Loading

datadog-dd-trace-py-rkomorn bot commented Jan 22, 2025 • edited Loading

Datadog Report

pr-commenter bot commented Jan 22, 2025 • edited Loading

Benchmarks

vitor-de-araujo commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 22, 2025 •

edited

Loading

pr-commenter bot commented Jan 22, 2025 •

edited

Loading