Fix: Instrumented tests are executed in Drone CI again#16612
Fix: Instrumented tests are executed in Drone CI again#16612
Conversation
The Drone CI job is running ./gradlew createGplayDebugCoverageReport This only executes tests with coverage. In 5fd2e29, the line testCoverageEnabled = project.hasProperty("coverage") was replaced with enableUnitTestCoverage = project.hasProperty("coverage") which only left the unit tests with coverage Enabling the coverage for androidTest (the instrumented tests) makes Drone run them again. Closes #16350 Signed-off-by: Philipp Hasper <vcs@hasper.info>
The drone tests simply ended with the log line: stable-IT test failed, but no output was generated. Maybe a preliminary stage failed. The reason was that the script didn't use the correct folder Signed-off-by: Philipp Hasper <vcs@hasper.info>
|
blue-Light-Screenshot test failed, but no output was generated. Maybe a preliminary stage failed. |
|
APK file: https://www.kaminsky.me/nc-dev/android-artifacts/16612.apk |
|
|
||
| debug { | ||
| enableUnitTestCoverage = project.hasProperty("coverage") | ||
| enableAndroidTestCoverage = project.hasProperty("coverage") |
There was a problem hiding this comment.
@tobiasKaminsky with this, Drone now actually runs the instrumented tests again. See https://drone.nextcloud.com/nextcloud/android/28585/1/3 as a proof of failure, i.e. broken tests on master which were never detected.
I'll go ahead addressing the test failures. But there is several other things I noticed:
- We should have Drone only run the instrumentation tests. Currently, drone CI runs everything for which coverage is enabled. Why this is chosen as filter, I don't know. But what this means is that Drone wastes time on the unit tests which we already run elsewhere.
- Unit tests were the only tests Drone ran since Chore: Improve Gradle Configuration #15859. Another weird side-effect of that PR.
- Easiest for saving drone time would be to disable the coverage for unit tests in the line above, but I guess this is not desired?
- Actually, what about the coverage reporting? It doesn't appear anymore. It looks broken, in several ways: - I am not seeing the coverage report on any PR I have recently checked. Don't know when this stopped. Many of the pulls on Codecov just show errors: https://app.codecov.io/github/nextcloud/android/pulls
- The unit test job seems to still upload to Codecov, but prints out wrong links in the log, e.g.https://app.codecov.io/github/***/android/commit/2c1438fba2fda500d3ec2e330d2aa62284ef8e2c (log) .
- You need to replace the
***withnextcloud, then it works. Is this redaction of secrets at play - if so, I surely hope, no secret token has simply "nextcloud" as value?
- Coverage reporting should also be extended by the instrumentation tests. I don't see them uploading anything
| if [ $TYPE = "IT" ]; then | ||
| FOLDER=app/build/reports/androidTests/connected/flavors/gplay | ||
| FOLDER=app/build/reports/androidTests/connected/debug/flavors/gplay |
There was a problem hiding this comment.
@tobiasKaminsky these adjustments are needed so the report files are actually found. But reporting still is broken, which you can see e.g. in the log of https://drone.nextcloud.com/nextcloud/android/28585/1/3:
It does upload the report to https://www.kaminsky.me/nc-dev/android-integrationTests/28585-IT-stable-10-44 but it fails to create the expected Github comment notifying the developer of this.
The error is:
{
"message": "Must have admin rights to Repository.",
"documentation_url": "https://docs.github.com/rest/issues/comments#create-an-issue-comment",
"status": "403"
}
The Drone CI job is running ./gradlew createGplayDebugCoverageReport This only executes tests with coverage.
In 5fd2e29 (PR #15859), the line
was replaced with
which only left the unit tests with coverage
Enabling the coverage for androidTest (the instrumented tests) makes Drone run them again.
Closes #16350
🏁 Checklist