Skip to content

feat: allow citgm job comparison#455

Merged
codebytere merged 8 commits intonodejs:masterfrom
codebytere:allow-citgm-comparison
Jul 23, 2020
Merged

feat: allow citgm job comparison#455
codebytere merged 8 commits intonodejs:masterfrom
codebytere:allow-citgm-comparison

Conversation

@codebytere
Copy link
Member

@codebytere codebytere commented Jul 10, 2020 •
edited
Loading

Follow-up to #454.
Closes #352.

Extends ncu-ci cigtm <jobid> to ncu-cu cigtm <jobid> [jobid2], allowing for direct comparisons between two CITGM runs to see if new failures were introduced.

I also did a bit of a refactor here, since there were a million classes in the result parser file and it was getting wildly unwieldy. I tried to do the minimum necessary to make this work but I plan to follow it up with the remainder of the the class split-out work.

Display:

Developer ❯ ncu-ci citgm 2392 2390                                             11:06PM
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2392
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2392/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x
Commit     [feed95cd4c2c] Working on v12.18.1
Date       2020-06-02 20:27:47 +0200
Author     Michaël Zasso <targos@protonmail.com>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2390/
Source     https://github.com/nodejs/node/pull/33811/
Commit     [9a60117875dd] 2020-06-16, Version 12.18.1 'Erbium' (LTS)
Date       2020-06-09 20:23:09 -0700
Author     Shelley Vohr <shelley.vohr@gmail.com>
----------------------------------- Results ------------------------------------
┌────────────────────────┬───────────────────────────┬────────────────────────────┬─────────────────────────┐
│        (index)         │             0             │             1              │            2            │
├────────────────────────┼───────────────────────────┼────────────────────────────┼─────────────────────────┤
│       debian9-64       │ 'express-session-v1.17.1' │ 'yeoman-generator-v4.10.1' │                         │
│     centos7-ppcle      │    'ember-cli-v3.18.0'    │      'multer-v1.4.2'       │ 'torrent-stream-v1.2.0' │
│      rhel7-s390x       │  'torrent-stream-v1.2.0'  │                            │                         │
│   fedora-latest-x64    │    'spawn-wrap-v2.0.0'    │                            │                         │
│     ubuntu1804-64      │                           │                            │                         │
│ fedora-last-latest-x64 │                           │                            │                         │
│     ubuntu1604-64      │                           │                            │                         │
└────────────────────────┴───────────────────────────┴────────────────────────────┴─────────────────────────┘
Sample JSON Output 
[
  {
    "baseBuild": {
      "source": "https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x",
      "upstream": "https://ci.nodejs.org/job/citgm-smoker/2390/"
    },
    "comparisonBuild": {
      "source": "https://github.com/nodejs/node/pull/33811/",
      "upstream": "https://ci.nodejs.org/job/citgm-smoker/2392/"
    },
    "debian9-64": [
      {
        "name": "express-session-v1.17.1",
        "status": "FAILED"
      },
      {
        "name": "yeoman-generator-v4.10.1",
        "status": "FAILED"
      }
    ],
    "centos7-ppcle": [
      {
        "name": "ember-cli-v3.18.0",
        "status": "FAILED"
      },
      {
        "name": "multer-v1.4.2",
        "status": "FAILED"
      },
      {
        "name": "torrent-stream-v1.2.0",
        "status": "FAILED"
      }
    ],
    "rhel7-s390x": [
      {
        "name": "torrent-stream-v1.2.0",
        "status": "FAILED"
      }
    ],
    "fedora-latest-x64": [
      {
        "name": "spawn-wrap-v2.0.0",
        "status": "FAILED"
      }
    ],
    "ubuntu1804-64": [],
    "fedora-last-latest-x64": [],
    "ubuntu1604-64": []
  }
]
Sample Markdown Output 
# CITGM Data for [#2392](https://ci.nodejs.org/job/citgm-smoker/2392/) - [#2390](https://ci.nodejs.org/job/citgm-smoker/2390/)

## New Failures in job [#2390](https://ci.nodejs.org/job/citgm-smoker/2390/)

### debian9-64

* express-session-v1.17.1
* yeoman-generator-v4.10.1

### centos7-ppcle

* ember-cli-v3.18.0
* multer-v1.4.2
* torrent-stream-v1.2.0

### rhel7-s390x

* torrent-stream-v1.2.0

### fedora-latest-x64

* spawn-wrap-v2.0.0

### ubuntu1804-64

None.

### fedora-last-latest-x64

None.

### ubuntu1604-64

None.

cc @richardlau (since i can't request your review)

@codecov
Copy link

codecov bot commented Jul 10, 2020 •
edited
Loading

Codecov Report

Merging #455 into master will decrease coverage by 0.25%.
The diff coverage is 69.56%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #455      +/-   ##
==========================================
- Coverage   76.66%   76.41%   -0.26%     
==========================================
  Files          21       27       +6     
  Lines        1603     1717     +114     
==========================================
+ Hits         1229     1312      +83     
- Misses        374      405      +31
Impacted Files Coverage Δ
lib/ci/build-types/test_build.js 49.52% <49.52%> (ø)
lib/ci/build-types/citgm_comparison_build.js 64.93% <64.93%> (ø)
lib/ci/build-types/citgm_build.js 79.26% <79.26%> (ø)
lib/ci/ci_result_parser.js 42.54% <79.31%> (-9.14%) ⬇️
lib/ci/build-types/job.js 81.63% <81.63%> (ø)
lib/ci/ci_utils.js 100.00% <100.00%> (ø)
lib/ci/jenkins_constants.js 100.00% <100.00%> (ø)
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4b189b...3f4fcb1. Read the comment docs.

@codebytere codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from dd48ae2 to 3cd93b9 Compare July 10, 2020 06:44
@MylesBorins
Copy link
Contributor

MylesBorins commented Jul 15, 2020

Just realized that we might want to compare both citgm + citgm-nobuild jobs. Is that possible with this implementation
?

@codebytere
Copy link
Member Author

codebytere commented Jul 15, 2020 •
edited
Loading

@MylesBorins it does not at present - this pulls citgm-smoker jobs at the API level only so i'd need to extend this by adding the ability to fetch jobs at:

  • https://ci.nodejs.org/job/citgm-smoker-nobuild/${id}/api/json
  • https://ci.nodejs.org/job/citgm-smoker-nobuild/${id}/testReport/api/json

I'll make an issue and add that as a followup TODO, if that sounds good to you!

@codebytere
Copy link
Member Author

codebytere commented Jul 16, 2020 •
edited
Loading

Also - for more effective reviewing, just d22d5e1 is the actual new change introduced in this PR.

richardlau
richardlau reviewed Jul 20, 2020
View reviewed changes
@codebytere codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from 8b99bba to 1fc9c06 Compare July 20, 2020 18:33
@richardlau
Copy link
Member

richardlau commented Jul 20, 2020

Trying to make some sense of this:

-bash-4.2$ npx -p codebytere/node-core-utils#allow-citgm-comparison ncu-ci citgm 2421 2417
npx: installed 349 in 18.908s
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2421
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2421/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.x
Commit     [0bd44b4a88b7] Working on v10.21.1
Date       2020-06-02 18:57:33 +0100
Author     Bethany.Griggs <Bethany.Griggs@uk.ibm.com>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2417/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.22.0-proposal
Commit     [c5215d0b3da0] 2020-07-21, Version 10.22.0 'Dubnium' (LTS)
Date       2020-07-15 11:27:18 -0400
Author     Richard Lau <riclau@uk.ibm.com>
----------------------------------- Results ------------------------------------
┌────────────────────────┬─────────────────────────┬─────────────────────┬───────────────────────────────┬─────────────────────┬─────────────────────┬──────────────────┬──────────────────────┬─────────────────┬──────────────────────┐
│        (index)         │            0            │          1          │               2               │          3          │          4          │        5         │          6           │        7        │          8           │
├────────────────────────┼─────────────────────────┼─────────────────────┼───────────────────────────────┼─────────────────────┼─────────────────────┼──────────────────┼──────────────────────┼─────────────────┼──────────────────────┤
│      aix71-ppc64       │ 'path-to-regexp-v6.1.0' │    'pug-v3.0.0'     │         'uuid-v8.2.0'         │                     │                     │                  │                      │                 │                      │
│        osx1015         │     'acorn-v7.3.1'      │    'ava-v3.10.1'    │        'clinic-v6.0.2'        │ 'ember-cli-v3.19.0' │  'react-v16.13.1'   │ 'semver-v7.3.2'  │ 'underscore-v1.10.2' │                 │                      │
│   fedora-latest-x64    │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│        osx1014         │     'acorn-v7.3.1'      │    'ava-v3.10.1'    │ 'eslint-plugin-jest-v23.18.0' │   'clinic-v6.0.2'   │ 'ember-cli-v3.19.0' │ 'fastify-v3.0.3' │   'react-v16.13.1'   │ 'semver-v7.3.2' │ 'underscore-v1.10.2' │
│ fedora-last-latest-x64 │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1804-64      │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│     centos7-ppcle      │     'clinic-v6.0.2'     │ 'ember-cli-v3.19.0' │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1604-64      │                         │                     │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1404-64      │                         │                     │                               │                     │                     │                  │                      │                 │                      │
│       debian9-64       │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│      rhel7-s390x       │ 'torrent-stream-v1.2.0' │                     │                               │                     │                     │                  │                      │                 │                      │
└────────────────────────┴─────────────────────────┴─────────────────────┴───────────────────────────────┴─────────────────────┴─────────────────────┴──────────────────┴──────────────────────┴─────────────────┴──────────────────────┘
-bash-4.2$

This shows, for example, 'torrent-stream-v1.2.0' for rhel-s390x but that module appears as failing in both citgm runs.
image
image

@codebytere
Copy link
Member Author

codebytere commented Jul 20, 2020

@richardlau spotted the error - fix incoming!

@codebytere
Copy link
Member Author

codebytere commented Jul 21, 2020 •
edited
Loading

@richardlau you should see:

node-core-utils on git:allow-citgm-comparison ❯ ncu-ci citgm 2421 2417
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2421
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2421/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.x
Commit     [0bd44b4a88b7] Working on v10.21.1
Date       2020-06-02 18:57:33 +0100
Author     Bethany.Griggs <Bethany.Griggs@uk.ibm.com>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2417/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.22.0-proposal
Commit     [c5215d0b3da0] 2020-07-21, Version 10.22.0 'Dubnium' (LTS)
Date       2020-07-15 11:27:18 -0400
Author     Richard Lau <riclau@uk.ibm.com>
----------------------------------- Results ------------------------------------



FAILURE: 9 failures in 2417 not present in 2421


┌────────────────────────┬─────────────────────┬──────────────────┐
│        (index)         │          0          │        1         │
├────────────────────────┼─────────────────────┼──────────────────┤
│      aix71-ppc64       │ 'ember-cli-v3.19.0' │                  │
│     ubuntu1804-64      │                     │                  │
│ fedora-last-latest-x64 │ 'spawn-wrap-v2.0.0' │ 'winston-v3.3.3' │
│        osx1014         │  'fastify-v3.0.3'   │                  │
│     ubuntu1404-64      │                     │                  │
│      rhel7-s390x       │ 'uglify-js-v3.10.0' │                  │
│        osx1015         │                     │                  │
│     centos7-ppcle      │ 'ember-cli-v3.19.0' │ 'multer-v1.4.2'  │
│     ubuntu1604-64      │   'clinic-v6.0.2'   │ 'yargs-v15.4.1'  │
│       debian9-64       │                     │                  │
│   fedora-latest-x64    │                     │                  │
└────────────────────────┴─────────────────────┴──────────────────┘

now which makes far more sense!

@codebytere codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from 843b259 to 0dbe375 Compare July 21, 2020 01:11
@richardlau richardlau mentioned this pull request Jul 21, 2020
Merged
4 tasks
@codebytere codebytere force-pushed the allow-citgm-comparison branch from 0ec2b1f to 91f107e Compare July 22, 2020 19:06
@MylesBorins
Copy link
Contributor

MylesBorins commented Jul 23, 2020

At a cursory glance it seems like the --no-build flag is a binary flag for the entire tool, not for independent jobs... so there would not be a way to compare a regular CITGM job to a no-build job, is that accurate?

Could we potentially also allow a small grammar here, rather than a flag... perhaps ${jobType}:${jobNumber}

e.g.

1337 ~ citgm job 1337
citgm:1337 ~ citgm job 1337
nobuild:1337 ~ nobuild job 1337

thoughts?

@codebytere
Copy link
Member Author

codebytere commented Jul 23, 2020

@MylesBorins the way i've written it at present is that for comparison jobs, if --nobuild is passed that would apply to the base job:

https://github.com/nodejs/node-core-utils/pull/455/files#diff-5bdf2b1736f8191ae8dc6098a8f25651R6-R21

so if one passed nci-ci citgm 880 2433 --nobuild that would compare 880, a nobuild job, to 2433, a regular job. However, i do agree that that might be a little confusing 🤔

@MylesBorins
Copy link
Contributor

MylesBorins commented Jul 23, 2020

@codebytere as long as it is documented and possible that seems like a good way to move forward... and we could revisit later if we want to improve the dx

@codebytere codebytere merged commit 336dc7f into nodejs:master Jul 23, 2020
@codebytere codebytere deleted the allow-citgm-comparison branch July 23, 2020 19:19
@codebytere codebytere mentioned this pull request Aug 4, 2020
Merged
@mmarchini mmarchini mentioned this pull request Aug 31, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@richardlau richardlau richardlau approved these changes

@targos targos Awaiting requested review from targos

@BridgeAR BridgeAR Awaiting requested review from BridgeAR

+1 more reviewer

@mmarchini mmarchini mmarchini approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ncu-ci: parse CITGM results

4 participants

@codebytere @MylesBorins @richardlau @mmarchini