Fix Smoke Test Failures #1692
Merged
Fix Smoke Test Failures #1692
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
LGTM for code -- kicking tests |
Contributor
|
@blueorangutan package |
|
@jburwell a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-29 |
Contributor
|
@blueorangutan test |
|
@jburwell a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests |
Contributor
|
Looks like the blueorangutan build didn't complete properly. I am re-kicking it now. @blueorangutan test matrix |
|
@jburwell a Trillian-Jenkins matrix job (centos6 mgmt + xs56sp1, centos7 mgmt + vmware55u3, ubuntu mgmt + kvmcentos7) has been kicked to run smoke tests |
This was referenced Sep 28, 2016
Member
|
A failure at the backend was detected, I've rekicked the test jobs. We should expect results in next ~7.5 hours. |
|
Trillian test result (tid-64)
|
|
Trillian test result (tid-63)
|
Contributor
|
This PR has 39 test errors/failures. We are investigating the cause of each failure to determine the root cause (environmental, test defect, or CloudStack defect). I will update this PR when we have more information to report. |
|
Trillian test result (tid-68)
|
Member
|
I'm investigating failures on this PR, likely most of them (esp related to oobm) are env related as udp traffic seems to be blocked. There also seems to be an error with the log upload script where file incorrectly gets uploaded resulting in few byte sized zip files (I've fixed it as well). |
This was referenced Sep 28, 2016
|
Trillian test result (tid-76)
|
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-37 |
Member
|
On investigation, it was found that Trillian was falling back on master branch's tests which is why the test failures were high. Some of the feature such as oobm don't exist in 4.8. I've rekicked a new trillian env for testing now. |
|
Trillian test result (tid-78)
|
Contributor
|
@borisstoyanov could you please amend the first line of commit 62f4826 to be |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-41 |
Member
|
Thanks @serg38 so does that mean a +1/LGTM on this one? |
|
Yes +1 LGTM
|
Contributor
|
@karuturi can you provide a test LGTM from CloudMonger? |
Member
|
@karuturi can you help run BVT tests on this PR? |
Member
|
All, we need a LGTM on this one to get the ball moving on release work. Pinging several -- @karuturi @swill @wido @NuxRo @nvazquez @serg38 @GaborApatiNagy @harikrishna-patnala @JayapalUradi @K0zka @rafaelweingartner @terbolous @pyr @syed |
Contributor
|
Sorry guys, am unable to test currently and cannot LGTM code that I obviously don't understand. |
Member
|
@NuxRo np, I was pinging most of the active people on the list who may have the time/motivation to contribute towards next releases. Thanks. |
Contributor
|
@PaulAngus I will test this out on my XenServer7 setup. I would need some help to understand how you are running these. I'll PM you on Slack |
|
We will post results later today for Vmware 5.5/6 tests |
Contributor
|
If the three test failures don't represent regressions, we should probably get this merged per Rohit's suggestion. |
Contributor
|
I'm running the smoke tests on my setup. I will post the results as soon as they have completed. |
Contributor
|
My tests have finished. I had run into some issues where one of the tests got stuck when cleaning everything. I have yet to determine the root cause. I've had 7 test cases that have failed. Here is the list of the tests |
Contributor
|
@PaulAngus were there any tests that were expected to fail? If so I can exclude them and try to debug the ones that were supposed to run correctly. |
Contributor
|
@syed Thanks for testing out. Tests related VPC private gateway, redundant VPC are still failing as noted [1]. I am working on fix. We want to proceed with this PR with those failing tests are known issues. If the rest of the tests are passing can we have LGTM? [1]http://markmail.org/message/n2p2jziywo7kjzx7#query:+page:1+mid:rvv4kqkim2nme35h+state:results |
Contributor
Member
|
@pdion891 thanks, this PR fixes marvin tests. Blueorangutan (bo) is just a bot that facilitates running the marvin tests and sharing the results. It uses Trillian to deploy an environment and run smoke tests using this script: Is that a LGTM/+1? |
Contributor
|
Thanks @murali-reddy for confirming this. Based on that this PR LGTM. Thank you guys for fixing the smoke tests. This is a very needed requirement for getting stable releases 👍 |
|
We finished testing on vmware 5.5 and 6. Overall much better state. We have still failure in execution of the following tests: All of the failure seem to be either environmental or due to incorrect ACL order . We had to make one change that I propose to include. In test_service_offereing.py line 389 switch from So this is another testing LGTM for vmware 5.5 and 6. I will post test results in few min. |
|
Successful tests. RHEL 6 management server, advanced networking, Vmware 5.5/6 hypervisors |
|
Failed tests Exceptions: |
Member
|
@serg38 thanks for sharing the test results. I can see why all the out-of-band management tests may fail, these tests require that the mgmt server is able to send/receive UDP traffic from/to the management server to the server where you're running the marvin tests. In case of any iptables rules dropping UDP packets, most/all the oobm tests will fail as they launch a The other failures related to redundant VRs and VPC are known issues. |
|
@rhtyd We run marvin tests on management server. Do we need any special configuration in test data for out-of-band-managment ? ERROR: Tests out-of-band management issue power soft actionTraceback (most recent call last): ERROR: Tests out-of-band management issue power actionTraceback (most recent call last): |
Member
|
@serg38 okay I see, the issue is that the marvin you've installed is old and does not have the necessary API definitions (cmd, response classes). I recommend doing a clean build of the branch you're testing and then install the generated marvin, something like: |
|
Thanks @rhtyd It didi it. Now two of them still fail but the rest are successful |
Member
|
@serg38 Thanks, both of them are environment related failure and ignore-able. I'm not sure about the first error, most like environment related. The second one could be related to ping timeout (the timeout used to check mgmt server health by others in the cluster), you can reduce them in the global settings, restart mgmt server based on following config (used by Travis): |
Member
|
Thanks @serg38 @syed @mlsorensen @pdion891 for your feedback, comments, review and tests. I think we've enough reviews, tests/lgtms for merging the PRs now. I'll proceed with merging this PR, based subsequently fwd-merge this on 4.9, master branch based on results/discussions/review from PR #1703 #1708. |
asfgit
pushed a commit
that referenced
this pull request
Oct 21, 2016
Fix Smoke Test Failuresfix ping tests to properly recognise successful ping test (on 4.8 branch) * pr/1692: CLOUDSTACK-9529: Cleanup resources after marvin test completes CLOUDSTACK-9533: gateway of public IP is not handled correctly when parsing the cmd_line.json to create ips.json databag CLOUDSTACK-9532: Use macchinina as a template for failing tests CLOUDSTACK-9527: test_01_test_vm_volume_snapshot making test negative again CLOUDSTACK-9531: Try template teardown without failure CLOUDSTACK-9527: Skip tests not supported for hypervisor CLOUDSTACK-9524: Check router hypervisor before ssh to VR CLOUDSTACK-9522: Check for available attribute in marvin response CLOUDSTACK-9526: Marvin test_deploy_vgpu_enabled_vm.py - Fix a hardcoded username and password CLOUDSTACK-9515: internal LB vm is not handled when parsing cmd_line.json, resulting in internal LB vm not come up CLOUDSTACK-9161: move quota test to plugins Marvin Tests: Fix VPC network offering selection fix macchinina template specificied for vmware in Marvin tests fix ping tests to properly recognise successful ping test CLOUDSTACK-9514: Making the credentials of the host to be picked up from the CLOUDSTACK-9511: fix test_privategw_acl.py to handle multiple physical networks Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix ping tests to properly recognise successful ping test (on 4.8 branch)