Codecov Report

❗ No coverage uploaded for pull request base (master@905e9a9). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #4195   +/-   ##
=========================================
  Coverage          ?   95.14%           
=========================================
  Files             ?      262           
  Lines             ?    23169           
  Branches          ?     1674           
=========================================
  Hits              ?    22043           
  Misses            ?      907           
  Partials          ?      219

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 905e9a9...2acee52. Read the comment docs.

I got the same results in Jenkins that I did locally:

        if browser_profile:
>           capabilities["moz:firefoxOptions"]["profile"] = browser_profile.encoded
E           KeyError: 'moz:firefoxOptions'

Bedrock appears to be using Selenium 3 - I'll see if there are configuration changes needed for our environment.

I am wrong. There is an PR for updating to Selenium 3 (mozilla/bedrock#4472) but it isn't merged, and appears stale. I'll try to get this to a test case for an upstream issue.

@jwhitlock Selenium Python client 3.4.1 was released today, which should solve your issue. https://pypi.python.org/pypi/selenium/3.4.1

Still fails with selenium 3.4.1.

Opened SeleniumHQ/selenium#3927

The upstream bug is marked as fixed, but is not integrated in a new release, so we're still blocked. I'm watching the hub and node-firefox Docker images, and will test with the next release.

There's a new release, and I've upgraded the packages and docker images. However, it is still not working. It appears there is a problem with settings w3c_capabilities, tracked in SeleniumHQ/selenium#3808.

Selenium team believes the issue is fixed in master, waiting for Selenium 3.5 release to test again.

Selenium 3.5 is released, with a new Python selenium library and pytest plugin. The tracking issue SeleniumHQ/selenium#3808 is closed. However, there are no 3.5.x images for hub or node-firefox yet.

I've rebased and installed today's libraries, and the tests immediately fail in check_response:

>       raise exception_class(message, screen, stacktrace)
E       WebDriverException: Message: None
E       Stacktrace:
E           at java.util.HashMap.putMapEntries (HashMap.java:500)
E           at java.util.HashMap.putAll (HashMap.java:784)
E           at org.openqa.selenium.remote.DesiredCapabilities.<init> (DesiredCapabilities.java:55)
...

Full output in https://gist.github.com/jwhitlock/167dc3ed2d0f8f654a9fe3f05cc4c924.

New 3.5.0 images are released for hub or node-firefox. I've rebased on master and updated the script to use the new tags. I'm also preparing the script to be used in the future, because I think it will be useful.

The nodes no longer fail with an exception, but instead run the tests. Progress! Unfortunately, our functional tests are broken (bug 1394480), so I can't see if they pass yet. Working on that next.

Sorry, I'm having trouble getting that branch installed in my Docker image. Here's the current state:

With pytest-dev/pytest-selenium#137

• Local run, Chrome 61, chromedriver 2.32 - Good but flaky tests
  • Run 1: 212 passed, 1 failed (tests/functional/test_search.py::test_search_zero_results on wait_for_page_to_load)
  • Run 2: 212 passed, 1 failed (tests/functional/test_article.py::test_header_feedback_submenu on show_feedback_submenu)
• Local run, Firefox 56.0.2, geckodriver 0.19.1 👍
  • Run 1: 213 passed
  • Run 2: 213 passed

With pytest-selenium 1.11.1 (current release):

• Local run, Chrome 62, chromedriver 2.33: 👍
  • Run 1: 213 passed
• Local run, Firefox 56.0.2, geckdriver 0.19.1 - Good but flaky tests, server crying
  • Run 1: 211 passed, 2 failed (tests/functional/test_dashboard.py::test_dashboard, is_first_details_diff_displayed, and tests/headless/test_redirects.py::test_legacy_urls[url5], 504 status code)
  • Run 2: tests/headless/test_redirects.py::test_legacy_urls[url15], 504 status code)

I think the differences are due to server weather rather than pytest-dev/pytest-selenium#136.

With scripts/run_selenium_hub.sh (just functional tests):

• Chrome
  • Run 1: 37 pass
  • Run 2: 37 pass
• Firefox
  • Run 1: 31 pass, 6 fail
  • Run 2: 31 pass, 6 fail
  • Failures:
    1. tests/functional/test_article.py::test_header_tech_submenu, page.header.show_tech_submenu()
    2. tests/functional/test_article.py::test_header_feedback_submenu,
       page.header.show_feedback_submenu()
    3. tests/functional/test_dashboard.py::test_dashboard, page.is_first_details_diff_displayed,
    4. tests/functional/test_home.py::test_header_tech_submenu, page.header.show_tech_submenu()
    5. tests/functional/test_report.py::test_report_content, page.header.show_feedback_submenu()
    6. tests/functional/test_report.py::test_report_bug, page.header.show_feedback_submenu()

In Jenkins (just Firefox): 30 pass, same 6 failed tests as scripts/run_selenium_hub.sh

I'm considering:

1. Update dependencies and test in Jenkins on current Chrome 62 (current version), rather than Firefox 42 (oldest supported by Selenium 2.x)
2. Support Chrome or Firefox for local tests
3. Continue testing Firefox in Jenkins with new Selenium releases

@jwhitlock pytest-selenium 1.11.2 is available now, which should help with the "Unable to find a matching set of capabilities" issue.

The problem went away with me with Selenium 3.7.1 Docker images. I've updated to pytest-selenium 1.11.2, and I'm having some trouble with Chrome in Jenkins. It may just be server weather, or maybe I need to adjust those tests. There's lots of moving parts, and none of them are particularly reliable.

First bit of progress in a while. I updated to the 3.8.1 Docker images, and had the same results as 3.7.1. I tried some interactive testing in pdb, and the --html output, and then guessed that there was an issue with move_to_element when running Firefox remotely, to emulate a mouse hover.

This was the hover code:

            hover = ActionChains(self.selenium).move_to_element(submenu_trigger)
            hover.perform()

If I first move the mouse to just left of the submenu, and then onto the submenu, I can get the hover action to occur:

            hover = (ActionChains(self.selenium)
                     .move_to_element_with_offset(submenu_trigger, -5, 0)
                     .move_to_element(submenu_trigger))
            hover.perform()

My guess is that, for the remote Firefox, there's no current mouse position. The first move teleports the cursor to the element, and doesn't give the browser a chance to register it as a hover. Moving to a outside point, and then onto the element, allows Firefox to register the element getting focus and activates the hover. This trick isn't needed when running against Chrome or local Firefox, just when running against dockerized Firefox with the Remote driver.

I'll continue tomorrow and see if I can get the remaining tests working locally with similar tricks, and then take next steps toward working in Jenkins and a mergeable PR.

I've gotten the integration tests running successfully with Selenium Grid (hub + chrome node + firefox node). I've opened a few PRs:

• PR bug 1402950: Run tests more reliably with Firefox and Remote driver in Selenium 3.x #4592 - Update the functional tests either 1) work with Remote Firefox or 2) xfail for known issues
• PR bug 1402950: Add Chrome to integration tests #4593 - Switch integration tests to run Chrome, Firefox, and Headless in parallel, with many options to avoid failures from flakiness

I'll rebase when / if those are merged, and clean up. That would leave:

• The tool updates (selenium 3 + pytest packages)
• The scripts/run_selenium_docker.sh script, to run tests against dockerized stand-alone browsers. I find this allows me to work on tests while doing other things with my computer, so I may keep it. This version is also better for interactive debugging.
• The scripts/run_selenium_hub.sh script, to run tests with dockerized hub / node browsers. This is a closer match to what runs in Jenkins, and can reproduce other errors. It may be more productive to use this rather than "commit and pray" development for Selenium errors.

I think I will try to get the scripts into a seperate PR as well, so that this becomes just the tool update.

I merged the two scripts scripts/run_selenium_docker.sh and scripts/run_selenium_hub.sh into one horrible bash script that can also run against the development environment (almost). I've put that work into PR #4600.

I rearranged this PR to build on top of the other PRs. I'll wait for those to be merged, and then I should be able to reduce this to a one-commit PR.

This pull request is ready! 🎉

The functional test test_dashboard failed in Firefox when it ran the full test procedure, but when it was broken up into a series of tests, it passed locally and in Jenkins. 🤷‍♂️

There's an extra commit to allow the tests to run in Jenkins, and if I remove it, it will delete the test run in Jenkins. I can remove the commit before merging, or I can delete Jenkinsfiles/upgrade_pytest_1308322.yml in a new PR.