GitHub started installing "Chrome for Testing" builds on Windows and
Mac. Unfortunately, these builds don't have support for H264 and AAC
codecs, which we need in our tests.
This fixes the issue by explicitly installing an official build of
Chrome on Windows, to overwrite the "Chrome for Testing" build.
This caches Babel's transpiler output for reuse, and should speed up all
test runs in theory, but the effect is most noticeable on local test
runs.
This uses a fork of karma-babel-preprocessor, which contains
https://github.com/babel/karma-babel-preprocessor/pull/77. If/when that
PR is merged, we can move back to the upstream module.
Local runs will start faster because only modified source files will be
re-processed through Babel when the tests start up.
In the Selenium workflow, Babel output and node_modules will both be
computed by the singular build-shaka job, stored, and then reused by all
the Selenium lab matrix jobs.
On my workstation (3.3 GHz cores, 32GB RAM, spinning platter disk), I
see tests start about ~60 seconds faster. In the lab (2.1-4.7 GHz cores,
64GB RAM, solid-state disk), I see tests start about ~10 seconds faster.
As I was building packages for shaka-lab services in the shaka-project/shaka-lab repo, it occurred to me that this KARMA_PORT variable was too specific to the shaka-player project. Instead, the new shaka-lab package for self-hosted runners will have a more generic ALLOCATED_PORT variable. This updates the CI workflow to use that new name.
When we first deployed certificates from letsencrypt, we did not know
about the "live" folder, which contains helpful symlinks to the latest
certificate. Instead, we were copying the entire "archive" folder, which
contains even old certs, then searching that folder for the latest
version. We should be using the "live" folder instead, letting certbot
maintain the symlinks. This means there is no need for
--lets-encrypt-folder in test.py.
- In karma.conf.js, give ourselves the ability to merge Selenium configs together without clobbering argument lists, for better deduplication in shaka-lab.yaml. This is based on https://github.com/shaka-project/karma-local-wd-launcher/pull/50
- Refactor shaka-lab.yaml to take advantage of this new ability. This removes a bunch of duplicate command line arguments for various versions of Chrome.
- Remove a TODO about a bug that Chrome decided they would never fix.
- Sort the list of devices and platforms, with comments to mark out the sections.
- Remove IE11, which we are well past supporting or testing.
This should be cherry-picked to all active branches.
This is a much simplified version of the appspot job from the release
workflow, but with control given to the human caller instead of computed
from release details. It can be used to correct deployment issues.
Due to limits in the number of versions you can have deployed at once on
App Engine, we can no longer keep every version of the Shaka Player Demo
deployed. So the version index needs to be aware of which versions are
deployed, and only link to versions that are still available.
This makes the index generator aware of App Engine deployments.
The initial commit was untested, but having the workflow in the main
branch allowed me to iterate on the workflow and test it in a branch.
The following changes were made:
- Standardize how workflow inputs are read (using inputs, which works
for all triggers, instead of github.event.inputs, which only works for
some triggers)
- Improve the description of test_filter in selenium-lab-tests.
- Add missing fields in the workflow_call trigger of
selenium-lab-tests.yaml. (Unfortunately, the two triggers need to have
the inputs defined separately, and with slightly different fields.)
- Log the chosen ref in selenium-lab-tests, to help debug inputs.
- Fix variable expansion in selenium-lab-tests so that test_filter can
contain spaces.
- Fix ignore_test_status, which did not work at all and broken the
nightly tests last night.
- Fix dependencies between jobs in update-screenshots.
- Do not set status for the subordinate selenium-lab-tests job if
started from update-screenshots.
- Always upload screenshots from selenium-lab-tests if started from
update-screenshots (normally only uploaded on failure).
- Set the PR status from upload-screenshots.
- Fix unpacking of screenshots from artifacts in upload-screenshots.
- Fix updating of changed screenshots.
- Fix handling of a lack of screenshot changes.
- Split up the jobs for easier debugging in upload-screenshots.
Still to do:
- Fix setting of final PR status
- Use a personal access token so that tests can be triggered from the PR
push
When the VTT size setting is used, the horizontal positioning was wrong
in both native and UI display.
The native display is wrong on Chrome and Edge because of a layout bug
in Chrome, where the shadow DOM for the cue box has conflicting
(redundant) styles. For example, these VTT settings:
`line: 85% position: 50% size: 63%`
result in these styles in the shadow DOM:
`top: 85%; left: 18.5%; transform: translate(-18.5%, -85%)`.
The `translate` style is what breaks the positioning. Unfortunately,
there is no way to fix that in JavaScript.
The UI display, however, was buggy for different reasons and is fixable.
The styles `left: 0; top: 0;` were applied by default, and then `top:
85%;` and `width: 63%;` were set based on the cue settings. The default
of `left: 0` was what broke the positioning. Removing this leaves `left`
set implicitly to `auto`, which is correct.
No other test cases were broken (or fixed) by this change.
This also adds a filter parameter to the lab workflow to run a subset of
tests for quicker results. This is useful for updating screenshots.
Closes b/259121343
This will upload screenshots when the Selenium lab tests fail. This can
be used to get screenshot changes either for examination or for updating
the test expectations.
There was a potential type error in an edge case in lib/player.js. It
would have been caught by the latest compiler, so this upgrades the
compiler and fixes another type error in one other place, as well.
This removes the overly-complex docker-compose system and rewrites the
Dockerfile. It also changes the docker commands to those that specify a
user ID, so that build outputs are written as the current user.
Closes#4387 (failed docker build not detected in CI)
Issue #4619 (failed to build w/ docker)
Because our release PRs are created with the default GitHub token, they
do not trigger test runs. This is a feature of GitHub that prevents
recursive loops of workflows.
This change makes it so that marking a PR "ready for review" will
trigger the tests to run again. This gives us an easy way to trigger
test runs on a release PR, by marking it a draft, then marking it
"ready" again.
We were misreading the computed ref for tests (missing "outputs" in the
variable name), which led us to check out and test main instead of the
PR's head. This also led us to set status on the latest commit in main,
instead of on the PR.
Once the variable was read correctly and the correct ref was checked
out, I found that the ref argument to set-commit-status was actually not
usable, since "git rev-parse refs/pull/4528/head" would fail. But now
that we are checking out the correct commit, the ref argument turns out
to be unnecessary. Whatever we checked out is what we should report
status on.
Player integration tests use the compiled bundle by default. This
prevents us from getting coverage data in these tests, since the
instrumentation was added to the uncompiled library only.
By collecting coverage data in uncompiled mode, we can fix this.
Since Safari 16 became the default on macos-11 images in GitHub, we are
updating our homedir installation of Safari 14 to Safari 15 instead.
This also renames that entry in the matrix from Safari-14 to Safari-old,
in recognition of the fact that we will probably have to continue
updating this.
This workflow will trigger after the nightly Selenium tests. If those tests failed, this "deflake" workflow will automatically re-run the failed jobs up to 3 times (total, including the original scheduled run).
This should help cut down on noise and help highlight actual platform-specific failures. The flakiness of our tests and infrastructure will be tracked separately and addressed at a somewhat lower priority than test greenness.
This requires multiple self-hosted runners to be set up, but will
allow us to more easily read results and to re-run individual failed
jobs per browser.
This change also includes several optimizations that were motivated by
the split:
- stop installing the GitHub CLI (now installed on newer self-hosted
images, saves ~20 seconds)
- cache node_modules (saves about ~1 minute if package-lock.json
hasn't changed)
- pre-build Shaka once in a GitHub VM instead of in parallel on
multiple runner containers hosted on the same machine (saves ~45
minutes w/ 12 runners in parallel)
The initial implementation had a bug where a class declaration
"statement" would be seen as covering all the lines of all the methods
in that class. This change fixes that case, so that the lines of
methods and inner statements are not "part" of their outer containers.
This generates a more detailed JSON coverage file containing maps of
exactly what statements were executed by tests. This also extracts
the data from this file and updates the PR with a comment showing how
much of the changed code is covered by tests.
This will attach a code coverage report to PRs and to nightly tests,
both specifically from Chrome on Linux. Reports will be attached to
workflow runs whether or not the tests pass.
The available package versions of Safari have changed. This updates
to what is currently available. This also updates to expect the
current format of the packages (pkg inside zip).
The v3.2.7 and v3.3.5 releases went out without updating the version
number in lib/player.js. This fixes the workflow to match the
release-please branch name, and changes the workflow to fail if the
branch name changes again in the future.
Having both v4.0.0 and v4.0.0-main tags refer to the same commit
confuses some of our build scripts because git describe can produce
either tag. So we should only create -main tags when the release is
from a branch.
This changes the default behavior of the test runner. Now tests that
require access to license servers will run by default. To disable DRM
tests (for example, if there is no connection to the outside
internet), use --no-drm.
These tags should be attached to the head of the main branch, so that
is what we should check out during the tagging job.
In v3.1.7, v3.2.5, and v3.3.3, we accidentally tagged the heads of the
release branches. The `-main` tags for those releases had to be
corrected manually.
If another instance of the build-and-test workflow is started for the
same PR, cancel the old one. If a PR is updated and a new test run is
started, the old test run will be cancelled automatically to conserve
resources.
Includes a YAML config file for our lab, and documentation on how to
create a custom config for another Selenium grid.
A workflow will run tests nightly in the Shaka lab, using a
self-hosted runner with access to our private grid.
The workflow can also be triggered manually by maintainers to test a
PR in the lab. This will report status back to the PR.
Joey Parrish
Álvaro Velad Galván
Shaka Bot