summaryrefslogtreecommitdiff
path: root/runner
AgeCommit message (Collapse)Author
2018-10-23runner: Add "root" also to the "totals" objectPetri Latvala
Turns out the same information is looked up in different places in different code paths. Piglit's summary module looks up total counts in ['totals']['root'], CI looks up ['totals']['']. The latter is inherited from piglit, so this has probably changed at some point. Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=108486 Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Andi Shyti <andi.shyti@intel.com> Tested-by: Andi Shyti <andi.shyti@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-19runner: Add --overall-timeoutPetri Latvala
With --overall-timeout $foo, the runner will stop executing new tests when $foo seconds have already been used. A resumed run will start over with no time used, using the same timeout. This allows for executing a long list of tests piecemeal, in about $foo length executions. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=106127 Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-15runner: Fix results when there's warns and dmesg-warnsPetri Latvala
After setting the result object text, the string retrieved from the old object is invalid. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2018-10-04runner: Add unit test for resume when enough subtests are already runPetri Latvala
Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-04runner: Don't try to execute an empty set of subtestsPetri Latvala
If we resume a test run with igt_resume, or if resume is done automatically from a test timeout, the runner will execute the last attempted test with the subtest selection set to original set minus the subtests already journaled to have started. If this results in an empty set, we get a harmless but misleading message from the test saying "igt_core-WARNING: Unknown subtest: subtest-name,!subtest-name" If the journal already contains as many subtests as we have requested (when we know the set), assume we have them all already and move to the next job list entry instead. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-03runner: Add a unit test that checks for fd leaksPetri Latvala
Pretty much needed, as proven. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-03runner: Plug more fd leaksPetri Latvala
Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-10-03runner: Add time_elapsed field to resultsPetri Latvala
When starting a test run, drop a timestamp file. Do the same when ending a run. Slap those timestamps directly into the time_elapsed field in results.json. Using timestamps instead of measuring actual elapsed time goes against the naming of the field, but the name is chosen by piglit. Even though piglit itself uses timestamps. Corner cases: On incomplete test runs, the end timestamp will be missing. The time_elapsed field will only have the start timestamp. This matches piglit behaviour exactly. On incomplete but resumed test runs, the end timestamp will be the time when the resume finishes. Piglit doesn't do this, and instead leaves the end timestamp missing. Discussing which behaviour is better is left as an exercise to the readers. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-09-28runner: Plug an fd leakPetri Latvala
Make sure comparefd gets closed in dump_dmesg(). Otherwise we run out of descriptors after a bit over 1000 tests executed... Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-09-27runner: Avoid null characters in results.jsonPetri Latvala
CI pipeline (namely, cibuglog) doesn't cope well with strings that have \0 in them. If null characters appear in output files, pretend the output stops at the first such character. Well behaving tests should not print them anyway. The case in CI happened due to some hang/crash/explosion/solar flare that corrupted the output of a test. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Acked-by: Martin Peres <martin.peres@linux.intel.com> Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com>
2018-09-21runner: Support -t and -x with a --test-listPetri Latvala
Actually implement what was already commented to work. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com> #irc
2018-09-21runner: Lowercase the name used for matching -t and -xPetri Latvala
And thus make it possible to run -t basic-s3 for example. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com> #irc
2018-09-21runner: Make generate_piglit_name non-staticPetri Latvala
and move it to job_list.c Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com> #irc
2018-09-19runner: Flush igt_runner outputs before forkingPetri Latvala
If the output of igt_runner is piped or redirected, buffered prints could be left lingering and read as test executable output if execv() fails. This can happen easily if CI for example generates a testlist with an incorrect binary name, or an optional test binary (say, kms_chamelium) is not built for the deployment in question. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-09-17runner: Increase buffer size for reading outputsPetri Latvala
The law of chosen magic numbers: The number selected is wrong. Chose another magic number for the size of the buffer used to read test outputs and kernel log records. It's now 2048, up from 256. Also added a warning print if that's still not enough for kernel logs. The lesson to learn here is that the /dev/kmsg interface does not give you a truncated log record as initially thought, but reports an undocumented EINVAL instead. Subsequent reads give the next record, so the failsafe added will make sure any future EINVALs will only drop the record that is too long instead of everything from that point onwards. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-09-12runner: Seek to the end to get to the end of kmsgPetri Latvala
When draining the rest of kmsg records, read the compare record from the end of kmsg or you get incomplete dmesg fields in the results. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-09-12runner/resultgen: More checking if subtests should existPetri Latvala
If a test with subtests just exits immediately, or the test binary doesn't exist at all (as is sometimes the case with kms_chamelium), the existence of subtests doesn't end up in the execution journal. As was done for timeouts in a797cbf6918a ("runner/resultgen: Be more robust with incomplete tests"), check if we were attempting to run a subtest before attributing a 'notrun' result to an incorrect field. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-08-31runner: Introduce --piglit-style-dmesgPetri Latvala
With the flag, dmesg handling is done exactly as piglit does it: Level 5 (info) and higher dmesg lines, if they match a regexp, cause test result to change to dmesg-*. The default is false (use new method). Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Acked-by: Martin Peres <martin.peres@linux.intel.com>
2018-08-31runner/resultgen: Add a new field to carry overall runtimesPetri Latvala
Previously, the total runtime of binary foo with subtests bar and quz was accumulated to the tests field under 'igt@foo' with just a TimeAttribute field. This confuses piglit-derived scripts deep down the CI pipeline, so move the overall binary runtime to a new field 'runtimes', with TimeAttribute fields for 'igt@foo'. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Martin Peres <martin.peres@linux.intel.com> Acked-by: Martin Peres <martin.peres@linux.intel.com>
2018-08-17runner/resultgen: Implement the totals fieldPetri Latvala
The totals field in the results json lists the total amount of particular test results, both overall and by binary. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-08-17runner/resultgen: Be more robust with incomplete testsPetri Latvala
If a test is incomplete and didn't have time to print that it's entering a subtest, the generated results will think the test binary does not have subtests. If that case is known, make sure to attribute blame correctly. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-08-16runner: Match the piglit name with -t and -x optionsPetri Latvala
Instead of just matching the binary/subtest name. Originally not implemented to get the runner landed faster. Turned out to be simple enough. Signed-off-by: Petri Latvala <petri.latvala@intel.com> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-08-13runner/meson: Add support for use_rpath optionArkadiusz Hiler
Cc: Petri Latvala <petri.latvala@intel.com> Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Petri Latvala <petri.latvala@intel.com>
2018-08-13runner: Fix path handlingArkadiusz Hiler
absolute_path() tends to return NULL if more than the last element of the path is nonexistent. That behavior is confusing the callers, which use NULL as a convention for something not being set at all. Let's fix that by sprinkling a little bit of recursion onto absolute_path() and wrapping POSIX in some sanity. Cc: Petri Latvala <petri.latvala@intel.com> Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Reviewed-by: Petri Latvala <petri.latvala@intel.com>
2018-08-09runner: Unit tests for the runnerPetri Latvala
TODO: Unit tests for the results.json file contents. v2: - Avoid writing the nul character to mock files - Properly set up tmpdirs v3: - Restore the resume-related changes that were lost in rebase v4: - Better teardown for temporary directories - Build with autotools Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
2018-08-09runner: New test runnerPetri Latvala
This is a new test runner to replace piglit. Piglit has been very useful as a test runner, but certain improvements have been very difficult if possible at all in a generic test running framework. Important improvements over piglit: - Faster to launch. Being able to make assumptions about what we're executing makes it possible to save significant amounts of time. For example, a testlist file's line "igt@somebinary@somesubtest" already has all the information we need to construct the correct command line to execute that particular subtest, instead of listing all subtests of all test binaries and mapping them to command lines. Same goes for the regexp filters command line flags -t and -x; If we use -x somebinaryname, we don't need to list subtests from somebinaryname, we already know none of them will get executed. - Logs of incomplete tests. Piglit collects test output to memory and dumps them to a file when the test is complete. The new runner writes all output to disk immediately. - Ability to execute multiple subtests in one binary execution. This was possible with piglit, but its semantics made it very hard to implement in practice. For example, having a testlist file not only selected a subset of tests to run, but also mandated that they be executed in the same order. - Flexible timeout support. Instead of mandating a time tests cannot exceed, the new runner has a timeout on inactivity. Activity is any output on the test's stdout or stderr, or kernel activity via /dev/kmsg. The runner is fairly piglit compatible. The command line is very similar, with a few additions. IGT_TEST_ROOT environment flag is still supported, but can also be set via command line (in place of igt.py in piglit command line). The results are a set of log files, processed into a piglit-compatible results.json file (BZ2 compression TODO). There are some new fields in the json for extra information: - "igt-version" contains the IGT version line. In multiple-subtests-mode the version information is only printed once, so it needs to be duplicated to all subtest results this way. - "dmesg-warnings" contains the dmesg lines that triggered a dmesg-warn/dmesg-fail state. - Runtime information will be different. Piglit takes a timestamp at the beginning and at the end of execution for runtime. The new runner uses the subtest output text. The binary execution time will also be included; The key "igt@somebinary" will have the runtime of the binary "somebinary", whereas "igt@somebinary@a" etc will have the runtime of the subtests. Substracting the subtest runtimes from the binary runtime yields the total time spent doing setup in igt_fixture blocks. v2: - use clock handling from igt_core instead of copypaste - install results binary - less magic numbers - scanf doesn't give empty strings after all - use designated array initialization with _F_JOURNAL and pals - add more comments to dump_dmesg - use signal in kill_child instead of bool - use more 'usual' return values for execute_entry - use signal number instead of magic integers - use IGT_EXIT_INVALID instead of magic 79 - properly remove files in clear_test_result_directory() - remove magic numbers - warn if results directory contains extra files - fix naming in matches_any - construct command line in a cleaner way in add_subtests() - clarify error in filtered_job_list - replace single string fprintfs with fputs - use getline() more sanely - refactor string constants to a shared header - explain non-nul-terminated string handling in resultgen - saner line parsing - rename gen_igt_name to generate_piglit_name - clean up parse_result_string - explain what we're parsing in resultgen - explain the runtime accumulation in add_runtime - refactor result overriding - stop passing needle sizes to find_line functions - refactor stdout/stderr parsing - fix regex whitelist compiling - add TODO for suppressions.txt - refactor dmesg parsing - fill_from_journal returns void - explain missing result fields with TODO comments - log_level parsing with typeof - pass stdout/stderr to usage() instead of a bool - fix absolute_path overflow - refactor settings serialization - remove maybe_strdup function - refactor job list serialization - refactor resuming, add new resume binary - catch mmap failure correctly v3: - rename runner to igt_runner, etc - add meson option for building the runner - use UPPER_CASE names for string constants - add TODO comments for future refactoring - add a midding close() - const correctness where applicable - also build with autotools Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>