| Age | Commit message (Collapse) | Author |
|---|
|
This switch allows users to select which dmesg log level is treated as
warning resulting in overriding the test results to
dmesg-fail/dmesg-warn.
Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Petri Latvala <petri.latvala@intel.com>
|
|
To aid testing function parsing metadata.txt is split into outer helper
that operates on dirfd and inner function that operates on FILE*.
This allows us to test the parsing using fmemopen(), limiting the amount
of necessary boilerplate.
Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Petri Latvala <petri.latvala@intel.com>
|
|
Which regexp gets compiled is settings specific, depending whether we
run piglit-style or not.
If it's optimized to be initialized only once and it is a global
variable, it will be "stuck" in the mode we have selected with the first
run, which may break tests.
Let's remove this optimization and initialize it each time, as it takes
less 0.002s on my hardware.
Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Simon Ser <simon.ser@intel.com>
|
|
We use the timeout status for when the runner had to kill a testcase,
which indicates a more sever issue than an operation failing that we
expected to complete within seconds.
Since it's unused, drop it.
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
|
|
When possible, all tests we know we were going to attempt to execute
now appear in the results as "notrun". The only known case where it's
not possible to add an explicit "notrun" is when running in
multiple-mode, because "no subtests" and "run all subtests, we didn't
list them beforehand" are represented the same.
v2: Rebase and adjust to already landed json changes
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Acked-by: Martin Peres <martin.peres@linux.intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
New piglit bumped its results_version to 10, making glxinfo and pals
optional in practice, not just by accident. Unfortunately reading
results with newer piglit attempts to convert the results to version
10, reading glxinfo and pals, and thus fails. In a hilarious summary:
A commit to piglit making glxinfo optional makes it mandatory for us.
v2: json unit tests confirmed to be working...
Reported-by: Andi Shyti <andi.shyti@intel.com>
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Andi Shyti <andi.shyti@intel.com>
Tested-by: Andi Shyti <andi.shyti@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
Characters in kernel logs, when read from /dev/kmsg, are escaped as
\xNN if they are not between 32 and 127 of if they are "\". Decode
what we can when creating results.json.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
Accidentally pushed, believing this was the kill(child, 0) switcheroo.
This reverts commit 0be3f3e7c1613dcaf27267fce778025ea46a36c1.
Acked-by: Petri Latvala <petri.latvala@intel.com>
|
|
I have whinged on for ages about the dmesg-warnings being an expected
part of kernel testing (where else is the kernel meant to log its
errors?) and should be treated the same as our stderr for the test. That
is if a test fails, it fails and does not need to be conflated with
whether or not there was a dmesg warning (just as the test saying why it
failed on stderr does not need flagging), and that a passing test with a
dmesg warning is simply a warn.
The effect is that we simply remove the "dmesg-" flagging from results
names, as the err/dmesg output is simply collated for the error report
already.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Petri Latvala <petri.latvala@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
|
|
Deviating a bit from the piglit command line flag, igt_runner takes an
optional comma-separated list as an argument to
--abort-on-monitored-error for the list of conditions to abort
on. Without a list all possible conditions will be checked.
Two conditions implemented:
- "taint" checks the kernel taint level for TAINT_PAGE, TAINT_DIE and
TAINT_OOPS
- "lockdep" checks the kernel lockdep status
Checking is done after every test binary execution, and if an abort
condition is met, the reason is printed to stderr (unless log level is
quiet) and the runner doesn't execute any further tests. Aborting
between subtests (when running in --multiple-mode) is not done.
v2:
- Remember to fclose
- Taints are unsigned long (Chris)
- Use getline instead of fgets (Chris)
v3:
- Fix brainfart with lockdep
v4:
- Rebase
- Refactor the abort condition checking to pass down strings
- Present the abort result in results.json as a pseudo test result
- Unit tests for the pseudo result
v5:
- Refactors (Chris)
- Don't claim lockdep was triggered if debug_locks is not on
anymore. Just say it's not active.
- Dump lockdep_stats when aborting due to lockdep (Chris)
- Use igt@runner@aborted instead for the pseudo result (Martin)
v6:
- If aborting after a test, generate results.json. Like was already
done for aborting at startup.
- Print the test that would be executed next as well when aborting,
as requested by Tomi.
v7:
- Remove the resolved TODO item from commit message
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
This allows testing to skip the file writing.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
Turns out the same information is looked up in different places in
different code paths. Piglit's summary module looks up total counts in
['totals']['root'], CI looks up ['totals']['']. The latter is
inherited from piglit, so this has probably changed at some point.
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=108486
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Andi Shyti <andi.shyti@intel.com>
Tested-by: Andi Shyti <andi.shyti@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
After setting the result object text, the string retrieved from the
old object is invalid.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
|
|
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
When starting a test run, drop a timestamp file. Do the same when
ending a run. Slap those timestamps directly into the time_elapsed
field in results.json.
Using timestamps instead of measuring actual elapsed time goes against
the naming of the field, but the name is chosen by piglit. Even though
piglit itself uses timestamps.
Corner cases:
On incomplete test runs, the end timestamp will be missing. The
time_elapsed field will only have the start timestamp. This matches
piglit behaviour exactly.
On incomplete but resumed test runs, the end timestamp will be the
time when the resume finishes. Piglit doesn't do this, and instead
leaves the end timestamp missing. Discussing which behaviour is better
is left as an exercise to the readers.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
CI pipeline (namely, cibuglog) doesn't cope well with strings that
have \0 in them. If null characters appear in output files, pretend
the output stops at the first such character. Well behaving tests
should not print them anyway.
The case in CI happened due to some hang/crash/explosion/solar flare
that corrupted the output of a test.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Acked-by: Martin Peres <martin.peres@linux.intel.com>
Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com>
|
|
and move it to job_list.c
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Acked-by: Tomi Sarvela <tomi.p.sarvela@intel.com> #irc
|
|
If a test with subtests just exits immediately, or the test binary
doesn't exist at all (as is sometimes the case with kms_chamelium),
the existence of subtests doesn't end up in the execution journal. As
was done for timeouts in a797cbf6918a ("runner/resultgen: Be more
robust with incomplete tests"), check if we were attempting to run a
subtest before attributing a 'notrun' result to an incorrect field.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
With the flag, dmesg handling is done exactly as piglit does it: Level
5 (info) and higher dmesg lines, if they match a regexp, cause test
result to change to dmesg-*.
The default is false (use new method).
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Acked-by: Martin Peres <martin.peres@linux.intel.com>
|
|
Previously, the total runtime of binary foo with subtests bar and quz
was accumulated to the tests field under 'igt@foo' with just a
TimeAttribute field. This confuses piglit-derived scripts deep down
the CI pipeline, so move the overall binary runtime to a new field
'runtimes', with TimeAttribute fields for 'igt@foo'.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Acked-by: Martin Peres <martin.peres@linux.intel.com>
|
|
The totals field in the results json lists the total amount of
particular test results, both overall and by binary.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
If a test is incomplete and didn't have time to print that it's
entering a subtest, the generated results will think the test binary
does not have subtests. If that case is known, make sure to attribute
blame correctly.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
|
This is a new test runner to replace piglit. Piglit has been very
useful as a test runner, but certain improvements have been very
difficult if possible at all in a generic test running framework.
Important improvements over piglit:
- Faster to launch. Being able to make assumptions about what we're
executing makes it possible to save significant amounts of time. For
example, a testlist file's line "igt@somebinary@somesubtest" already
has all the information we need to construct the correct command
line to execute that particular subtest, instead of listing all
subtests of all test binaries and mapping them to command
lines. Same goes for the regexp filters command line flags -t and
-x; If we use -x somebinaryname, we don't need to list subtests from
somebinaryname, we already know none of them will get executed.
- Logs of incomplete tests. Piglit collects test output to memory and
dumps them to a file when the test is complete. The new runner
writes all output to disk immediately.
- Ability to execute multiple subtests in one binary execution. This
was possible with piglit, but its semantics made it very hard to
implement in practice. For example, having a testlist file not only
selected a subset of tests to run, but also mandated that they be
executed in the same order.
- Flexible timeout support. Instead of mandating a time tests cannot
exceed, the new runner has a timeout on inactivity. Activity is
any output on the test's stdout or stderr, or kernel activity via
/dev/kmsg.
The runner is fairly piglit compatible. The command line is very
similar, with a few additions. IGT_TEST_ROOT environment flag is still
supported, but can also be set via command line (in place of igt.py in
piglit command line).
The results are a set of log files, processed into a piglit-compatible
results.json file (BZ2 compression TODO). There are some new fields in
the json for extra information:
- "igt-version" contains the IGT version line. In
multiple-subtests-mode the version information is only printed once,
so it needs to be duplicated to all subtest results this way.
- "dmesg-warnings" contains the dmesg lines that triggered a
dmesg-warn/dmesg-fail state.
- Runtime information will be different. Piglit takes a timestamp at
the beginning and at the end of execution for runtime. The new
runner uses the subtest output text. The binary execution time will
also be included; The key "igt@somebinary" will have the runtime of
the binary "somebinary", whereas "igt@somebinary@a" etc will have
the runtime of the subtests. Substracting the subtest runtimes from
the binary runtime yields the total time spent doing setup in
igt_fixture blocks.
v2:
- use clock handling from igt_core instead of copypaste
- install results binary
- less magic numbers
- scanf doesn't give empty strings after all
- use designated array initialization with _F_JOURNAL and pals
- add more comments to dump_dmesg
- use signal in kill_child instead of bool
- use more 'usual' return values for execute_entry
- use signal number instead of magic integers
- use IGT_EXIT_INVALID instead of magic 79
- properly remove files in clear_test_result_directory()
- remove magic numbers
- warn if results directory contains extra files
- fix naming in matches_any
- construct command line in a cleaner way in add_subtests()
- clarify error in filtered_job_list
- replace single string fprintfs with fputs
- use getline() more sanely
- refactor string constants to a shared header
- explain non-nul-terminated string handling in resultgen
- saner line parsing
- rename gen_igt_name to generate_piglit_name
- clean up parse_result_string
- explain what we're parsing in resultgen
- explain the runtime accumulation in add_runtime
- refactor result overriding
- stop passing needle sizes to find_line functions
- refactor stdout/stderr parsing
- fix regex whitelist compiling
- add TODO for suppressions.txt
- refactor dmesg parsing
- fill_from_journal returns void
- explain missing result fields with TODO comments
- log_level parsing with typeof
- pass stdout/stderr to usage() instead of a bool
- fix absolute_path overflow
- refactor settings serialization
- remove maybe_strdup function
- refactor job list serialization
- refactor resuming, add new resume binary
- catch mmap failure correctly
v3:
- rename runner to igt_runner, etc
- add meson option for building the runner
- use UPPER_CASE names for string constants
- add TODO comments for future refactoring
- add a midding close()
- const correctness where applicable
- also build with autotools
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
|
