There are a number of benefits tied to the upstream / third-party checkasm
version, including:
- Improved long-term maintainability, code reuse with other projects, etc.
- Vastly improved overall performance / runtime for benchmarking, due
primarily to the ability to scale the runtime of each test to that test's
complexity.
- Much more robust statistical analysis of benchmarking results; including
robust outlier rejection, an estimation of the histogram, and the ability
to report the variance / stddev in addition to the (trimmed) mean.
- Interactive HTML and JSON output formats in addition to CSV/TSV.
- More readable and user-friendly output across the board, especially for
failures and data dumps (e.g. also showing errors inside padding bytes).
- Better cross-platform support, including dynamic fallback of timer
implementations on ARM platforms, a better RISC-V harness, and more.
There are multiple approaches to how we can solve the problem of integrating
this third party checkasm into dav1d, but I think the hybrid approach of
loading it as an external dependency, falling back to a meson wrap file,
provides the best overall compromise. This avoids the messiness of git e.g.
git submodules, while still allowing us to pin individual tags.
In 16 bpc, the pixels are 16 bit integers, but valid pixels only
are up to 12 bits, and the scaling buffer only contains 4096
elements.
The src pixels are, normally, supposed to be valid pixels, but when
processing blocks of 32 pixels at a time, it can operate on
uninitialized pixels past the right edge.
Before: Cortex A53 A72 A73 Apple M1
fgy_32x32xn_16bpc_neon: 10372.5 8194.4 8612.1 24.2
After:
fgy_32x32xn_16bpc_neon: 10837.9 8469.5 8885.1 24.6
When benchmarking, the functions are called with a fixed width
of 64x32 or 32x16, while the test itself is run with a random size
in the range up to 128x32.
In 16 bpc mode, the source pixels must be within the valid range,
because they otherwise cause accesses out of bounds in the scaling
array.
Niklas Haas