The 16-bit kernel is dispatched for every non-8-bit pixel format
(9/10/12/16-bit content, all stored in uint16_t). It's supposed to
undo the Q16 scaling that set_filter_param() applies to `amount`:
fp->amount = amount * 65536.0;
but the shift written in the kernel is `>> (8+nbits)`, which for the
nbits=16 instantiation of the macro comes out to `>> 24` instead of
`>> 16`. Because of this, on any non-8-bit input, unsharp applies ~1/256
of the user's requested strength and is effectively a no-op. The
8-bit kernel (nbits=8) happens to be correct because 8+8 == 16.
This commit also widens the intermediate product to int64 before the
shift, to avoid a potential overflow. Take a 16-bit pixel at the
edge of a sharp white/black region, with the user-facing `amount`
set to its declared maximum of 5.0.
*srx = 65535
blur = 32768
diff = *srx - blur = 32767
amount_q16 = 5.0 * 65536 = 327680
Then the kernel computes:
product = diff * amount_q16
= 32767 * 327680 = 10,737,090,560 (~1.07e10)
which overflows INT32_MAX. Widening to int64 keeps the
multiplication in range; the subsequent `>> 16` brings it back to
sample range and the final cast to int32 is then safe. The widening
is a semantic no-op for 8/9/10/12-bit content where the product
always fits in int32 (worst case at 12-bit: 4095 * 327680 ~ 1.34e9).
Introduced by ee792ebe08 (2019-11-08, "avfilter/vf_unsharp: add 10bit
support"). The fate-filter-unsharp-yuv420p10 reference added in the
same series was generated from the broken kernel and is regenerated
here. fate-filter-unsharp (8-bit) is unaffected.
Repro:
python3 -c "import numpy as np; y=np.tile(np.where(np.arange(128)//8 & 1, 512, 256).astype('<u2'), (128,1)); c=np.full((64,64), 512, '<u2'); open('in.yuv','wb').write(y.tobytes()+c.tobytes()*2)"
ffmpeg -f rawvideo -pix_fmt yuv420p10le -s 128x128 -i in.yuv \
-lavfi "split=2[a][b];[b]unsharp=la=1[bs];[a][bs]psnr" \
-f null - 2>&1 | grep PSNR
Before: `PSNR y:66.50 ...` -- the filter is effectively a no-op,
so the sharpened output matches the input almost exactly.
After: `PSNR y:28.27 ...` -- the filter actually sharpens, so
output and input differ as expected.
Signed-off-by: Nil Fons Miret <nilf@netflix.com>
Made-with: Cursor
Nil Fons Miret
e294b390a0