patch 9.1.0681: tests: Analyzing failed screendumps is hard Commit: https://github.com/vim/vim/commit/6bff6a2fa449b9149eb2db4851e0b62a173b8748 Author: Aliaksei Budavei <0x000c70@gmail.com> Date: Mon Aug 19 21:33:26 2024 +0200 patch 9.1.0681: tests: Analyzing failed screendumps is hard Problem: tests: Analyzing failed screendumps is hard Solution: Facilitate the viewing of rendered screendumps under src/ add some documentation on how to use the viewdumps.vim script (Aliaksei Budavei) With the submitted "viewdumps.vim" script, a few manual steps in typical workflows (see below) can be automated. The updated "README.txt" contains additional information. ============================================================ Reviewing LOCAL failed screendump tests can be arranged as follows: 1) Run tests and generate screendumps: ------------------------------------------------------------ cd /path/to/fork/src/testdir make ------------------------------------------------------------ 2) Examine the screendumps from the "failed" directory: ------------------------------------------------------------ ../vim -u NONE -S viewdumps.vim ------------------------------------------------------------ ============================================================ Reviewing UPLOADED failed screendump tests can be arranged as follows (it can be further locally scripted): 1) Fetch an artifact with failed screendumps from "github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID". 2) Extract the archived files: ------------------------------------------------------------ unzip /tmp/failed-tests.zip -d /tmp ------------------------------------------------------------ 3) Set up the "dumps" directory. Create a symlink to "/path/to/fork/dirs/dumps" in the extracted directories so that term_dumpdiff() can be used. (The lookup algorithm resolves "dumps" for every loaded filename. So, with "/tmp/src/testdir/failed/*.dump" files passed as script arguments, the algorithm will make the files in "/tmp/src/testdir/dumps" queried.) ------------------------------------------------------------ cd /path/to/fork ln -s $(pwd)/src/testdir/dumps /tmp/src/testdir/dumps ------------------------------------------------------------ 4) Examine the extracted screendumps: ------------------------------------------------------------ ./src/vim -u NONE -S src/testdir/viewdumps.vim \ /tmp/src/testdir/failed/*.dump ------------------------------------------------------------ 5) Clean up: ------------------------------------------------------------ unlink /tmp/src/testdir/dumps rm -rf /tmp/src ------------------------------------------------------------ ============================================================ Reviewing SUBMITTED FOR PULL REQUEST screendump tests can be arranged as follows (it can be further locally scripted): 1) List the fetched changeset and write the changed "dumps" filenames to "/tmp/filelist": ------------------------------------------------------------ cd /path/to/fork git switch prs/1234 git diff-index --relative=src/testdir/dumps/ \ --name-only prs/1234~1 > /tmp/filelist ------------------------------------------------------------ 2) Reconcile relative filepaths, and copy next-to-be-updated "dumps" files in the "failed" directory (note the missing new screendumps, if any): ------------------------------------------------------------ git switch master cd src/testdir/dumps test -d ../failed || mkdir ../failed cp -t ../failed $(cat /tmp/filelist) ------------------------------------------------------------ 3) Remember about the introduced INVERTED relation between "dumps" and "failed", i.e. the files to be committed are in "dumps" already and their previous versions are in "failed"; therefore, copy the missing new screendumps from "dumps" to "failed" (otherwise these won't be shown): ------------------------------------------------------------ git switch prs/1234 cp -t ../failed foo_10.dump foo_11.dump foo_12.dump ------------------------------------------------------------ 4) Examine the screendumps from the "failed" directory (new screendumps will be shown with no difference between their versions): ------------------------------------------------------------ cd .. ../vim -u NONE -S viewdumps.vim ------------------------------------------------------------ closes: #15515 Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com> Signed-off-by: Christian Brabandt <cb@256bit.org>