Mercurial > vim

annotate src/testdir/screendump.vim @ 19687:c3f506e24eab v8.2.0400

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
patch 8.2.0400: not all tests using a terminal are in the list of flaky tests Commit: https://github.com/vim/vim/commit/3cdcb090a664e182078f7cc0ca3e4e5a94bbb784 Author: Bram Moolenaar <Bram@vim.org> Date: Wed Mar 18 19:18:10 2020 +0100 patch 8.2.0400: not all tests using a terminal are in the list of flaky tests Problem: Not all tests using a terminal are in the list of flaky tests. Solution: Introduce the test_is_flaky flag.
author Bram Moolenaar <Bram@vim.org>
date Wed, 18 Mar 2020 19:30:04 +0100
parents 9606c0adc148
children 60b5abfc4897
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
1 " Functions shared by tests making screen dumps.
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
2
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
3 " Only load this script once.
17164
7927cf327396 patch 8.1.1581: shared functions for testing are disorganised
Bram Moolenaar <Bram@vim.org>
parents: 17113
diff changeset
4 if exists('*VerifyScreenDump')
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
5 finish
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
6 endif
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
7
17164
7927cf327396 patch 8.1.1581: shared functions for testing are disorganised
Bram Moolenaar <Bram@vim.org>
parents: 17113
diff changeset
8 source shared.vim
7927cf327396 patch 8.1.1581: shared functions for testing are disorganised
Bram Moolenaar <Bram@vim.org>
parents: 17113
diff changeset
9 source term_util.vim
14019
dc67449d648c patch 8.1.0027: difficult to make a plugin that feeds a line to a job
Christian Brabandt <cb@256bit.org>
parents: 13845
diff changeset
10
dc67449d648c patch 8.1.0027: difficult to make a plugin that feeds a line to a job
Christian Brabandt <cb@256bit.org>
parents: 13845
diff changeset
11 " Skip the rest if there is no terminal feature at all.
dc67449d648c patch 8.1.0027: difficult to make a plugin that feeds a line to a job
Christian Brabandt <cb@256bit.org>
parents: 13845
diff changeset
12 if !has('terminal')
13329
424321d6eea7 patch 8.0.1539: no test for the popup menu positioning
Christian Brabandt <cb@256bit.org>
parents: 13322
diff changeset
13 finish
424321d6eea7 patch 8.0.1539: no test for the popup menu positioning
Christian Brabandt <cb@256bit.org>
parents: 13322
diff changeset
14 endif
424321d6eea7 patch 8.0.1539: no test for the popup menu positioning
Christian Brabandt <cb@256bit.org>
parents: 13322
diff changeset
15
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
16 " Verify that Vim running in terminal buffer "buf" matches the screen dump.
13329
424321d6eea7 patch 8.0.1539: no test for the popup menu positioning
Christian Brabandt <cb@256bit.org>
parents: 13322
diff changeset
17 " "options" is passed to term_dumpwrite().
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
18 " The file name used is "dumps/{filename}.dump".
14693
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
19 " Optionally an extra argument can be passed which is prepended to the error
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
20 " message. Use this when using the same dump file with different options.
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
21 " Will wait for up to a second for the screen dump to match.
14693
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
22 " Returns non-zero when verification fails.
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
23 func VerifyScreenDump(buf, filename, options, ...)
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
24 let reference = 'dumps/' . a:filename . '.dump'
16150
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
25 let testfile = 'failed/' . a:filename . '.dump'
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
26
19687
c3f506e24eab patch 8.2.0400: not all tests using a terminal are in the list of flaky tests
Bram Moolenaar <Bram@vim.org>
parents: 17918
diff changeset
27 " Starting a terminal to make a screendump is always considered flaky.
c3f506e24eab patch 8.2.0400: not all tests using a terminal are in the list of flaky tests
Bram Moolenaar <Bram@vim.org>
parents: 17918
diff changeset
28 let test_is_flaky = 1
c3f506e24eab patch 8.2.0400: not all tests using a terminal are in the list of flaky tests
Bram Moolenaar <Bram@vim.org>
parents: 17918
diff changeset
29
17918
9606c0adc148 patch 8.1.1955: tests contain typos
Bram Moolenaar <Bram@vim.org>
parents: 17164
diff changeset
30 " Redraw to execute the code that updates the screen. Otherwise we get the
16326
1679b9f54946 patch 8.1.1168: not all screen update code of terminal is executed in tests
Bram Moolenaar <Bram@vim.org>
parents: 16314
diff changeset
31 " text and attributes only from the internal buffer.
1679b9f54946 patch 8.1.1168: not all screen update code of terminal is executed in tests
Bram Moolenaar <Bram@vim.org>
parents: 16314
diff changeset
32 redraw
1679b9f54946 patch 8.1.1168: not all screen update code of terminal is executed in tests
Bram Moolenaar <Bram@vim.org>
parents: 16314
diff changeset
33
16150
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
34 let did_mkdir = 0
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
35 if !isdirectory('failed')
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
36 let did_mkdir = 1
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
37 call mkdir('failed')
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
38 endif
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
39
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
40 let i = 0
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
41 while 1
15103
9339601e7a31 patch 8.1.0562: parsing of 'diffopt' is slightly wrong
Bram Moolenaar <Bram@vim.org>
parents: 14693
diff changeset
42 " leave some time for updating the original window
9339601e7a31 patch 8.1.0562: parsing of 'diffopt' is slightly wrong
Bram Moolenaar <Bram@vim.org>
parents: 14693
diff changeset
43 sleep 10m
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
44 call delete(testfile)
13329
424321d6eea7 patch 8.0.1539: no test for the popup menu positioning
Christian Brabandt <cb@256bit.org>
parents: 13322
diff changeset
45 call term_dumpwrite(a:buf, testfile, a:options)
15916
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
46 let testdump = readfile(testfile)
16249
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
47 if filereadable(reference)
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
48 let refdump = readfile(reference)
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
49 else
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
50 " Must be a new screendump, always fail
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
51 let refdump = []
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
52 endif
15916
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
53 if refdump == testdump
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
54 call delete(testfile)
16150
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
55 if did_mkdir
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
56 call delete('failed', 'd')
b23048205589 patch 8.1.1080: when a screendump test fails, moving the file is a hassle
Bram Moolenaar <Bram@vim.org>
parents: 15916
diff changeset
57 endif
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
58 break
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
59 endif
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
60 if i == 100
16249
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
61 " Leave the failed dump around for inspection.
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
62 if filereadable(reference)
17113
18ec0b92f431 patch 8.1.1556: command for failing screenshot does not include directory
Bram Moolenaar <Bram@vim.org>
parents: 16477
diff changeset
63 let msg = 'See dump file difference: call term_dumpdiff("testdir/' .. testfile .. '", "testdir/' .. reference .. '")'
16249
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
64 if a:0 == 1
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
65 let msg = a:1 . ': ' . msg
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
66 endif
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
67 if len(testdump) != len(refdump)
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
68 let msg = msg . '; line count is ' . len(testdump) . ' instead of ' . len(refdump)
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
69 endif
67e43b01497f patch 8.1.1129: when making a new screendump test have to create the file
Bram Moolenaar <Bram@vim.org>
parents: 16150
diff changeset
70 else
17113
18ec0b92f431 patch 8.1.1556: command for failing screenshot does not include directory
Bram Moolenaar <Bram@vim.org>
parents: 16477
diff changeset
71 let msg = 'See new dump file: call term_dumpload("testdir/' .. testfile .. '")'
15916
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
72 endif
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
73 for i in range(len(refdump))
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
74 if i >= len(testdump)
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
75 break
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
76 endif
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
77 if testdump[i] != refdump[i]
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
78 let msg = msg . '; difference in line ' . (i + 1) . ': "' . testdump[i] . '"'
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
79 endif
95648dc0124f patch 8.1.0964: cannot see in CI why a screenshot test failed
Bram Moolenaar <Bram@vim.org>
parents: 15653
diff changeset
80 endfor
14693
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
81 call assert_report(msg)
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
82 return 1
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
83 endif
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
84 let i += 1
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
85 endwhile
14693
156a7c5328a7 patch 8.1.0359: no clue what test failed when using a screendump twice
Christian Brabandt <cb@256bit.org>
parents: 14031
diff changeset
86 return 0
13304
013c44d9dc09 patch 8.0.1526: no test using a screen dump yet
Christian Brabandt <cb@256bit.org>
parents:
diff changeset
87 endfunc