1. Summary
  2. Files
  3. Support
  4. Report Spam
  5. Create account
  6. Log in

Proposed Test Suite Design: test_suite_spec.lyx

File test_suite_spec.lyx, 14.2 KB (added by borutr, 9 months ago)

Proposed Test Suite Design (LyX)

Line 
1#LyX 1.6.4 created this file. For more info see http://www.lyx.org/
2\lyxformat 345
3\begin_document
4\begin_header
5\textclass article
6\begin_preamble
7\pdfoptionpdfminorversion=3
8\usepackage[
9  pdftitle={Proposed Test Suite Design},
10  pdfauthor={Michael Hope},
11  pdfkeywords={c case compiler framework GPL regression SDCC suite test},
12  colorlinks=true,
13  linkcolor=blue] {hyperref}
14\end_preamble
15\use_default_options false
16\language english
17\inputencoding auto
18\font_roman times
19\font_sans default
20\font_typewriter default
21\font_default_family default
22\font_sc false
23\font_osf false
24\font_sf_scale 100
25\font_tt_scale 100
26
27\graphics default
28\paperfontsize default
29\spacing single
30\use_hyperref false
31\papersize default
32\use_geometry false
33\use_amsmath 1
34\use_esint 0
35\cite_engine basic
36\use_bibtopic false
37\paperorientation portrait
38\secnumdepth 3
39\tocdepth 3
40\paragraph_separation indent
41\defskip medskip
42\quotes_language english
43\papercolumns 1
44\papersides 1
45\paperpagestyle default
46\tracking_changes false
47\output_changes false
48\author ""
49\author ""
50\end_header
51
52\begin_body
53
54\begin_layout Title
55Proposed Test Suite Design
56\begin_inset ERT
57status open
58
59\begin_layout Plain Layout
60
61
62\backslash
63date{2001-07-13}
64\end_layout
65
66\end_inset
67
68
69\end_layout
70
71\begin_layout Author
72Michael Hope (michaelh @ juju.net.nz)
73\end_layout
74
75\begin_layout Abstract
76This article describes the goals, requirements, and suggested specification
77 for a test suite for the output of the Small Device C Compiler (sdcc).
78 Also included is a short list of existing works.
79 
80\end_layout
81
82\begin_layout Section
83Goals
84\end_layout
85
86\begin_layout Standard
87The main goals of a test suite for sdcc are
88\end_layout
89
90\begin_layout Enumerate
91To allow developers to run regression tests to check that core changes do
92 not break any of the many ports.
93 
94\end_layout
95
96\begin_layout Enumerate
97To verify the core.
98 
99\end_layout
100
101\begin_layout Enumerate
102To allow developers to verify individual ports.
103 
104\end_layout
105
106\begin_layout Enumerate
107To allow developers to test port changes.
108 
109\end_layout
110
111\begin_layout Standard
112This design only covers the generated code.
113 It does not cover a test/unit test framework for the sdcc application itself,
114 which may be useful.
115\end_layout
116
117\begin_layout Standard
118One side effect of (1) is that it requires that the individual ports pass
119 the tests originally.
120 This may be too hard.
121 See the section on Exceptions below.
122\end_layout
123
124\begin_layout Section
125Requirements
126\end_layout
127
128\begin_layout Subsection
129Coverage
130\end_layout
131
132\begin_layout Standard
133The suite is intended to cover language features only.
134 Hardware specific libraries are explicitly not covered.
135\end_layout
136
137\begin_layout Subsection
138Permutations
139\end_layout
140
141\begin_layout Standard
142The ports often generate different code for handling different types (Byte,
143 Word, DWord, and the signed forms).
144 Meta information could be used to permute the different test cases across
145 the different types.
146\end_layout
147
148\begin_layout Subsection
149Exceptions
150\end_layout
151
152\begin_layout Standard
153The different ports are all at different levels of development.
154 Test cases must be able to be disabled on a per port basis.
155 Permutations also must be able to be disabled on a port level for unsupported
156 cases.
157 Disabling, as opposed to enabling, on a per port basis seems more maintainable.
158\end_layout
159
160\begin_layout Subsection
161Running
162\end_layout
163
164\begin_layout Standard
165The tests must be able to run unaided.
166 The test suite must run on all platforms that sdcc runs on.
167 A good minimum may be a subset of Unix command set and common tools, provided
168 by default on a Unix host and provided through cygwin on a Windows host.
169\end_layout
170
171\begin_layout Standard
172The tests suits should be able to be sub-divided, so that the failing or
173 interesting tests may be run separately.
174\end_layout
175
176\begin_layout Subsection
177Artifacts
178\end_layout
179
180\begin_layout Standard
181The test code within the test cases should not generate artifacts.
182 An artifact occurs when the test code itself interferes with the test and
183 generates an erroneous result.
184\end_layout
185
186\begin_layout Subsection
187Emulators
188\end_layout
189
190\begin_layout Standard
191sdcc is a cross compiling compiler.
192 As such, an emulator is needed for each port to run the tests.
193\end_layout
194
195\begin_layout Section
196Existing works
197\end_layout
198
199\begin_layout Subsection
200DejaGnu
201\end_layout
202
203\begin_layout Standard
204DejaGnu is a toolkit written in Expect designed to test an interactive program.
205 It provides a way of specifying an interface to the program, and given
206 that interface a way of stimulating the program and interpreting the results.
207 It was originally written by Cygnus Solutions for running against development
208 boards.
209 I believe the gcc test suite is written against DejaGnu, perhaps partly
210 to test the Cygnus ports of gcc on target systems.
211\end_layout
212
213\begin_layout Subsection
214gcc test suite
215\end_layout
216
217\begin_layout Standard
218I don't know much about the gcc test suite.
219 It was recently removed from the gcc distribution due to issues with copyright
220 ownership.
221 The code I saw from older distributions seemed more concerned with esoteric
222 features of the language.
223\end_layout
224
225\begin_layout Subsection
226xUnit
227\end_layout
228
229\begin_layout Standard
230The xUnit family, in particular JUnit, is a library of in test assertions,
231 test wrappers, and test suite wrappers designed mainly for unit testing.
232 PENDING: More.
233\end_layout
234
235\begin_layout Subsection
236CoreLinux++ Assertion framework
237\end_layout
238
239\begin_layout Standard
240While not a test suite system, the assertion framework is an interesting
241 model for the types of assertions that could be used.
242 They include pre-condition, post-condition, invariants, conditional assertions,
243 unconditional assertions, and methods for checking conditions.
244\end_layout
245
246\begin_layout Section
247Specification
248\end_layout
249
250\begin_layout Standard
251This specification borrows from the JUnit style of unit testing and the
252 CoreLinux++ style of assertions.
253 The emphasis is on maintainability and ease of writing the test cases.
254\end_layout
255
256\begin_layout Subsection
257Terms
258\end_layout
259
260\begin_layout Standard
261PENDING: Align these terms with the rest of the world.
262\end_layout
263
264\begin_layout Itemize
265An
266\emph on
267assertion
268\emph default
269 is a statement of how things should be.
270 PENDING: Better description, an example.
271 
272\end_layout
273
274\begin_layout Itemize
275A
276\emph on
277test point
278\emph default
279 is the smallest unit of a test suite, and consists of a single assertion
280 that passes if the test passes.
281 
282\end_layout
283
284\begin_layout Itemize
285A
286\emph on
287test case
288\emph default
289 is a set of test points that test a certain feature.
290 
291\end_layout
292
293\begin_layout Itemize
294A
295\emph on
296test suite
297\emph default
298 is a set of test cases that test a certain set of features.
299 
300\end_layout
301
302\begin_layout Subsection
303Test cases
304\end_layout
305
306\begin_layout Standard
307Test cases shall be contained in their own C file, along with the meta data
308 on the test.
309 Test cases shall be contained within functions whose names start with 'test'
310 and which are descriptive of the test case.
311 Any function that starts with 'test' will be automatically run in the test
312 suite.
313\end_layout
314
315\begin_layout Standard
316To make the automatic code generation easier, the C code shall have this
317 format
318\end_layout
319
320\begin_layout Itemize
321Test functions shall start with 'test' to allow automatic detection.
322 
323\end_layout
324
325\begin_layout Itemize
326Test functions shall follow the K&R intention style for ease of detection.
327 i.e.
328 the function name shall start in the left column on a new line below the
329 return specification.
330 
331\end_layout
332
333\begin_layout Subsection
334Assertions
335\end_layout
336
337\begin_layout Standard
338All assertions shall log the line number, function name, and test case file
339 when they fail.
340 Most assertions can have a more descriptive message attached to them.
341 Assertions will be implemented through macros to get at the line information.
342 This may cause trouble with artifacts.
343\end_layout
344
345\begin_layout Standard
346The following definitions use C++ style default arguments where optional
347 messages may be inserted.
348 All assertions use double opening and closing brackets in the macros to
349 allow them to be compiled out without any side effects.
350 While this is not required for a test suite, they are there in case any
351 of this code is incorporated into the main product.
352\end_layout
353
354\begin_layout Standard
355Borrowing from JUnit, the assertions shall include
356\end_layout
357
358\begin_layout Itemize
359FAIL((String msg =
360\begin_inset Quotes eld
361\end_inset
362
363Failed
364\begin_inset Quotes erd
365\end_inset
366
367)).
368 Used when execution should not get here.
369 
370\end_layout
371
372\begin_layout Itemize
373ASSERT((Boolean cond, String msg =
374\begin_inset Quotes eld
375\end_inset
376
377Assertion failed
378\begin_inset Quotes erd
379\end_inset
380
381).
382 Fails if cond is false.
383 Parent to REQUIRE and ENSURE.
384 
385\end_layout
386
387\begin_layout Standard
388JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
389, and assertSame.
390\end_layout
391
392\begin_layout Standard
393CoreLinux++ includes the extra assertions
394\end_layout
395
396\begin_layout Itemize
397REQUIRE((Boolean cond, String msg =
398\begin_inset Quotes eld
399\end_inset
400
401Precondition failed
402\begin_inset Quotes erd
403\end_inset
404
405).
406 Checks preconditions.
407 
408\end_layout
409
410\begin_layout Itemize
411ENSURE((Boolean cond, String msg =
412\begin_inset Quotes eld
413\end_inset
414
415Postcondition failed
416\begin_inset Quotes erd
417\end_inset
418
419).
420 Checks post conditions.
421 
422\end_layout
423
424\begin_layout Itemize
425CHECK((Boolean cond, String msg =
426\begin_inset Quotes eld
427\end_inset
428
429Check failed
430\begin_inset Quotes erd
431\end_inset
432
433)).
434 Used to call a function and to check that the return value is as expected.
435 i.e.
436 CHECK((fread(in, buf, 10) != -1)).
437 Very similar to ASSERT, but the function still gets called in a release
438 build.
439 
440\end_layout
441
442\begin_layout Itemize
443FORALL and EXISTS.
444 Used to check conditions within part of the code.
445 For example, can be used to check that a list is still sorted inside each
446 loop of a sort routine.
447 
448\end_layout
449
450\begin_layout Standard
451All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
452\end_layout
453
454\begin_layout Subsection
455Meta data
456\end_layout
457
458\begin_layout Standard
459PENDING: It's not really meta data.
460\end_layout
461
462\begin_layout Standard
463Meta data includes permutation information, exception information, and permutati
464on exceptions.
465\end_layout
466
467\begin_layout Standard
468Meta data shall be global to the file.
469 Meta data names consist of the lower case alphanumerics.
470 Test case specific meta data (fields) shall be stored in a comment block
471 at the start of the file.
472 This is only due to style.
473\end_layout
474
475\begin_layout Standard
476A field definition shall consist of
477\end_layout
478
479\begin_layout Itemize
480The field name
481\end_layout
482
483\begin_layout Itemize
484A colon.
485 
486\end_layout
487
488\begin_layout Itemize
489A comma separated list of values.
490 
491\end_layout
492
493\begin_layout Standard
494The values shall be stripped of leading and trailing white space.
495\end_layout
496
497\begin_layout Standard
498Permutation exceptions are by port only.
499 Exceptions to a field are specified by a modified field definition.
500 An exception definition consists of
501\end_layout
502
503\begin_layout Itemize
504The field name.
505 
506\end_layout
507
508\begin_layout Itemize
509An opening square bracket.
510 
511\end_layout
512
513\begin_layout Itemize
514A comma separated list of ports the exception applies for.
515 
516\end_layout
517
518\begin_layout Itemize
519A closing square bracket.
520 
521\end_layout
522
523\begin_layout Itemize
524A colon.
525 
526\end_layout
527
528\begin_layout Itemize
529The values to use for this field for these ports.
530 
531\end_layout
532
533\begin_layout Standard
534An instance of the test case shall be generated for each permutation of
535 the test case specific meta data fields.
536\end_layout
537
538\begin_layout Standard
539The runtime meta fields are
540\end_layout
541
542\begin_layout Itemize
543port - The port this test is running on.
544 
545\end_layout
546
547\begin_layout Itemize
548testcase - The name of this test case.
549 
550\end_layout
551
552\begin_layout Itemize
553function - The name of the current function.
554 
555\end_layout
556
557\begin_layout Standard
558Most of the runtime fields are not very usable.
559 They are there for completeness.
560\end_layout
561
562\begin_layout Standard
563Meta fields may be accessed inside the test case by enclosing them in curly
564 brackets.
565 The curly brackets will be interpreted anywhere inside the test case, including
566 inside quoted strings.
567 Field names that are not recognised will be passed through including the
568 brackets.
569 Note that it is therefore impossible to use some strings within the test
570 case.
571\end_layout
572
573\begin_layout Standard
574Test case function names should include the permuted fields in the name
575 to reduce name collisions.
576\end_layout
577
578\begin_layout Subsection
579An example
580\end_layout
581
582\begin_layout Standard
583I don't know how to do pre-formatted text in LaTeX.
584 Sigh.
585\end_layout
586
587\begin_layout Standard
588The following code generates a simple increment test for all combinations
589 of the storage classes and all combinations of the data sizes.
590 This is a bad example as the optimiser will often remove most of this code.
591\begin_inset Newline newline
592\end_inset
593
594
595\end_layout
596
597\begin_layout Verse
598
599\family typewriter
600/** Test for increment.
601 
602\begin_inset Newline newline
603\end_inset
604
605
606\begin_inset space ~
607\end_inset
608
609
610\begin_inset space ~
611\end_inset
612
613type: char, int, long
614\begin_inset Newline newline
615\end_inset
616
617
618\begin_inset space ~
619\end_inset
620
621
622\begin_inset space ~
623\end_inset
624
625Z80 port does not fully support longs (4 byte)
626\begin_inset Newline newline
627\end_inset
628
629
630\begin_inset space ~
631\end_inset
632
633
634\begin_inset space ~
635\end_inset
636
637type[z80]: char, int
638\begin_inset Newline newline
639\end_inset
640
641
642\begin_inset space ~
643\end_inset
644
645
646\begin_inset space ~
647\end_inset
648
649class:
650\begin_inset Quotes eld
651\end_inset
652
653
654\begin_inset Quotes erd
655\end_inset
656
657, register, static */
658\begin_inset Newline newline
659\end_inset
660
661 
662\begin_inset Newline newline
663\end_inset
664
665static void
666\begin_inset Newline newline
667\end_inset
668
669testInc{class}{types}(void)
670\begin_inset Newline newline
671\end_inset
672
673{
674\begin_inset Newline newline
675\end_inset
676
677
678\begin_inset space ~
679\end_inset
680
681
682\begin_inset space ~
683\end_inset
684
685{class} {type} i = 0;
686\begin_inset Newline newline
687\end_inset
688
689
690\begin_inset space ~
691\end_inset
692
693
694\begin_inset space ~
695\end_inset
696
697i = i + 1;
698\begin_inset Newline newline
699\end_inset
700
701
702\begin_inset space ~
703\end_inset
704
705
706\begin_inset space ~
707\end_inset
708
709ASSERT((i == 1));
710\begin_inset Newline newline
711\end_inset
712
713}
714\end_layout
715
716\end_body
717\end_document