The Foundations of Item Generation for Mass Testing

Item Generation for Test Development

Volumn , Issue , 2013, Pages 3-34

  Irvine, Sidney H ^a  

^a BRITISH PSYCHOLOGICAL SOCIETY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85142554566     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9781410602145-6     Document Type: Chapter

References (119)

1. Armstrong, R. D., Jones, D. H., and Wang, Z. (1994). Automated parallel test construction using classical test theory. Journal of EducationalStatistics, 19, 73-90.
2. Baddeley, A. D. (1968). A three-minute reasoning test based on grammatical transformation. Psychonomic Science, 10, 341-342.
3. Baddeley, A. D., and Hitch, G. (1974). Working memory. In G. H. Bower, (Ed.), The Psychology of learning and motivation (Vol. 8, pp. 47-90).
4. Bartram, D. (1987). The development of an automated testing system for pilot selection: The Micropat project. Applied Psychology: International Review, 36, 279-298.
5. Bejar, I. I. (1986a). The psychometrics of mental rotation (RR-86-19). Princeton, NJ: Educational Testing Service.
6. Bejar, I. I. (1986b). Analysis and generation of Hidden Figure items: A cognitive approach to Psychometric Modeling (RR-86-20). Princeton, NJ: Educational Testing Service.
7. Bejar, I. I. (1986c). Final Report: Adaptive testing of spatial abilities (ONR 150 531). Princeton, NJ: Educational Testing Service.
8. Bejar, I., and Yocom, P. (1991). A generative approach to the modelling of isomorphic hidden-figure items. Applied Psychological Measurement, 15(2), 129-137.
9. th British Psychological Society Annual Conference, Swansea, Wales.
10. Bongers, S. H., and Greig, J. E. (1997). An Australian trial of the British Army Recruit Battery - Part 2. Report of The Air Force Office, Mawson ACT 2607, Australia.
11. Bunderson, C. V., Inouye, D. K., and Olsen, J. B. (1988). The four generations of computerised educational measurement. Research Report (RR88-35) Princeton, NJ: Educational Testing Service.
12. Carroll, J. B. (1976). Psychometric tests as cognitive tasks: A new “Structure of Intellect.” In L. B. Resnick (Ed.), The nature of intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates.
13. Carroll, J. B. (1980). Individual difference relations in psychometric and experimental cognitive tasks (Report No. 163) Thurstone Psychometric Laboratory, University of North Carolina, Chapel HiU, NC.
14. Carroll, J. B. (1983). The difficulty of a test and its factor composition revisited. In H. Wainer and S. Messick (Eds.), Principals of modern psychological measurement. Hillsdale, NJ:Lawrence Erlbaum Associates.
15. Carroll, J. B. (1986). Defining abilities through the person characteristic function. In S. E. Newstead, S. H. Irvine, and P. L. Dann (Eds.), Human assessment: Cognition and motivation. Dordrecht, Netherlands: Nijhoff.
16. Carroll, J. B. (1987). New perspectives in the analysis of abilities. In R. R. Ronning, J. A. Glover, J. C. Conoley, and J. C. Witt (Eds.), The influence of cognitive psychology on testing. Hillsdale, NJ: Lawrence Erlbaum Associates.
17. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York:Cambridge University Press.
18. Carroll, J. B., Meade, A., and Johnson, E. S. (1991). Test analysis with the person characteristic function: implications for defining abilities. In R. E. Snow and D. E. Wiley (Eds.), Improving inquiry in education, psychology and social science: a book in honour of Lee J. Cronbach (pp. 109-143). Hillsdale, NJ: Lawrence Erlbaum Associates.
19. Christal, R. E. (1984) New cognitive tests being evaluated by TTCP services. Report to the Technical Cooperation Program Meeting of 1984, USAF Armstrong Laboratory, Brooks AFB, San Antonio, Texas.
20. Christal, R. E. (1987). A factor-analytic study of tests of working memory. Unpublished Report; Human Resources Division, USAF Armstrong Laboratory, Brooks AFB, San Antonio, Texas.
21. Christal, R. E. (1990). Comparative validities of ASVAB and LAMP tests for Logic Gates learning Technical Report, USAF Armstrong Laboratory, Brooks AFB San Antonio, Texas.
22. Clark, H. H. (1969). Linguistic processes in deductive reasoning. Psychological Review, 76, 387-404.
23. Clark, H. H (1970). Comprehending comparatives. In G. B. Flores D’Arcais and W. J. M. Levelt (Eds.), Advances inpsycholinguistics. Amsterdam: North Holland.
24. Clark, H. H., and Chase, W. G. (1972). On the process of comparing sentences against pictures. Cognitive Psychology, 3, 472-517.
25. Collis, J. M., and Irvine, S. H. (1991a). Predictive Validity and Utility of the ABC Battery with Royal Navy Officers under Training. SP(N) Report TR 261, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
26. Collis, J. M., and Irvine, S. H. (1991b). The Plymouth ABC Battery for Artificer Apprentice Entrants. Validity and reliability studies. SP(N) Report TR 265, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
27. Collis, J. M., and Irvine, S. H. (1991c). The Plymouth ABC Battery for EN Non-Technician Ratings under Training. Validity and reliability studies. SP(N) Report TR 266, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
28. Collis, J. M., and Irvine, S. H. (1991d). The Plymouth ABC Battery for WRNS Rating Entrants. Validity and reliability studies. SP(N) Report TR 267, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
29. Collis, J. M., and Irvine, S. H. (1991e). The Plymouth ABC Battery for RN/WRNS Ratings andRM other Ranks under Training. Validity and reliability studies: Summary report. SP(N) Report TR 271, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
30. Collis, J. M., and Irvine, S. H. (1993a). The ABC Combined Battery for Artificer Apprentices. Further validity studies. SP(N) Report TR 311, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
31. Collis, J. M., and Irvine, S. H. (1993b). The ABC Combined Battery for Royal Marine Apprentices. Further validity studies. SP(N) Report TR 312, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
32. Collis, J. M., and Irvine, S. H. (1993c). The ABC Combined Battery for RN/WRNS Non-Technicians. Further validity studies. SP(N) Report TR 313, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
33. Collis, J. M., and Irvine, S. H. (1993d). The effects of pre-knowledge, retest andtypes of test administration on computer generated cognitive tasks in a group of royal navy and royal marine entrants. SP(N) Report TR 314, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
34. Collis, J. M., and Irvine, S. H. (1994). A new generation of ability tests for selection and training The Navy Personnel Series. Technical Report 1:1994, Human Assessment Laboratory, University of Plymouth, Plymouth UK.
35. Collis, J. M., Tapsfield, P. G. C, Irvine, S. H., Dann, P. L., and Wright, D. (1995). The British Army Recruit Battery goes operational: From theory to practice in computer-based testing using item-generation techniques. International Journal of Selection and Assessment, 3, 96-103.
36. Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684.
37. Dann, P. L., and Irvine, S. H. (1986). Handbook of computer-based cognitive tasks. Centre for Computer-Based Assessment, University of Plymouth, Plymouth, UK.
38. Dennis, I. (1993). The development of an item generative test of spatial orientation closely related to Test SP80a. SP(N) Report TR 307, Office of the Senior Psychologist (Naval), Ministry of Defence, London.
39. Dennis, I. (1995). The structure and development of numeracy and literacy tests in the Navy Personnel Series. Technical Report for the Office of the Senior Psychologist (Naval). Human Assessment Laboratory, University of Plymouth, Plymouth UK.
40. th International Military Testing Association Conference, Toronto, Canada.
41. Dennis, I., and Evans, J. St. B. T. (1989). System architecture for computerised assessment. Report for The Army Personnel Research Establishment (Contract 2021/12). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
42. Dennis, I., and Evans, J. St. B. T. (1996). The speed-error trade off problem in psychometric testing. British Journal of Psychology, 87, 105-129.
43. Embretson, S. E. (1995). Working memory capacity versus general control processes in abstract reasoning. Intelligence, 20, 169-189.
44. Embretson, S. E. (1996). Multidimensional latent trait models in measuring fundamental aspects of intelligence. In I. Dennis and P. G. C. Tapsfield (Eds.) Human abilities, their nature and measurement. Mahwah, NJ: Lawrence Erlbaum Associates.
45. Evans, J. St. B. T. (1982). The psychology of deductive reasoning. London: Routledge.
46. Evans, J. St. B. T., and Wright, D. E. (1992) The transitive inference task (Tech. Rep. 2-1992 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
47. Evans, J. St. B. T., and Wright, D.E. (1993) The properties of fixed-time tests: A simulation study, (Tech. Rep. 3-1993 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
48. Furneaux, W. D. (1952). Some speed, error and difficulty relationships within a problemsolving situation. Nature, 170, 3.
49. Goeters, K-M, (1979). Die anderung derpsychometrischen kennewerte und der faktorenstruktur als folge derubung von tests. Unpublished Doctoral Dissertation, University of Hamburg, Hamburg.
50. Goeters, K-M., and Rathje, H. (1992). Computer-generiete parallel-tests für die fahigeitsmessung in der eignungsauswahl von operationellem luftfahrtpersonal. DLR Institut für Flugmedizin Abteilung Luft-und Raumfahrtpsychologie, Hamburg.
51. Greig, J. E., and Bongers, S. H. (1996). An Australian trial of the British Army Recruit Battery. Proceedings of the 38th Annual Conference of the International Military Testing Association. San Antonio, November, 1996.
52. Greig, J. E., and Bongers, S. H. (1997). An Australian trial of the British Army Recruit Battery-Part 3: Validity coefficients and factor structure. Report of The Air Force Office, Mawson; ACT 2607, Australia.
53. Grenzebach, A. P., and McDonald, J. E. (1992). Alphabetic sequence decisions for letter pairs with separations of one to three letters, Journal of Experimental Psychology Learning, Memory, and Cognition; 18, 865-872
54. Groen, G. J., and Parkman, J. M. (1972). A chronometric analysis of simple addition. Psychological Review, 79, 329-343.
55. Harris, R. L., and Tapsfield, P. G. C. (1995). The British Army Recruit Battery Trials of Pre-Test Booklets. Human Assessment Laboratory Technical Report 10-1995, University of Plymouth, Plymouth, UK.
56. Hockey, G.R. J., and Maclean, A. (1986). Direct temporal analysis of individual differences in cognitive skill. In S. E. Newstead, S. H. Irvine and P. L. Dann (Eds.), Human assessment:cognition and motivation. Dordrecht: Nijhoff.
57. Hockey, G. R. J., Maclean, A., and Hamilton, P. (1981). State changes and the temporal patterning of component resources. In J. Long and A. D. Baddeley, (Eds.), Attention and performance, Vol. 9. Hillsdale, NJ: Lawrence Erlbaum Associates.
58. Holroyd, S. R., Atherton, R. M., and Wright, D. E. (1995a October). The criterion related validity of the British Army Recruit Battery. Proceedings of the 37th Annual Conference of the International Military Testing Association, Toronto.
59. Holroyd, S. R., Atherton, R. M., and Wright, D. E. (1995b). Validation of the British Army Recruit Battery against measures of performance in basic military training. Centre for Human Sciences, Report DRAJCHS/HS3/CR95019/1.0. Defence Evaluation and Research Agency, Famborough.
60. Hornke, L. F., and Habon, M. W. (1986). Rule-based item bank construction and evaluation within the linear logistic framework. Applied Psychological Measurement, 10, 369-380.
61. Hough, P. V. C. (1962). Method and means for recognising complex patterns. U.S. Patent 3, 069, 654.
62. Hunt, E., Lunneborg, C, and Lewis, J. (1975). What does it mean to be high verbal? Cognitive Psychology, 7, 194-227.
63. Irvine, C. D., and Irvine, S. H. (1996). Effects of antihypertensive treatment on cognitive function of older patients: Effect is not proved. British Medical Journal, 313, 166.
64. Irvine, S. H. (1998a). The computer-generation of ability tests for adaptive testing in selection and training:A report in the form of a technical handbook. USAF Air Force Laboratory, Brooks AFB, San Antonio, Texas.
65. Irvine, S. H. (1998b). New tests for recruitment: standardisation and validation. Final Report for The Royal Ulster Constabulary. Inpsych Ltd. Berwick-upon-Tweed, UK.
66. Irvine, S. H., and Christal, R. E. (1994). The primacy of working memory in learning to identify electronic logic gates (Tech. Rep. 4-1994). Human Assessment Laboratory, University of Plymouth, Plymouth UK.
67. Irvine, S. H., Dann, P. L., and Anderson, J. D. (1990). Towards a theory of algorithmdetermined cognitive test construction. British Journal of Psychology, 81, 173-195.
68. Irvine, S. H., Dann, P. L., and Evans, J. St.B. T. (1987). Item generative approaches for computerbased testing: A prospectus for research. Report for the Army Personnel Research Establishment. Human Assessment Laboratory, University of Plymouth, Plymouth, Devon UK.
69. Irvine, S. H, Dann, P. L., Evans, J. St. B. T., Dennis, I., Collis, J., Thacker, C, and Anderson, J. D. (1989). Another generation of personnel selection tests: Stages in a new theory of computer-based test construction. Report for the Army Personnel Research Establishment. Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
70. Irvine, S. H., Kutschke, T., and Walker, R. F. (2000). Screening conscripts in Germany Using Item-Generative Tests (Report 01/00). Bundesministerium der Verteidigung: Bonn, Deutschland.
71. Irvine, S. H., and Reuning, H. (1981). Perceptual speed and cognitive controls. Journal of Cross-Cultural Psychology, 12, 425-444.
72. Irvine, S. H., Schoeman, A., and Prinsloo, W. (1988). Putting cognitive theory to the test:group testing reassessed using the cross-cultural method. In G. K. Verma and C. Bagley (Eds.), Cross-cultural studies of personality, attitudes and cognition. London: Macmillan.
73. Jacobs, N. R. (1996). Validation of the British Army Recruit Battery (BARB) against phase 2 military training performance measures. Centre for Hunan Sciences Report PLSD/CHS/fiS3/CR96049/1.0 Defence Evaluation and Research Agency, Famborough, UK.
74. Jacobs, N. R., Cape, L. T., and Lawton, D. H. (1997). Validation of the British Army Recruit Battery (BARB) against phase 2 military training performance measures. Centre for Human Sciences, Report PLSD/CHS/HS3/CR97018/1.0. Defence Evaluation and Research Agency, Famborough, UK.
75. Jensen, A. R. (1982). Reaction time and psychometric g. In H. J. Eysenck (Ed.), A model for intelligence. New York: Springer-Verlag.
76. Jensen, A. R. (1988). Speed of information-processing and population differences. In S. H. Irvine, and J. W. Berry (Eds.), Human abilities in cultural context. New York: Cambridge University Press.
77. Just, M. A., and Carpenter, P. A. (1985). Cognitive co-ordinate systems: Accounts of mental rotation and individual differences in spatial ability. Psychological Review, 92, 137-172.
78. Kirsch, H. (1971). Der Wegfiguren test als auswhlinstrument. Aviation Psychology. (Quoted in Goeters, K.-M. 1979, p. 125.)
79. Kitson, N., and Elshaw, C. C. (1996). A comparison of the British Army Recruit Battery and the RAF Ground Trades Test Battery (Report DRA/CHS/HS3/CR96060/1.0). Defence Research and Evaluation Agency, Famborough, UK.
80. Kornbrot, D. E. (1988). Random walk models of binary choice: The effect of deadlines in the presence of asymmetric payoffs. Acta Psychologica, 69, 109-127.
81. Kornbrot, D. E. (1989). Organisation of keying skills: the effect of motor complexity and number of units. Acta Psychologica, 70, 19-41.
82. Kornbrot, D. E. (1997). Information accrual models of cognitive processes: evidence from the shape of reaction-time distributions. Manuscript submitted for publication, Department of Psychology, University of Hertfordshire, UK.
83. Kyllonen, P. C. (1996). Is working memory capacity Spearman’s g? In I. Dennis and P. G. C. Tapsfield (Eds.), Human abilities, their nature and measurement (pp. 49-74). Mahwah, NJ:Lawrence Erlbaum Associates.
84. Kyllonen, P. C, and Christal, R. E. (1988). Cognitive modelling of learning abilities: A status report of LAMP (AFHRL-TP-87-66). Manpower and Personnel Division, Air Force Human Resources Laboratory, Brooks AFB, Texas.
85. Kyllonen, P. C, and Christal, R. E. (1989). Cognitive modelling of learning abilities. In R. Dillon and J. W. Pellegrino (Eds.) Testing: Theoretical and applied issues (pp. 143-173). San Francisco: Freeman.
86. Kyllonen, P. C, and Christal, R. E. (1990). Reasoning ability is (little more than) working memory capacity?! Intelligence, 14, 389-433.
87. Kyllonen, P. C, and Woltz, D. J. (1988 April). Role of cognitive factors in the acquisition of cognitive skill. Paper delivered at Minnesota Symposium on Learning and Individual Differences:University of Minnesota, Minneapolis.
88. Lohman, D. F. (1994). Component scores as residual variation or why the intercept correlates best. Intelligence; 19, 1-11.
89. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ:Lawrence Erlbaum Associates.
90. Miller, G. A., and McKean, K. E. (1964). A chronometric study of some relations between sentences. Quarterly Journal of Experimental Psychology, 16, 297-308.
91. Mislevy, R. J., and Sheehan, K. M. (1988). The role of collateral information about examinees in item parameter estimation. ETS Research Report (RR-88-55-ONR). Princeton, NJ: Educational Testing Service.
92. Mislevy, R. J., and Wingersky, M. S., Irvine, S. H., and Dann, P. L. (1991). Resolving mixtures of strategies in spatial visualisation tasks. British Journal of Mathematical and Statistical Psychology, 44, 265-288.
93. Moyer, R. S., and Landauer, T. K. (1967). Time required for judgements of numerical inequality. Nature, 215, 1519-1520.
94. Neimark, E.D., and Estes, W. K. (1967). Stimulus sampling theory. San Francisco: Holder.
95. Parkman, J. M. (1972). Temporal aspects of simple multiplication and comparison. Journal of Experimental Psychology, 95, 437-444.
96. Posner, M. L., Boies, S. J., Eichelman, W. H., and Taylor, R. J. (1969). Retention of visual name codes of single letters. Journal oj’Experimental’Psychology Monographs, 79, 1-16.
97. Prince, M. J., Bird, A. S., Blizzard, R. A., and Mann, A. H. (1996). Is the cognitive function of older patients affected by hypertensive treatment? British Medical Journal, 312, 801-808.
98. Restle, F., and Davis, J. H. (1962). Success and speed of problem-solving by individuals and groups. Psychological Review, 69, 520-536.
99. Royer, F. L. (1971). Information processing of visual figures in the Digit Symbol Substitution Task. Journal of Experimental Psychology, 87, 344-342.
100. Shepard, R. N., and Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171, 701-703.
101. Sternberg, R. J. (1977). Intelligence, information processing, and analogical reasoning: The componential analysis of human abilities. Hillsdale, NJ: Lawrence Erlbaum Associates.
102. Sternberg, S. (1966). High speed scanning in human memory. Science, 153, 652-654.
103. Tapsfield, P. G. C. (1993a). The British Army Recruit Battery: Test-retest reliability (Tech. Rep.: 5-1993 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
104. Tapsfield, P. G. C. (1993b). The British Army Recruit Battery. 1993 applicant norms (Tech. Rep.:6-1993 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
105. Tapsfield, P. G. C, and Wright, D. E. (1993). A preliminary analysis of summary data arising from the operational use of the British Army Recruit Battery (Tech. Rep.: 3-1993 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
106. Tatsuoka, K. M., and Tatsuoka, M. M. (1978). Time-score analysis in criterion-referenced tests (CERL Report E-l). Report of the Computer-Based Education Research Laboratory, University of Illinois, Urbana, 111.
107. Thurstone, L. L. (1938). Primary mental abilities. Chicago: University of Chicago Press and latterly as Medway Reprint (1975) Chicago: University of Chicago Press.
108. Vernon, P. A. (1983). Speed of information processing and general intelligence. Intelligence, 7, 53-70.
109. Vernon, P. A., and Jensen, A. R. (1985). Individual and group differences in intelligence and speed of information processing. Personality and Individual Differences, 5, 411-423.
110. Wainer, H., and Messick, S. J. (Eds.), (1983). Principals of modern psychological measurement. Hillsdale, NJ: Lawrence Erlbaum Associates.
111. Weibull, W. (1951). A statistical distribution of wide application. Journal of Applied Mechanics, 18, 293-297.
112. White, P. O. (1982). Some major components in general intelligence. In H. J. Eysenck, (Ed.), A model for intelligence. New York: Springer-Verlag.
113. Woltz, D. J. (1987). Activation and decay of semantic memory: An individual differences investigation of working memory. Manuscript submitted for publication. Manpower and Personnel Division, Air Force Human Resources Laboratory: Brooks AFB, Texas.
114. Wright, D. E. (1990). Item response and theory for item generation: Comment and developments (Tech. Rep. 3-1990 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
115. Wright, D. E. (1992). IRT modelling using latent variable generalised linear models (Tech. Rep.: 3-1992 Army Personnel Research Establishment). Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
116. Wright, D. E., and Dennis, I. (1992). Development of a test of mental cube folding for use in officer selection. (Tech. Rep. for Science Air, 3), Ministry of defence, London. Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
117. Wright, D. E., Irvine, S. H., and Tapsfield, P. G. C. (1992). Test Lengths and Reliabilities. (HAL Tech. Rep.: HAL1-1992) Army Personnel Research Establishment.
118. Wright, D. E. (1993). BARB and the measurement of individual differences: Departing from traditional models. Paper presented at the International Military Association Conference, Williamsburg, Virginia, November 1993 also as Tech. Rep., 2-1994 (Army Personnel Research Establishment) Human Assessment Laboratory, University of Plymouth, Plymouth, UK.
119. Wright, D. E. (1997, October). Exploiting the speed-accuracy trade-off. Paper presented at a symposium on The Future of Learning and Individual Differences, University of Minnesota, Minneapolis.