The A*StarSM Audit Service

The A*StarSM Audit service is a comprehensive review of all classroom groups based on collected test results. Each class is measured according to the normal variation of its peers and a detailed analysis is performed on each irregular group. The audit is processed by sophisticated, patented, computer software. The review procedure and report elements are described below.

Review Procedure

Review of the Test

The Audit review evaluates the test, not for what it measures, but for how well it makes the measurement. In this regard, the review examines the internal consistency and range of measurement; the suitability of the measurement for critical decisions (pass/fail, certification, program assignment, etc.); and the capability of the measurement to support an analysis of the standardization of the test administration.

Review of the Test Administration

The Audit review applies an innovative, patented method of group test item response pattern analysis to evaluate the consistency of the test administration with the standardized procedures established by the test publisher. The review determines normative group response patterns and normal pattern variation. Response patterns of all individual test taker groups are evaluated against the appropriate norm and measured in terms of the normal variation. Exceptional groups are further evaluated for the cause of their deviation from the norm.

Review of Irregular Groups

Test taker groups (by classroom, school, test center, etc.) with unusual test item response patterns are evaluated for the likelihood of inadvertent errors in test administration and potentially improper influence on test taker responses.

Irregular Groups & Subject Groups

The response patterns of irregular groups are highly unlikely to occur in a normal test administration. Most have subset, 'subject groups' that indicate the cause of the irregularity. The members of subject groups are identified and their item responses subjected to a probability analysis. The subject group response pattern is then compared to response patterns known to arise from particular patterns of proctor influence.

The A*StarSM Audit Report

The A*StarSM Audit report is provided as both a written document and as an electronic file with user friendly software to support further search and analysis.

The Written Report

The written report provides a psychometrician's analysis of the test, the test administration, and the significance of any problems found by the review. The report further provides a comprehensive listing of all proctor groups, the individual group Audit scores, and the group standings on the test and several measures of conformity with expected test performance. Individual client options determine areas of further analysis and the depth of the written review.

The Electronic File

The electronic file provides a versatile resource for the client to access the audit information for organizational use. The file includes a comprehensive listing of all test administrations and test taker groups along with A*StarSM measures of reliability and, where warranted, estimation of assessment problems. The client may utilize the electronic file to initiate a review of client selected test taker groups by group identification, by group characteristics or by group test results. Individual test takers may be identified whose scores are unlikely to support critical (i.e. pass/fail) decisions. The client may merge other client file information to permit, for example, a review of new proctors, or other organizational information. With multiple reviews conducted over time, the Report file may provide evaluation of test taker gains, or changes in instruction, recruitment, selection or other organizational programs.

Characteristic Patterns
Pattern
Indication
Higher than expected, yet variable, performance over a limited test section

Often indicates proctor explanation or instruction regarding test content.

Higher than expected performance over the later half of test items, yet consistent with the norm pattern of item difficulty

Often indicates extension of test time limits.

Higher than expected, uniform performance over the later test items

Often indicates adding correct answers following the test administration. The added correct answers may be limited to answers left blank by lower performing students, resulting in no erasures.

Higher performance over easier questions and lower performance over difficult questions

Often indicates students copying answers, achieving the same answers right and the same answers wrong. Sometimes the result of seating students close together. Students may be seated in multiple clusters, resulting in greater variation in the copying and in the irregularity of the test response pattern.

Dramatic deviation from the norm over the length of the test

Indicates pervasive proctor control over students responses.