Sixth in a series on NCCCO's Test Integrity
January 2012 - The National Commission for the Certification of Crane Operators (NCCCO) recently held two workshops designed to ensure NCCCO Practical Exam Auditors continue to conduct audits of Practical Examiners in a consistent and accurate manner.
Nineteen Auditors participated in classroom and field training at the workshop held in September at H&E Equipment’s permanent test facility in Salt Lake City, UT, while nine more attended similar sessions in November at Crane Operators Certification and Inspection (COCI) in Stanley, NC. NCCCO's Manager of Test Integrity and a staff investigator also attended the Salt Lake City sessions.
“NCCCO audits ensure the quality, integrity, and fairness of CCO practical examinations,” said Audit Program Manager, Phillip Kinser, who conducted the workshops. “The psychometric validity of NCCCO’s practical tests depends on its certification exams being administered with the highest degree of standardization and uniformity.”
NCCCO conducts audits of Practical Test Sites and Practical Examiners in accordance with the accreditation requirements of the National Commission for Certifying Agencies (NCCA) and the American National Standards Institute (ANSI). These mandatory annual Auditor updates are also required to maintain NCCCO programs’ accreditation to international standards, as defined by ISO 17024 General Requirements for Bodies Operating Certification Systems of Persons.
Example of an issue covered during Auditor training: The pole bases at the end of a corridor have been placed (incorrectly) on the plywood where the circle is painted rather on the ground as stated in the test site set-up procedure. As a result, tennis balls are prone to being knocked off the poles when the test weight lands on the plywood causing the candidate to lose points inappropriately. The oversized plywood is also obscuring the string line that runs under the pole bases, an error that could also negatively impact the scoring process.
In addition to participating in the workshops, Auditors must administer at least ten Practical Exams and/or audit ten or more Examiners every year. Auditors must also continue to abide by NCCCO certification requirements and remain in good standing as NCCCO Practical Examiners.
Since Auditors are now also required to have signalperson certification, a signalperson inter-rater reliability exercise was conducted along with a more detailed review of all program updates and clarifications and committee decisions. Other topics covered included refined observational skills, how to recognize discrepancies, and identification of typical audit issues.
During an audit, Auditors first ensure that the Practical Test Sites are set up correctly according to the test site layout CAD. To ensure maximum inter-rater reliability, the Auditor also scores candidates’ practical exams alongside the Practical Examiner; both score sheets are then compared to ensure that the Practical Examiner is scoring the exams correctly. “Any discrepancies noted are based on the Auditor’s score sheet,” said NCCCO’s Kinser, “so it is vital that Auditors understand all the details and nuances of setting up the courses and scoring candidates properly.”
After review of the Auditor’s report by NCCCO staff, the Practical Examiner is notified in writing of any discrepancies noted either in the test site set-up or any aspect of the practical test. This includes any inconsistencies between the Auditor’s score sheet and that completed by the Examiner. Corrective actions, along with references to all pertinent NCCCO policies regarding the Practical Exam process, are also included in the follow-up letter. Although the auditing process seeks to avoid disruption of the Practical Exam process, an Auditor can stop the test and provide corrective action if he or she believes the integrity of the test is being compromised.
Read the other stories in this series:
Crane Operators Respond to Nationwide Survey
NCCCO Audit Process Puts Examiners to the Test
NCCCO Management Committees Continually Monitor Test Performance Statistics
CCO Code of Ethics Reflects Certificants' Commitment to Safety
NCCCO Commissioners Represent Diverse Industry Stakeholders