Formative and Summative Assessments (Standard 5.1)
What is the difference between formative and summative assessments, and what does CAA expect from programs in the area of student learning outcomes?
The CAA added clear definitions in the implementation language [PDF] to distinguish formative (ongoing measures to assess acquisition of knowledge and skills) from summative (comprehensive evaluation of student learning outcomes) assessments, both of which are required. Other requirements of the program are detailed in the implementation language of this standard.
*Note: As one of the most frequently cited standards for partial or non-compliance, programs should ensure that a process has been developed and implemented for the tracking and documentation of summative and formative assessments of student performance, along with a documented process for the implementation of remediation plans.
Student Progress Documentation (Standard 5.2)
*Note: As one of the most frequently cited standards for partial or non-compliance, programs should ensure that there is a formal process in place for the tracking and documentation of student progress toward the degree.
Does the program still need to keep track of supervisor qualifications (e.g., CCC status)?
Yes. If the program’s mission and goals include preparing students for entry into professional practice and thereby to earn credentials to do so, then there must be evidence of supervisor information consistent with those requirements. Although CAA does not require programs to provide CCC status of clinical supervisors as part of its review, the program still needs to track this type of data to support a student’s application for ASHA certification along with clinical hours supervised by these individuals. Similarly, if state licensing or teacher certification procedures require supervisors to hold comparable credentials, programs would need to monitor that consistent with those expectations.
Do site visitors verify or check student hours when they review student files on-site?
CAA site visitors are not verifying the clinical hours for each student to meet credentials. They are reviewing student files to ensure that there is a tracking mechanism in place to document student acquisition of knowledge and skills, based on the program’s stated mission and goals, and to determine if it is consistently used.
Program Effectiveness (Standard 5.3)
*Note: As one of the most frequently cited standards for partial or non-compliance, programs should ensure that they conduct ongoing assessments of program effectiveness and use those results for program improvement.
If a university offers accredited programs in both audiology and speech-language pathology, should they provide both sets of student achievement data, one for each program area of study?
Yes. The CAA examines data specific to each professional area’s program to support compliance with the accreditation standards, so it is important to be able to see the distinct data. Additionally, if the program is offered via multiple modalities (e.g., distance education) those data would need to be collected and analyzed separately to ensure program effectiveness in each modality.
If a program is submitting an application for initial accreditation (e.g., a new doctoral audiology program), then it may not have full sets of student achievement data for the program. If the program has not graduated 3 years of students, it would then leave the question blank and provide a written explanation.
Does CAA have different expectations for student achievement when a program is offered through multiple delivery modes (i.e., residential, satellite, distance education)?
No. The CAA applies the same benchmarks for student achievement regardless of the mode of educational delivery. The CAA requests data specific to each delivery mode be provided in response to this standard, which allows the effectiveness of each delivery mode that comprise the program’s accreditation to be verified. The CAA determined that programs should be held to the same benchmark and thereby not disadvantaging students participating in one component over another. CAA, however, will assess whether the threshold for each student achievement measure has been met for the program in its entirety (all modalities) to determine compliance with the standard.
When reporting data to the CAA regarding the Praxis examination pass rates, should programs report results of students’ first attempt? Or is it expected to report only the final outcome of the test-takers efforts?
The CAA expects programs to report and publish student achievement data for the last 3 completed academic years. When reporting the Praxis examination pass rates, programs are required to report results only once for test-takers who took the exam multiple times in a single examination reporting period.
The CAA does not require programs provide the results from first-time test takers, but the program may collect and analyze these data to support its own self-assessment activities. There is value in the program considering those data, and its ability to determine what improvements can be made as part of the overall ongoing evaluation of the program’s effectiveness.
What should a program do if test-takers’ pass rate on the Praxis examination, as averaged over the 3 most recently completed academic years, falls below the CAA-established threshold?
Pass rates on the Praxis examination are one measure of a program’s success in the area of student achievement. The program must use the results of its assessments towards continuous program improvement. If the program’s pass rates as averaged over the 3 most recently completed academic years fall below the CAA-established threshold, then the program should undergo further self-assessment to determine what may be causing the poor performance, and look to key indicators such as curriculum, faculty qualifications, clinical offerings, and overall assessment of opportunities for students to gain the requisite knowledge and skills needed for entry into independent practice. Once key indicators are identified, a plan for additional review, needed modifications to the program, and further follow-up should be developed.
The program may wish to consider the timing of when most test-takers are sitting for the exam, and whether that may be a contributing factor to why test-takers are not passing the Praxis examination. ETS [PDF] developed and validated the examination to be taken at a point when the test-taker was expected to have acquired the appropriate knowledge and skills needed to enter independent professional practice. If timing is determined to be a factor, the program may wish to review and modify its instructions to students and graduates as to optimal timing to take the exam (e.g., after all course work and most of a student’s clinical experiences have been completed).
When is the best time to collect employment data?
It is entirely up to the program, based on data collection and/or alumni survey policies in place at the program or institution level. Some programs find that it is easiest to collect employment data (sometimes referred to as job placement information) at the time of graduation because response rates decline as time passes. However, the implementation language indicates that programs are to report the employment rate one year post-graduation for its graduates, so graduates have one year to secure employment or pursue further education. Collecting employment data before the full year has elapsed may not provide an accurate picture of graduates’ abilities to secure employment.
Also, CAA does not prescribe the format or mechanism for data collection, thus does not require that data collection occur through traditional/formal survey methods or that data be collected at a single point in time. Some programs prefer to collect information throughout students’ final semester and during the first year post-graduation, updating data as each individual secures employment or acceptance in an advanced education program. Such an approach allows programs to present more accurate data, as some students secure employment or admission to an education program before graduating and others may take additional time after graduation to do so.
Should the data submitted for reporting student achievement represent the same student cohort?
Reporting the required student achievement data is not intended to reflect the same student cohort for each of the 3 outcome measures. Data reported for employment rates may be for an earlier group/year than the data reported for program completion rates (e.g., employment rates may be reported in 2015–2016 for the 2014–2015 graduates, but the program completion rates reported in 2015-2016 for the 2015-2016 graduates).
Faculty Evaluation (Standard 5.4)
Should a program conduct evaluations of faculty members in a certain fashion? What if the program does not have faculty evaluate the program director?
The Standards for Accreditation indicate the need for an evaluation process to occur, but do not prescribe exactly what that process must include for faculty. As long as some form of feedback mechanism is developed, implementation can be documented, affected individuals are aware of the process and outcomes, and the evaluations are conducted on a regular schedule, then that would meet the intent of standard.
Also see information about the evaluation of the program director under Standard 1.5.
Does the evaluation of ALL faculty include tenured faculty, even if the university has a union or collective bargaining agreements that would prevent/limit evaluations of faculty?
Yes. The CAA would expect the program to describe whatever manner is available and appropriate to support program goals and improvement. The program must describe the mechanism for regular evaluation of its faculty by program leadership in accordance with institutional policy and guidelines; if no formal evaluations are being done the program must indicate the circumstances.
*Note: As one of the most frequently cited standards for partial or non-compliance, programs should ensure that there is a documented process for evaluation of faculty members, that all faculty members are aware of the process, and that results of these evaluations or assessment are used to improve the program.