TTUHSC School of Pharmacy
Home SOP Faculty

Faculty & Staff

Self Governance » Assessment of Learning Committee

Agenda

MEETING OF 12/12/01

In attendance were Drs. Robert Supernaw, Mark Haase, Jim Stoll, Thomas Thekkumkara, Michelle Condren, Quentin Smith, Cindy Raehl, and Craig Cox attended via conference call. Dr. Tom Abbruscato. Chairman of the Credentialing Committee attended on behalf of the Credentialing Committee at the invitation of the Assessment of Learning Committee.

The meeting was called to order by Chairman, Mark Haase. Dr. Haase commented briefly on the cut scoring project performed on December 7 and 8. He thanked all of the committee members how participated. It was a very successful project we had a panel of faculty and residents the first day and former students, faculty and residents on Saturday.

The first order of business was the OAC's "Proposed Action for Poor Performance on the Annual Assessment of Learning Exam" that will be presented at the faculty meeting on December 17. This proposal has been reviewed, edited and approved by the OAC members. Chairman Haase previously e-mailed this proposal to the faculty for the response and input. Dr. Abbruscato stated that the Credentialing Committee wanted clarification on numbers 4 and 5 on Student failure of overall exam. The Credentialing committee thought that was a bit too progressive to have the students who failed appear before the committee as no procedures have been developed for remediation of such failure. It was explained that "appearing before committee" means just that. Any student who fails the retake exam will have the opportunity to be present and plead his/her case to the credentialing committee on why they did not pass the exam and what they feel they could have done to have performed better. The credentialing committee had concerns regarding those students who pass all of their courses but failed the Assessment exam. Dr. Haase explained that this will only be a Credentialing Committee/progression issue if a student fails the Assessment exam and then fails the retake of the exam. Dr. Supernaw stated that last year the only failures were P4 students who rushed through the exam so that they would go on a ski trip to New Mexico and also had the old carry-over attitudes that the Assessment exam did not mean anything. There was no action taken against these students. Now that we have refined out cut scoring competency program we are certain of the validity and reliability of our exam we are now ready to put some importance on it and propose formal consequences for failure. It was decided that we should make this a student authored and guided plan to be presented at the student's appearance before the Credentialing Committee. Dr. Haase suggested that Dr. Abbruscato take this information back to his committee prior to the faculty meeting and explain the Assessment Committee's role and the Credentialing Committee's role in instituting this proposal for consequences and progression based on failure of the retake exam and in student progression or lack thereof based on poor performance. Dr. Supernaw reiterated that hopefully this will be a moot point if we do not have any student who fail to meet the competency standard of the Assessment exam. If we have no failures then this will not be a consideration.

Dr. Haase advised the committee that the middle section on proposal is the long range goals for the 2003 and 2004 exam and this has been included in the proposal at the request of Dean Nelson for the proposed long-range plans for the OAC.

The committee decided to divide the "Proposed actions for Poor Performance on Annual Assessment of Learning Exam" into three parts and ask for approval on each separate section rather than try to get the whole proposal approved at this time.

The third section of the proposal is to provide for recognition for outstanding performance on the Assessment Exam. Dr. Haase suggested we have some sort of Dean's Award that would go to the top ten percent performers in each grade level and this award would be in the form of a certificate prepared by the Assessment of Learning Office.

Dr. Haase also suggested that the committee members review the document previously provided them entitled "Assessment - An Overview" prior to the faculty meeting in anticipation of questions that may be asked by faculty. This document was prepared by Dr. Supernaw and it covers the most frequently asked questions about our program and the exam. Attached hereto and incorporated herein by reference.

The last order of business is the actual Assessment exam which is scheduled for January 11, 2002. It was decided that instead of going to talk to the classes and explain what the Assessment exam is, like we have done in the past, we would prepare something in writing and distribute it to the students. It was decided that this year's exam would begin at 10:00 - 12:00 with a lunch break and resume at 1:00 - 3:00 for the P4s. Each class level has sixty questions and will be given an hour to take the exam. The P1s will test 10:00-11:00. P2s 10:00-12:00. P3s 10:00-12:00 lunch break 1:00 - 2:00. Dr. Haase requested volunteers to proctor and most all of the committee including visitor Dr. Abbruscato volunteered to proctor. The class rooms where the exam will be administered will be included in the instructions to proctors and the instructions to students that are being prepared and will be distributed to each soon.

The meeting was adjourned.

Assessment of Learning - An Overview


Three major objectives comprise the rationale for the Assessment of Learning Program at TTUHSC School of Pharmacy. First, faculty have a need to know how well students are learning. Second, students have the need for awareness of their knowledge, abilities, and skills strengths and weaknesses on an annual basis programmatic and learning. Third, assessment is an accreditation standard. And fourth, the public demands, and the faculty need a bonafide measure of the quality of the curriculum and its outcomes.

The School of Pharmacy Assessment of Learning Committee at Texas Tech University developed a strategy for providing students an analysis of ability mastery on absolute as well as relative scales. Previously, annual assessment scores were provided on a relative, class-standing basis (i.e., quartile scores). For the 2001 Assessment of Learning, using a modified Angoff system of cut score determination, composite cut scores defining performance standards for the "minimally competent student" were determined for ability sets related to (i) basic sciences, (ii) problem prevention and solving, (iii) pharmaceutical care, (iv) moral, ethical, and professional reasoning, (v) management, and (vii) dispensing skills. Overall scores as well as ability set scores were provided to students and faculty advisors. Advisors were trained to help the students interpret the scores.. Students who do not meet the ability expectations were given specific definitions of deficiencies. A detailed summary of scores, including absolute competency determinations, and relative scores (i.e., class means, class ranges, percentile scores, and Z-scores) were tabulated, and faculty advisors were charged with individual explanation for the significance of the reports to their student advisees.

The Most Important Questions (and Answers) About Our Annual Assessment Program


  1. Is the test knowledge, skills, and ability-anchored?
    • Yes. Each item has been written specifically to test the mastery of a specific knowledge, skill, or ability statement.
  2. Are test items quality controlled?
    • Yes. Each item is written by a faculty member, then reviewed by a panel, then reviewed by the test administrator and committee chair, then by a panel of 7 faculty (7 for 2001; 9 for 2002) who review every question on the assessment for accuracy, format, and merit in testing its corresponding ability statement.
  3. Is the assessment subjected to psychometric analysis?
    • Yes. The test, as a whole, and each item are scrutinized - the test in terms of reliability (pxt), and each item in terms of degree of difficulty (pi), and point-biserial (pbis). Items identified as problematic are reviewed for consideration of deletion.
  4. Do the scores derived from the annual assessment indicate mastery or lack of mastery of the skills expected for students in each level of the curriculum?
    • Yes. The faculty has developed a psychometrically-valid process (the Angoff method, modified) for the determination of cut-scores required for each item on the exam. The aggregate cut-score for the entire exam then defines the minimally competent student for each level of the curriculum.
  5. Is the assessment reliable?
    • Yes. For our purposes, reliability is defined as the ability of a single test to be predictive of scores achieved on an alternate examination that test the same abilities. In other words, reliability is the indication of the variance in scores achieved if a student were to be retested. Most psychometricians would agree that an examination reliability approaching +0.6 indicates excellent reliability (+1.0 is perfect). For the overall examination (P1 - P4 questions) administered in January 2001, pxt = 0.745. This indicates that 86.3% of the variance in scores can be attributed to true knowledge, skills, and abilities (true scores) rather than observed scores.
  6. Is the assessment valid?
    • Yes. For our purposes, validity is the support for the inferences drawn from the test scores. The assessment is valid in terms of face validity and content validity. However, it does not, at present, possess criterion-related validity. Face validity is the ability of an evaluation to test what it purports to test. Content validity indicates that the test did, in fact, measure the mastery of specific abilities developed by experts (pharmacy faculty, in our case). There are four specific tests of content validity, all of which have been satisfied by our annual assessment process. Criterion-related validity indicates the conclusions drawn from test scores and projected onto performance behaviors are valid. Clearly, we do not suggest that our test scores indicate anything about performance of clinical activities. What is most important in maintaining the validity of our annual assessment is to report scores for what they are - an accurate assessment of a student's mastery of specific abilities defined by the faculty.
  7. Do the assessment scores correlate with anything?
    • Yes...
    • The annual assessment scores correlate with overall grades. This correlation is 0.427 (p < 0.001).
    • Overall grades correlate with: (i) pharmaceutical care, (ii) management, (iii) dispensing, and (iv) basic science annual assessment scores. Overall grades DO NOT correlate with (i) problem solving and (ii) moral, ethical, and legal judgment annual assessment scores.
    • The annual assessment scores correlate with overall pre-P4 rotation grades. This correlation is 0.521 (p < 0.001).
    • The annual assessment scores correlate with summer clerkship grades. This correlation is 0.372 (p = 0.010).
    • The annual assessment scores correlate with fall clerkship grades. This correlation is 0.459 (p = 0.001).
    • The annual assessment scores do NOT correlate with spring clerkship grades. This correlation is 0.242 (p = 0.102). These spring clerkship grades do NOT correlate with NAPLEX scores, either.
    • The annual assessment scores correlate with NAPLEX scores. This correlation is 0.485 (p = 0.001).
    • The annual assessment scores do NOT correlate with NAPLEX Jurisprudence scores. This correlation is 0.278 (p = 0.058).
  8. Do the assessment scores indicate any regional differences in our students' mastery of knowledge, skills, and abilities?
    • Yes, but not much...
    • The Amarillo P4 students performed significantly better than the Dallas students (p = 0.016), but not significantly better than the Lubbock students.
    • The Lubbock P4 students performed significantly better than the Dallas students (p = 0.029), but not significantly better than the Amarillo students.
    • In terms of clerkship grades, there were no significant differences in any set of rotations (summer, fall, spring) among the three campuses.
    • NAPLEX scores for the Lubbock graduates were best (mean = 111.14), second Amarillo (mean = 110.88), followed by Dallas (mean = 104.3). However, there were NO SIGNIFICANT DIFFERENCES in NAPLEX scores among the three campuses.
    • NAPLEX Jurisprudence scores for the Lubbock graduates were best (mean = 88.07), second Dallas (mean = 85.19), followed by Amarillo (mean = 84.18). The Lubbock graduates scores significantly better than the Amarillo graduates, but not the Dallas graduates. There were no significant differences between the Amarillo and Dallas graduates.
  9. Are there any other interesting descriptive statistics?
    • Yes...
    • Overall GPA of male students = 82.4; female students = 82.9 (no statistical difference - p = 0.524).
    • Annual Assessment scores of male students = 75.6; female students = 76.4 (no statistical difference --p = 0.559).
    • Looking at the most interesting class (P3 class) secondary to its three cohorts (regular students, irregular set-back students, and J-class students), there were NO significant differences in assessment scores between any two cohorts.

Back to Top