TTUHSC School of Pharmacy
Home SOP Administration

Administration

Assessment of Learning

Frequently Asked Questions

The Most Important Questions (and Answers) About the Annual Assessment

  1. Is the test knowledge, skills, and ability-anchored?
    • Yes. Each item has been written specifically to test the mastery of a specific knowledge, skill, or ability statement.
  2. Are test items quality controlled?
    • Yes. Each item is written by a faculty member, then reviewed by a panel, then reviewed by the test administrator and committee chair, then reviewed by a panel of 7-9 faculty who review every question on the assessment for accuracy, format, and merit in testing its corresponding ability statement.
  3. Is the assessment subjected to psychometric analysis?
    • Yes. The test, as a whole, and each item are scrutinized - the test in terms of reliability (pxt), and each item in terms of degree of difficulty (pi), and point-biserial (pbis). Items identified as problematic are reviewed for consideration of deletion.
  4. Do the scores derived from the annual assessment indicate mastery or lack of mastery of the skills expected for students in each level of the curriculum?
    • Yes. The faculty has developed a psychometrically-valid process (the Angoff method, modified) for the determination of cut-scores required for each item on the exam. The aggregate cut-score for the entire exam then defines the minimally competent student for each level of the curriculum.
  5. Is the assessment reliable?
    • Yes. For our purposes, reliability is defined as the ability of a single test to be predictive of scores achieved on an alternate examination that test the same abilities. In other words, reliability is the indication of the variance in scores achieved if a student were to be retested. Most psychometricians would agree that an examination reliability approaching +0.6 indicates excellent reliability (+1.0 is perfect). For the overall examination (P1 - P4 questions) administered in January 2001, pxt = 0.745.
  6. Is the assessment valid?
    • Yes. For our purposes, validity is the support for the inferences drawn from the test scores. The assessment is valid in terms of face validity and content validity. However, it does not, at present, possess criterion-related validity. Face validity is the ability of an evaluation to test what it purports to test. Content validity indicates that the test did, in fact, measure the mastery of specific abilities developed by experts (pharmacy faculty, in our case). There are four specific tests of content validity, all of which have been satisfied by our annual assessment process. Criterion-related validity indicates the conclusions drawn from test scores and projected onto performance behaviors are valid. Clearly, we do not suggest that our test scores indicate anything about performance of clinical activities. What is most important in maintaining the validity of our annual assessment is to report scores for what they are - an accurate assessment of a student's mastery of specific abilities defined by the faculty.