Evaluation Items
Evaluation items are the elements of every program for which the evaluator will provide a response. Although very similar in nature, different names are used to refer to evaluation items depending on the program type.
Program Type |
Evaluation Item Nomenclature |
---|---|
Performance |
Key Performance Indicator (KPI) |
Compliance |
Requirement |
Risk |
Risk Item |
Information Management and Robo |
Data Item |
Evaluation items share certain common characteristics:
- Inclusion in a category of one or more related evaluation items (e.g., financial performance KPIs may be grouped under a category “Financial”).
- A specialized category called “Critical Gate” changes the behavior of the evaluation such that a rating of these items that is below minimum acceptable values will result in the failure or rejection of the entire evaluation: effectively, if a Critical Gate evaluation item fails, the whole evaluation fails.
- Unique identification number – allows unambiguous reference and ordering of evaluation items.
- Unique description – provides sufficient explanation to the evaluator to understand what data are being requested.
- Frequency - specifies the periodicity of the evaluation (annual, semi-annual, quarterly, monthly, one-time).
- Score type - specifies the scale on which the evaluation item will be rated (10 pt Scale, 5 pt Scale, Info Only, Letter (A-E), Numeric, Percent, R/Y/G, Yes/No).
- Rating scale parameters (depend on the score type, and may allow specification of minimum, target, and maximum values associated with an evaluation item).
- Weight - provides a means to emphasize one evaluation item in comparison to another.
-
Designing Evaluation Items
There are some key differences in evaluation items for different types of programs:
- Compliance requirements are generally yes/no questions, and phrased such that a positive or “yes” response is a preferred status for that requirement.
- “Has the supplier provided evidence of employment eligibility for all employees and sub-contractors who may work on-site in the delivery of services under this contract?”
- Performance KPIs are typically expressed as a factor for which a numeric rating can be provided.
- “Average on-time completion rate for all projects under management during current quarter.”
Regardless of these differences, there are some fundamental best-practice guidelines for all evaluation items that should be considered:
- Evaluation items should be specific.
- Good: “Percent of projects delivered within budgeted amount ± 5%”
- Could be Better: “Level of financial performance”
- Evaluation items should be measurable.
- Good: “Number and cumulative duration of system outages within acceptable limits established by SLA”
- Could be Better: “System performed adequately during the period”
- Data should be available to the evaluator with which to answer the evaluation item.
- The total number of evaluation items should be reasonable for the circumstances of the evaluation.
- Two evaluation items for a ten-year IT outsourcing contract for a Fortune 100 company’s data centers worldwide are probably too few or too vague.
- Three-hundred unique KPIs for a $50,000 contract to provide paper recycling services are probably too many or too detailed.