Evaluation Items

Evaluation items are the elements of every program for which the evaluator will provide a response. Although very similar in nature, different names are used to refer to evaluation items depending on the program type.

Evaluation items share certain common characteristics:

• Inclusion in a category of one or more related evaluation items (e.g., financial performance KPIs may be grouped under a category “Financial”).
  • A specialized category called “Critical Gate” changes the behavior of the evaluation such that a rating of these items that is below minimum acceptable values will result in the failure or rejection of the entire evaluation: effectively, if a Critical Gate evaluation item fails, the whole evaluation fails.
• Unique identification number – allows unambiguous reference and ordering of evaluation items.
• Unique description – provides sufficient explanation to the evaluator to understand what data are being requested.
• Frequency - specifies the periodicity of the evaluation (annual, semi-annual, quarterly, monthly, one-time).
• Score type - specifies the scale on which the evaluation item will be rated (10 pt Scale, 5 pt Scale, Info Only, Letter (A-E), Numeric, Percent, R/Y/G, Yes/No).
• Rating scale parameters (depend on the score type, and may allow specification of minimum, target, and maximum values associated with an evaluation item).
• Weight - provides a means to emphasize one evaluation item in comparison to another.

• Designing Evaluation Items

There are some key differences in evaluation items for different types of programs:

• Compliance requirements are generally yes/no questions, and phrased such that a positive or “yes” response is a preferred status for that requirement.
  • “Has the supplier provided evidence of employment eligibility for all employees and sub-contractors who may work on-site in the delivery of services under this contract?”
• Performance KPIs are typically expressed as a factor for which a numeric rating can be provided.
  • “Average on-time completion rate for all projects under management during current quarter.”

Regardless of these differences, there are some fundamental best-practice guidelines for all evaluation items that should be considered:

• Evaluation items should be specific.
  • Good: “Percent of projects delivered within budgeted amount ± 5%”
  • Could be Better: “Level of financial performance”
• Evaluation items should be measurable.
  • Good: “Number and cumulative duration of system outages within acceptable limits established by SLA”
  • Could be Better: “System performed adequately during the period”
• Data should be available to the evaluator with which to answer the evaluation item.
• The total number of evaluation items should be reasonable for the circumstances of the evaluation.
  • Two evaluation items for a ten-year IT outsourcing contract for a Fortune 100 company’s data centers worldwide are probably too few or too vague.
  • Three-hundred unique KPIs for a $50,000 contract to provide paper recycling services are probably too many or too detailed.