Action Overview: Evaluate the Effectiveness of the Data System

What are the essential features of effectiveness in a decision support data system?

In an effective decision support data system, data is used by a team to answer important questions and make decisions to support continuous improvement. These characteristics of data must be present:

• accessible
• accurate
• relevant
• reliable
• timely
• usable
• valid  

Why evaluate effectiveness of the data system?

Evaluating data systems helps organizations assess and manage performances so the right improvements can be made to support continuous improvement in a standards-based system.  Benefits include:

• Determining if data is accessible, accurate, reliable, timely, and usable.
• Keeping  the continuous improvement process transparent, and allowing for open and frank conversations that are needed to effect change. 
• By celebrating successes as well as identifying and resolving potential threats to accurate, timely, and usable data, leaders are supporting data based decision making.  

What are common threats to an effective data system?

One of the most important but often challenging steps in evaluating effectiveness is ensuring that teams take the time to discuss how easy, accurate and efficient it is to have data in time for meetings. The most common threats to an effective data system are: 

• using available data rather than specific data to inform identified questions or needs
• inaccurate data collection (assessing students off grade-level, inconsistent test modifications, ineffective fidelity tool, etc.)
• inaccurate or ill-defined data entry procedures (double entry, ambiguous definitions or coding, etc.)
• reliance on a single individual to enter and pull data reports (if an individual is on leave, the team will not have the data, data has to be pulled from multiple systems and graphs created, etc.).
• financial, structural, or other challenges outside the team’s control that hinder improvements to the data system.

Suggested Activities

How do educators use Portal resources to evaluate the effectiveness of a data system?

The portal provides an activity to guide leadership teams in the Activity: Analyze Data System for Effectiveness.

Related Content

A supplemental activity related to data analysis Activity Directions: Preparing for Analyzing and Responding to Data teams that need more background knowledge on data usage prior to evaluating data system effectiveness.

Resources

 National Implementation Research Network (NIRN). Handout 28: Decision Support Data System (DSDS). retrieved 10/3/19 from https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/AIHub-Handout28-DriversEd-DSDS.pdf.