National Human Services Training Evaluation Symposium
On May 23rd, 2018, California’s research project on predictive risk modeling (PRM) was presented at the National Human Services Training Evaluation Symposium in Berkeley, California. Dr. Emily Putnam-Hornstein from the Children’s Data Network (CDN) and Ms. Mary Lau and Ms. Jennie Feria from the Los Angeles County Department of Children and Family Services (DCFS) gave the research keynote address, Predictive Risk Modeling: A Tool for Data-Driven Decision Making. These collaborators shared how PRM can bridge the gap between research and practice. A recording of the keynote address can be viewed here.
Ms. Lau is the manager of Outcome and Analytics Section and has spearheaded efforts to shape DCFS’ Data Driven Decision Making efforts both within the department and on a national platform. Ms. Feria is the Division Chief of the Child Protection Hotline Division and manages a staff of 240. Los Angeles County shared how they are creating an agency culture for data-driven decision-making and incorporating research and data into the organization’s daily practice at the Hotline Calls Division.
The presenters shared lessons learned from their respective work in data driven decision-making. Lessons learned from the PRM research project show that that openness and transparency should be the rule, not the exception. This means public agencies, not vendors, should own model code and weights, and that independent validation of model performance and independent evaluation of its implementation and impact should be features of all initial PRM efforts. Algorithmic accuracy should be assessed for children of different races/ethnicities and unwarranted disparities should be addressed. Communities should demand (and receive) information concerning how the model is being used and how the model performs. PRM should be initially implemented to augment and complement clinical judgement and other approaches to assessment, not as a substitute. Once implemented, models can and should be re-weighted and re-validated on an ongoing basis in order to maintain accuracy. Finally, PRM must be efficient and simple to administer and agency culture must be supportive.
We are proud to have presented our work alongside our county partners to underscore the value of data in improving practices that support families.
