PROJECTS

ASSESSING AND DEVELOPING WORKPLACE ESSENTIAL SKILLS WITH ESAT – PHASE I & II

May 2021

After being awarded a seed grant from the Future Skills Centre, an initial study into our Employability Skills Assessment Tool (ESAT) conducted by Futureworx and Blueprint allowed us to better understand how ESAT is used, how it is perceived by users, and the types of outcomes that would be expected and could feasibly be measured. The initial research, Phase 1, found that:

Staff perceive positive outcomes for participants, especially related to improvements in self-awareness of social and emotional skills (S&E skills), S&E skills, and well-being.
Staff identify several implementation factors as key to success: the importance of having staff buy-in, resourcing additional capacity for staff to dedicate toward ESAT use, including more than one perspective in the staff-assessment process, integrating S&E skills content across programming, and debriefing results with participants.
There is ample variability in ESAT use and implementation, with some programs following Futureworx’ recommended approach, but many others that do not.
Staff agree on many of ESAT’s core strengths for their participants, including that it is easy to use, helps staff establish a common language around employability skills, and eases tension during potentially difficult conversations with participants.

Based on the results of Phase I, The Future Skills Centre provided funding to conduct Phase II.

LEARNING GOALS: The learning goals of Phase II are:

Clearly define the minimum implementation conditions to maximize ESAT’s effectiveness

Strengthen the current version of ESAT by applying some of staff’s suggestions from Phase I, and perform some light-touch validity testing
Examine how ESAT is implemented across partner organizations to understand readiness factors, organizational barriers, and whether/why adaptations are made to implementation
Examine which target populations are being reached and whether participants are satisfied
Gather externally generated evidence for post-program outcomes (e.g., whether participants have better labour market outcomes) to compare these with ESAT scores; and

Understand whether the ESAT is a valid measure of S&E skills
Contribute to the evidence base on how to assess S&E skills, and how to validate a S&E skills assessment tool within a workforce development context

TIME FRAME: Completion Sept 2023

ACTIVITIES: The project activities include:

Part A: Prepare for evaluation

The approach to evaluation proposed for Phase II has several components and will therefore involve several preparation activities before evaluation activities can begin:

Clearly define ideal conditions for implementation: Building on the ESAT User Guide, Futureworx will clearly outline the minimum implementation conditions for PDPs.
Partner selection and onboarding: Futureworx will recruit and onboard program delivery partners (PDP) that are interested in taking part in the research and that meet all project requirements.
Support refinements to the evaluation plan: Once partners are onboarded, Futureworx and Blueprint will work with partners to refine and adapt the evaluation plan laid out below.

Part B: Strengthen ESAT

Before ESAT is used within this research project, Futureworx will work with Blueprint to apply lessons learned from Phase I alongside some preliminary light-touch validity testing to improve self-assessment questions and exemplar descriptions. This will include the following activities:

Create a Theory of Change (TOC): Ensure that the theories and assumptions underlying ESAT align with findings from Phase I and can be used to strengthen the tool.
Support preliminary light-touch validity testing: Provide Blueprint with the ESAT assessment data required to test the correlation between individual items for each skill.
Revise content of ESAT self-assessment questions and exemplar behaviours: Incorporate recommendations from Blueprint (based on validity testing and findings from Phase I) to revise the ESAT tool.

Part C: Evaluate ESAT

Three components of the evaluation will happen in tandem over the course of Phase II.

Part C.1: Implementation supports and evaluation

Futureworx and Blueprint will engage an external consultant with expertise in implementation science to support partners in tackling the delicate balance between delivering the program according to implementation requirements (i.e., fidelity of the model) and adaptations that are required for their context. Activities to be led by the implementation consultant and supported by both Futureworx and Blueprint will include:

Readiness Assessment: Identifying and assessing factors that affect a PDP’s readiness for implementation
Barriers and Facilitators Assessment: Assessing PDP’s perceptions of potential challenges and success factors that could be proactively mitigated or leveraged to improve evaluation results

Tracking Fidelity and Adaptations: Understanding how ESAT is implemented across sites including whether and why adaptations are made

Part C.2: Outcomes Evaluation 

This part of the evaluation will help us understand what outcomes are achieved by participants of each ESAT-using program and whether there is a relationship between ESAT scores and outcomes. Futureworx will lead or support Blueprint with the following activities:

Define outcomes of interest: Identify the key participant outcomes to be tracked during the evaluation (e.g., improved skills, self-awareness, employment outcomes, education outcomes).
Review data collection strategy and tools: Ensure that research instruments are accurately tailored to PDPs by providing feedback to Blueprint on surveys, interview protocols, etc.

Support deployment of data collection strategy: Liaise between delivery partners and Blueprint to support the roll out and sequencing of all data collection.

Part C.3: Process Evaluation  

The final component of the evaluation will help understand ESAT users’ experience with the tool, focusing primarily on the perceptions of participants and employers. Futureworx will support Blueprint with the following activities:

Support design and deployment of a data collection strategy: Liaise between delivery partners and Blueprint to support interviews with partner staff, participants, and employers.

Part D: Validity testing

The final component of Phase II will involve a second, more robust round of validity testing, aiming to understand whether ESAT provides a valid measure of participants’ employability skills. Data collected during the evaluation will provide information on the necessary outcomes to make inferences about validity. Futureworx will support Blueprint as required with the following activities:

Concurrent validity testing: Testing the extent to which ESAT is related to an existing, previously validated measure.
Predictive validity testing: Testing the degree to which ESAT scores accurately predict desired outcomes.
Inter-item reliability testing: Testing whether each revised question is aligned with the skill it seeks to measure.

PROGRAM DELIVERY PARTNERS: Futureworx is inviting existing and new ESAT users to participate in this project as program delivery partners (PDPs). Funding supports and incentives will be provided to PDPs. Existing ESAT site managers can apply in ESAT. New users can apply by contacting Futureworx.

CONTACT: For more information about, or to express interest in participating in this project, please Contact Paul Brinkhurst, Futureworx, 587-226-7800, paulb@futureworx.ca.

______________________________________________________________________________________________________________________________________________________

Futureworx is proud to release its report – “Building a PAN CANADIAN SOFT SKILLS FRAMEWORK”

This project was funded to explore the need for, and how best to create, a pan-Canadian soft skills framework to support the development of a productive,
adaptable, globally competitive workforce capable of meeting the rapidly changing needs of work and the workplace.

Funded by the Government of Canada’s Adult Learning, Literacy and Essential Skills Program.

REPORT: Soft Skills Framework Report – PDF (5MB)

We welcome your feedback and questions – simply CONTACT US.

INTEGRATING SOFT SKILLS INTO THE ESSENTIAL SKILLS FRAMEWORK – BUILDING A NATIONAL CONSENSUS

OBJECTIVE: The objectives of this project are:

To develop a plan to create a common assessment and development approach for ‘soft’ skills in Canada;
To propose their integration into the existing essential skills framework;
To identify and develop common terminology.

These objectives will be achieved by:

Identifying existing ‘soft’ skills taxonomies, frameworks, efforts, assessments and resources being used and/or supported; and
Developing a plan for moving forward on a collaborative impact effort to expand the essential skills framework to include ‘soft’ skills.

TIME FRAME: Completion Nov 2018

ACTIVITIES: The project has four main activities:

Conduct a literature review of work on ‘soft’ skills across Canada to capture current thinking and efforts on ‘soft’ skill development;
Develop and implement a survey targeting a broad set of stakeholders across Canada (e.g. employers, service providers, educators, trainers, governments) to understand current practices, language, and views on ‘soft’ skills development;
Select a representative group of stakeholders (15-20), to be involved in a planning session to explore possibilities for the creation of a Canadian “soft skills” framework and its integration with the existing essential skills framework (stakeholders to be trained in consensus-based decision making and Collective Impact as part of the process); and
Write a report that summarizes the results from the project and providing recommendations for reaching consensus on the development of a Canadian “soft skills” framework.

OUTPUTS: Principal outputs for the project include:

A summary of the existing ESDC Competency-based “soft” skills taxonomy;
An initial draft set of skills suited for initiating consensus discussions;
Recommendations for integrating “soft skills” into the existing essential skills framework;
A plan for developing a national consensus on ‘soft’ skills development and assessment;
Recommendations on how to render the taxonomy of practical use to front line application; and
A final report to ESDC.

FINAL REPORT: Final Report – Sharing Perspectives on Soft Skills Survey 06-11-2018 – PDF (427KB)

CONTACT: For more information about, or to express interest in participating in this project, please Contact Paul Brinkhurst, Futureworx, 587-226-7800, paulb@futureworx.ca.