The comparison group was not eligible for WorkAdvance services. Comparable services were available in the community, such as targeted sector training from community colleges and other community providers.
The total program cost per participant for St. Nick’s Alliance WorkAdvance over the three years from 2011 to 2014, was $7,696 (2013–2014 dollars), which included management costs, services, and occupational skills training costs. The study also estimated costs from the second year of operations of $6,666 (2013 dollars) per participant. The second-year costs were lower because they no longer included start-up costs from the first year, and several program participants had completed the occupational skills training and moved on to lower-cost post-employment services.
A separate cost-benefit analysis focused on a 63-month observation period after enrollment. It examined costs and benefits to the participant, the government, and society. (These figures are presented in terms of 2018 dollars.)
- The participant. The St. Nick’s Alliance intervention resulted in a net financial gain of $1,623 to the participant; this means the sum of net benefits from earnings, fringe benefits, TANF benefits, and unemployment insurance was estimated to be $1,623 higher than the work-related expenditures, value of the time they would otherwise have available outside work, decrease in SNAP, disability and housing assistance benefits, and increase in their tax payments.
- The government. The St. Nick’s Alliance intervention resulted in a net financial loss of $4,130 to the government; this means that the sum of net increases paid by the government in TANF benefits, unemployment insurance payments, and employment and training costs was estimated to be $4,130 higher than the sum of the net gains in tax payments and reductions in SNAP, disability, and housing assistance benefits.
- Society. The St. Nick’s Alliance intervention resulted in a net financial loss of $3,293 to society as a whole; this means that the sum of the net increase in earnings and fringe benefits received by participants, and the decrease in administrative costs of transfer programs, was estimated to be $3,293 lower than the sum of the losses. The losses encompassed the net increase to the government in work-related expenditures and provision of employment and training, the value to participants of the time that they would otherwise have available outside work, and deadweight loss because of taxation or transfer payments (that is, costs from inefficiencies that occur when taxes or transfer payments affect individuals' economic decisions)
The analysis also developed estimates over a 10-year period that included the 63 months of observation and projections for the remainder of the 10 years. The 10-year estimate instead found a benefit to society of $13,742, reflecting the study’s findings that earning impacts were growing at the end of the observation period.
St. Nick’s Alliance WorkAdvance intervention began in 2011 and ended in 2014. The site enrolled study participants from October 2011 to June 2013. Participants received WorkAdvance services for up to two years after enrollment and took a follow-up survey about 18 to 24 months after random assignment. Evaluators examined outcomes from quarterly administrative data through 2018.
The St. Nick’s Alliance WorkAdvance intervention was a sector program in environmental remediation and related occupations with wraparound services. Its goal was to help residents in the Williamsburg-Greenpoint neighborhoods of Brooklyn advance into careers that paid well, offered opportunities for advancement, and helped protect the environment and improve the quality of life of residents. St. Nick’s Alliance’s model focused on training first, which placed participants into training before engaging them in employment services.
St. Nick’s Alliance also adapted its services to meet industry and employer needs while keeping its focus on participants’ long-term career advancement and upward mobility.
During intake, staff thoroughly screened applicants, including through use of a drug test, to identify the candidates they believed were the most suitable for the intervention. To ensure participants were prepared and qualified for the skills training and potential occupations, staff evaluated participants’ level of motivation to be in the program and their likelihood of completion, assessed their reading and math literacy skills, and determined whether they met income eligibility requirements. About six months into implementation, St. Nick’s Alliance refined their screening process based on employer input by adding a criminal background check. Once participants enrolled in the program, they received the following core services:
- Work-readiness activities and soft-skills training. Participants completed sector-specific orientations and received coaching to help them develop goals and maintain progress toward their career advancement objectives. During a classroom-based career-readiness training, staff helped participants develop a résumé and cover letter and addressed transportation barriers. Participants also received training in interview skills and soft skills. Staff also showed them examples of performance reviews to help them understand how employers would evaluate their work once they found a job.
- Occupational or sectoral training. Participants could receive occupational skills training in sectors such as environmental remediation and pest control and could obtain up to five credentials. St. Nick’s Alliance offered the pest control training on-site and partnered with private schools that offered the other trainings off-site. Participants in the training program attended full-time classes on weekdays in cohort style groups and received occupational skills training concurrently with work readiness activities.
- Job development and job placement. Upon completing training, participants received assistance from two on-site job developers who facilitated one-on-one job matching of participants to employers within their designated industries and distributed their résumés in batches to employers in related fields.
- Employment retention and advancement services. Participants received various post-employment services to help them maintain and advance in their job, such as coaching on advancement goals, instruction on how to negotiate for higher wages, and case management to tackle barriers to retention. Staff developed a follow-up plan to maintain communication with participants and provided them with feedback from employers on their performance.
Staff dedicated substantial time to building rapport with participants by routinely visiting participants at their training sites and following up via email or phone to check on their progress. Staff also adjusted their approach to training when building relationships with employers by remaining attuned to their recommendations and feedback. For example, in response to the high demand for mold remediation skills after Hurricane Sandy, the service provider shifted its environmental remediation certificate to offer mold remediation technician training. St. Nick’s Alliance staff aimed to strengthen relationships with employers by ensuring participants had all the certifications required for any job to help the employer save time and resources. Staff maintained communication with employers weekly to identify job openings, conduct job matching, and follow up on previous hires. Employers also engaged in the WorkAdvance program by providing counseling on curricula and sector trends, serving as guest speakers in career-readiness classes, participating in mock interviews, and hosting visits by participants in their workplace.
The initial screening helped St. Nick’s Alliance identify applicants they believed were most adept, committed, and determined to be in the program. Six months into the program, the enrollment screening criteria became more stringent for the Environmental Remediation Technician program after adding a criminal background check and screening out those with . prior criminal justice involvement . This change was based on employers’ input. After the program experienced barriers to employment for participants with prior criminal justice system involvement, St. Nick’s Alliance changed some other program components during implementation. For instance, the provider received technical assistance from MDRC, a non-profit organization, to enhance its services and make the program more hands-on. The provider also began incorporating employer role-playing scenarios to help participants understand employer perspectives and improve their behavior and decision-making in the workplace. In addition, St. Nick’s Alliance began delivering messaging and language focused on advancement at the beginning of program services instead of waiting to talk about advancement until participants were employed.
Challenges. St. Nick’s Alliance had limited experience with sectoral approaches, which affected the intervention’s implementation. The organization needed considerable MDRC technical assistance for more than a year to implement the program and enhance staff capacity. St. Nick’s Alliance previously offered some skills training on environmental remediation but not as extensively and robustly as needed for the WorkAdvance program. The organization had to expand its occupational skills training program to include a broader set of careers that emphasized advancement, including pest control technician and hazmat driver. When it came to advancement, job developers had difficulty finding permanent employment with advancement potential for participants in environmental remediation because the industry had a high concentration of short-term, project-based work. Moreover, the service provider found it challenging to engage study participants in the career-readiness classes because of the prolonged wait time (up to three months or more) between the time of enrollment and the start of the class, which was provided simultaneously with the occupational skills training courses. It took time for St. Nick’s Alliance to find the appropriate training providers that met employers’ needs and to establish a formal partnership to enroll participants into their training courses. This delay caused a bottleneck in service delivery from participants waiting to start training and increased drop-off rates.
The study did not discuss any tools to measure fidelity to the intervention model.
The WorkAdvance program received funding from various private foundations. The federal SIF matched the private dollars. The SIF is part of the Corporation for National and Community Service, and it uses public and private resources to fund established programs and provide technical assistance to support their evaluation efforts.
The intervention took place in New York City. The city’s economy was improving from the Great Recession during the study, but growth of the environmental remediation industry was slow during the follow-up period. Hurricane Sandy occurred during this period, which increased the demand for particular workforce training programs.
WorkAdvance was implemented by a multiservice community-based organization named St. Nick’s Alliance that offers housing, health care, youth services, and other supportive services in Brooklyn, NY. The organization houses a workforce development unit that operated the WorkAdvance program.
The WorkAdvance Managing Collaborative was a key stakeholder for St. Nick’s Alliance. The collaborative included the following partners:
- The New York City Center for Economic Opportunity managed the contracts with the WorkAdvance providers, including St. Nick’s Alliance; helped raise matching funds required for the Social Innovation Fund (SIF) (funding described further under “Funding sources”); operated the SIF network, through which the provider could share best practices and experiences with other WorkAdvance providers; and monitored the performance of the project.
- MDRC provided technical assistance to St. Nick’s Alliance while implementing the WorkAdvance model, monitored the performance of the program, and led the evaluation.
The study population encompassed unemployed or working adults who earned less than $15 per hour and had a family income below 200 percent of the federal poverty level. The program required that participants be eligible to work legally in the United States, have a minimum 8th-grade-level math and reading assessment score, and not have participated in a St. Nick’s Alliance occupational skills training program two years before enrollment.
Participants of St. Nick’s Alliance WorkAdvance program were mostly single (71 percent), male (85 percent), and Black or African American (63 percent) or Latino or Hispanic (23 percent). Participants were an average age of 35. Almost half (45 percent) had at least one child. Eighty-eight percent of participants had their high school diploma or GED. Eleven percent of participants were employed at enrollment and, among those, 65 percent worked full time and earned an average hourly wage of $10.84. Thirty-eight percent of participants were previously incarcerated or convicted of a crime. Twenty-five percent of participants received unemployment insurance benefits, and 42 percent received Supplemental Nutrition Assistance Program (SNAP) benefits.
St. Nick’s Alliance coaches provided one-on-one preemployment coaching concurrent with occupational skills training and followed up with participants two to three times per week during their job search. The occupational skills training ranged from 5 weeks to 12 weeks depending on the occupation type, and the career-readiness training included nine sessions for four hours each. After participants obtained a job, job developers checked in with them weekly for the first 30 days, after which career coaches contacted participants monthly.
According to data collected over a three-month span from April to July 2014, participants who engaged in an advancement coaching meeting attended about two coaching meetings, on average, with staff. Most coaching was conducted by phone.
Participants could receive up to two years of WorkAdvance program services and follow-up services thereafter.
The on-site staff dedicated to implementing the WorkAdvance program at St. Nick’s Alliance consisted of the following:
- The part-time managing director and one full-time project coordinator who managed the project
- A full-time recruiter who recruited participants
- A full-time administrative assistant and a full-time retention specialist who were responsible for intake, career-readiness instruction, and other staff support
- Two full-time job developers tasked with managing the relationships with employers and linking participants to jobs
- Two case managers (one part-time and one full-time) and a full-time career coach or senior vocational counselor who provided case management and coaching services
- A part-time data manager
The program experienced changes to staffing throughout the study. At the start of the program, St. Nick’s Alliance leaders moved existing staff onto WorkAdvance to provide coaching and share recruitment responsibilities. The existing staff had to take on additional obligations beyond their initial role for the WorkAdvance program that became onerous, and eventually St. Nick’s Alliance hired a WorkAdvance project coordinator and separate career coaches to deliver the pre- and post-employment advancement services. The existing case managers at St. Nick’s Alliance were retrained in advancement-focused coaching to help participants progress with their career advancement while they addressed barriers.
The study authors did not include information on the degrees or certification requirements of St. Nick’s Alliance staff.