Purpose: The purpose of this evaluation is to document and explicate the process of community mobilization, planning, and collaboration that has taken place before and during the Safe Kids/Safe Streets awards; to inform program staff of performance levels on an ongoing basis; and to determine the effectiveness of the implemented programs in achieving the goals of the Safe Kids/Safe Streets program.

Background: This program will evaluate the Safe Kids/Safe Streets program described in this Application Kit. The Safe Kids/Safe Streets program has three goals: (1) to encourage localities to restructure and strengthen the criminal and juvenile justice systems to be more comprehensive and proactive in helping children and adolescents and their families who have been or are at risk of abuse and neglect; (2) to implement or strengthen coordinated manage-ment of abuse and neglect cases by improving policy and practice of the criminal and juvenile justice systems and the child welfare, family services, and related systems; and (3) to develop comprehensive communitywide, cross-agency strategies to reduce child and adolescent abuse, neglect, and resulting child fatalities.

The program strategy through which grantees under the Safe Kids/Safe Streets program are to achieve these goals consists of four major components. The first component is reform of the justice, child welfare, family services, and related systems' handling of child maltreatment and establishment of a system of accountability. The second is to implement a continuum of services to protect children and support families. Third, sites will collect data, provide for sharing of data to support proper adjudication of abuse and neglect cases, and evaluate their progress. Finally, sites will conduct prevention education and public information campaigns.

The evaluation of the Safe Kids/Safe Streets program will be conducted in two phases. The first phase will encompass the design and implementation of a process and formative evalua-tion and the design of an outcome evaluation. The second phase, implementation of the impact evaluation, may be funded through supplemental awards upon successful completion of the first phase. Both the process and the impact evaluation designs will be developed in collaboration with the local evaluation team and local project staff. The process evaluation should start early, after approval of the process design, and continue as long as there are program activities to evaluate.

Applicants for the Evaluation of the Safe Kids/Safe Streets program should review the Safe Kids/Safe Streets program announcement, paying special attention to the evaluation require-ments. The evaluator chosen will be expected to provide leadership regarding the evaluation efforts taking place at the funded sites and bring together data and evaluation results into a national evaluation of the basic concepts of the Safe Kids/Safe Streets program.

Goals: The goals for Phase I of the Evaluation of the Safe Kids/Safe Streets program are:

• To understand the process of effective implementation of the Safe Kids/Safe Streets program in order to strengthen and refine the program for future replication.

• To identify factors that contribute to or impede the successful implementation of the program. It is essential to know not only whether the program is successful or unsuccessful and the degree to which it is successful or not, but also why or how it was successful or unsuccessful.

• To help develop or improve the capability and utility of local data systems that track at-risk youth, including victims of child neglect or abuse.

• To build an understanding of the general effectiveness of the Safe Kids/Safe Streets program and of selected program components and to determine its impact in the participating communities.

• To help develop the capacity of Safe Kids/Safe Streets sites to evaluate what works in their communities.

Objectives: The objectives of this evaluation are:

1. To develop a detailed design, including data collection instruments, for a process evaluation of the Safe Kids/Safe Streets program for implementation in collaboration with all sites.

2. To develop templates for capturing the data necessary for the national process evaluation and to make those templates available for implementation at the sites.

3. To provide evaluation training and technical assistance for--and to collaborate with--grantees at each of the sites to implement a process evaluation of the development and implementation of each Safe Kids/Safe Streets program.

4. To compile and analyze results and provide routine feedback to the sites on the planning, program development, and implementation process. The methods of the formative evaluation should be used to provide feedback to project staff on the adequacy of data sharing among the components of the system in the effort to prevent and prosecute child abuse and neglect.

5. To develop a research design to determine outcomes and the impact of the overall strategy and individual program interventions on delinquency, violence, and related behaviors and risk factors. The design must meet scientifically rigorous standards for evaluation and acknowledge the differences in local circumstances and strategies.

Program Strategy: Applicants should become familiar with the OJP Safe Kids/Safe Streets program and associated literature. Applicants' program strategies should demonstrate an understanding of the collaborative efforts that will be necessary to bring together project implementation staff, project evaluation staff, and the evaluation grantee.

The applicants' strategies must provide a process evaluation that includes all participating sites. Project designs must also include provision for simultaneously assessing appropriate strategies for a future impact evaluation.

Applicants should provide a discussion of research questions for the process evaluation, which will serve as a basis for the data collection plans and instruments. Depending on the funded sites' experiences, the process evaluation design may focus on the following levels of activities:

• Community-level efforts in planning, policymaking, resource allocation, and guidance.

• Program-level responses, experiences, and achievements in program implementation.

Because it is important to know the mechanism through which the changes achieved by the program were accomplished, it is expected that a detailed logic model will be developed at each site in collaboration with program site and evaluation staff and also in a more general form at the national level.

Applicants should include, but are not limited to, the following issues in the evaluation:

• What factors contribute to (or inhibit) changes in the communities in the planning for and delivery of the Safe Kids/Safe Streets program?

• What lessons can be drawn from local communities for Federal and/or State policies, program planning efforts, and local service delivery of intensive services for youth?

• What factors contribute to effective implementation and what changes occur as a result of implementation?

• What planning and implementation strategies (for example, coordination, consultation, use of OJP-provided technical assistance) are used at local levels, and what is the effect of their use?

Applicants must propose a process evaluation that includes the stages of the implementation process, beginning with the selection of targeted sites in the community. Applicants must discuss how they will join with the sites in a collaborative effort and describe the planning and the implementation process to be used at local levels. In addition, applicants should explain how these plans will be implemented and how technical assistance will contribute to the implementation process. Applicants should also discuss the applicability of qualitative methods in this evaluation.

Applicants must include a discussion of their process to develop a preliminary outcome evaluation research design and a description of their methods, including qualitative methods such as ethnography, to conduct an impact evaluation. This description must include an assessment of program goals, measurable impact objectives, data elements and sources for measuring impact, the need for sampling designs and strategies, and a clear statement of the ability of the demonstration sites to support a rigorous impact evaluation. The use of the logic model to relate program activities to program outcomes should be part of this process. In addition to developing one overarching logic model, the grantee should assist each site in developing its own logic model to guide program development and measurement.

Products:

1. Final design of the process evaluation. This design will incorporate modifications recommended by OJP after the award process.

2. Draft comprehensive final report. This report will contain two parts: (1) recommenda-tions regarding the feasibility of conducting an impact evaluation and, if appropriate, a research design, and (2) a detailed account of the process evaluation including overall findings and an analysis of the factors that contributed to or impeded successful imple-mentation.

3. Final report on the process evaluation. This final report will incorporate modifications recommended by OJP and the project advisors, as appropriate.

Eligibility Requirements: OJP invites applications from public and private agencies, organi-zations, institutions, or individuals. Applicants must demonstrate that they have experience in the design and implementation of this type program. Private, for-profit organizations must agree to waive any profit or fee. Joint applications from two or more eligible applicants are welcome, as long as one is designated primary applicant and any others co-applicants.

Selection Criteria: Applications will be evaluated and rated by a peer review panel according to the selection criteria outlined below.

Problem(s) To Be Addressed (15 points)

Applicants must include a clear and concise statement of the problem and demonstrate an understanding of the Safe Kids/Safe Streets program. Applicants also should discuss how to apply state-of-the-art evaluation methods, including qualitative methods, to achieve OJP evaluation objectives and overcome potential problems associated with evaluating these types of programs.

Goals and Objectives (10 points)

Applicants must define goals and objectives for this evaluation program that are clearly defined, measurable, and attainable.

Project Design (35 points)

Applicants must present a clear research design for the conduct of a process evaluation and the formulation of a strategy to carry out a multisite impact evaluation. The design must be sound, feasible, and capable of achieving the objectives set forth in this solicitation.

Management and Organizational Capability (35 points)

Applicants' management structure and staffing must be adequate and appropriate for the successful implementation of the project. Applicants must present a workplan that identifies responsible individuals, their time commitment, major tasks, and milestones.

Applicants must document evidence of the organization's ability to conduct the project successfully. Organizational experience with multisite research and evaluation of youth in the juvenile justice and child welfare or criminal justice system is recommended. Key staff should have significant experience with multisite evaluation/research of juvenile or related criminal justice programs. Applicants must demonstrate the ability to conduct evaluations of compre-hensive, community-based multistrategy initiatives. They must also demonstrate the ability to work effectively with practitioners in resolving design, definition, and data collection and analysis issues and other requirements of the project. Staff r�sum�s should be included.

Budget (5 points)

Applicants must provide a proposed budget that is complete, detailed, reasonable, allowable, and cost effective in relation to the activities to be undertaken.

Award Period: This project will be funded initially for an 18-month budget period for Phase I of a 66-month project period. Funding in the second and subsequent budget periods will depend on grantee performance, availability of funds, and other criteria established at the time of award.

Award Amount: Up to $300,000 is available for the initial 18-month budget period.

Delivery Instructions: All application packages should be mailed or delivered to the Office of Juvenile Justice and Delinquency Prevention, c/o Juvenile Justice Resource Center, 1600 Research Boulevard, Mail Stop 2K, Rockville, MD 20850; 301 251 5535. Note: In the lower left-hand corner of the envelope, you must clearly write "Evaluation of the Safe Kids/Safe Streets Program."

Due Date: Applicants are responsible for ensuring that the original and five copies of the application package are received by 5 p.m. EDT on September 9, 1996.

Contact: For further information call Eric Peterson, Program Manager, Research and Program Development Division, 202 616 3644, or send an e-mail inquiry to [email protected].