Community Based Violence Intervention and Prevention Initiative Performance Measures and Reporting Training

- [Hiba] Hello everyone and welcome to today's Community Based Violence Intervention and Prevention Initiative Performance Measures and Reporting Training. My name is Hiba Sumitro and I am one of the Training and Technical Assistance Specialists and contract support on BJA's Planning, Performance, and Impact team. We are so happy you've joined us today and are excited to present this information as you progress through the July submission period. Please note that the CVIPI program has grantees from various components who are joined, here, with us today. We welcome all grantees to our session today coming from OJJDP, OVC, and BJA. If after today's training you still have questions or think you need additional help, we can provide you with one-on-one assistance. We will provide contact information for our team at the end of the presentation. Today, we will provide an overview of CVIPI reporting requirements. Specifically, we will address what is performance management, purpose of performance measures and benefits to grantees, the what, where, when, and how for reporting, the CVIPI performance measure questionnaire, data quality best practices, and then we will provide our contact information and various resources to assist you with reporting. Then, we will wrap up with a question and answer session. The work you do as grantees is vitally important. Performance measurement helps to tell the story of you, your organizations, and your programs, through data and assessment. Performance measurement is the process by which grantees regularly collect data on grant activities to determine whether they are implementing the program as intended, while achieving the desired goals and objectives. Performance measures are used to capture inputs, outputs, and outcomes over time. Performance measures also allow for pre- and post-program analysis to assess change over time. Performance measures, in the form of questionnaires, have been developed for each grant program. The CVIPI measures are at the link shown, here, and are included in your program solicitation. These measures may be updated throughout the life of your program and we will also drop it into the chat. Understanding the importance of performance measurement ensure strong program development and implementation. Performance measurement provides an opportunity to evaluate raw outputs of program data to improve existing programs and create ideas for new ones. Performance measurement allows you, as grantees, and OJP to answer the who, what, where, and when of grant programs. Performance measures provide a holistic view of your programs to identify areas of success and potential opportunities for improvement. This information allows OJP to develop targeted training and technical assistance to help you as the grantee as you continue through the performance measurement process. Additionally, collecting performance measure data also helps the Department of Justice comply with the reporting requirements of federal law. This includes the Government Performance and Results Modernization Act of 2010, the Digital Accountability and Transparency Act of 2014, the Foundations for Evidence-Based Policymaking Act of 2018, and the Grant Reporting Efficiency and Agreements Transparency Act of 2019. Performance measurement provides a tremendous benefit to you as a grantee. It can help you proactively address challenges and generate evidence that you are meeting your goals and allows you to highlight the impacts of your program and organization over time. Now, let's talk about the what, where, when, and how of reporting grant activity for the CVIPI program. Prime grantees are responsible for managing their subrecipients, communicating with their subs about reporting requirements, and ensuring the quality of subrecipient data. Grantees are also responsible for setting up and managing subrecipient accounts and data reports. For more information on these roles and responsibilities, please visit the link provided, here, and we will also drop it into the chat box. This table outlines the type of data you'll report for CVIPI for each reporting period, when your reports are due in the PMT, and when you will upload your PMT reports to JustGrants. As you can see from the alternating values under the types of data required column, you'll only report on the narrative questions in January, July, and during your last reporting period of grant activity, regardless of where that falls on the schedule. You will also report on closeout questions during your last reporting period. Reporting is based on award start date. During the submission period, grantees will be required to complete three quarterly performance reports in the PMT, one for each quarter that they have not yet reported: October to December, January to March, April to June. Grantees are encouraged to use the Excel data spreadsheet as a job aid to encourage report data and to accurately report data for those three quarters in the PMT by July 30th. Please note that the reports can be completed in any order, but please pay close attention to enter data into the appropriate reporting period when the grant activity occurred. Please note that this reporting cadence is just for the CVIPI program and reports are due 30 days after the end of a reporting period. If no grant funding has been expended, answer "No" to the question, "Was there grant activity during the reporting period?" This will allow you to provide an explanation and skip the measures. The PMT is only open for data entry during the month after a reporting period closes. The PMT is accessible year-round for you to review and edit your data, generate reports, and more. Please also note that if you ever need to go back and revise previously completed reports, you'll need to contact the PMT Helpdesk to help you unlock those reports prior to making revisions. Twice each year in January and July, CVIPI grantees are required to download a PDF of their PMT report and upload it into JustGrants. We will explain more of what this means in just a few minutes. There are three types of questions you will need to respond to: performance measures, narrative questions, and closeout questions. Performance measures are a series of questions that measure outcomes of grant activities and demonstrate the accomplishment of goals and objectives of OJP programs. Grantees report on performance measures during each reporting period. Narrative questions are a series of narrative questions related to the grantee's specific goals. Objectives, barriers, and successes are reported only in January and July. Closeout questions are a series of questions you need to respond to only when activities have been completed and the grant is ending. The performance measure questionnaire outlines all of the questions that grantees are required to respond to. The link to the CVIPI performance measures is shown on your screen and will be dropped into the chat now. We will walk through the performance measure questionnaire in more depth shortly, but please be aware that, as you collect your data in preparation for reporting, we suggest using this questionnaire as a guide to track your responses in your files in the time interval that is easiest for you. When it is time to report in the PMT, this will help ensure that you can add those figures to get the most accurate reporting period totals. As always, if you have questions on any of the performance measures, please don't hesitate to contact the PMT Helpdesk. You'll receive a copy of today's slides with all of these links included in a follow-up email after today's session. The performance measurement tool or PMT is the online legacy reporting system for many OJP components. All performance reporting will eventually move to the Justice Grants or JustGrants system. The timeline for the transition is currently unknown. However, as of today, CVIPI grantees are expected to report grant activity on a quarterly basis in the PMT until further notice. To access the PMT, go to the link on the screen. And if you are a first-time user or your account is inactive, please contact the appropriate PMT Helpdesk for assistance using the addresses shown, here. I would now like to turn the presentation over to my colleague, Jeannine, who will discuss how to report in the PMT.

- [Jeannine] Thank you, Hiba. Now that we've reviewed the what, where, and when of reporting data, I would like to transition into the how I report data. As mentioned previously, you will report your data in the PMT. To access the PMT, visit the PMT single sign-on page at ojpsso.ojp.gov. The single sign-on page allows multiple OJP grant recipients to access all of their awards using one username and password. So, for example, if your organization receives funding from both BJA and the Office of Victims of Crime, you can log in with one username and password to access both of those PMT applications. The point of contact listed in the PMT is the person who will first have contact with the PMT system. If this person is not already a registered PMT user, they will receive an automated email from the PMT with instructions on how to receive access to the PMT. The point of contact may then add additional users who will also receive a registration link via email, to create their own unique account to access the system. Once your unique account is created, enter your registered email address as your username along with your password to log in. We will provide some additional resources on this slide to assist you with your account creation, also adding users and data entry. Once you're logged in, you may see multiple options available to you if your organization receives funding from various OJP components. The CVIPI module is located under BJA PMT. The first profile page of the PMT displays the various awards your organization receives from a particular OJP program. If your organization receives funds from various sources, you'll have access to multiple profiles. The picture on the slide depicts an organization who receives funding directly from BJA, making them a grantee, and who also receives funding from another potential state agency, making them a subgrantee. To continue to enter or edit data, select the grantee or subgrantee profile you want to report on by clicking on the plus sign and then your organization's name. As a note, you will not see this page if you only have one PMT profile or award, but instead be brought right into the performance measurement tool. If you are logging into the PMT for the first time, the system will direct you to the profile page. This information comes from the PMT. If any of it is incorrect, please be sure to contact the PMT Helpdesk or your state policy advisor to update the contact information in the system. Make sure you also inform the PMT Helpdesk along with your state policy advisor or your grant manager of any necessary changes. Having accurate contact information listed in the PMT is extremely important so that you will receive email updates, reminders, notices, and other necessary user materials. If others within your organization also need access to enter or edit data, go to the Manage User page to add them. Only add users who need access to the BJA PMT to complete data entry and reporting. Adding a user automatically sends that person an email with a link to create and complete a user account. We will provide some additional resources in the chat to assist you further in the PMT. Remember, you can always contact the PMT Helpdesk for assistance. To begin the data entry process, go to the Federal Awards page. The purpose of the Federal Awards page is to give an overview of data entry for all awards. Find the reporting period you wish to enter data for and click the red Begin Reporting Process button. If you have any questions about the PMT, the PMT Helpdesk's toll-free number and email address are on the bottom of every page. As mentioned previously, during the submission period, grantees will be required to complete three quarterly performance reports: one of each quarter that they have not yet reported- October to December, January to March, April to June. On this slide is a view of what your CVIPI Federal Awards page will look like. All three reports circled, here, in this slide will need to be completed. Once you click Begin Reporting Process on the Federal Awards page, you'll be brought to the Award Administration page. Depending on whether or not this is your first time reporting or if your program allows subgrantees, you will see one of the two options on the following pages. This page will display if this is your first time entering data for a federal award. The purpose of this page is to tell the system how funds are administered. As you can see, there are three options. Since reporting is tied to funding, if only the recipient uses grant dollars, then they will be the only ones reporting in the PMT for this award. If both the prime grantee and the subrecipient receive and use funds, they will be required to report in the PMT. Prime grantees who use grant funds solely for administration should select option one. Prime grantees with subrecipients will either select option one or option three. Select either the first or third option and then continue. If you are not a new grantee or if your program does not allow for subrecipients, then this page will display upon clicking Begin Reporting Process. On this page, grantees are only provided one option for reporting the allocation of grant funds. In addition, the PMT will also ask whether this is the last reporting period of data you have to report. Select "No" to create a quarterly report and select "Yes" to create a final PMT report. The final report is the last report you will generate in the PMT, so grantees should only create a final report if all funds have been expended prior to the last day of the reporting period. Otherwise, select "No" to proceed with your quarterly report. Every 6 months, in January and July, you will need to generate a PDF of the PMT report, save it to your computer, and upload it in JustGrants as an attachment to the awards performance report. PDF reports cannot be created until all past data entry and report status are complete. Once data entry for all three reporting periods is complete, grantees will then create and save the PDF reports during the October to December 2022 reporting period and the April to June 2023 reporting period. Please note that April to June 2023 report will auto-generate the semiannual January to June 2023 report. Grantees should then attach both PDFs to the corresponding Award Performance reports in JustGrants by July 30th. The PDF function is not currently available, but will be before the end of submission period. You may want to enter grant activity now and then in the coming weeks, log back in and generate the PDF reports. The JustGrants team developed a training that provides step-by-step overview of this process. The link can be found on this screen. Since JustGrants is a completely separate system and managed by a different team, we encourage you to reach out to the JustGrants Support Team for any issues or questions related to the JustGrants system. A phone number and email are listed, here, on this slide. In the JustGrants system, go to the Performance Report page of your grant award. Type any relevant comments in the Comments field. In the Attachment section, select Upload to upload the semiannual PMT PDF report that you just saved. Be sure to align the reporting period dates in the PMT PDF reports to the same reporting period performance report dates in JustGrants. Also, be sure to select Submit when submitting a report. Selecting Save does not submit your report. Once submitted, the report is routed to the Grant Manager for review. I'm now going to turn the presentation over to Angie for an overview of each of the sections in the performance measures.

- [Angie] Thank you. Now, let's take a look at CVIPI's unique questionnaire structure. As mentioned earlier, the CVIPI questionnaire contains performance measures, narrative questions, and closeout questions. Listed on this slide are the various sections of questions that will be in the questionnaire. Remember that you can access a PDF version of the questionnaire to assist you for when it is time to report your data. Within the questionnaire, there are three types of questions you may encounter: multiple choice questions, where grantees should select the response that best reflects their activities; multiple response questions, which allow grantees to select all responses that reflect their activities; and finally, open-text questions, where grantees may be requested to input numeric or text responses. Since this is a new program and the first-time grantees are entering data, all grantees should answer "No" to question 1, "Is this the last reporting period for which the award will have data?" For question 2, "Was there grant activity during the reporting period?", grantees should answer "Yes" if they have obligated, expended, or drawn down grant funds to implement objectives that were proposed in the approved grant application. If no grant funding has been expended during the reporting period for which you are responding to, answer "No" to the question. This will allow you to provide an explanation for no activity and skip the measures. The next section is the General Agency and Award Information. Grantees will select their CVI grant award categories. Categories 1 and 2 are for CVIPI Planning and Implementation awards. Categories 3 and 4 are for CVIPI Expansion and Enhancement awards. Categories 1 through 4 will complete the survey. Categories 5 through 7 report in the Training and Technical Assistance portal, and then upload their report to the JustGrants system. Categories 3 and 4 will complete an additional section related to the evaluation of CVIPI enhancement and expansion. You'll also indicate your primary target area of your CVIPI program, the name of the jurisdiction or jurisdictions, and the population. In the next section, Violence Reduction Strategic Planning, grantees will need to describe the status of their Violence Reduction Strategic Planning activities. Since grantees will be back reporting, only report of the status of your planning activities during the reporting period for which you are reporting and not the current status of the activities. You'll have an open-text field to describe the activities that you conducted as part of your strategic plan. You will then enter the number of community resident members consulted or involved in your strategic planning, and select key partners in the multiple response checklist. In the Data Analysis and Planning Activities section, grantees will complete this, if they conducted data collection or analysis to inform their violence reduction strategic plan. Answering "No" will skip this set of questions and lead you to the next section. The CVIPI Policies, Procedures, and Training section focuses on new CVIPI staff and partners. Grantees will enter the number of new CVIPI staff or partners in question 20 and the total number of CVIPI staff or partners in question 21 during the reporting period. If all CVIPI staff and partners were new during the reporting period, responses to question 20 and 21 would be the same. The Community Collaboration and Engagement section focuses on community engagement activities. Community engagement refers to building or maintaining relationships within the community of focus, including community members and community-based organizations. Grantees will select, "Not Applicable," "Do Not Know," "Daily," "Weekly," "Monthly," or "Quarterly," for each of the activity options in question 24: "How often was your CVIPI team involved in the following community activities during the reporting period?" In the next section, grantees will need to input the number of individuals from each type of group that were represented on their multidisciplinary team. Grantees should enter zero if no individuals were represented from a particular group. The PMT will not allow you to complete your report if a number is not entered. Grantees will be prompted to return to the question and enter data. In the Targeted Violence Interventions and Supports for Highest-Needs Groups section, you will indicate the types of CVIPI strategies that were employed during the reporting period such as hospital-based violence intervention programs or violence interrupters. These strategies may change or remain consistent throughout your grant award. Grantees will then enter the number of at-risk people engaged and the number of credible messengers that were part of the program during the reporting period. OJP works with TTA programs in order to meet the unique needs of grantees and other criminal justice practitioners and agencies. The objective of OJP's TTA services are to provide information to grantees and criminal justice professionals, to replicate models and approaches, increase knowledge and the use of best practices, emerging technologies and new models, support agencies and jurisdictions to identify and address operational and programmatic needs. In order to help with tracking, the Training and Technical Assistance section of the performance measures should be answered if any OJP-funded TTA services were rendered during the reporting period. The Evaluation of CVIPI Enhancement and Expansion Awards section only applies to categories 3 and 4 that have a research partner as part of their program. Categories 1 and 2 do not respond to this section. Finally, the Semiannual Narrative Questions section are required to be reported on in January, July, and during the last period of grant activities. Based on the grant-funded activities, grantees will need to identify their top three accomplishments and goals during the program throughout the reporting period. This is an opportunity for grantees to highlight their successes. It can also allow you to identify barriers and lessons learned. At the end of your award, grantees will be required to answer a series of questions related to sustainability and impacts. This section will only be visible when grantees answer "Yes" to question 1, "Is this last reporting period for which you'll have data to report?" I'll now turn it back over to Jeannine to close up best practices.

- [Jeannine] Thank you, Angie. For the narrative questions, we want to set you up for success and offer a few tips and lessons learned. It is important to write well-defined goals and objectives to clarify your priorities and highlight criteria for success from the very beginning. The SMART mnemonic walks users through important dimensions of good goal and objectives. Your goals should be specific, measurable, achievable, relevant, and time-bound. We recommend that you reevaluate your goals twice a year to determine whether they should be updated. Use your data to drive this process. Here are a few examples of well-defined SMART goals that will set this program up for success. While the goals on the right are great goals, they're difficult to measure and not specific enough to measure success. The goals on the left are specific, measurable, achievable, and relevant to the program. Be sure to ask yourself, what is the program seeking to accomplish? How will we measure the progress? Is it reasonable? Does it align with the overall goal of the program's objective? And is there enough time to accomplish the goal? We would also like to share some tips to improve data quality that your program can do from the very beginning. It is recommended that a designated staff person coordinates all performance measure data collection and entry to ensure consistency. If this is the first time you will report data, make sure you're familiar with the data you will need to collect and report. Do this by reviewing the PDF version of the questionnaire. Ensure a backup person is aware of the data collection and reporting process so they can fill in if the designated staff person is unavailable or leaves the role. Consider available data collection methods including case management system or other databases, spreadsheets, or tracking intake forms. If partner organizations are included in your program design, be sure to engage them from the start in the planning process. Determine if formalized agreements are needed to ensure the necessary data is collected and the program meets its goals and objectives. Lastly, ensure all data you receive from contracted service providers is reviewed and validated before completing data entry. I'd like to turn the presentation back over to Hiba for some resources and other helpful information.

- [Hiba] Thank you so much, Jeannine. To sum up our discussion today, here are some of the great resources we already have available to assist you with reporting grant activity. The BJA Performance Measures webpage, for example, contains some foundational information about performance reporting at BJA, along with various resources, including links to the PMT and the JustGrants reporting system, contact information for both the PMT and JustGrants Helpdesks, as well as the BJA YouTube and multimedia pages. Here, you can also find archive trainings on program-specific performance measures, performance report submission, and subgrantee management. Please note that resources are also available to grantees specifically joining us from OVC and OJJDP that have likely also been received or will be received in the upcoming days or in upcoming TTA products unique to those programs. I also encourage you to view our subgrantee reporting video found, here, as prime grantee on your program subgrantees should direct the award questions to you, the prime, before contacting the PMT Helpdesk. The PMT Helpdesk has limited capabilities to help subgrantees outside of creating or unlocking accounts. So, if you have any specific questions about your subawards or subgrantee reporting, please don't hesitate to contact your state policy advisor or Grant Manager or the PMT Helpdesk. Since CVIPI grantees will be expected to submit a copy of the reports into JustGrants by the end of July 2023, here is the JustGrants support team's contact information. As a reminder, this team will be able to support any challenges with submitting performance reports, reporting, or navigating the JustGrants system overall. Before we begin our Q&A session, we would like to take a moment to, once again, thank you all so much for your time, dedication, and commitment to performance reporting and for joining us today. If you have follow-up questions after today's presentation, please contact us through the PMT Helpdesk. Our office hours are Monday through Friday 8:30 a.m. to 5:00 p.m. Eastern Time except for federal holidays. We strive to return every email within 24 hours, so please don't hesitate to send us an email even after business hours. As mentioned previously, you'll receive a PDF copy for today's presentation with links in the days ahead. At this time, we will transition to our Q&A session. If you have not already done so, please use the Q&A panel to submit your questions to all panelists. We just received a question from Royal, "I don't have a profile to select to complete when I'm logged into the PMT to be able to complete a report. Is this what we have to to wait a few days for?" I think, Liz, that would be a good question for you.

- [Liz] Royal, I can work on setting a user account up for you currently if you don't already have one. You'll get an automated email with a link to complete registration, so be on the lookout for that in the next few minutes, sometime today.

- [Hiba] We also received another question from Catherine, "If our reports are due July 30th, a couple of weeks to wait for the slideshow seems problematic." Yes, so I understand why that would be a concern. We hope to send the slideshow actually in a day or two, so you should receive a follow-up email by the end of this week at latest with a copy of the slide deck, so you will have that available to you very soon. And if there are any other questions that come up or concerns that need immediate attention, you are, of course, welcome to contact the helpdesk and they can help you along your way. Summer, I see your question. Liz, I think you will be sending all of those links out shortly after today, is that correct?

- [Liz] Yeah. If you already have a user account in the PMT for another award you were reporting on, it should already be linked to your existing account. If not and you're a completely new user, if we haven't already sent you an automated email either yesterday or this morning, you will get those today.

- [Hiba] Perfect. Royal, it looks like you have another question, "I have got the email to log in, but when following the steps, I don't have any reports to complete on the pages." I would definitely recommend contacting the helpdesk for that sort of question. Sometimes it just requires things that need to happen behind the scenes for that to show up. But Liz, is there anything else you would like to add on that?

- [Liz] Yeah, Royal, I just fixed your account. If you would log out and try again, you should have access to the system now.

- [Hiba] Thank you, Liz. We will go ahead and stop recording at this time.