Byrne State Crisis Intervention Program (SCIP) Performance Measures and Reporting Training

[Christina] My name is Christina Mariscal and I'm the task lead providing contract support on BJA's Planning, Performance, and Impact Team. We're happy that you've joined us today. A PDF copy of today's slides will be distributed to you following the presentation. This training is also being recorded and will be posted on BJA's YouTube channel in the coming weeks. If you run into any technical issues, please reach out to Sarashaniece Coleman, our Webex system facilitator for today. I encourage you to send her a private message via the chat function or send her an email to the address listed, here. All attendees are muted to prevent background noise during today's presentation. Please note that we'll respond to any questions you have via the Q and A panel. We will try to answer questions as they come in throughout the session, or we will respond during the facilitated Q and A session at the end. If you do not see Q and A on the right side of your screen, navigate to the bottom right of the Webex window. You should see the symbol with the three dots, and if you click on that, you can add the Q and A panel. To submit a question, please use the Q and A panel by clicking on the tab labeled "Q & A" located on the bottom right side of the Webex window and choose "All Panelists" to share the question with the larger group. If we are unable to answer your question during the allotted time today, we will follow up with you individually. Additionally, if you need more help after today's training, we can provide you with one-on-one assistance. We will provide our contact information at the end of the presentation. We will use the chat box to share links and resources with you during the session, so please make sure to check the chat box and then submit your questions using only the Q and A panel. During today's presentation, we will provide an overview of Byrne SCIP reporting requirements. Specifically, we will address performance management at the Bureau of Justice Assistance, so what, where, when, and how for reporting; the Byrne SCIP performance measure questionnaire; reporting in the performance measurement tool; data quality best practices. We will also provide our contact information and various resources to assist you with reporting and we will conclude with a question and answer session. We encourage you to drop your questions in the Q and A panel as they come up throughout the training today. The work you do as grantees is vitally important and performance measurement helps to tell the story of you, your organizations, and your program through data and assessment. Performance measurement is the process by which grantees regularly collect data on grant activities to determine whether they're implementing the program as intended and achieving the desired goals and objectives. Performance measures are used to capture inputs, outputs, and outcomes over time. Performance measures also allow for pre- and post-program analysis that can be used to assess change over time. BJA has established performance measures for each grant program. These measures can be found at the link shown, here, and are included in your program's solicitation. These measures may also be updated throughout the life of your program. Understanding the importance of performance measurement ensures strong program development and implementation. Performance measurement provides an opportunity to evaluate raw outputs of program data to improve existing programs and create ideas for new ones. Performance measurement allows you, as a grantee, and BJA to answer the who, what, where, and when of your grant program. For BJA, performance measures provide a holistic view of your program to identify areas of success and potential opportunities for improvements. This information allows BJA to develop targeted training and technical assistance to help you as a grantee as you continue through the performance measurement process. Additionally, collecting performance measure data also helps the Department of Justice comply with the reporting requirements of federal law. This includes the Government Performance and Results Modernization Act of 2010, the Digital Accountability and Transparency Act of 2014, the Foundations for Evidence-Based Policy Making Act of 2018, and the Grant Reporting Efficiency and Agreements Transparency Act of 2019. Performance measurement provides a tremendous benefit to you as a grantee and can help you proactively address challenges and generate evidence that you are meeting your goals, and it also allows you to show the impact of your program and your organization. Now, let's talk about the what, where, when, and how of reporting grant activity for the Byrne State Crisis Intervention Program. Prime grantees are responsible for managing their subrecipients, communicating with their subrecipients about reporting requirements, and ensuring the quality of subrecipient data. Prime grantees are also responsible for setting up and managing subrecipient accounts and data reports. For more information on the prime and subrecipient roles and responsibilities, please visit the link provided, here, for the OJP Subawards Toolkit. There are two types of questions you'll need to respond to: performance measures and narrative questions. The performance measures are a series of questions that measure outcomes of grant activities and demonstrate the accomplishment of goals and objectives for BJA programs. Grantees report on performance measures during each quarterly reporting period. A series of narrative questions related to the grantee's specific goals, objectives, barriers, and successes are reported on semiannually in January and July. The Performance Measure Questionnaire outlines all of the questions that grantees are required to respond to. The link to the Byrne SCIP performance measures is shown on your screen, here, and we'll also drop it in the chat. As a reminder, you'll receive a copy of this presentation with all of the links following this session. We will also do a walkthrough of the Performance Measure Questionnaire in a few moments. We suggest using the questionnaire as a guide to track relevant data in your files, in the time interval that's easiest for you. Then when it's time to report in the PMT, you can add those figures and get your reporting period totals. If you have questions on any of the performance measures, please reach out to the BJA PMT Helpdesk and the email address is listed, here. The performance measurement tool or PMT is the online legacy reporting system for BJA and other OJP components. All performance reporting will eventually move to the Justice Grant or JustGrants system. The timeline for the transition is currently unknown, however, Byrne SCIP grantees report grant activity in the PMT until further notice on a quarterly basis. To access the PMT, visit the link on the screen. If you are a first-time user or your account is inactive, please contact the PMT Helpdesk for assistance. Every 6 months, in January and July, you'll generate a PDF of the PMT report, save it to your computer, and then upload it into JustGrants as an attachment to your awards performance report. For Byrne SCIP grantees, the performance report module in JustGrants is still in development. Therefore, you will not need to upload a PDF of the PMT report into JustGrants until July of this year. At that time, you'll be asked to report on data from October 1st, 2022, through June 30th, 2023. As you are reporting in April, you should save two PDFs from PMT: one for the October to December period of 2022, and the second for the January through March 2023 period and attach both to the performance report in JustGrants to cover the period once it is due in July. The JustGrants team also developed a training that provides a step-by-step overview of the process for uploading PMT reports in JustGrants, and the link can be found, here, at the bottom of your screen. Since JustGrants is a completely separate system and managed by a different team, we encourage you to reach out to the JustGrants support team for any issues or questions related to the JustGrants system and we will provide contact information later on in the presentation for the JustGrants Support Team. When you're submitting your report in JustGrants, you'll go to the performance report page of your grant award. You'll type in any relevant comment in the comments field. In the attachment section, you'll select "Upload" to upload the semiannual PMT PDF report that you just saved. You should only attach one PMT PDF report. Be sure to align the reporting period dates in the PMT PDF report to the same reporting period in JustGrants. Also be sure to select "Submit" when you are submitting a report, as selecting "Save" does not submit your report. Once your report is submitted, it will be routed to your grant manager for a review. This table outlines the type of data you'll report for each reporting period, when your reports are due in the PMT, and when you'll upload your PMT reports to JustGrants. As you can see in the second column, you'll only report on narrative questions in January, July, and during your last reporting period of grant activity, regardless of where that falls on the schedule. Note that reports are due 30 days after the end of a reporting period. Also, reporting is based on award start date. Although Byrne SCIP awards were made in 2023, grantees will need to report for two periods ahead of the April 30th deadline, since the award started on October 1st, 2022. Grantees will need to enter data for October 1st through December 31st and January 1st through March 31st, even if there was no grant activity. If no grant funding has been expended, answer "No" to the question, "Was there grant activity during the reporting period?" and this will allow you to provide an explanation and skip the performance measures. Please note the PMT is only open for data entry during the month after a reporting period closes. The PMT itself is accessible year round for you to review and edit your data, generate reports, and more, but you will need to contact the PMT Helpdesk to help you unlock reports when you need to revise the reports that were previously submitted. Now, I'd like to transition to my colleague Jeannine Bulbulia, who will walk you through how to report on your grant activity.

[Jeannine] Now that we have reviewed the what, where, and when of reporting data, I would like to transition into the how do I report data. As mentioned previously, you'll report your data in the PMT. To access the PMT, visit the PMT's single sign-on page at ojpsso.ojp.gov. The single sign-on page allows multiple OJP grant recipients to access all of their awards using one username and password. So, for example, if your organization receives funding from both BJA and the Office of Victims of Crime, you can log in with one username and password and access both of those PMT applications. The point of contact listed in the PMT is the person who will first have contact with the PMT system. If this person is not already a registered PMT user, they will receive an email from the PMT with instructions on how to create a new account when contact information is updated in the PMT. The point of contact may then add additional users who will also receive a registration link via email to create their own unique account to access the system. Once your unique account is created, enter your registered email address as your username along with your password to log in. We have provided some additional resources on this slide to assist you with account creation, adding users, and data entry. The first profile page of the PMT displays the various awards your organization receives from a particular OJP program. If your organization receives funds from various sources within BJA, you'll have access to multiple profiles. This picture on the slide depicts an organization who receives funding directly from BJA, making them a grantee, and who also receives funding from another potential state agency, making them a subgrantee. To continue to enter or edit data, select the grantee or subgrantee profile you want to report on by clicking on the "plus" sign and then your organization's name. As a note, you will not see this page if you only have one PMT profile or award, but instead, be brought right into the PMT. To begin the data entry process, go to the Federal Awards page. The purpose of the Federal Awards page is to give an overview of data entry for all awards. Find the reporting period you wish to enter data for and click the red "Begin Reporting Process" button. If you have any questions about the PMT, the PMT Helpdesk's toll-free number and email address are at the bottom of every page. Once you click Begin Reporting Process on the Federal Awards page, you'll be brought to the Award Administration page. Depending on whether or not this is your first time reporting and if your program allows subgrantees, you'll see one of two of the following pages. This page will display if this is your first time entering data for a federal award. The purpose of this page is to tell the system how funds are administered. As you can see, there are three options. Since reporting is tied to funding, if only the subrecipient uses grant dollars, they will be the only ones reporting in the PMT for this award. If both the prime guarantee and the subrecipient receive and use the funds, then both will be required to report in the PMT. Prime grantees and subrecipients will either select option one or option three. Select either the first or third option and select "Continue." If you are not a new grantee or if the program does not allow for subrecipients, then this page will display upon clicking Begin Reporting Process. On this page, grantees are only provided one option for reporting the allocation of grant funding. In addition, the PMTs will also ask whether this is a last reporting period of data you have reported. Select "No" to create a quarterly report. Select "Yes" to create a final report in the PMT. The final report is the last report you will generate in the PMT, so grantees should only create a final report if all funds have been expended prior to the last day of the reporting period. Otherwise, select "No" to proceed with your quarterly report. If you are logging into the PMT for the first time, the system will direct you to the profile page. This information comes from the PMT. If any of it is incorrect, please be sure to contact the PMT Helpdesk or your State Policy Advisor to update the contact information in the system. Make sure you also inform the PMT Helpdesk, along with your BJA State Policy Advisor, of any necessary changes. Having accurate contact information listed in the PMT is extremely important so that you receive email updates, reminders, notices, and other necessary user materials. If others within your organization also need access to enter or edit data, go to the Manage Users page and add them. Only add users who need access to the BJA PMT to complete data entry and reporting. Adding the user automatically sends that person an email with a link to create and complete a user account. We'll provide some additional resources in the chat to assist you further in the PMT. Remember, you can also contact the PMT Helpdesk for assistance. Now, let's take a look at the Byrne SCIP unique questionnaire structure. As mentioned earlier, the Byrne SCIP questionnaire contains performance measures and narrative questions. Listed on this slide are the categories that will be in the questionnaire. Remember, you can access the PDF version of the questionnaire to assist you when it's time for you to report your data. Within the questionnaire, there are three types of questions: multiple choice questions, where grantees should select the response that best reflects their activities; multiple response questions, which allows grantees to select all responses that reflect their activities; and lastly, open text questions where grantees may be requested to input numeric and text responses. The first unique section to Byrne SCIP I would like to review is the Partnership Activities section. As part of the award, Byrne SCIP grantees are required to establish an advisory board. Grantees must form a diverse board that includes, but is not limited to, representatives from law enforcement, the community court, prosecution, behavioral health providers, victim services, and/or legal counsel. In the Partnership Activities section of the performance measures, grantees will have to report whether the advisory board has been established. If grantees have not established an advisory board, they'll be required to explain what steps are being taken to establish one. We want to make a note of an error in the language in the existing measures. Question three, in the Partnership Activities section, should read, "have you established the required Crisis Intervention Program Advisory Board to help guide your program?" We'll be making the correction in an upcoming fix in the PMT. However, we wanted to make you aware that this is referring to the required advisory board mentioned in the solicitation. In the Crisis Intervention Programs section, grantees will identify new and existing crises intervention program activities that are using program funds. Provide additional program activities in the other section, if program funds were used by program activities not listed. Grantees will also identify if the grant funded measures to safeguard the constitutional rights of an individual subject. In the Extreme Risk Protection Orders section, if grantees did not use program funds to carry out ERPOs, the PMT will automatically skip the section. For example, question 14, in the Extreme Risk Protection Orders asks, "During the reporting period, were program funds used to carry out ERPOs?" If you answer "No," you will not be prompted to answer question 15, which asks the type of ERPO activities funded. This ensures grantees only answer questions pertaining to the activities conducted that reporting period. Grantees are not required to answer the ERPO questions if they did not use funds to carry them out. The final section I would like to review is the Semiannual Narrative Questions section. The Semiannual Narrative Questions are required to be reported in January, July, and during the last reporting period of grant activity. Based on the grant-funded activities, grantees will need to identify the top three accomplishments of the program during the reporting period. This is where grantees get to highlight their successes. This is also the opportunity for grantees to identify barriers and lessons learned. Now, let's go over some best practices. For the narrative section, we want to set you up for success and offer a few tips and lessons learned. It's important to write well-defined goals and objectives to clarify your priorities and highlight criteria for success from the very beginning. The SMART mnemonic walks users through important dimension of good goals and objectives. Your goals should be specific, measurable, achievable, relevant, and time bound. We recommend that you reevaluate your goals twice a year to determine whether they should be updated. Use your data to drive this process. Here are a few examples of well-defined smart goals that will set this program up for success. While the goals on the right are great goals, they're difficult to measure and not specific enough to measure success. The goals on the left are specific, measurable, achievable, and relevant to the program. Be sure to ask yourself, what is the program seeking to accomplish? How will we measure the progress? Is it reasonable? Does it align with the overall goal of the program's objective? And finally, is there enough time to accomplish this goal? We would also like to share some tips to improve data quality that your program can do from the very beginning. It's recommended that a designated staff person coordinates all performance measures, data collection, and entry to ensure consistency. If this is the first time you will report data, make sure you're familiar with the data you will need to collect and report. Do this by reviewing the PDF version of the questionnaire. To ensure a backup person is aware of the data collection and reporting process so they can fill in if the designated person is unavailable or leaves the role. Consider available data collection methods, including case management systems or other databases, spreadsheets, or tracking intake forms. If partner organizations are included in your program design, be sure to engage them from the start in the planning process. Determine if formalized agreements are needed to ensure the necessary data is collected and the program meets its goals and objectives. Lastly, ensure all data you receive from contracted service providers is reviewed and validated before completing data entry. I'd like to turn the presentation back over to Christina for resources and other helpful information.

[Christina] Hi, thanks Jeannine. Before we transition to our Q and A session, I just want to point out a few resources that may be helpful to you. And this is sort of a cumulative list of some of the things we've mentioned throughout the presentation today. The one thing I want to call out is the BJA Performance Measures webpage or website. That's the first link here that contains a lot of foundational information about performance reporting at BJA, and you'll also be able to navigate to your performance measures from there, you'll be able to navigate directly to the performance measurement tool and the JustGrants system, as well as to BJA's YouTube page. And that is where this training will be archived, on BJA's YouTube page and the multimedia page as well. We also want to share the contact information for the JustGrants Support Team. As mentioned previously, please reach out to the JustGrants team for issues related to reporting in JustGrants. And this includes getting set up with an account, updating users in JustGrants, so anything JustGrants related, please reach out to the JustGrants Support Team. And here is our contact information. Our office hours are Monday through Friday, 8:30 AM to 5:00 PM Eastern Time, except for federal holidays. We strive to return every email within 24 hours, so please don't hesitate to send us an email even if it's after business hours. And as mentioned, you'll receive a PDF copy of today's presentation and you'll have the ability to click on all the live links in that. We'll send it via a PDF so that you can use that as a resource as well. And our email address is here: [email protected]. So, we will transition to the Q and A. The first question that we have, one user is saying that, in the PMT, only the October through December report is showing and there is no report for January through March. The January through March period is live in the PMT. We did check this specific user account, it's appearing there when we're looking at it. You may want to refresh your webpage. Also, it's ideal if you use Chrome as your browser, but if you go through these steps and still do not see it, please reach out to us and we can do a screen share or walk you through how to get access to that. The next question is regarding that only one PDF should be uploaded to JustGrants for the semiannual report. However, with the development not being ready yet, will the July report be an exception? Will we need to upload multiple PDFs for that first October 2022 to June 2023 period? And the response here from one of our team members, the PDF function should be available by the July reporting period. And at that time, you'll be able to upload two different PDFs into JustGrants. One report will be for the October to December 2022 period, and then you'll be able to create one PDF semiannual report including both the January through March 2023 and April through June 2023 reporting periods. So, technically each of those would be uploaded separately. So, two separate PDF reports uploaded separately, not under one performance report in JustGrants. Okay, it looks like maybe Michelle, we need to follow up with you if October to December is the only reporting period that you're seeing. And if anyone else is having that issue, feel free to let us know that as well, but it may be just an isolated incident here. Okay. Question, will there be any modules over funding like there are in other BJA progress reports? I'm not sure if Angie or anyone else on the team has an answer to that, but I don't think there are plans right now to revise these measures. So, what you're seeing is what you're reporting on. I know that there are funding modules in JAG and for other programs, but there will not be for Byrne SCIP at this time. I'm not sure if there's a plan to change that. Anyone feel free to jump in or raise hand. I can unmute you. But the performance measure questionnaire is what you should follow for right now. Okay, under the Award Administration page, do you select "funds are being used by grantee and subrecipient" if, as a grantee or prime recipient, you are only using administrative funds, or do you check that all funds are being used by subrecipients?

[Jeannine] This is Jeannine. From my understanding, you're asking if you use like 10 percent of your funding for administration, you should be reporting that administration cost. So, it would be both. It would not be all funds are being used by subrecipients.

[Christina] Thanks, Jeannine. Okay, and it looks like another grantee is having the same issue as Michelle, so we can check on that and follow up with you. Like I said, if anyone else is having a similar issue, please let us know. We are able to, on the backend, sort of shadow your account so we can essentially log in as you and see what you're seeing, and when we do that, it does look like it's there, so I'm not sure why it's not appearing on your end. So, we may have to troubleshoot that. But the best I can say, and it looks like Michelle tried but it didn't work, but to use Google Chrome and sort of refresh your browser. Okay, looks like we have another question for the team here. If we are using admin funds now as allowed but no activity other than that, will we say no activity yet? Jeannine or Angie?

[Angie] So, I believe that if you do not have grant activity aside from the allowable administrative funds, please do respond, "no grant activity for the reporting period."

[Christina] Okay. So, a question just to confirm, the first report is due in July, covering October to March and March to June. We're talking about two separate things. So, you're going to report your grant activity in the PMT, there will be two separate reports due in the PMT. One for October through December of 2022, and then one for January through March of 2023. That's what you're going to see now, or should see now in April. And those are due ahead of the April 30th deadlines. When you go to report in JustGrants, because that module is currently, for this specific award, currently in development, the PR, the performance reports are currently in development in JustGrants, you will need to go back and you'll report from October 2022 through June. That's correct. Nothing is due in JustGrants at this time. So, you do not need to submit the under-construction report. If there are other grantees who when you log into the PMT and you're not seeing the January through March period in the PMT, if you could please let us know, that would be great, and then we can troubleshoot. But it should be live and ready to go, and we are seeing it on most everyone's account, so we just have to figure out why you aren't seeing it on your end. Okay, it looks like there might be, sounds like there might be a bug since this module was just rolled out and released, so we will troubleshoot. Thank you so much everyone. So, our email address is here. Yep. I would share ours since they don't need to report in JustGrants just yet. We're trying to triage some of these concerns with the PMT and the January through March report. So, we will do that and we will let you all know what we find and we will do that ASAP. If you have any additional questions, please feel free to share them with us. At any point, if we aren't able to respond, we'll get it to someone who can. And we thank you for your time and wish you happy reporting.