Edward Byrne Memorial Justice Assistance Grant (JAG) Program Performance Measure Training

MICHELLE GARCIA: Welcome to everyone, thank you for joining us today to discuss the Edward Byrne Memorial Justice Assistance Grant Program Performance Measures. Next slide, please. 

Today we'll provide an overview of the process that we engaged in with many of you on this webinar to revise the Byrne JAG Performance Measures. We'll go through what is now going to be collected, the what, where, when, and how of it. We'll demonstrate how you will submit your information. And then as was already noted, we've built in a significant amount of time at the end to answer any questions that you might have about the new measures and the process. Next slide. 

Most of today's session will be provided by Angie Balchi and James Steyee. And before we kick off the nuts and bolts of the new performance measures, I'd like to welcome and turn it over to the BJA Director Karhlton Moore for some opening remarks, Karhlton? 

KARHLTON MOORE: Thank you, Michelle. And let me thank everyone for joining us and certainly the entire team here at BJA and our partners for the work that has gone into putting this webinar on, but also the work that has gone into getting to the point where we've been able to significantly reduce the amount of information that we're requesting. And you'll all have a chance to learn about that through the training. 

I think for many of you, you may know that prior to coming to BJA, I was the director of the Ohio Office of Criminal Justice Services, and there I was responsible for submitting PMT data to BJA. And I heard from my team who were the people who were actually tasked with doing this work and the challenges, the timeframes, the amount of information, the number of subgrantees who we were gathering information from in order to submit to BJA in a timely fashion. And so when I got to BJA, this is one of the things that I was really focused on in making sure that we were asking for things that we needed, not just things that would be neat to know or neat to have. And certainly when I got to BJA, my team here was also very much focused on trying to streamline this process to make it more manageable and more meaningful that the data that we would collect from each and every state would be information that we could rely on even more so. 

So, we went and started through this process, and I just want to say that the experience from Michelle, who was the head of an SAA office, and Jimmy who also worked at an SAA office, and certainly other people who are part of the BJA team, I found an incredibly willing group of people who wanted to reduce the burden that PMT was presenting to people in the field. And so they were anxious and eager to begin this work, and we really did this the way that things are supposed to get done. We didn't just sit in Washington D.C. and decide that these are the most important things or this is the way we're going to change things. But we reached out and we heard from many of you. We worked with partners in the field in order to make sure that we were doing this the right way with feedback and input and guidance from people in the field, also recognizing the incredible knowledge and experience and expertise that was at BJA. And I think what all of you, all of us collectively have come up with, is a great advancement in PMT that will result in more efficient use of your time, but also information that we can rely on, even, even more. So I want to thank everyone involved for being a part of this important process. JAG is where we're starting, but it's certainly not where we are ending, and we'll continue to look at other grant programs in order to make sure that we are collecting information that is necessary and needed, but at the same time, we are doing it in a way that is efficient, and information that we can utilize in order to make the case for the incredible work that you all are doing across the country. So thank you to everyone who is joining us here today. I can't imagine that there are other things that you'd rather be doing than being a part of a webinar on PMT. That was sarcasm, by the way, but I appreciate your presence here and I look forward to working with each and every one of you, Michelle? 

MICHELLE GARCIA: Thank you so much, Karhlton. We know everybody is thrilled to know that there are going to be fewer questions, and so let's dive in now to talk about both our process and where we've ended up. And with that, I'll turn it over to Jimmy. 

JAMES STEYEE: Thank you, Michelle and also Karhlton. We appreciate your time and comments and I'll just echo a lot of what Karhlton said. So thank you for making my job a lot easier. Next slide, please. 

Looks like my video may have frozen, but I apologize for that. I'll just keep moving forward. The JAG performance measures review process took place over the last year and involved direct feedback from JAG grantees and stakeholders much like yourselves. The goal of the process was to revise and develop performance measures that are streamlined while being informative to all of the various stakeholders that BJA has while also being responsive to federal reporting laws and requirements. The intent of the revisions was to make reporting less burdensome and to improve your experiences with data collection and reporting, hopefully saving you time and resources. Next slide, please. 

BJA tried to gain as much information and feedback from stakeholders as possible as we developed the new JAG performance measures. The review process included three forms of feedback: a steering committee, an online feedback form, and an analysis of the current reporting. Steering committees with JAG SAAs and local grantees were developed with 10 members each. Six meetings took place with the steering committee members to present information and gather feedback. An online feedback form containing 12 multiple-choice and open-ended questions on current JAG reporting was sent to the SAAs and local direct awards. A total of 61 responses were received and analyzed. We used the analysis of the current reporting mechanisms to understand questions that are often not used and find gaps in information we're often asked in trends and current reporting such as commonly used categories, common categories of purchased equipment, and other things. Next slide, please. 

Based on the feedback we received through the form and in a series of meetings with our steering committees, we synthesized all of that information into a few recommendations, which I'll quickly cover here. You'll see how we've incorporated them into our revised reporting structure in the coming slides. 

First, we wanted to improve the user experience. While we know reporting in the PMT and then having to upload reports into JustGrants is cumbersome, this will not change. But at the time, at this current time, the PMT is the best tool we have that allows for subgrantee reporting. However, we have reduced the number of measures and ensured that we have clearer logic and skip flows to navigate through the measures. Further, we have reduced the number of sections so that you're not navigating through as many unrelated questions that do not apply to you. We also wanted to ensure the relevancy of the questions. Related to our efforts to streamline the number of questions and sections, we wanted to ensure the data that is collected is usable and important to BJA and other stakeholders. 

Similarly, we wanted to ensure data are actually utilized. BJA thought through the types of data that we need and that we will use. With that, we came up with a list of reports we intend to create with the data we collect from grantees. Next slide, please. 

The revisions have led us to a more streamlined and usable questionnaire. The old framework had 14 sections, one for each activity area, and up to 285 questions depending on which sections were applicable to you. The new framework has a maximum of 60 questions depending on your awarded activities. And we took care to change the activity areas to the program areas that are statutorily defined and included in the JAG solicitation. We also include more space for narrative descriptions of activities so that grantees can more easily define the impacts that their programs are having in their communities. The old framework conflated grant activities that were funding programs versus grant activities that were simply purchases. The new framework helps to clarify that issue while allowing grantees who are only making purchases or using funds for overtime or personnel to a shortened questionnaire. The new framework should reduce the burden on grantees. I'm now going to turn it over to Angie, where she will walk you through the specific details of the new measures. 

ANGIE BALCHI: Thank you, Jimmy. Now we're going to get into the what, where, when, and how of the new JAG reporting requirements. We'll review each of these in the coming slides. 

There are two required sources of data that you'll report on: performance measures and Project Progress questions. Performance measures are a series of questions that are related to the outputs and outcomes attributable to grant activities. Grantees report on performance measures during every quarterly reporting period. Project Progress questions are a set of open-ended narrative questions related to your specific goals and objectives and your project's progress including successes, barriers, and next steps. These are reported on twice a year in January and July. 

The revised JAG performance measures can be found through the link on the screen, the first link there. We encourage each of you to become familiar with the performance measures in this PDF document before you report on them in the first time in January. Please feel free to reach out to us with questions as you begin reviewing. You can also use the PDF questionnaire as a guide to track relevant data so that when it's time to report, you have the information readily available to you. For more information on performance measures overall, you can visit the OJP performance measure webpage also provided on this screen. 

The Performance Measurement Tool, or PMT, is one of the OJP'S data collection tools for grant recipients and will continue to be the reporting tool for JAG grantees. You can access the PMT by following the link on this page. The PMT also provides additional resources and tools to assist you with reporting. 

Like Jimmy mentioned, twice a year in January and July, JAG grantees will continue to report grant activity in the PMT, and then upload a PDF report of their PMT directly into JustGrants to fulfill your reporting requirements. The PDF document is a summary of your answers in the PMT with subrecipient data aggregated. When creating a report, you will also be given a chance to leave any comments about your data submission that you feel are relevant. Save the PDF onto your computer so it can be uploaded to JustGrants to fulfill your semi-annual performance report. An instructional video is provided through the link on the screen through JustGrants. 

Also, when you're generating your PDF report, you can also download an Excel file of your federal awards' individual level PMT data for your own use and records. 

This table outlines the type of data you'll be reporting each reporting period when your reports are due in the PMT and whether you also have to upload them into JustGrants. JAG grantees will be begin reporting on the revised performance measures starting the January 2025 reporting period for October 1 through December 31, 2024. Remember that reports are due 30 days after the end of the reporting period with the first deadline being January 30, 2025. You will also be reporting on your narrative Project Progress questions during this January reporting period, so be sure to create a PDF of your PMT report as discussed on the previous slide and upload that into JustGrants. Also, remember that the PMT is only open for data entry during the reporting month. So please contact the PMT help desk if you need to unlock reports to revise previously completed reports. 

We'll now discuss how to report on the revised performance measures. 

This slide shows a roadmap of the revised JAG questionnaire. You can see that the nine JAG program areas in the blue box on the left, these program areas are aligned to the statutory program areas in the JAG Notice of Funding Opportunity. So you can see that some program areas that were previously separated, like prosecution and court or corrections and community corrections, are now considered the same program area. A single questionnaire is to be completed for each of these separate JAG program areas or initiatives within the nine areas. Answer all the questions in the questionnaire that are specific to that program area. 

First, you'll be responding to basic expenditure questions like funding, usage on equipment, consultants and personnel. If your project only uses funds for these expenditure areas, you will then receive a shortened Project Progress section to complete in January and July. If your project has additional components like carrying out program-based activities or services, trainings, or has a crisis intervention program or research component, you will be guided to those additional sets of questions. We'll explain this more in the coming slides. 

In the first page of the PMT, you will see the Award Administration page. Select begin reporting for your JAG federal award that you are completing. This page is to be completed by prime recipients during the first reporting period to tell the system how JAG awards are sub-awarded. Note that allowable administrative set-aside funds are not applicable for reporting on the performance measures. Prime recipients that are using administrative funds will be required to complete Project Progress questions semi-annually. 

There are three options of how grant funds can be distributed and used. First is by the prime recipient for a single JAG program area. Only select this option if all funds are used by the prime recipient and you're using funds for a single JAG program area. If the prime recipient is awarding subrecipients in one or more JAG program areas, select option B. And if funds are distributed only to subrecipients for one or more JAG program areas, select option C. As a reminder, the JAG program areas are the same as those aligned with the JAG solicitation. We'll get more into the JAG program areas in the coming slides. 

A major difference with the new JAG measures is that reporting is done at the project initiative level rather than a grantee subrecipient level. That means that a single organization or entity that is receiving JAG funding may be funding separate program areas, meaning that they will have more than one performance report to complete. Additionally, multiple organizations may be funding the same project or initiative. There's only a need to complete a single performance report based on the program area that it is supporting. If the prime recipient is supporting more than one program area, select Add a Subrecipient to create sub-accounts for each additional project or initiative. You'll be creating sub-awards based on the program area that your award is supporting. You'll also create a project initiative titled to identify that sub-account. If funding supports more than one program area, multiple sub-awards will be created. 

For example, this slide shows in the orange box: Subrecipient BJA test 2 has one line for data entry because their funding is supporting a single program area with the project title Juvenile Delinquency Prevention and Intervention Program. Compare this to the red box where subrecipient JP test has two lines of data in red because their funding is supporting two distinct JAG program areas: one, a county drug task force initiative, and the second a community corrections and reentry program. When the prime recipient has funding that is associated to two different projects, they must create a sub-award for themselves so that they can enter their second program area. 

To create a sub-award, under the subrecipient page, you'll select add a sub-award, adding sub-awards for each project initiative that is supporting a JAG program area. Create a unique sub-award number for each sub-award. Enter the project period, start, and end dates. Then, enter the amount of the sub-award specific to that project or initiative. 

The program area selection is the first page of the JAG questionnaire. You will answer these questions during the first reporting period, and these can't be changed unless you reach out to the PMT help desk. The purpose of this question is to inform the system, again, which program area is the project initiative serving. Remember that you're completing a questionnaire solely related to the expenditures and activities for this program area within the project initiative. You can only select a single JAG program area per project or initiative, so select the option that best fits. 

So for example, if a task force initiative with a prevention component can be reported entirely under a law enforcement program. However, if your JAG funding is supporting two separate program areas, for example, a task force and a crisis intervention program, a sub-award account must be created for each program, thereby creating a new questionnaire. The next question on this page indicates if you are a law enforcement agency, you will be prompted to answer additional questions during the first reporting period that you may already be familiar with including JAG policies, procedures, and trainings, and additional training information. 

As you may already know, the general award information tab indicates your grant status. Remember that once you select Yes for was there program activity during the reporting period, your program becomes operational and you should answer Yes every reporting period until the grant closes out. Remember, again, that administrative set-aside funds are not applicable for reporting. You will only complete your Project Progress narrative section. 

This section asks about your JAG expenditures within the reporting period only. If you use or plan to use funds for things like equipment, consultants, or personnel but did not expend funds during the current reporting period, you will respond No to these questions. 

The first question asked if you expended funds on equipment, technology or supplies during the reporting period. If you answer Yes, a table will appear for you to complete. Type in all the equipment and supplies that fit within each equipment category area. Indicate the total quantity that you purchase and your total funds spent. 

The next section of expenditures will ask additional questions on consultants and contractual supports with text box descriptions. Also, if you expended any money on personnel over time or salary pay, describe the type of personnel support and enter the total amount expended during the reporting period. 

The next question asked if you use JAG funds solely for those expenditures you just reported or if funds are part of an ongoing project initiative. So if you only spend funds on equipment, personnel, or contracts only, you'll be prompted to complete a shortened Project Progress section. However, if JAG funds were used to support ongoing services or activities, select the other uses below. You can see that that will generate additional tabs at the top of the questionnaire for you to complete. 

Be sure that you don't have to set up additional program area reports at this time by going back to your federal award administration page and adding another sub-award. Remember that you're completing these additional sets of questions based on the project initiative title at the beginning of the questionnaire. 

There are two versions of open-ended narrative questions to be able to describe your project's status and accomplishments. All grantees are required to complete a version of Project Progress in January, July, and at the close of their award. If your JAG award is focused on purchasing equipment or supplies or paying for personnel or contracts, you'll respond to this shortened version of the narrative questions. Also, if your award is new and you have no grant activity yet, you'll respond to these reduced Project Progress questions until your award is active. Here, you will provide a brief description of any JAG funding expenditures or activities conducted and how it impacts your agency. Describe any challenges and how BJA can provide assistance. Finally, describe major activities in the next six months. 

For sub-awards that are operating programs or activities, you will respond to a more comprehensive set of open-ended narrative questions. During the first reporting period only, describe what your JAG award is being used for. Include the target population, problem of focus, goals and objectives within your program area. Highlight one to three accomplishments based on your goals and objectives that were accomplished during the reporting period. Make sure to use data to describe your impacts as well as anecdotes. Finally, we encourage grantees to submit a JAG Success Story, which will be through the link in the questionnaire to go to the BJA webpage. 

When you're responding to your Project Progress questions, you'll need to make sure that you have well-defined goals and objectives for your sub-award. It is important to keep in mind that goals should be SMART, meaning: specific, measurable, attainable, relevant, and time-based. It is good to review and reevaluate your goals semi-annually as you are responding to your Project Progress to see if changes are necessary and be able to respond to your narrative questions completely. 

Now, let's take a look at the narrative questions in more depth. Again, if your project is only used for expenditures, each quarter you'll provide a description of what you have purchased, what activities have been undertaken to achieve those purchases, and how those purchases will have an impact on your agency. If your project is activity or program-based, be sure to provide a description of your overall award, including the population, problem of focus, goals and objectives. When describing your program's accomplishments, be sure to outline any action taken by your agency in the implementation of your award. Try to describe one to three accomplishments each reporting period. 

In the example goal here, the goal of the program is enhancing crime analysis capabilities and strategies. In the reporting period, the grantee established a working group which assessed gaps and developed a timeline for the project. 

Every reporting period, be sure to describe any challenges that you encountered even if you were able to overcome the challenge. 

We would like to share some tips for improving data quality that you can implement at the beginning of this new reporting process. First and foremost, we understand that setting up your subrecipients and sub-awards may be a change from prior reporting. Please contact the PMT help desk if you need assistance to set up your awards and your program areas in the PMT. This is to ensure that you are completing a single questionnaire for each project initiative that supports a program area. It is recommended that a staff is designated to coordinate all performance measures and ensure consistency. SAAs should also provide assistance to subrecipients. If this is the first time you'll be reporting data, make sure you're familiar with what you will need to collect and report by reviewing the PDF version on the performance measure webpage. Consider other tools of data collection methods, databases, spreadsheets that may assist you. Lastly, ensure that all subrecipient data is reviewed and validated before completing data entry. Note that the PMT system will alert you on any questions missed and the PMT help desk may reach out to you if we have questions about your data. 

We will now do a short demo of the performance measure tool changes. One second while I switch my screen share. In the first scenario, SAA is awarded $1,052, to fund a county court to implement a drug court program and a local law enforcement agency to support purchases of body-worn camera equipment. You can see on the federal award page that there are two sub-awards created for each of the program and the associated project initiative title, 600,000 to a county drug court and 400,000 for a Body Worn Camera Program. Select Begin Reporting. You can see that the project initiative title is carried over at the top of the page. You'll be completing the questionnaire based on that program. Select the JAG program area that best fits for the award administration, in this case, prosecution and court program. Remember that you can only select one program area per project initiative that best fits. Next, this program is not a law enforcement agency, so they select No. Click Save & Continue to go to the next page. 

As mentioned, there is grant activity during the reporting period, and this is not the final reporting period. This program did not expend any JAG funds for equipment technology and supplies. However, they did contract with a counseling service organization to provide CBT and peer mentoring during the drug court program. Finally, they also provided personnel salary for a part-time project coordinator. Enter the amount of funding expended during the reporting period for any personnel or pay. 

Next, this JAG program did use funds to support an ongoing project initiative, which is the drug court program. Select program services or any other additional activities that may apply such as training, research, and assessment. This will generate additional tabs at the top of the page. Click Save & Continue to be moved to the next page. 

Here select services and activities that you provided with the JAG program. you can select multiple services or activities within each program area even if they overlap. 

Next, enter whether you're conducting an evaluation or assessment of this program. Enter in a description of the impact, how you're measuring your impact and success for the program, including data sources that you will be using. 

Finally, if you have any outcomes for the program, describe it using raw data like percentages in achieving your objectives. 

If there was a research and assessment component, provide a brief description of your project question, outcome measures, and goals. 

Next, add the Project Progress narrative questions during the January and July reporting periods. The brief description, question one, will only need to be entered during the first reporting period. Then, enter one to three accomplishments during the reporting period as well as challenges. Be sure to say if there is any assistance that BJA or a TTA provider can help with any challenges. Here's where you can select the link to submit a JAG Success Story on the BJA webpage. 

Next, review all of your responses here and the alerts column will notify you if there are additional questions that were missed. You can click on that required question to take you back to see the questions missed. 

For the next questionnaire, Body Worn Camera Program, select Enter/Edit Data. Again, the Body Worn Camera Program title will carry over. Ensure that you're responding to the questionnaire for this program only. This is a law enforcement program and is a law enforcement agency, which will generate an additional tab of questions. 

Again, there is grant activity during the reporting period. This program did expend funds on equipment, technology, or supplies. Find the category area that best fits the items that were purchased during the reporting period. You can separate them in the open text box fields by adding commas. See the examples listed within each category section. This is not an exhaustive list, so use your best judgment on which items fit within the category area. Then after you type each item, enter the total quantity purchased and total JAG funds spent. You can also draw down the items so that you can see your list more clearly. Please only use the other category option if your items do not fit within one of the already existing categories. 

This program did not use funds on consultants, contracts, or pay. However, there was a training component and programs and services conducted. 

You may already be familiar with this law enforcement specific section. You should complete this during your first reporting period only. Indicate whether specific procedures or strategies were implemented with or without JAG funding. Then, enter the training topics. Again, you will only have to do this during your first reporting period. 

On the next page, select the types of services or activities. This program did have a training component on BWC best practices and accountability. 

Next, enter your Project Progress in January and July describing achievement towards your goals and objectives. Review your data to save and submit. 

In the second scenario, a local JAG award is administered for $36,200 to fund a state crime lab to purchase forensic equipment and supplies. There's a sub-award set up for the project initiative title, State Crime Lab. Click Enter/Edit Data to begin reporting. 

Again, you can see that the project initiative title carries over so you know which program area you were reporting for. This is a law enforcement program, but they are not a law enforcement agency. There was grant activity during the reporting period. This program only made expenditures within equipment, technology, and supplies. Again, remember to insert the items within each category that best fits. Enter the aggregate number of quantity spent and amount. They did not expend any funds on consultants or contractual supports and JAG funds were used for these expenditure items only. 

Once you click Save & Continue, you'll be taken to the Project Progress expenditure only narrative questions. Again, this is a slightly reduced version where you will briefly describe any funds spent during the reporting period and how that may impact your agency. Be sure to indicate if there's any assistance that BJA can provide and major activities for the next six months. Click Save & Continue to review your information. 

In the third scenario, a local award is distributed to a sheriff's department for $69,400 to purchase radio equipment and curriculum. They also hired a project director to run a crisis intervention program. You can see that this project initiative title was carried over from the previous page. Select the program area that best fits. This is a law enforcement agency as they have an ORI number. 

Click Save & Continue to continue reporting on expenditures. Again, separate each item purchase within the overall category by commas. You can drag down to continue entering more information. This program did not use any funds on consultants or contracts, and they did expend funds for overtime pay to hire a project director. Enter the total amount during the reporting period. This program also had a crisis intervention component. Complete the Law Enforcement Agency section during the first reporting period. 

Next, respond to the Crisis Intervention Program section. You will be indicating which program activities were taken and whether they are new or existing programs in addition to open-ended questions. Reminder to complete your Project Progress and review all of your data upon completion.