One of my jobs is to administer the Teacher As Researcher grant. As someone who has been involved in writing several grant proposals, I must say that I've enjoyed being on the other end of the process. Composing the RFP was fun, and reading the varied applications was fascinating, but the actual scoring was surprisingly difficult.
Let me make it clear that I did not officially score any of the proposals. Since I know several of the applicants, we decided early on that only the external review team would score the grants. When the review team met, I asked them to reflect on the process of reading the grants. Here are excerpts from the reflections and my musings on what can be done to improve the process.
REFLECTIONS from Grant Scorers
What worked well tonight?
"·Nice preparation for filling out forms
·Good discussion at the conclusion of scoring with whole group
·Well Run session"
"Meeting new people.
Being exposed to grants for ideas.
Food/Paid parking"
"I liked the chart Donna made.
I learned something-Quinn's 6 questions.
Thanks for the food."
"Loved the discussion with other readers.
Liked the immediate feedback"
"The grant proposals were well laid out and easy to score. The process was informative and has given me motivation to write my own proposals."
Are there any weaknesses or confusing portions of the grant that need to be addressed before next year?
"I hope we have more to read next time. There is a lot of need out there."
[I hope we have more, too. From conversations I had with prospective applicants, I was expecting 14-15 applications instead of just 8. I need to follow up and see why people did not apply. Did they run out of time? Couldn't get a group together? . . . Also, with a regular CFG Coach newsletter and updated contact lists for next year, we can be sure that more people are getting the RFP.]
"I think that the grant should indicate how specific they want the analysis of data. Do they want quantifiable data? or can the applicant write a grant based on research data from other studies? Did not ask how the applicant will evaluate the grant. What will he/she consider successful?"
[This reflection brings up important points about data analysis and evaluation. What is success? It will be something different for each group. I will bring up this idea when I meet with the awardees in early June. I think that this reflection also shows that I need to do a better job about training the scorers about what makes action research different from more traditional research.]
"I have a concern about the rubric. The questions did not seem to help me score the grant. Perhaps there should be one initial question before moving onto the rubric: Does this proposal meet the RFP general request? (If not, then there is no need to score.)
I wish the CFG's could just have a focus rather than require research."
[I agree that the rubric needs changing. Next year, I'll add a section for the executive summary where the scorers can judge if the proposal meets the imperatives of the RFP.]
"As most seemed to be weak in the data portion, I am wondering how to clarify expectations of initial data and ending data as well as a control group. What is the measurable impact? How much will the data % increase? idea: find guidelines for writing School Improvement Plans (SIP).
[Again, this shows that I need to do a better job of explaining to the scorers what makes this particular grant special. Much of the data in action research may be qualitative instead of quantitative. Journals, reflections, etc. are perfectly valid forms of data--although they still need to go through some type of analysis.]
"·RFP needs to specify that "research" must include core areas.
·RFP should address multiple grant requests. [I agree! One external partner was listed on three different applications. I think that seeing the same names on more than one application made the scorers question the commitment to any one group.]
·RFP should clearly state that committee needs to see evidence of new work and research!"
I also had an opportunity to read some Focused Impact Grants a couple of weeks ago. I liked how the rubric for that grant had the points divided into Excellent, Good, Passing, and Not Passing. If I had done that with the TAR rubric, there probably would not have been such a huge point spread among the different readers.
From reading Focused Impact Grants, I also saw that I need to add points to the rubric for the commitment letters. They should clearly outline the ongoing roles and responsibilities of members for this grant. The external partners should show a commitment to implementing the project.
Finally, one of my fellow readers for the Focused Impact Grant mentioned that he reads the budget before reading the rationale and methods. By "following the money" he gets a better sense of what the grant might accomplish than if he starts by reading the jargonny prose.