p. 1

development of a wholistic accountability framework for the native literacy stream final report peterborough action research team acknowledgements the report was developed as part of the national action research project connecting the dots improving accountability in the adult literacy field in canada funding for the project was provided by the office of literacy and essential skills oles human resources and skills development canada.

[close]

p. 2

introduction context the project manager and lead researcher of the peterborough action research team art have twenty-one years of combined experience working in ontario literacy and basic skills lbs native stream.1 as colleagues and trainers of twenty-six native literacy programs in ontario we have seen and heard about many performance-related issues in our sister programs including programs being placed on probation programs being taken over by mainstream sponsors sponsors giving up on programs reductions in the funding of program core budgets we hypothesized that these issues were fundamentally connected to non-compliance with funding accountability requirements and the ripple effects of not meeting performance measures we were concerned about native literacy programs accountability record with ontario s literacy funder the ministry of training colleges and universities mtcu however we were also aware of a number of serious challenges faced by native literacy programs funding resources and staff are in short supply while community support and appreciation of the value of the programs are limited in relation to accountability in particular staff struggle to articulate what they do well and what needs work within their programs they have difficulty using the required reporting forms and employing mtcu terminology in discussing their programs.2 it seems that they can tell the story about what they do and how well it works for the learners but cannot fit into the template provided by mtcu research question the peterborough art was determined to find a better way to help native literacy staff capture their successes and challenges we hypothesized that an alternative more appropriate accountability framework would enable native literacy programs to better articulate demonstrate and discuss with their field consultants how they strive to meet the funder s standards the improvements that need to be made to the program and the potential obstacles to making those improvements there are four streams of adult literacy in ontario anglophone francophone deaf and native 2 in current practice the mtcu s program monitoring report the main form that captures and evaluates program performance is sent out in advance of the site visit for completion by the program coordinator if time allows most often it is filled in at the site visit by the field consultant who asks to see or hear proof of certain standards being met development of a wholistic accountability framework for the native literacy stream 1 page 2

[close]

p. 3

the alternative framework to be developed as part of the connecting the dots improving accountability in the adult literacy field in canada ctd project would serve a number of functions interpret ministry terminology explain why the ministry requires information provide opportunities for telling native literacy program stories suggest other practical uses of the information gathered for instance the data could be used when reporting about program results and challenges to chief and council or to the board of directors of the friendship centre to produce the new accountability framework or tool the peterborough art would adapt the program monitoring report pmr used by mtcu project goals the goal of this initiative was to make it easier for native literacy programs to describe and discuss their activities at site visits our tool is designed to allow programs to explain the context within which they operate we were confident that an adaptation of the pmr along with training for the participating programs in its use would lead to improved documentation of program performance potential outcomes included an improved understanding by native literacy providers of mtcu accountability expectations a better understanding of pmr terminology greater confidence in articulating the work being done stronger evidence of compliance with funder regulations our conception of what we wanted to achieve aligned directly with the goals 2 and 3 of the ctd project which were to develop a common language between the community and government/funders to talk about accountability build on this knowledge and use the language to develop innovative models that can satisfy the needs and requirements of both providers and funders and improve accountability in positive ways project challenges satisfying three levels of governance there are three levels of governance to consider and satisfy on a reserve the proposed tool therefore needed to provide ways for coordinators to tell their stories and share this information with mtcu field consultants at program monitoring visits ministry the chief and council or the friendship centre board of directors at regular reporting intervals sponsor the community/learners at presentations or open house events development of a wholistic accountability framework for the native literacy stream page 3

[close]

p. 4

1 the ministry the core funder the core funding from mtcu requires accountability that is spelled out in the contract and guidelines.3 core funding is about $50,000 annually which is insufficient to cover all program costs programs thus depend on additional support from sponsors i.e the band or friendship centre 2 the sponsor the secondary funder the band or friendship centre support the program by providing physical space and logistical support including supplies technology support and bookkeeping/payroll services administrative fees collected from the program by the sponsor do not cover all costs often in exchange for local support another level of accountability and/or expected service is placed on the literacy coordinator accountability expected by the band is often quite different from what is laid out in mtcu s literacy and basic skills lbs guidelines for example a literacy worker under a friendship centre sponsorship must participate in joint activities such as fundraisers front-desk duty responding to community needs or closing down at the same time as all other programs if not in line with lbs-defined services these activities can place a program s funding in jeopardy at the same time however unless the practitioner complies with band in council resolutions he/she can be replaced additionally elections on reserve are held every two years so the key players keep changing 3 community and learners the community expects programs to be accountable to its needs which may differ from what the other two levels of governance require the community elects chief and council and can have a strong influence on them when it comes to what happens at the local program level for example during the summer many community people want to enrol their children in native literacy programs so their skills can be improved or maintained however funding is for adult literacy and cannot support a summer school for children if the program does not offer what the community expects complaints are made to chief and council or the band administrator as a result program workers might be required to offer the services demanded by the community even though these cannot be delivered using lbs funding assuring cultural relevance one of our importance concerns with pmr was that it did not take into account culture which is a critical component of a native literacy program from our communications with native literacy providers and as trainers in the field we knew that these programs support positive outcomes for aboriginal learners that were often missed when funders looked at activities and outcomes solely using the pmr 3 appendix b provides a list of the requirements set out for literacy and basic skills programs funded under mtcu development of a wholistic accountability framework for the native literacy stream page 4

[close]

p. 5

in searching for culturally relevant performance measurements we investigated the aboriginal public education field and the research of david bell and george fulford.4 bell and fulford led teams in 2004 and 2007 that studied aboriginal school success the participating schools in the bell and fulford studies were located in aboriginal communities were aboriginal run and supported and their focus was on education delivered with active community involvement although native literacy programs are not schools the learners refer to our programs as their schools and communities consider us to be alternative schools for these reasons we decided that it was appropriate to apply some of bell and fulford findings to our project using data gathered in twenty school settings across canada bell and fulford compiled eight key indicators of successful aboriginal schools these include governance and leadership school climate curriculum/programs teaching/learning aboriginal language culture and history and assessment for learning and outcomes comparing these indicators and the pmr we noted similarities including the assessment categories used by each however we also noted the absence of several of the indicators from the pmr the focus and activities of many of the native literacy programs address the missing indicators but there is no opportunity to report on them using the pmr the pmr examines program accountability within the following categories administrative and financial accountability program delivery ­ assessment training plans training follow up community links ­ information and referral local co-ordination partnerships learners feedback significantly there are no categories for aboriginal language culture and history the art tool our adapted accountability framework method hereafter referred to as the art tool blends the eight indicators identified by bell and fulford with the pmr s categories david bell et al 2004 sharing our success ten case studies in aboriginal schooling society for the advancement of excellence in education/saee kelowna bc and george fulford 2007 sharing our success more case studies in aboriginal schooling society for the advancement of excellence in education/saee kelowna bc development of a wholistic accountability framework for the native literacy stream 4 page 5

[close]

p. 6

bell and fulford s 8 key indicators of aboriginal school success governance and leadership assessment for learning teaching and learning culture and history curriculum pmr categories administrative and financial accountability program delivery some native literacy programs embed culture and history within the outcomes learners must achieve in the reading writing and numeracy domains programs are not assessed for addressing this topic on the mtcu form community links/learners feedback no comparable category mtcu does not fund aboriginal language literacy programs school climate aboriginal language mtcu places administrative and financial accountability at the top of the list on the pmr and the accompanying checklists for this category encompass most of the report s pages we hypothesized that this is the area where many of our programs fall short not only in performing the required activities of this category but also in articulating the situations that hinder performance or success identifying steps toward improving in the administrative/financial area finding solutions methods time human resources to help effect timely improvements the art tool allows programs to describe the ways in which they are accountable to the multiple stakeholders to whom they report the funder mtcu the sponsor chief and council or friendship centre board of directors and the community/learners we developed sections within the art tool see appendix c that explain the terminology being used and provide suggestions on how the programs can demonstrate that they have delivered what the funder is looking for other sections encourage the literacy coordinator to describe all support received by the program this includes in-kind support such as space provided by the community at program monitoring time often a period of stress for the coordinators the written reminders in the art tool remind the coordinator to report contributions from all stakeholders and the accompanying expectations that might accompany such contributions development of a wholistic accountability framework for the native literacy stream page 6

[close]

p. 7

project participants to all of the following chi meegwetch for your contributions participating programs wasauksing adult learning centre moosonee native literacy program chippewas of georgina island native literacy ken das win native learning centre team members and duties lead researcher karen mcclain drafted the literature review on successful aboriginal schools bell 2004 and fulford 2007 applied the eight key success indicators to the pmr in order to draft the art tool visited programs all four initial visits and facilitated sharing circles trained the participants in the use of the art tool drafted the final report liaised with the research friend reported to the board of directors on project progress research assistant/project manager pat powell owen worked with the lead researcher on developing the art tool consulted with the ctd project manager at all stages of the project consulted with ministry representative and research friend organized initial visits to programs visited two programs three times two programs twice consulted with the lead researcher on findings and best ways to present them provided hands-on support at program visits and email/phone support to program staff between visits wrote monthly reports and submitted invoices to the project manager redrafted and finalized the final report research friend susan hardie offered consultation and advice regarding releases for photos and intellectual property offered feedback at all steps of the project initial interim final provided samples of logic models photo releases research briefs funder representative harold alden offered feedback on use of the pmr administrative assistant angela wynne assisted with office coverage while researchers were on program visits front desk/tech support bryant owen set up return visits to programs provided tech support including photo formatting and map development development of a wholistic accountability framework for the native literacy stream page 7

[close]

p. 8

project summary programs selected we hoped to enrol four programs in our research using an email invitation that included a quick overview of our project s purpose we invited eight programs known to us that were delivering lbs on reserve the first three to respond were selected we also included moosonee which is not a first nation but experiences similar challenges because of isolation the literacy needs of moosonee are unique and over 90 of the population is aboriginal all four of the programs receive approximately $50,000.00 annually from mtcu there are no other sources of funding coming into the programs except for significant in-kind support from the first nation or friendship centre sponsor that can be valued at $10,000 to $20,000 a year in-kind support includes space for the program a computer technician receptionist central phone system etc when we first visited two of the programs had fewer than five learners and one had no learners at all learner recruitment and retention was a key concern of all of these program workers as the programs were in non-compliance with ministry requirements on the number of participants and hours needed to maintain core funding the fourth program had a participation rate of eight learners per day these learners were picked up and dropped off and coordinator also provided lunch the community is spread out and apparently this was the only way the coordinator was able to get the learners to come and stay for any length of time money for their transportation and meals came from the band most learners were working on communication skills cultural studies computer basics and numeracy learners goals included continuing their education and training or becoming self-employed all tutoring was provided one-on-one or in small groups at least three days per week some evening small group work was also provided at one program site visits during the first site visit to the four programs we introduced our research project reviewed the draft art tool and trained participants on using it throughout the project and after the research ended engaged the native literacy program sponsors chiefs and councils friendship centre executive directors and communities as well as the literacy staff through the facilitation of sharing circles5provided give back 6 5 a sharing circle may be used in aboriginal organizations and gatherings to gain each person s perspective there are no right or wrong answers each participant has something to bring and the circle configuration provides encouragement to do so 6 giving back is a cultural mannerly response to those who have given something to us program staff and other stakeholders gave us a great deal by agreeing to participate in the development of a wholistic accountability framework for the native literacy stream page 8

[close]

p. 9

there was also ongoing support from art members between visits through telephone and email check-ins assistance given to the programs was mainly administrative in nature examples of assistance include help in producing forms to capture contact time explanations of processes database operation management practices and sharing learner and volunteer recruitment ideas summary of activities four programs participated in the research two programs were visited three times each one was visited twice and one was visited once between september 2008 and june 2009 three programs fully engaged in the process the fourth program decided not to remain in the project because it felt that it had a full understanding of accountability and was fully supported by all levels of governance i.e the funder the local band and the community eighty-seven contacts were made by email or telephone between visits to assist the programs with questions related to accountability set up visits and follow up on suggestions made at visits working together as a team successes researchers and native literacy coordinators were on the same page about the challenges of accountability processes we are native literacy workers ourselves researchers had an existing rapport with their literacy colleagues so research participants quickly opened up and shared information researchers were used to juggling projects with the day-to-day work in literacy so the additional workload this project presented was not daunting to us responses from the research friend and the ctd project manager to the researchers questions were timely and helpful all team members accepted and fulfilled their responsibilities to the benefit of the project research participants were comfortable enough to send requests for help and to ask questions that greatly assisted the research challenges the needs of the participating programs were much greater than expected necessitating extensive support between visits requests for information and documentation from the programs required several reminders the extent to which the programs needed to become more accountable was underestimated while we have no authority to do so we felt it necessary to help research project and by giving up several days out of their busy schedules in return we offered an additional day of our time at each visit to take action in areas where the program workers indicated they needed assistance in order to improve their levels of accountability with the funder development of a wholistic accountability framework for the native literacy stream page 9

[close]

p. 10

them understand the importance of meeting ministry standards we worked hard to help them understand that putting the systems we suggested in place would make life easier for them while bringing the programs into compliance with funder expectations increased service demands at the peterborough native learning program meant that juggling both the day-to-day literacy program tasks and art-related work was difficult during the last four months questions to the research friend generated additional questions a face-to-face meetings could have helped but she moved to another province the funder s representative changed due to an overlap in roles we received a request from the funder that we not make additions to the province s pmr as it is an official document in an effort to comply with the request and to clarify our intent we took a step back and devised a logic model to help describe our process and outcomes this was shared with the funding representative and helped us stay on track we also made adjustments to the tool so that the participating programs would understand that it was not a replacement for the pmr some of the expectations of our funder ctd were unclear or were revised when work had already been completed for example the outline for this report attending scheduled teleconferences with the other four arts to provide updates was not always possible because of time constraints sometimes the other projects did not join the calls and it felt more productive to do the check-in/updates directly with the ctd s project manager collaboration between team and funder there was not large-scale collaboration with the funder during the peterborough art project but we believe there will be other opportunities post-project to share and implement the tool within the native stream of literacy as mentioned earlier the funder expressed concerns about the adaptation of an official government form the mtcu pmr by the end of the project it was clear to all involved that the tool we created is a support document to be used in conjunction with the pmr the ministry representative has indicated that our tool has the potential to assist the programs with interpreting what the ministry is looking for he has endorsed the use of our tool commenting that the tool provides constructive suggestions that will help native stream agencies to recognize and to present evidence of responsive and flexible lbs delivery in their communities it provides the guidance that agencies would need.7 our hope is that the programs will have the tool at their fingertips at monitoring time to help them remember to speak to the successes and challenges of their programs programs may use it to identify areas of improvement and strength aiding in future 7 quote from harold alden senior policy analyst mtcu and funder representative peteroborough art excerpt from email to peterborough art project manager of september 9 2009 development of a wholistic accountability framework for the native literacy stream page 10

[close]

p. 11

planning processes and/or to form the basis of reports to other stakeholders field consultants who have visited the participating programs told us they have noticed positive changes related to accountability accomplishments the peterborough art tool helped the programs improve their understanding of funder accountability how to use the pmr and why and how mtcu collects data in the designated categories previously many of the native literacy staff did not understand the terminology or the rationale behind the requirements spelled out in the mtcu s pmr consequently they had difficulty complying with the regulations increased understanding by community members about native literacy services we discovered through the sharing circles at the first meetings that the community members and in two cases the direct supervisors of the programs did not have a full understanding of what native literacy programs can offer many had never made a referral to the program because they thought it was for non-readers the impression of the community representatives is that first nation members know how to read so they do not need native literacy services this certainly provided one explanation for the small number of learners enrolled in two of the programs it also demonstrated to us how much work needs to be done in terms of promoting greater public awareness of the broader definition of literacy training and what native literacy programs have to offer greater confidence and improved communication with ministry staff in talking about their work native literacy workers expressed that they had more knowledge now of what was expected why it was expected and how to describe their work.8 three of the four told us they were reticent to talk about missing targets with the funder the program staff came to realize that as long as they had a viable plan in place to begin to address the shortfalls their communication with the field consultant could be positive the ideas we shared with programs about learner recruitment and raising the programs profiles in the communities are bringing positive results at the outset of the project all art participant programs were falling short of meeting the contact hours and number of learners required by the funder we saw positive changes in this area over the course of the project by the end of the data collection period end of june 2009 we heard from two of the participating programs that they believe they are well on their way to meeting mtcu targets for numbers and hours the field consultants own awareness has been raised regarding the complexities of operating native literacy programs on reserve and in isolation two field consultants shared that they noticed that the attitude and understanding of the native literacy program staff toward the required visits and monitoring tools changed significantly one of these field consultants wondered what we had done at our research visit to 8 project manager s notes from art impact follow up visit february 2009 page 11 development of a wholistic accountability framework for the native literacy stream

[close]

p. 12

produce such a difference in the confidence level of the literacy coordinator 9 the other voiced an appreciation for the context within which the native literacy coordinator works i had no idea there were so many demands on [the literacy coordinator from the community and that the number of learners who would come was so limited 10 the field consultants who monitor the programs involved in this study see the value of having someone trusted by the native literacy field who is also well versed in the accountability requirements of the funder visit the programs to assist with questions and offer practical help the ontario native literacy coalition has tried to offer such services online and the results were negligible native literacy staff respond more positively to meeting face-to-face within their programs and communities our visits with phone and email follow-ups confirmed this team learning during the course of the project we learned that the programs operate in an environment where opportunities for referral and outreach are very limited two are located on islands these programs also have limited resources staff materials internet access software etc program coordinators maintain that these factors account in part for the low numbers and lack of interest the participants were working very hard but not efficiently because they didn t know where to ask for certain pieces of information that would help them with their administrative responsibilities and accountability program coordinators can be called away for days at a time to address a sponsor group or community need they are pulled in many different directions they worry about acknowledging this situation to the ministry preferring to say nothing which is a problem sponsors and communities expect the native literacy program to address culture history and language and do not understand why program funding cannot be used for this coordinators therefore find themselves in a bind as a result of this learning we devised innovative community-appropriate strategies to help increase participant numbers and contact hours revised the art tool to reflect what the programs reported to us about in-kind support and activities that they are expected to engage in some of this information was new to us really listened and put aside preconceived ideas that we thought would help bring the programs in line with funder expectations 9 comment made by a field consultant by telephone to the project manager after a site visit november 2008 10 comment made by a field consultant to the project manager at a discussion at the onlc s fall conference in keene on nov 2008 development of a wholistic accountability framework for the native literacy stream page 12

[close]

p. 13

encouraged literacy coordinators to be more open with their field consultants reassuring them that asking the funder for help is acceptable and in no way a sign of weakness or incompetence reminded coordinators that other people and organizations we provided lists are willing and even mandated to help so they should reach out without hesitating next steps we suggest that the peterborough art findings could alter the discussion about accountability in adult literacy in three areas the need to interpret the funder s expectations our approach to investigating accountability implementing an intervention through use of the art tool and providing practical support worked well our findings suggest that interpreting accountability frameworks and face-to face assistance with meeting and/or articulating standards of accountability in the native stream of literacy is highly beneficial the ontario native literacy coalition onlc11 has had ongoing discussions with the mtcu about this need the project has confirmed that the need is real we believe that this project can help further propel the accountability discussion if we have the opportunity to present the results to mtcu and the ontario native literacy coalition for many years the coalition has seen the need for a `field worker but did not have actual data to support a request for mtcu funding the project provides concrete proof that meeting face-to-face with native stream staff to address accountability and mentoring result in benefits to both program workers and the funding body we will seek out an opportunity to share our findings with both the onlc and the ministry with the hope that the field worker position idea might be revisited the need for practical hands-on community-engaged assistance in achieving accountability with staff turnover in the native stream an ongoing issue a team approach to meeting performance indicators including a representative of the sponsor group and a reliable committed program volunteer would be helpful if the literacy program coordinator moves on which frequently happens the sponsor group can and will support the continuation of the program until such time as replacement staff can be hired discussion about the importance of community involvement through the sharing circles was opened up there is a need for the discussion to continue and for action to occur for example volunteers coming forward to help the one person running the native literacy agency expansion of the categories measuring performance to better reflect the expectations and priorities of aboriginal communities indicators of success that are very important to aboriginal people are missing from the 11 ontario native literacy coalition onlc is the provincial literacy network responsible for training advocacy and support for its native literacy program members it also acts as a liaison between the programs and mtcu development of a wholistic accountability framework for the native literacy stream page 13

[close]

p. 14

current framework for measuring the performance of native literacy programs these performance indicators focus on culture and history and aboriginal language community members expect their local native literacy program to address these subjects at the sharing circles attended by native literacy program staff learners sister program staff community members board members and the direct supervisors of the native literacy programs we asked participants what native literacy meant to them they indicated that they expect native literacy to include or primarily address literacy in the first languages and that local aboriginal culture and history should be an integral part of the programs a priority according to native literacy practitioners some learners leave and the community won t support native literacy programs because there is no native to it the programs we visited answer to their sponsors and communities as much as they answer to the funder the expectation that they deliver language culture and history training is loud and clear however mtcu funding does not cover such learning therefore does not measure performance in these areas clearly there is a disconnection in terms of service expectations from the stakeholders that has placed the program staff in an awkward position development of a wholistic accountability framework for the native literacy stream page 14

[close]

p. 15

appendix a profiles of the participating programs development of a wholistic accountability framework for the native literacy stream page 15

[close]