Program Development through Evaluation: Family Literacy at Simmons College
The Scott/Ross Center for Community Service at Simmons College typically employs 150 to 160 students per semester in community service Federal Work-Study programs, representing 33% of the institution’s FWS spending. Programs focus on family literacy, with an emphasis not only on helping children acquire literacy skills but also on creating opportunities to engage families in the process. In the past year, program leaders focused on improving the program in three areas: student development (including a review of four student development frameworks), student evaluation, and community partner evaluation.
Jeremy Poenhert and Andrea Miller
Scott/Ross Center for Community Service
In any given semester, the Scott/Ross Center for Community Service at Simmons College typically employs 150 to 160 students in community service Federal Work-Study (FWS) programs. In the academic year 2004-2005, Simmons College spent 33% of the college’s FWS funds on these programs. This represented approximately 22,000 hours of community work, averaging 5-8 hours per student.
While the community service FWS programs change slightly from year to year, depending on community need and Scott/Ross Center goals, certain elements are consistent across programs. Programs focus on family literacy, with an emphasis not only on helping children acquire literacy skills but also on creating opportunities to engage families in the process. All programs also have a strong student leadership component, with student coordinators who oversee daily program operations. Student leaders are supervised by Scott/Ross Center staff.
Throughout the history of community service FWS at Simmons, programs have continually changed. New programs are developed in response to community need and student interest. Some programs have been discontinued because of changes in community partnerships. In the past three years, the Scott/Ross Center placed a special emphasis on deepening training, developing new educational materials, building collaboration across programs, and collaborating with community partners and campus offices to continually strengthen the programs.
Overview of Current Community Service FWS Programs
|Program||Student Staff*||Children Participating*||Description|
|Afterschool Program at Simmons||15||15||Children visit Simmons campus twice a week for tutoring|
|America Counts||25||25||Math tutoring|
|America Reads||25||25||Literacy tutoring|
|Curriculum enrichment||15||30||Tutors lead literacy activities around 8 specific subjects|
|Farragut After-School Program||15||30||Tutors provide homework help and lead enrichment activities|
|Jumpstart||46||46+||Simmons is a site for the national Jumpstart program|
|Special Projects||10||Varies||Students work on an array of special projects in support of Scott/Ross programs|
|Steps to Success||4||22||Students provide weekly tutoring and 5 college visits to participants|
|Strong Women, Strong Girls||5||25||Simmons is a site for the national Strong Women, Strong Girls program|
*These numbers represent typical enrollment, and vary from semester to semester.
Continuous Program Improvement
The Scott/ Ross Center is focused on program improvement in three areas: intentionally incorporating an understanding of student development into the programs; evaluating student experiences; and evaluating partnerships with community organizations. The Center recognized that by focusing on these three areas, which often get overlooked in the day-to-day operations of community service FWS programs, we can significantly benefit our programs, students, and partners.
While we have only begun tracking student retention rates, there is observational evidence that many students participate in our community service FWS programs for multiple years, with a significant number remaining involved for three or four years. This allows us to create active and sustained learning opportunities for students in our programs.
In 2005, we set specific learning outcomes for participating students that we will use to refine the training, support, dynamics and evaluation of our programs. The ultimate goal of this process is to clearly establish programs that not only benefit the community but also engage student staff in actively applying their community experiences to their own learning and development.
Our first step was to gather and review student development models from other higher education service programs. Our research did not find any models specifically intended for FWS students, but we did find a number of programs with established goals for participants in a range of service programs. We found four programs that had models that were especially applicable to our goals:
- Tufts University, the University College of Citizenship and Public Service
- Bonners Scholars Program
- California State University Monterey Bay
- City Year
The first model, from California State University Monterey Bay, identifies eight learning objectives: academic learning, democratic citizenship learning, diversity learning, political learning, leadership learning, inter- and intra-personal learning, social responsibility learning, and social justice learning. Each has its own set of knowledge, skills, and attitudes to be acquired. The Bonner Scholar model identifies skills, roles, and commitments for student development for a community service scholarship program at 27 colleges and universities across the United States. It also looks specifically at a student’s progression over the course of several years.
The Tufts University College of Citizenship and Public Service Student Learning Outcomes model adapts Bloom’s Taxonomy for outcomes and student learning objectives, classified into three dimensions: cognitive-based, skill-based, and affective-based. City Year, a year-long service program, drafted a set of competencies that corps members are expected to develop over time through involvement in a Civic Leadership Program. The City Year model of leadership development identifies a set of 12 competencies (learning objectives) that are increasingly mastered through service, feedback, reflection, and training.
Each of the frameworks above has a unique approach to establishing and describing learning outcomes for their participants. Using their models, Simmons College Scott/Ross Center identified three components to include in a student development model for FWS students:
- Learning objectives
Our draft student learning objectives are below. We found California State University Monterey Bay’s format especially useful for describing the learning objectives, and used their model to craft a framework that fit our programs. We also list outcomes, which are examples of specific ways students can demonstrate and apply the knowledge, skills, and attitudes of the learning objective to our FWS programs. Any direct references or inspiration from the four models we researched are noted.
|Diversity learning (1)||Understanding individual vs. institutional dimension; understanding “isms” (2)||Developing cross-cultural communication skills (2)||A diverse array of voices is necessary to make sound community decisions (2*)||Knowledge:
Students are familiar with the complex diversity issues in the greater Boston K-8 school system.
Students are able to engage in constructive dialogue about diversity in K-8 education.
Addressing diversity issues is essential to understanding urban K-8 education.
|Leadership learning (2)||Understanding multiple theories and styles of leadership (3)||Developing skills and techniques to respond to complex leadership challenges (3)||Leadership is a multi-faceted skill that anyone can develop (3)||Knowledge:
Awareness of their own leadership styles and how that impacts their work.
Ability to apply different leadership techniques in the schools.
Different situations encountered in the program require different approaches to leadership.
|Inter- and intra-personal learning (2)||Understanding the concept of multiple social identities (2*)||Developing skills to work with a broad array of people (3)||Valuing multiple social styles (3)||Knowledge:
Awareness of the different social styles of children and coworkers in the programs.
Ability to respond to different styles of parents, children and coworkers in the program.
Working in the community requires an ability to respond to different styles.
|Social/professional responsibility learning (2*)||How individuals in certain professions act in socially responsible ways (2)||Determine how to apply one’s professional skills to the betterment of society (2)||Responsibility to others applies to those pursuing all kinds of careers (2)|
|Social justice learning (2)||Knowledge of systemic inequities (2)||Skills to organize actions for social change (2)||Commitment (will) to act for social change (2)|
|Civic engagement (4)||Understand the many layer of being an engaged citizen (JP)||Ability to actively engage in the democratic process (JP)||Individuals and groups have the right and responsibility to engage in the democratic process (JP)|
|Community building (Bonner)||Understanding core concepts in building a vibrant community (Bonner)||Techniques for building and supporting engaged communities||Effective and positive communities require commitment and effort|
|International perspective (Bonner)||Understanding of the complex relationship between community issues and global society||Ability to analyze community issues from a global perspective||Global and local issues are interconnected|
|Critical thinking (CSUMB and Tufts) (Pascarella, Terenzini)||Understanding multiple approaches to analyzing complex social issues.||The ability to identify, analyze and respond to data, issues and arguments. (P&T)||It is important to ask challenging questions and follow the reasons and evidence wherever they lead. (P&T)|
- Multiple models
- Created by our working committee
- Bonner Scholars
* adapted from the cited model
Note: The City Year model was the final model researched, so it has not yet been integrated into the chart above.
In 2005, the Scott/Ross Center conducted its first student evaluation across all of the Center’s FWS programs. The Center created a student position of Evaluation Coordinator who worked with Center staff to develop the survey tool, gather responses, and compile the results. She consulted with two faculty members for advice on designing effective surveys and piloted the questionnaire with a number of students to gauge its effectiveness. The survey was distributed as a paper document with 18 multiple choice questions and 6 open-ended questions. Questions gauged student attitudes and perceptions of a wide range of programmatic issues, including diversity, community impact, and relevance to student personal and career goals.
Of the 150 students involved in the programs, 97 completed and returned the survey, a 65% return rate. The student Evaluation Coordinator compiled the information in a written report for the Center. The overall results were positive, with high rates of student satisfaction. The results also identified areas where the programs could be strengthened.
The 2006 survey was available both on paper and online. A special effort was made to reach students who visited the office less often through emails, multiple announcements and working closely with student leaders in each program. This effort paid off: the response rate was up from 65% (97 students) in 2005 to 77% (116 students) in 2006. The addition of the online version of the survey provided a better forum for students to share their feedback; more students responded to open-ended questions, and their responses were longer.
In future years, we would like to offer a pre-survey to students before they begin their positions, as a comparison to their end of the year results. Finally, the results will be shared more widely with the students, as an opportunity to spark dialogue and discussion about improving and strengthening programs.
Highlights from the spring 2005 survey are below:
|Statement||Percent Agreeing or Strongly Agreeing||Percent Neutral||Percent Disagreeing or Strongly Disagreeing|
|My position was rewarding||93%||7%||0%|
|My position has helped me develop valuable work skills||86%||13%||0%|
|My position has helped me to clarify my academic and career goals||46%||46%||8%|
|I have been able to apply my classroom studies to my work||50%||40%||10%|
|I was exposed to people of races and ethnicities other than my own||100%||0%||0%|
|My work benefited the children I worked with||95%||5%||0%|
|My work benefited the families of the children in the program||73%||27%||0%|
|I have a greater understanding of diversity issues in the community||80%||19%||1%|
|I developed valuable leadership skills||72%||28%||0%|
|I got along well with my co-workers||95%||3%||2%|
Community Partnership Evaluation
In addition to student evaluations, the Scott/ Ross Center conducts an evaluation of its partnerships with community organizations. Among other goals, the assessment asks community partners for feedback on the current status of their partnership with the college, how the partnership might improve, and how community partners could play a role in supporting college student development. While the assessment includes all Scott/Ross Center partners (including those that work with volunteers and service learning students), a special emphasis is placed on gathering feedback from sites that work with community service FWS students.
While developing the survey in 2005, staff reviewed theoretical models of successful community partnerships and frameworks of student development and leadership, as well as informal feedback from community partners and experts in higher education. In consulting relevant research on college student development and campus/community partnerships, the Scott/ Ross Center identified a significant gap. Though there were many frameworks for both college student development and campus/community partnerships, there was little that incorporated student development into the structures of campus/community partnerships. The assessment was a first step toward better understanding the current state of our community partnerships and the role of community partners in student development. By paying close attention to community partnerships and college student development, we can maximize the learning experiences for the FWS students, and, by extension, maximize the students’ impact on community members.
The community partner survey was administered online, and consisted of 27 questions. Partners were contacted via email and phone; hard copies of the survey were available for those who chose not to complete the survey online. In total, 20 of 45 possible Simmons partner sites responded to the survey. Overall, the survey revealed that community partners are satisfied with the state of the partnerships. Partners reported that the benefits of the partnership outweighed the challenges, and the partnerships are mutually beneficial to the organization and the students. Over 95% of partners were “satisfied” or “very satisfied” with the skills and knowledge of students, and 95% “agreed” or “strongly agreed” that “Simmons is sensitive to the needs and concerns of its community partners.” Nearly 85% know more about the programs and services offered by the college as a result of the partnership.
Some results related to specific actions that the Scott/ Ross Center can take to improve our partnerships with community organizations. Most related to training, student development, or communication. Many community partners (85%) reported that they would like students to gain a greater understanding of the community and the social issues addressed by the organization and “would like students to gain useful skills and knowledge as a result of their work” with the organization (83%).
Although partners were pleased with students’ knowledge and skills, they were less satisfied with their awareness of social justice issues. Also, more than half of the partners reported that the goals of the partnership were written and had been agreed upon by all members, which was interesting because the Scott/ Ross Center has few formalized, documented agreements with community partners. The Scott/Ross Center therefore has an opportunity to further engage community partners in student development through training and increased communication.
Finally, partners described their own ideas for supporting college student development, including:
- Student learning agreements between college and organization;
- Documenting the goals of both organizations;
- Meeting regularly with the Simmons coordinators/office staff to assess plans;
- Creation of a Student Leadership Team to develop a mission and vision for the partnership;
- Providing internships or volunteer positions focused on individual students’ interests; and
- Allowing students to take charge of and manage programs.
The results of this assessment will be used improve Simmons’ existing community partnerships. A committee will meet to discuss recommendations and draft an action plan. Furthermore, the Scott/ Ross Center will conduct focus groups to follow up with the liaisons at the community organizations for more information on specific questions and strategies. We will also contact FWS students to participate in focus groups to discuss the findings of the study.
These three areas of focus — student development, student evaluations, and partner evaluations — will be used to improve the Scott/ Ross Center’s existing community service FWS programs, trainings, and community partnerships. A student development framework will link our work with college undergraduate students to well researched student development theory and practice. This framework can also be incorporated into future student evaluations, allowing the Center to measure whether students have met the specified learning objectives.
The existing student evaluation will continue to provide significant information regarding student outcomes at different points in their undergraduate careers. Information about the needs of community partners and the current state of the partnerships will be used in combination with the student development and evaluation research to enhance the overall experience of the students and the community partners. By supporting the development of college students, responding to community needs, and cultivating meaningful partnerships with community organizations, Simmons can develop students who are great tutors, mentors and volunteers today, and who are lifelong active citizens in their communities.