Assessment Committee: Christine Elliott (Library), Burton Callicott (Library), Chris Korey (FYE), Liza Wood (FYE)
N = ~800 Spring Semester FYE Students
n = 40 randomly selected assignments for piloting of assessment and rubric optimization
Rubric Development: 40 randomly selected assignments were blinded and examined in order to optimize our rubric and to examine the assessment criteria to ensure inter-rater reliability. The optimized rubric was created in two sessions, which also allowed us to norm our samples.
Measure 1: An ideas and keyword concept map activity will be administered to students who complete the FYE library session to assess their ability to generate keywords related to a research topic in order to search library databases for supporting evidence. A rubric will be used to assess student responses.
Performance Target: Set to baseline.
Results: The 40 concept map/keyword activities were analyzed with the optimized rubric. Each assignment was reviewed by two independent reviewers. Scores that differed by a factor of 2 were discussed by the group as a whole and the scores were then updated after that discussion. The two scores were averaged for each rubric category and then all 40 assignments were averaged to provide an overall score for each particular category of analysis. The rubric was scaled from 0 to 3.
Concept Map Score: 2.17
Keyword and Search Concepts: 2.55
Discussion of the Results: The assessment committee found that students needed more instruction on how to create a concept map that would best help them generate more specific keywords from broader search topics. These more specific keywords would allow them to use library databases more effectively. The results indicated that the librarians and faculty needed to focus more instruction on the overall approach to search processes that will best yield the most relevant information. These results will serve as the baseline for the 2016-17 assessment process.
Measure 2: A source identification activity will be administered to students who complete the FYE library session to assess their ability to identify and summarize appropriate sources. A rubric will be used to assess student responses.
Performance Target: Set to baseline.
Results: The 40 source identification activities were analyzed with the optimized rubric. Each assignment was reviewed by two independent reviewers. Scores that differed by a factor of 2 were discussed by the group as a whole and the scores were then updated after that discussion. The two scores were averaged for each rubric category and then all 40 assignments were averaged to provide an overall score for each particular category of analysis. The rubric was scaled from 0 to 3.
Book Choice: 2.3
Book Quality Statement: 1.725
Article Choice: 2.2
Article Quality Statement: 1.725
Discussion of the Results: The assessment committee found that while students were able to identify a book or article, they needed more instruction on how to better describe the relevance of their selected book or article. This is the first step on the way to an annotated bibliography that asks them to describe the choice of a source in more detail. Our results indicated that the librarians and faculty needed to focus more instruction on how best to summarize the quality of a particular source in relation to the research project they are working on. These results will serve as the baseline for the 2016-17 assessment process.
Changes for the 2016-17 Assessment Cycle:
FYE Embedded Librarian Training: Updates will be made to the training to help librarians improve their approaches to helping students create a better search process. There will also be a focus on helping students to create more specific statements about the quality of their identified resources. Finally a few additions will be made to the activities themselves to clear some student confusion in regards to designing and using the concept map.
In March of 2017, the FYE program underwent a SACSCOC external review. Aspects of the Executive Summary that touch on Information Literacy instruction are included here:
Information Literacy:
The librarians have a strong desire to work with the FYE faculty to ensure a positive learning experience for the students as well as enhance the development of student Information Literacy (IL) skills. Our review finds that the librarians are highly qualified and very competent for their role in the FYE program. They have excellent lesson plans, active learning strategies, and thorough library guides for students. The librarians follow the American Library Association (ALA) Information Literacy Competency Standards for Higher Education which is a “best practice” (4). The librarians provide detailed instructions in their preparation for the sessions and offer resources that have been developed for the courses. We reviewed examples of typical class sessions as well as the library guides that were developed to support the “general” sessions for FYE courses. It was noted that these guides were general in that they did not have a particular paper/project developed for specific course projects.
The major discussion in our time with the librarians was a concern over whether the learning objectives for the FYE Information Literacy (IL) sessions are realistic to meet in 50 minutes “stand alone” library sessions, particularly if those sessions have no link back to a paper or project in the FYE course. While the session importantly allows students exposure to library resources, staff, and how to develop basic IL skills, the time is not sufficient to meet the current learning objectives for the FYE course. Some of the learning objectives could possibly be moved to other courses such as writing or upper level curriculum courses.
We have several recommendations for the IL component of the FYE/FYSS courses:
1) It would be ideal if all FYE sections had some type of project or paper embedded within the course syllabus that would require students to utilize IL skills throughout the course. The library faculty have had much more success with sections where there is a direct course project/paper versus a general session where students may not utilize the research skills for the remainder of the semester. Again, while the exposure to the library resources is valuable, the lack of application after the session brings into question whether the learning objectives are being met. The librarians mentioned that they had facilitated additional sessions for a few FYE sections that had course specific projects, but by and large, the “one shot” sessions were most common. Thus, the recommendation to link the IL session to the FYE course would enhance the learning experiences for the students.
2) As an extension of this recommendation, the College may want to consider a stronger commitment to IL skills in the FYE. One of the evaluation team members has extensive experience with FYE courses where a course project is adopted by the FYE faculty member in cooperation with a librarian, which allows for more direct “hands-on” experience in the first IL session and related library guides. In these models, FYE courses have additional sessions (up to 4 total) to meet the IL outcomes. While this type of program may involve hiring of an additional librarian(s), the results are not only good for the students, but also for the FYE program. Many of the enhanced IL course projects have resulted in “active learning” experiences for the students such as campus improvement projects, mock government councils, poster presentations, simulated business presentations, etc., which bring positive exposure to FYE program as well as develop a strong base of IL skills at the start of the college career.
3) We recommend the use of “on-line” modules/materials for students to view prior to attending an IL session, provided there is sufficient grade linkage between the FYSS and the FYE to help students understand the importance of completing this activity. The librarians have already developed some on-line modules which can be viewed prior to a class session, particularly if 50 minutes is the only option. The librarians and the review team would NOT recommend that on-line modules be the only source of library contact. The time with the librarians (even if limited) is still important for introducing students to the library resources and faculty.
4) Pending which of these recommendations are feasible for the College, it seems that the overall learning objectives for the FYSS/FYE library time should be scaled back to focus on building the basic IL skills from the ALA of “research as inquiry” and “scholarship as conversation” (4). This change might allow for moving some of the objectives to writing and upper level courses, particularly the portions of Learning Objective 2 in the College’s IL objectives that relate to recognizing and classifying information as well as ethical and legal concerns. To fully achieve competence in these areas, more than a 50 minute session is needed.
