September 17, 2012

Executive Summary

Background

During the spring 2012 semester, UW-Madison and four peer institutions participated in a “quick turnaround” eText pilot based on a business model adopted by Indiana University’s eText initiative. The pilot was sponsored by Internet2 and involved McGraw Hill, an eTextbook provider, and Courseload, a software company that provides a feature-rich eReader platform for aggregating eTextbooks. The Internet2 contractual agreement involved a $20,000 flat fee which enabled UW-Madison to provide free eTextbooks to over 700 students in five different courses. Students and faculty in the pilot courses could access their eTexts via most any Internet-capable device by logging into their Learn@UW course and launching the Courseload eReader. The pilot gave UW-Madison an opportunity to partially field test an enterprise-service eText model that has been successfully implemented at Indiana University. The pilot provided valuable experience in setting-up a technical and support infrastructure, working with a textbook publisher and eReader software vendor, evaluating student and faculty attitudes toward using eTexts, and investigating potential cost savings for students.

Implementation

Five high-level campus leaders served as project sponsors for the pilot. A project manager coordinated the implementation process involving 10 sub-teams of individuals from various campus units. The sub-teams completed the following implementation tasks:

  1. Faculty/Course selection
  2. Contracting and legal agreements
  3. Internal/external communications
  4. Technical infrastructure and LMS integration
  5. Help Desk and user support
  6. Communication and relationship with bookstores
  7. Faculty, staff and student training
  8. Accessibility
  9. Evaluation
  10. Looking forward – beyond the pilot

Evaluation of Project Goals

Four main project goals were originally identified. In retrospect, some of these goals were overly ambitious for a first “quick turn-around” pilot. Nonetheless, initial progress has been made and will continue in additional future pilots.

Goal 1: Recommend a sustainable University of Wisconsin-Madison business plan that shapes an eText publisher model that can be most advantageous to students.

Despite the fact that the contractual conditions of the Internet2-sponsored pilot limited our transactions with publishers and eReader vendors, the general approach of the Indiana eText business model seems feasible for adoption at UW-Madison. However, because we were not able to adequately assess student reactions to required fee-based textbooks, this goal should continue to be pursued in future pilots. Fully accomplishing this goal would also require explorations of alternative business models or at least adaptations of the Indiana University model.

Goal 2: Develop teaching and learning IT architecture that allows students and faculty to access eTexts and other open content via course management systems and set a timeline for future access from the university’s online portal.

We were able to technically implement the authentication, access, and support systems that underlie the Indiana model. There were concerns about sharing student data with third parties as well as D2L data transfer protocols used in the pilot. It appears these concerns can be surmounted, however, if the model were to be fully implemented at UW-Madison. Necessary conditions for access from the portal were discussed, as were architectural challenges and opportunities regarding open content. Of greater importance is the need to establish a mission-critical 24/7 Help and Support system when implementing a centralized campus eText service. A coordinated plan involving the DoIT Help Desk, DoIT’s System and Network Control Center (SNCC) Sys Ops, and the Learn@UW Support Team was essential even for this small-scale pilot implementation.

Goal 3: Work with publishers, faculty, and software companies to ensure that eText, open content, and eReaders are accessible.

During the pilot, it was discovered that Courseload was not 508-compliant. In an end-of-semester survey, several students identified themselves as having a visual disability, but only one out of 722 students requested and received special accommodations. Courseload was alerted to this issue by all of the colleges participating in the pilot and has subsequently accelerated development efforts for building an accessible eReader, which should be available for field testing in early 2013. Throughout the pilot, we also discovered that publishers, such as McGraw Hill, are not providing material in formats that can be made accessible by eText aggregator companies like Courseload. UW-Madison and several other institutions are willing to work with these vendors and have joined Courseload’s “Accessibility Advisory Board.”

Goal 4: Evaluate student and faculty attitudes toward using eTexts, impact on student learning, and cost savings for students.

During the pilot, the four participating institutions developed and shared common evaluation instruments and published the Internet2 eText Pilot Final Report. UW-Madison also performed additional evaluations which focused primarily on student and faculty satisfaction with eText usage, the results of which differed slightly from the multi-institutional evaluation. Key findings from the Madison survey indicate that UW students viewed eTextbooks less favorably than the results shown in the multi-institutional report. This is likely due to the fact that the Madison student survey instrument used direct comparisons between eTextbooks and paper textbooks, whereas the multi-institutional survey did not. Three noteworthy conclusions of the Madison study were:

  • Most students perceive paper texts as providing a slightly or significantly better learning experience than eTexts.
  • Most students consider price and portability as the two top considerations in choosing to purchase an eText; however, compatibility with mobile devices and tablets is of low importance.
  • If price is the same, most students would prefer paper texts; but a $30 price discount on eTexts reverses the preference.

Faculty attitudes toward eTexts were influenced largely by perceived benefits to students (cost, portability, and learning). Few if any faculty used the available Courseload eReader tools to engage students more deeply with textbook content; nor did they encourage students to use available eReader tools for highlighting, collaborating, and posting questions. Overall, faculty found the eText fairly easy to use, but in hindsight thought that additional training would have helped them utilize the instructor tools more effectively.

Moving Forward

This initial pilot suggested that eTexts offer considerable promise in a higher-education setting:

  • An eText business model similar to that used at Indiana University could be adopted at UW-Madison; however, specifics on costs and sustainability need to be further investigated.
  • Students and faculty were able to access eTexts through UW-Madison’s D2L course management system. The necessary authentication, access, and support systems were implemented but would need to be enhanced for a full-scale implementation.
  • Students saw the price and portability potential of eTexts. If eTexts are significantly cheaper than paper, students would likely prefer them.

The pilot also revealed gaps in our knowledge of eTexts and raised questions regarding institutional support of the technology:

  • The feasibility of the eText business model used in the pilot might change if students are required to actually purchase e-textbooks. This should be explored more fully in future pilots, along with other business models or variations of the Indiana University model.
  • Several IT infrastructure concerns need to be addressed, such as eText access through My UW-Madison, the university’s online portal, and 24/7 eText help and support. Open content will pose architectural challenges as well as potential opportunities. Concerns about ensuring the privacy of student data must also be addressed.
  • Our experience showed that eTexts need to be made accessible for students with disabilities. This can be addressed through continued collaboration with eReader vendors and publishers.
  • Most faculty did not use available eText tools to engage students with content. Additional training and encouragement might have helped them use the tools more effectively.

These gaps and concerns can be explored further through additional pilot projects. Future trials of this promising technology will also provide opportunities for more robust evaluation of potential benefits to students in terms of cost savings and learning outcomes.

Back to Top

Project Overview

Background

In late fall of 2011, the Internet2 organization initiated a “quick turn-around” eText pilot for the spring 2012 semester. Five institutions — UW-Madison, University of Minnesota, Cornell University, University of Virginia, and University of California, Berkeley — agreed to participate. After a limited pilot implementation, UC Berkeley elected not to participate in the evaluation. The remaining four universities developed and shared evaluation instruments.

Based on a business model adopted by Indiana University’s eText Initiative, the pilot provided a feasible way for individual institutions to explore eTexts without having to negotiate with publishers and eReader platform providers. Indiana University and Internet2 Net Plus Services negotiated a contract with McGraw-Hill and Courseload to provide publisher content and Courseload’s eReader platform for up to 1,000 students per institution for a flat fee of $20,000. The incentive was that students in the pilot courses would receive free eTexts. During the pilot, the four participating institutions developed and shared common evaluation instruments enabling them to pool data and generate a final report that includes survey results from more than 2,200 students and more than 20 faculty. The findings of this multi-institutional evaluation have been published in the Internet2 eText Pilot Final Report, which has been made available nationally. Individual institutions were also encouraged to conduct their own independent evaluations. This report provides in-depth information and analysis from the UW-Madison eText pilot experience and differs in many ways from the multi-institutional evaluation.

Goals

Advances in technology have presented a unique opportunity to evaluate the costs and benefits of providing electronic textbooks to students. The original goals for the UW-Madison pilot were:

  1. Recommend a sustainable UW-Madison business plan that shapes an eText publisher model that can be most advantageous to students.
  2. Develop teaching and learning IT architecture that allows students and faculty to access eTexts and other open content via course management systems and set a timeline for future access from the university’s online portal.
  3. Work with publishers, faculty, and software companies to ensure that eText, open content, and eReaders are accessible.
  4. Evaluate student and faculty attitudes toward using eTexts, impact on student learning, and cost savings for students.

Purpose of this Report

This report summarizes the UW-Madison eText pilot experience and includes data and findings not presented in the Internet2 Final Report. Essentially, this report tells the story of what the UW-Madison project team accomplished, how it was done, lessons learned, and evaluation findings and recommendations for moving forward. In retrospect, we realized that attainment of project goals through a single “quick-turn-around pilot” was somewhat unrealistic and at best a first step. We therefore have embarked on a second eText pilot for the fall 2012 semester which will enable us to more fully address the above goals.

Back to Top

Implementation Approach

Project Team Structure

A group of UW campus leaders initiated the eText pilot following discussions with Internet2 and Indiana University. Five individuals were asked to become project Sponsors to provide direction and review project status. Project sponsors included:

  1. Bruce Maas, CIO and Vice Provost for Information Technology
  2. Joanne Berg, Associate Vice Chancellor for Enrollment Management
  3. Ed Van Gemert, Director of Libraries
  4. Aaron Brower, Vice Provost for Teaching and Learning
  5. Linda Jorn, Director of Academic Technology

An Implementation Team was formed consisting of a project manager from the Division of Information Technology (DoIT) and approximately 12 individuals from various campus units including Enrollment Management (Registrar), Libraries, the McBurney Accessibility Center, and various other DoIT units. The team met weekly during the early pilot phases and as needed throughout the remainder of the semester. Two students from campus student government were also part of the initial team; however, they did not actively participate as the semester progressed.

Members of the Implementation Team were grouped into 10 task-focused sub-teams. A team leader from each sub-team regularly updated the full project team on developments within their group. The sub-teams focused on the following areas:

 

  1. Faculty/course selection
  2. Contracts and legal
  3. Internal/external communications
  4. Technical infrastructure and LMS integration
  5. Help desk and user support
  6. Communication and relationship with bookstores
  7. Faculty, staff, and student training
  8. Accessibility
  9. Evaluation
  10. Looking forward/beyond the pilot

 

Faculty/Course Selection

The Faculty/course selection sub-team had a maximum of two weeks to select courses for the pilot. The following criteria were used for selecting candidate courses:

  • Large class over 75 students
  • Course uses a McGraw-Hill textbook available in an eText format
  • Faculty/instructor is actively using the D2L Learning Management System (LMS)
  • Course is either online or hybrid
  • Course uses a relatively expensive textbook
  • Course is mainly for undergraduates (might include an upper level undergraduate course with graduate students)
  • Course is not cross-listed
  • A diversity of faculty and disciplines are represented

The selection process was challenging not only because of time constraints but also because of difficulties communicating with faculty during the December- January semester break. Within 10 days, six courses were selected and approved by the project Sponsors. Formal confirmations of participation were communicated to faculty and students in each course as well as with McGraw-Hill, Courseload, and D2L Utility support. Sub-team members helped faculty to notify students of the free eText option before the start of classes. This effort included student emails as well as postings in the online Course Guide, Faculty Center, and Student Center services and coordination with the bookstores. Unfortunately, one selected course was dropped a week before the start of classes, because McGraw-Hill was not able to provide an eText version of the textbook in time. The final pilot courses are listed in the table below.

  Faculty Course Students
1 Felix Elwert Sociology 120: Marriage and Family  293
2 Shawnika Hull Journalism 565: Effects-Mass Communication  75
3 Suzannah Sandrik Engineering, Mechanics & Astronautics 202: Dynamics  88
4 Michael Titelbaum Philosophy 211: Elementary Logic  83
5 Kevin McSweeney Environmental Science 101 (Soil Science) 183
6 Steve Ventura

Total students: 722

Contracts and Legal

A project team member from the CIO’s office worked with the Internet2 Net Plus Service and campus legal counsel to obtain a license for the right to use the Courseload product. Internet2 expedited the process by providing a standard contract that all four participating pilot institutions could use. The contract addressed issues important to UW-Madison related to FERPA and State of Wisconsin purchase requirements. The contract also specified conditions and constraints regarding how long students and faculty could have access to the eTexts as well as data generated surrounding their use.

Internal and External Communications

Several communication mechanisms were utilized throughout the pilot to facilitate information sharing among team members, faculty, the university community, and the general public. These included:

  • EText List Serv: Implementation team members utilized a listserv to communicate project developments, questions, resource information, news, and meeting notes.
  • EText Project Wiki: The entire project team used a project wiki to store documents and obtain in-depth updates and information on all aspects of the project. Each sub-team maintained their own section of the wiki.
  • EText Pilot Email Account: A separate email account was set up for team members to communicate with faculty, the university community, and the public. Project team members could use this email account, instead of their personal accounts, for all external communications during the pilot.
  • EText Project Web Site: A project web site was used to inform the university community and the public about the pilot. This site was developed and maintained by the Communications Director in DoIT, who established it under the Office of the CIO.
  • Weekly Sponsor Updates: At the close of every week, the project manager provided a weekly update to the Sponsors to summarize project developments, problems, and items of interest.

Technical Infrastructure and LMS Integration

Courseload’s web-based eReader application was used by students and faculty to access their McGraw-Hill eTexts through a variety of Internet devices. This required that the Courseload eReader be accessed through Learn@UW, UW-Madison’s D2L Learning Management System. The Learn@UW Utility team worked with Courseload to map each D2L course with the respective eTextbooks. The Learn@UW Support team set up hyperlinks in the top banner of each D2L course homepage, which, when clicked, opened the Courseload eText reader in a new browser window. Thus, students were able to seamlessly authenticate into Courseload to access their eTexts via a single sign-on through Learn@UW using a secure HTTPS connection.

Help Desk and User Support

The DoIT Help Desk provided technical support for all students and faculty using the Courseload eText reader. Two weeks prior to the beginning of the semester, Help Desk staff were trained by Courseload to support the tool. Help Desk support processes were reviewed and finalized. The updated support documents were published in the Help Desk Knowledgebase (KB). Service and routing information was added to WiscIT, the Help Desk’s call tracking software. The Help Desk worked collaboratively with other eText team members to develop support procedures. The Help Desk Leadership team worked on a communication plan to ensure that all staff were ready to support the service. A temporary 24/7 support plan was also developed to ensure that a support system was in place for students after-hours when the Help Desk was closed. In addition, paper copies of each pilot course textbook were made available at the reserve desks of College and Wendt Libraries as back-ups.

Communication and Relationship with Bookstores

UW-Madison has no affiliation with any particular bookstore; however, each of the instructors involved in the pilot used the University Book Store to stock and sell textbooks to their students. Therefore, it was essential to involve University Book Store, especially during the early phases of the pilot. A project team member from Enrollment Management was instrumental in working with University Book Store management.

The Book Store staff were cooperative and supportive in many ways, including:

  • Communicating regularly with the Registrar’s Office and Faculty Selection Team regarding the courses and texts involved in the pilot.
  • Helping to make students aware of the free eTexts.
  • Working with the McGraw-Hill sales representatives to ensure reduced supplies of hardcover books for the pilot courses and also maintaining a supply of hardcover texts for students who chose to opt-out of the pilot.
  • Placing signage on book shelves for participating pilot classes to notify students of the free eText.
  • Honoring and reinforcing a flexible refund policy for students who had already purchased a textbook and wanted to return it.

Faculty, Staff and Student Training

Courseload provided formal training for faculty and technology support consultants by scheduling several one-hour live webinars, each recorded for later on-demand access. About a dozen project support team members attended a Courseload orientation session in early January. Four of the six pilot faculty were able to participate in a one-hour live webinar event with Courseload trainers. One faculty member remotely accessed the webinar recording, and one did not partake in any training before the pilot. All faculty were provided with the Courseload user documentation; however, feedback indicated that this documentation went largely unused by faculty.

Formal student training was not provided, based on recommendations from Indiana University and Courseload. Courseload did, however, provide a one-page instruction document along with a “Click-Through Agreement” form when students accessed the eReader for the first time. According to Courseload, this rather spartan instructional aid would be sufficient for most students. Based on the limited number of Help Desk support calls received throughout the pilot period, it appears that students indeed did not require formal training in using the eReader.

Accessibility

During the implementation team’s orientation to the Courseload platform, it was learned that the Courseload eReader did not meet campus accessibility requirements. This meant that students using screen readers would not be able to access textbook content through the Courseload eReader. An Accessibility sub-team was immediately formed to address student access needs during the pilot and to ensure that subsequent eReader decisions included a consideration of accessibility. The Accessibility sub-team included staff from the campus disability resource center, campus IT services, and the library. The team met regularly throughout the semester.
Accessibility remained high on the agenda during the pilot period. In May, the Accessibility sub-team and the director of the campus disability resource center met with Courseload’s CTO and CEO to communicate campus priorities and hear about vendor plans for addressing accessibility. See later sections of this report for more details about accessibility issues related to Courseload.

Evaluation

The UW Evaluation Team consisted of three evaluators from DoIT’s Department of Academic Technology. Limited Institutional Review Board (IRB) approval was obtained to administer in-person paper surveys in each of the pilot courses. Evaluators favored a paper survey approach over an online survey in order to obtain greater response rates from students. UW evaluators agreed to use a set of student survey questions developed by Indiana University and the inter-institutional evaluation team. In addition to these common questions, the UW evaluators, in collaboration with a faculty member participating in the pilot, appended additional questions to the survey to gain more insight and address issues not covered by the shared survey questions. Detailed results of student surveys are provided later in this report.

Individual interviews with each faculty participant were also conducted using the “Faculty Discussion Questions” created by the multi-institutional evaluation team. Results of our faculty interview data are provided later in this report.

The Evaluation Team obtained weekly updates from the Help Desk regarding reported issues from students and faculty. In addition, an evaluator contacted faculty via telephone and emails throughout the semester to check in and note any problems.

Looking Forward/Beyond the Pilot

Recognizing a need to deal with both the pilot implementation and the more general eText project goals, a sub-group was formed to consider broader issues related to eTexts beyond the current pilot. This group collected and shared information about eText activity at other campuses. Group members participated in discussions with representatives from the University of Minnesota Open Text Catalog about the future of eTexts and participated in a regional conference on eTexts.

The broader context of eText use and development gained by this group is now being used to inform future pilot activity and evaluation planning for an upcoming Fall 2012 eText pilot.

Back to Top

Evaluation of Project Goals

The section summarizes progress in accomplishing the four original project goals and provides suggestions for continuation of this work in future pilots.


Goal 1: Recommend a sustainable University of Wisconsin-Madison business plan that shapes an eText publisher model that can be most advantageous to students.

This first “quick turn-around” pilot afforded an initial plunge into the world of eTexts for UW-Madison at an enterprise level. The project provided an opportunity to partially field test a business model pioneered by Indiana University within the context of D2L, UW-Madison’s LMS enterprise system. The pilot was limited, because it involved only one publisher (McGraw-Hill) and a single vendor-hosted eText aggregator-reader product (Courseload). More specifically, we were unable to adequately field test many aspects of the Indiana model. That is, students received free eTexts with no direct expenses. Although the Indiana model could likely be adopted at UW, our pilot was not able to directly assess student reactions to a required fee-based textbook, other than anonymous student responses to hypothetical survey questions. Furthermore, by forfeiting the experience of negotiating reduced textbook prices with publishers, we were not able to ascertain actual cost savings for students. Nor were we able to study how the model impacts decision-making behaviors of publishers, instructors, and students over time. Recent debates surrounding the Indiana model have raised questions about its long-term viability as a cost-saving option for students.

To adequately investigate and recommend an eText business model, our work could expand beyond evaluation of the Courseload-based Indiana model. Fully accomplishing this goal would require at least one or two more pilots that explore alternative business models. We recommend some of the following issues be investigated in future pilots:

  • Learn more about how our instructors currently select and use eTexts on their own and/or with publishers.
  • Engage in a more realistic pilot that does not include artificial costs, so we can experience negotiating with publishers, provide students with actual textbook purchase options, and investigate what they prefer and choose.
  • Investigate other major commercial alternatives to Courseload.
  • Collaborate with peer institutions to develop reliable measures of costs and cost savings associated with various eText business models.
  • Investigate in greater depth the features and functions of eTexts that the majority of students and faculty perceive as more valuable than paper texts.
  • Evaluate students’ long-term access to content and the access and ownership rights to student annotations or other student-instructor-generated content of various models

Goal 2: Develop teaching and learning IT architecture that allows students and faculty to access eTexts and other open content via course management systems and set a timeline for future access from the portal.

Within the constraints of this pilot, we launched Courseload only from Learn@UW (D2L) to access only McGraw-Hill content. With those constraints and the very aggressive timeline, this did not represent the ideal model for an actual implementation. We were able, however, to technically implement the authentication, access, and support systems that underlie the Indiana model. There were some concerns about sharing student data with third parties and about CMS data transfer protocols used for the pilot. However, it appears that these concerns can be surmounted if the model were to be fully implemented at UW-Madison.

The pilot also allowed us a few opportunities to explore directions moving forward. First, we could work with the IMS Global Learning Tools Interoperability (LTI) architect, Charles Severance, to propose a future direction for LTI that aligns it with SAML/Shibboleth (http://www.slideshare.net/csev/connecting-ims-lti-and-saml-draft). Second, we devoted some time in a pilot subgroup to articulate the necessary conditions for access from My UW-Madison, the university’s online portal. In a future pilot, we expect to follow up on these issues as well as on the architectural challenges and opportunities regarding open content.


Goal 3: Work with publishers, faculty, and software companies to ensure that eText, open content, and eReaders are accessible.

From the perspective of project team members, this goal was partially addressed. We now have some baseline experience working with an eText software vendor, a major book publisher, and other universities to better understand and deal with accessibility and 508-compliance issues related to eTexts. During the pilot, our Accessibility sub-team quickly realized that the Courseload eReader was not 508-compliant, but there was not much we could do about it.

In the student survey administered at the end of the semester, 10 students out of 722 participating in the pilot identified themselves as having disabilities that made the use of the eText difficult. For all of them, the nature of their disability was vision related. At the outset of the course, however, only one student requested special accommodations to access a course text. The disability resource center, working directly with the publisher, was able to provide an alternative text format within a reasonably short  time. Although this accommodation provided alternative access to the actual textbook material, it functioned independent of the Courseload eReader. Consequently, the student in question did not have access to highlighting, annotation, and collaboration tools as well as instructor notes, all of which are interactive affordances provided by the Courseload eReader tool. This student’s needs were limited and most of the above-mentioned eReader features were not widely used in his class, so his learning experience was not significantly disadvantaged compared to other students in the class. This incident underscores, however, the potential inaccessibility (and related Section 508 violations) that could occur if instructors encourage student use of the Courseload eReader features.

During spring 2012, University of Minnesota Disability Services conducted a series of explorations and focus groups to determine how eTextbook and supplemental course materials delivered through the Courseload eReader platform were accessed and used by students with disabilities. The report concluded:

“The Disability Services evaluators understand that Courseload has a desire to make their product and delivery of eTexts accessible. Courseload should certainly be applauded and encouraged for their efforts. However, the Disability Services team cannot support the adoption of any application or system that does not provide an equivalent learning environment for students with disabilities. Ongoing efforts are needed on three fronts. First, faculty and course developers must be made aware of accessibility and learning style considerations. They must also be given the resources and encouragement for understanding and implementing a universal design for learning approach. Second, universities, colleges, government entities, and other organizations need to collaborate in putting pressure on publishers to produce accessible eTexts. Thirdly, the tools used to deliver eText content must be accessible and not interfere with natively accessible materials.”

The full Minnesota report is available on request.


Goal 4: Evaluate student and faculty attitudes towards using eTexts, impact on student learning, and cost savings for students.

The UW-Madison evaluation team administered student surveys in each class and conducted end-of-semester interviews with faculty and TAs involved in the pilot courses. Because this pilot constituted our first study of eTexts usage on the UW campus, the need for solid baseline data regarding user acceptance, receptivity, and attitudes toward using eTexts superseded the need for in-depth analysis of learning outcomes. Consequently, our evaluation focused primarily on student and faculty satisfaction with eText usage and suggestions for improvement.

Student Surveys

During the early project planning phase, the UW-Madison evaluation team collaborated with evaluators from the other Internet2 pilot institutions to develop a common set of student survey and faculty evaluation questions. The UW evaluation team, in collaboration with a faculty member participating in the pilot, developed additional questions that were appended to the student survey questionnaire. These questions more directly solicited comparisons of eTextbooks to their paper textbook counterparts on several dimensions including price, perceived learning impact, and convenience. Detailed results of our UW student survey data are provided in the next section. It is important to highlight that the results of our UW student evaluations differed from those found in the Internet2 (I2) report in the following ways:

  • When directly comparing paper textbooks to eTextbooks, UW students viewed eTextbooks less favorably than the results shown in the I2 assessment. In part, this could be due to the fact that the I2 student survey instrument neglects to make a direct comparison between eTextbooks and paper textbooks.
  • The I2 findings assessed the usefulness of Courseload’s eReader features, but never assessed how often those features were used (if at all). Taking into account the frequency of use for a given feature provides useful additional information and changes the picture of the perceived value of specific features.
  • Price was explored in more depth in the UW-Madison assessment. This more direct examination of student price sensitivity around eTexts suggests that price benefits need to be significant in order for students to view eTexts favorably.

The findings summarized above are explained in more detail in the next section.
Overall, we’ve made some progress in evaluating general user satisfaction with eTexts but not in evaluating their impact on learning. An assessment of learning impact would involve surveys, learning analytics, coordination with peers, and closer work with IRB. Our eText evaluation team is well-positioned to make this happen in future pilots. However, the work done on evaluation to date raises some important issues that need to be considered before launching into larger-scope assessments on learning impact.

eText Usage Information from Vendors

General usage statistics from Courseload’s learner analytics database were provided by the vendor. The table below shows the total student eText usage activity for each of the pilot courses. Mark-ups listed in column 4 include student bookmarks, annotations, sticky notes, and highlights. By far the most frequently used mark-up tool for students was the highlighter. In one course alone (Soc. 120), students made 28,581 highlights, according to Courseload’s analytics.  Although a temporary eText download option was available, all students accessed the eText online via a live Internet connection.

Course Total Students Total Pages Viewed Total # of Mark-ups Pages Read off-line
Philosophy 211 92 7575 859 0
Journalism 565 79 19753 3402 0
Sociology 120 281 61748 31316 0
Envir. Studies 101 188 17455 1898 0
Engineering 202 74 7108 287 0
Totals 714 114340 37951 0

The numbers above should be interpreted with the understanding that students in the pilot courses were able to opt-out of using the eText material entirely or could use a combination of eText and paper text formats throughout the semester.

Numbers of paper texts purchases from McGraw-Hill were provided. According to McGraw-Hill, only 75 students purchased black/white paper versions of the textbooks at a reduced cost of $28 per book. This number does not provide an accurate count of hardback textbook utilization, however, because students were free to purchase new hardcover textbooks from the University Book Store, purchase used books from the used textbook market, or use a friend’s textbook, none of which we were able to track. An informal class survey conducted by one of the pilot instructors teaching a class of 280 students revealed that a third of his students claimed they were using paper textbooks instead of the free eText at least some of the time. The student survey data in the next section sheds more light on actual eText usage by students.

Faculty Interviews

The UW-Madison evaluation team collaborated with evaluators from the other pilot institutions to develop a common set of questions for conducting interviews with faculty. We found the group questions to be adequate and did not alter or add to them. Interview results from our UW faculty did not differ significantly from the other participating I2 institutions, as summarized in the final I2 Inter-institutional Report. Our report focuses mainly on faculty perceptions related to training, technical benefits/hurdles, use of eText in class, benefits to students, and misconceptions. Details of our faculty interview data are also provided in the next section.
The questions asked of faculty were:

  1. How easy was it for you to learn how to use the eText functionality? What additional training and support from UW would have been helpful?
  2. Did you make use of the enhanced features of the reader (e.g., marking up the text, highlighting), and encourage the use by your students? If so, how? If not, why not?
  3. Did you use the statistics feature to track how students are utilizing the eText? Example: open the eText, annotate, highlight, etc.
  4. What were the benefits and using the eText compared with printed textbooks? What were the disadvantages using the eText compared with printed textbooks?
  5. Did anyone in the class have web accessibility problems with the eText and associated materials? By web accessibility, we mean all disabilities that affect access to the web including visual, auditory, cognitive, speech, etc.
  6. How do you think using the eText affected student learning?
  7. How did using the eText affect the quality of your interactions with students?
  8. Was the eText useful in evaluating your students’ progress?
  9. If you had the option to order an eText for your course in the future, what would persuade you to do so? What would dissuade you?
  10. What do you think the criteria should be in determining the future use of eText at UW?

Back to Top

Student Evaluation Results and Findings

Overview

Students in the five pilot courses received an in-class paper survey at the end of the semester to assess their perception and use of the Courseload eTextbooks. A total of 563 students responded out of a potential of 722 (78% response rate). Of these 563 individuals, 73.9% (416 students) indicated they had actually used the eTextbook. Some questions were specifically asked of eTextbook users, and other questions were asked of all students.

The student assessment was organized into these five areas:

  1. Use and Perceived Utility of eReader Features
  2. Course Communication and Content
  3. Student Considerations for Selecting eTextbooks
  4. Some Detail on Price Sensitivity
  5. Overall Student Perceptions of eTexts

Before presenting the findings in detail, it is worth noting three specific findings:

  1. Initial Difficulty: 46.5% of student eText users indicated that they found the eTextbook difficult to use “the first few times.”
  2. Features and Navigation: 63.7% of users agreed /or strongly agreed that the features and navigation within Courseload were easy to use.
  3. Perceived value of eTextbooks: 80.0% of eText users (and 60.3% of non eText users) agreed / or strongly agreed that they “see the value of eTextbooks.”

These numbers provide an important lens for interpreting subsequent findings. Any perceptions are not inordinately shaped by negative attitudes toward technology, poor experiences with the interface, or a negative attitude about eTextbooks generally.

Use and Perceived Utility of eText Features

Of the eReader features assessed in this evaluation, most were never used by students. In fact, only 40.3% of eText users indicated that their instructor encouraged the use of annotation, highlighting, and note-sharing. This has some correspondent validity with what was gleaned from faculty participants. Faculty were interested in many features, but did not necessarily feel confident enough to widely use and promote those features to their students.

Figures 1a and 1b below provide a breakdown of feature use and perceived utility of four eText features. Of those who used the features, the utility is highest with features that allow instructors to provide commentary or add more material to a text. Interestingly, this is even perceived as more important than a student’s own ability to annotate text or the ability for students to see annotations and notes from other students.

eText-1-report_clip_image003

Figure 1a Student Use of eText Features
eText Feature %Never used % Used
Your highlights and annotations 45.9% 54.1%
Material the instructor added to the eText 55.5% 44.5%
Instructor highlights and annotations 59.0% 41.0%
Other student highlights and annotations 79.2% 20.8%

Note: Only thise who USED a feature were used in the computation of a feature’s perceived utility.

usefullness

Figure 1b: Student Perceived Utility of eText Features
eText Feature % Not At All Useful % Useful
Your highlights and annotations 50.0% 50.0%
Material the instructor add to the eText 23.1% 76.9%
Instructor highlights and annotations 29.9% 70.1%
Other student highlights and annotations 48.1% 51.9%

Recommendation: Given the high utility students place on instructor centered features, train faculty to use and be comfortable with these features.

Course Communication and Content

A big consideration at the outset of this pilot was whether eTexts would facilitate greater interaction with instructors and fellow students in a class. Given the relatively low frequency of feature use, these results were important, but not surprising. An additional consideration (or hope) would be that eTexts would facilitate greater engagement with course material. What was surprising was that students largely disagreed with the statement that “I read more of the assigned material than I would with paper textbooks.” This is particularly disconcerting given that the average amount of material read by students is just over 50%. Figures 2a, 2b and 2c outline these findings.

course-communication-2a

course-communication-2b

Figure 2b
Allowed me to interact more with my professor %
Not at all 57.2%
A little 24.0%
Quite a bit 11.3%
A great deal 7.4%

course-communication-2c

Figure 2c
Allowed me to interact more with my professor %
Strongly Disagree 25.4%
Disagree 33.2%
Agree 12.8%
Strongly Agree 5.1%

NOTE: The average percentage of total course material students reported reading was52.9%
(14.6% of students reported that they had completed ALL the reading in the course)

Recommendation: Consider what we realistically expect eTexts to do in the classroom. Students have a limited amount of time that they need to spend efficiently.

Student Considerations for Selecting eTextbooks

A consideration of eText features and potential impacts on a course raises the question ”What do students view as important considerations when they are deciding to use an e-textbook or not?” Perhaps not surprisingly, price is at the top of the list with over 86% of students moderately and strongly agreeing that an eText needed to cost less than a used or rented paper text. What was surprising is that students rated mobile use (use on tablets and cellphones) the least important of all the features they were asked to rate. Figure 3 below provides a breakdown of these feature ratings.
eText-1-report_clip_image024

 

Figure 3: Student considerations when selecting eTextbooks
Student considerations when selecting eTextbooks % Moderately agree % Strongly agree
Costs less than a used or rented textbook 20.3% 65.9%
Is more portable than tradition textbooks 30.6% 44.2%
Accessible without an internet connection 29.3% 42.0%
Is more environmentally friendly than traditional textbooks 25.1% 27.7%
Available for entire academic career 22.2% 25.0%
Includes bonus material (videos, self-assessments, etc.) 27.8% 19.6%
Capbility to share notes questions with instructor and students 26.1% 20.7%
Readable on a handheld device (iPhone, etc.) 16.3% 17.9%
Readable on a tablet(iPad, Galaxy Tab, etc.) 15.4% 16.3%
Note: Mobile availability is the least important consideration for students.

Recommendation: Price is a key feature for students. Additionally, if feature prioritization is important, compatibiity with the mobile devices is of lowest importance with students.

 

Student Price Sensitivity and eTextbooks

Given the importance of price to students, it seemed worthwhile to get a sense of a tipping point around textbook cost. Obviously, if eTextbooks were more expensive, students would opt for paper texts. But, what if the price were the same? What if there was a modest price break for eTexts? When the price is the same, more than 75% of respondents prefer paper texts. When there is a $30 price discount for an eText , this trend nearly reverses itself (in this condition 70.5% would purchase an eText). Figure 4 below provides detail around these key issues surrounding price.

eText-1-report_clip_image028

 

Figure 4 Sensitivity and eTextbooks question 1
If and eText and the paper text cost the same, how likely would you be to purchase the eText %
Very unlikely 50.5%
Somewhat unlikely 25.3%
Very likely 6.7%
Figure 4 Sensitivity and eTextbooks question 2
If an eText cost $30 less than the paper text, how likely would you be to purchase the eText? %
Very unlikely 8.4%
Somewhat unlikely 21.2%
Somewhat likely 32.5%
Very likely 38.0%

Recommendation: It is difficult to overstate the importance of price in the decision making of student in the use of eTextbooks. If eText were used on a wider scale, significant price incentives would definitely encourage student adoption.

Overall student Perception of eTexts

A key piece of information is an overall comparative assessment of eText versus paper texts. To this end, a final question was asked: “Overall, do you think eTexts or paper texts provide you a better learning experience?”The following choices were provided: eText significantly better, eText slightly better, no difference (between eText and paper), paper slightly better, and paper significantly better.

Figure 5 below highlights the distribution of this question across users of eTexts in this pilot and non-users of eTexts in this pilot. While there is some variation across these groups, the overwhelming perception across all participants was that paper provided a slightly or significantly better learning experience. In fact, less than 20% of eText users in the pilot view eTextbooks as providing a significantly or slightly better learning experience.

eText-1-report_clip_image030 

Figure 5: