Interviews and direct observations

Methodology for background research

After researching what current experts in the field said about best practices for building discussion in classrooms, we conducted interviews and directly observed instructors and students. To identify factors that helped facilitate class discussion, we:

• Interviewed instructors and students who were involved in distance-learning classes.
• Interviewed instructors and students who were involved in discussion-based classes, in both traditional and virtual classrooms.
• Observed instructors and students interacting in classrooms, in both traditional and virtual classrooms.

We used this information to form our hypothesis and to identify what variable to differ in our test groups.

Goals for interviews and direct observations

Our goals for the interviews and observations were:

• To learn the benefits of discussion and see how discussions are structured in the classroom.
• To understand the problems with and benefits of distance learning.
• To identify and investigate various methods of communication in distance-learning courses that seem to work well.
• To understand the level of community that students feel in various classes.
• To identify factors that help establish a sense of community.
• To identify how a sense of community is evidenced.

Participants for interviews and observations

Our subjects for the interviews and observations included students and instructors who have participated in discussion-based classes, and students and instructors who have participated in distance-learning courses or courses with supplemental online forums.

Summary of findings from interviews and observations

After synthesizing our findings from the interviews and observations, we noticed several patterns. Instructors and students surfaced a few key issues as being the most important in building classroom community. Table 1 shows the issues that we learned that we needed to consider when creating our prototypes.

Table 1. Issues instructors and students named as the most important in developing a sense of community

Development of prototypes

Incorporating preliminary findings into prototypes

We took the findings from our research, interviews, and direct observations and turned them into specific design requirements for our prototypes. Our goal was to create environments through our prototypes that would most effectively inspire participants to develop a sense of community. We planned to have one control group in which students could not interact with one another to help gauge the success of the community from the other prototypes. Table 2 shows the specific design decisions we made for the prototype as a result of our preliminary findings.

Table 2. How we incorporated design requirements into the prototypes

Description of the three prototypes

We developed three prototypes: two using tools that we thought could facilitate discussion between students and the instructor, and one control group that did not allow students to interact with one another. In the two discussion-based classes, we would try to simulate a sense of community similar to what we had observed in the more traditional discussion-based classrooms, and we would compare the sense of community in each of those groups with the sense of community found in the control group, which had no student interaction.

After examining the design requirements that stemmed from our preliminary research, interviews, and direct observations, we chose bulletin boards and listservs as the tools for our discussion-based classes. The students in the control group used only email to correspond directly with the instructor, but they could not interact with any of the other students.

Each class would have the same number of students (five), the same assignments, and the same amount of encouragement and correspondence from the instructor. Each class would have an internet homepage with the same basic layout, which included:

- Links to the same assignments.

- A link to the instructor's email. (The instructor's email address was different for each group.)

- For Prototypes A and B, a link to the appropriate forum (the bulletin board and listserv, respectively).

- For Prototypes A and B, a link to the personal paragraphs that students write about themselves.

The only major difference between the classes would be the means of communication within the virtual environment. By having this be the only variable, we hoped to isolate and compare the effects that each communication medium had on the students' sense of community.

Prototype A: Bulletin board

Prototype A allowed all of the students and the instructor to post new messages, read messages that others had posted, and append responses to existing messages. Students could follow a discussion thread through the various posted responses or could skip directly to the beginning of another discussion thread. The students and the instructor could communicate asynchronously by leaving and retrieving messages at different times. Students could refer to the bulletin board to read each other's posts and replies at any time. This type of format is called a bulletin board (or bboard, for short).

The bulletin board forum we chose for Prototype A is called EZBoards.com. We selected this bulletin board service because:

- Set up is relatively easy.

- It allowed us to control which users could access the forum.

- The bulletin board service is free.

Prototype B: Listserv

Prototype B allowed students to interact with each other, as well as with the instructor, as in Prototype A; however, the difference is that the students in Prototype B received all of the posts through email in their individual inboxes rather retrieving them through a bulletin board, eliminating the need for the students to repeatedly return to a website to read all of their peers' messages.

The listserv we chose for Prototype B is called egroups.com. We selected this listserv service because:

- One of us had previous experience with egroups.com and knew it to be reliable.

- It allowed us to register each user and adjust the preferences ahead of time so individual students would have very little set up.

- The listserv service is free.

Prototype C: Email with instructor only (control group)

Prototype C acted as the control group for the first two. Unlike in Prototypes A and B, the students using Prototype C were not aware of one another. Each student communicated solely to the instructor through email, and, therefore, all of the class discussions were held one-on-one between each student and the instructor.

User testing

Profile of users

Most of our users were in their 20s. All were comfortable with the internet and with email. Some had experience with either bulletin boards or listservs, but none had used the specific bulletin board or listserv forum that we had chosen for Prototypes A and B. Most of the users knew at least one of us before participating in this study. All participants had taken an introductory-level literature course.

Makeup of the three user groups

Because we wanted to measure the sense of community that participants felt with one another as a result of the class, we took steps to ensure that the instructor and students in each group did not know one another beforehand.

Instructor for each user group

Since there were three user groups and three of us, we each took on the role of instructor for a particular group and monitored a different prototype.

Because many of our users knew us personally, we did not want them to feel any personal bias towards their instructor, so we needed the instructor to have anonymity. Even though we were monitoring the classes separately, we all assumed the same online identity of a fictitious instructor, Professor Liz Kelly (who referred to herself as "Prof Liz"). We procured three temporary email addresses through Hotmail for the professor, one for each prototype; these email addresses hid our real names from users to ensure that no one knew who was really acting as the instructor.

We consulted one another throughout the week-long test to make sure Professor Liz spoke using the same tone and language and encouraged students at the same rate to participate. Because we had found in our preliminary research that emoticons might help build a sense of community, we each made sure that Professor Liz incorporated emoticons in her posts to the class throughout the week.

Students for each user group

We were careful to assign users to groups in which they would not know their classmates, so that any sense of community had to be created from scratch.

Assignments for the week-long class

Each of the three user groups had the same assignments. The week-long class began on a Monday. Table 3 shows the assignments that each student was asked to complete.

Table 3. Assignments and their due dates from each week-long class

Assignment 1: Student bios

While doing our preliminary research, we learned from prominent online instructional designers that "initial online contact must include attempts at group development before moving into content."[3] In an attempt to lay the groundwork for community before diving into class content, we asked students to share something about themselves with the rest of the class (or, in the case of the email-only control group, the students emailed their bios only to the instructor). To encourage students to post their bios, "Professor Liz" posted her profile and picture as an example for the users to follow. Professor Liz's profile and picture was identical across each user group.

Assignment 2: Three responses to short story

We didn't want the course subject matter to intimidate students, so we selected a short story that students might have had previous exposure to in high school or in a college fiction class: "Young Goodman Brown." We also chose this story because its imagery and themes are relatively obvious. To make sure that everyone read the same version of the story, the text for the story was available online through a link provided on the assignments page.

We didn't know how comfortable each of our participants was with commenting on literature, so we tried to aid the discussion as much as possible. Professor Liz posted some initial questions about the story to help the students get an idea of the types of things to possibly comment on.

The length for each post was purposefully vague so that we could examine how long users would typically respond without being prompted for a specific length. In our initial research, we read that "imposed guidelines that are too rigid will constrain discussion, causing participants to worry about the nature of their posts rather than to simply post."[4] In all three prototypes, Professor Liz posted a handful of encouraging responses to praise students for their comments; however, the instructor never became directly involved in the story's analysis.

In the bulletin board and listserv groups, we tried to encourage inter-student discussion by having Professor Liz remind students that they could respond to classmates' posts and count the response as one of their assignment posts. Collaborative learning enhances community and "equality of participant-facilitator and participant-participant interactions" must be stressed.[5]

Rate of user participation

Although we originally assigned five students per prototype, some of the users did not complete all of the assignments and some did not participate at all. In the questionnaire that the users answered after the user testing, they blamed their lack of participation on technical difficulties, time constraints, or personal circumstances. Each forum had at least three users that completed at least one of the assignments. Table 4 shows the student participation of each of the three user groups.

Table 4. Student participation in each user group

We noticed that most of the students in using Prototype B, the listserv, completed most of the assignments, while most students using Prototype A, the bulletin board, only completed one. The students using Prototype C, email only with the instructor, did not seem to have any clear pattern with assignment completion.

When we asked users in the final questionnaire about why some of them did not participate at all, one person in both the bulletin board and email only groups cited laziness as the reason for their lack of participation; the person in the listserv group who did not participate claimed that he or she had attempted to but had trouble accessing the website. As we learned while doing our preliminary research, "the possibility for encountering technical difficulties is very real" when distance learning relies so heavily on technological accessibility.[6]

We also kept track of when students used emoticons, as the instructor had in several of the posting. In Prototype A, the bulletin board had a mechanism built in where users could select graphical emoticons (not just made up of keyboard symbols) such as smiley faces and insert them into text. Table 5 shows the number of students who did use emoticons at least once during the week-long class. One student in the email only group commented in the survey that he or she would have used emoticons if he or she had noticed others using them. We noted that almost everyone in the listserv group used emoticons in their postings.

Table 5. Students who used emoticons at least once

Conclusions and recommendations

Results from the questionnaire

As the monitors of the three forums, we could observe indications of a development or lack of community, but we had no insight into our users' specific attitudes and opinions. To help us gauge and compare their overall sense of community while using the prototypes, we developed a questionnaire for them to complete at the end of the week after each week-long class was over.

This questionnaire gave the users a chance to explain their feelings about the forum they were assigned to and, for some, why they chose not to participate in some assignments. The questionnaire also asked them about their impressions of their classmates and instructor, and asked them about the sense of community. Because we were looking for qualitative data, such as explanations and reasons for certain actions or opinions, we chose to leave room for them to insert their thoughts instead of providing a scale for them to rate their answers (i.e., the ten point scale of the QUIS format).

Table 6 summarizes the results of the questions asking how students felt about their instructor, Professor Liz Kelly. Though the sample size is small, we were able to notice trends to help us draw conclusions. In both the bulletin board and listserv groups, only one student viewed his or her relationship with Professor Liz as positive. Whether or not they viewed the relationship as positive, most of the students in the listserv and email only groups were not intimidated to ask the instructor for help, while 2 out of 3 of the students in the bulletin board group were reluctant to approach the instructor for help. All groups showed a weak sense of community between the students and the instructor.

Table 6. Summary of students' answers about their instructor

Table 7 summarizes the results of the question asking how students felt about the sense of community between them and their classmates. No students in the bulletin board or email only groups cited feeling any sense of community with classmates. (It's not surprising that the students who only emailed the instructor directly did not feel any sense of community with their classmates since they weren't given any way to access those classmates.) In the listserv group, however, one student felt as though he or she had built some sense of community with the group, and two other students in the listserv group wrote in that they felt that community would have developed within the group if the class had been longer than a week.

Table 7. Summary of students' answers about community with classmates

Table 8 summarizes the results of the questions asking how students felt about the class forum. The students in the listserv and email only groups all claimed to enjoy the method of interaction, while only a third of the bulletin board users did. Everyone seemed to have no problem using the course homepage. (The homepage was almost identical for each group.) We probed users to find out what they thought of Professor Liz's and other classmates' use of emoticons. The students in the listserv group had a much higher rate of satisfaction with emoticons; this is not surprising since 3 out of the 4 students in the listserv group used emoticons themselves at some point during the class.

Table 8. Summary of students' answers about the class forum

Conclusions on the success of the prototypes

The results of our user testing have proven most of our expectations to be correct. The listserv seemed to be the best forum to creating a sense of community among our users. In the listserv group:

• The majority of the students completed all of the assignments, unlike in the other two groups which had less than half of the students complete all assignments.
• No students cited being unmotivated to participate, unlike in the other two groups which each had one student claim that they were too lazy to participate.
• The majority of students used personable emoticons while writing their posts, unlike in the other two groups which each had only one student use emoticons.
• All students felt comfortable enough with the instructor to approach her for help, unlike in the other two groups which each had one student who felt hesitant asking the instructor for help.
• One student felt a sense of community with the classmates, and two others commented that they thought community would have developed if given more time, unlike in the other two groups where no student felt a sense of community.

However, the bulletin board fell behind even our control group, which surprised us. The bulletin board group had only three users who filled out the questionnaire, so the lower sample size could be an issue. In the bulletin board group:

• Though every student completed at least one assignment, the majority of the students completed one assignment and then stopped, unlike the other two groups in which students tended to trail off on assignments toward the end.
• No student attempted to contact the instructor for any reason, unlike in the other two groups which each had at least one student contact the instructor with questions that were not part of the assignments.
• Only one student felt comfortable enough with the instructor to approach her for help, unlike the other two groups in which the majority or all of the students did not feel hesitant to ask the instructor for help.
• Only one student claimed to enjoy the method of interaction, unlike the other two groups in which all students enjoyed the communication medium.
• Though all five students completed at least one assignment, only three filled out the questionnaire, unlike the other two groups in which every student who completed at least one assignment filled out the entire questionnaire.

The listserv's ability to place all of the group's posts into each user's inbox might have been a key factor in the listserv group's success. Because only a few users had previous exposure to bulletin boards, users in the bulletin board group might have had difficulty adjusting to the new medium of communication. Since all of the users felt comfortable with email, email-based forums such as listservs should have been easy for users in the listserv group to get used to.

Also, bulletin boards required the highest level of technological effort. Students in the bulletin board group were required to log in to a website to read the posts, and very few of the users seemed willing to go to such lengths to participate. Maybe this is why the least complicated of the prototypes, the email-only control group, was more successful than the bulletin board group at getting users to participate.

The lack of motivation was a problem with our users and was the most cited reason for not participating. According to the California Distance Learning Project, students who sign up for distance learning are typically:

• Voluntarily seeking further education
• Motivated
• Self-disciplined
• Tend to be older than the average student
• Tend to possess a more serious attitude toward their courses.[7]

Because our users did not fit the profile of a typical distance-learning student, it is not surprising that many of the users did not feel compelled to participate in the course.

One assumption we made going into the user testing that was not proven correct was our belief that emoticons would encourage community with the instructor through friendliness. While some of the users did feel a sense of community with the instructor÷and many felt that they would have felt community with the entire group if given more time÷the instructor's use of emoticons did not seem to effect how each student viewed his or her relationship with the instructor. Though in each group the instructor used emoticons regularly, the students in the various groups felt little sense of community with the instructor.

Recommendations for future iterations of distance learning classes

Through our user testing, we came to several conclusions that could help better establish a sense of online community in future attempts. We suggest that a listserv and a straightforward homepage with as few links as possible make a good combination for the communication media in an online course. Even though the introductory assignment seemed to benefit some of the users, it might have worked better if we had asked users to respond to one another's profiles rather then asking them simply to write them. The researchers Palloff and Pratt have found that asking users to read and respond to one another's introductions reinforces the idea of their peers as people.[8]

Another suggestion that we have for future online courses is the development of a visual system of presence. While conducting our preliminary research, we ran across research that suggested that the visual impact of nonverbal communication is what is lacking most in online learning environments. We had hoped that infusing our posts with emoticons would portray some visual sense of the intent behind our words, but our study did not show that emoticons were strong enough. Even though the bulletin board had an option to use graphical emoticons (like smiley faces, etc), this option was largely ignored by all of the users except the instructor. For future iterations of online distance learning, we suggest incorporating stronger visual elements, such as avatars. Because our study shows that some users lack motivation, especially when dealing with technical complications, the system of visual representation should be as simple and automated as possible.

Also, the instructor needs to encourage and stress the importance of interaction between students. There was a distinct correlation between the frequency of peer interaction and the success of community establishment. In the listserv group, where users were most likely to respond to one another's post, most of the users indicated that they either already felt a sense of community or would expect to if given more time. The researchers Palloff and Pratt state that one of the most important elements to build into an online course is "the expectation that students will provide constructive and extensive feedback to each other."[9] While we told our users that responses to one another would count as one of their posts, we did not actually build that in as a requirement. To encourage more interaction, we could have given users double credit for each response to another classmate's post.

We found that time was also a problem that needs to be remedied in future tests. While we were aware that it would be nearly impossible to create a full sense of community in a week, we were pleased to find that several users felt that there would have been a community established given more time. This leads us to conclude that the methods we used might have worked in a longer user test. Because we only had a week, we forced our users to do four assignments in seven days, a requirement that caused many users to not fully participate. If given a longer period for testing, we could have avoided the density of assignments and allowed users more freedom to develop community. We suggest that taking at least a month to run user test involving a distance learning class.

Glossary

Analytic Thought (Independent Thinking) - Thought about course materials that is independent; analysis of materials sparked by personal interest.

Bulletin Board (bboard)/ Discussion Group - An online environment in which messages appear as posts. Messages can be "threaded" so that a related reply appears under the message replied to. People can add new posts or reply to existing post asynchronously.

Chat room - An online environment in which textual conversation takes place synchronously in real time.

Community - A state of unison achieved by being comfortable with a group of people; having an understanding of one another's personal experiences and beliefs.

Distance Learning - Any course where one or more participants are not present at the same time or in the same place (including mail, video, online course, etc.).

Listserv - An email mechanism that acts as a single email address and disseminates messages to all of the members of a specified group.

Presence - The general knowledge of other people sharing an experience with you (for example, when knowing someone else is online at the same time as you, knowing how many students are in a class with you, etc.).

Bibliography

Alphabetic Listing of Internet Resources by SOFWeb. http://203.12.60.74/resource/resalpha.htm Last accessed 5 March 2000.

The Carnegie Mellon TELab. http://www.cmu.edu:80/acs/telab/indexX.html Last accessed 5 March 2000.

Course Info (BlackBoard). Chatham College. http://courseinfo.chatham.edu/ Last accessed 5 March 2000.

EGroups. http://www.egroups.com Last Accessed 3 May 2000.

Ehrmann, Stephen C. "New Technology, Old Trap." The Educom Review. Vol. 30, No. 5. Sept/Oct. 1995, pp. 41-43.

EZBoards. http://www.ezboards.com Last Accessed 3 May 2000.

Fenton, Edwin. "Conducting Discussions." April 1991.

Free Chat Now. http://www.freechatnow.com/ Last accessed 5 March 2000.

Hansen, Forrest. "Laying the Groundwork for Class Discussions." The National Teaching & Learning Forum. Vol. 1 No. 3, 1992.

Hotmail. http://www.hotmail.com Last Accessed 3 May 2000.

HyperLiterature, Virginia Tech. http://ebbs.english.vt.edu/cl/hthl00/ Last accessed 5 March 2000.

Lane, Lisa. Personal interview. 25 February 2000.

The National Council for Educational Technology. http://203.12.60.74/lt/ncet.htm Updated 21 December 1999. Last accessed 5 March 2000.

Nielson, Jakob. Alertbox. http://www.useit.com/alertbox/9708b.html Created 15 August 1997. Last accessed 5 March 2000.

ONEList.com. http://www.onelist.com/chat/ReadersCafe Last accessed 5 March 2000.

The Palace. http://www.thepalace.com/welcome/index.html Last accessed 5 March 2000.

Palloff, Rena and Keith Pratt. Building Learning Communities in Cyberspace. San Francisco: Jossey-Bass Publishers, 1999.

The Raritan Valley Instructional Design Center Resources. http://rvcc2.raritanval.edu/~idc/IDCsourc.htm Last accessed 5 March 2000.

Wilkinson, James and Heather Dubrow. "Encouraging Independent Thinking." Education for Judgement: Artistry of Discussion Leadership. Boston: Harvard Business School Press, 1991.

Yahoo! Chat. The Hobby Lobby. http://chat.yahoo.com/?room=Hobby%20Lobby::1600326525&identity=chat_pf_1&client=Java Last accessed 5 March 2000.

Yahoo! Poker. Social Lounge. http://games.yahoo.com/games/login?game=Poker Last accessed 5 March 2000.

[1] Fenton, Edwin. "Conducting Discussions." April 1991.

[2] The U.S. Department of Education expected in 1997 that by 1998, 90 percent of all higher education institutions with enrollments of 10,000 or more would offer distance learning.

[3] Palloff, 19

[4] Palloff, 18

[5] Palloff, 19

[6] Palloff, 67

[7] Palloff, 8

[8] Palloff, 114

[9] Palloff, 123