Response 

4/22

Both Marshall and Rossman (2011) and Creswell (2009) offer refreshing tips on making an argument and anticipate ethical issues in the proposal. To argue for the soundness of qualitative research, Marshall and Rossman introduce the emic perspective (p. 258), which is centered on how local people think (Kottak, 2006, p. 47). Compared with the etic approach which emphasizes what a researcher thinks, the emic approach allows immersive experience in participants’ community and the validity of participants’ point of view. The emic focus lays a foundation for long-time observational, participatory, and experimental studies in the qualitative strand. This community-situated perspective also makes sense for small-scale investigations, such as case studies, focus groups, and interviews in such domains as cultural studies, linguistics, literary studies, political science, and women’s studies (Marshall & Rossman, p. 264). As the epistemological assumptions of these domains suggest, what matters in the emic approach is the constructivist conceptualization of knowledge-making, especially recognition of the participant’s ability to know. To me, this epistemological difference between the emic and the etic approach might serve as the major validity argument for qualitative research.

An equally important consideration of a qualitative research’s soundness is its pathos: the research’s appeal to the reviewer and policymaker. As Marshall and Rossman (2011) contend, an understanding of the reviewer, advisor, and policymaker is a sure way to persuade (pp. 259, 263, 274). What strikes me as relevant is the consideration of how much reviewers outside our field might have known or need to know about what we do and what our methodology is capable of doing. Discourse analysis, for example, is a method that people from other fields might question. I find it particularly important for discourse analyst to provide a valid answer to the challenge that different readers might interpret a discourse in different ways. This challenge, however, is also a place where discourse researchers might offer a counterargument: it is the multiplicity of interpretation that warrants subjective coding of discourses.

Additionally, researchers can corroborate the validity of their research in the writing. Marshall and Rossman (2011) suggest that qualitative researchers may incorporate thick description of the methodology and design to provide an “audit trail” (pp. 253, 261). This is at least one thing the researcher can do to assist reviewers to assess the validity of a research. On the one hand, including intensive notes in the writing (p. 273) and extensive references in the proposal (p. 261) will create an opportunity for the reader to make their own decisions (p. 254). On the other hand, thick description, among other ways to make the research transparent, will present the researcher as passionate about the research and conscious of the audience, which increases the ethos and persuasiveness. The 60-page proposal on page 261 wouldn’t have the persuasive power if not for the thick description.

Along with the going-thick strategy of writing, good writing practices suggested by both Marshall and Rossman (2011) and Creswell (2009) also add to the argumentative effect of a proposal. Relevant to the purpose to convince reviewers are two more important considerations: the pilot study and ethical issues. The pilot study is a chance for researchers to argue for the qualitative approach and its applicability. For one thing, a pilot study may help test out the participant-specific issues that might affect the replicability of the research. As qualitative research is often questioned regarding its generalizability to other groups, the pilot study might help identify issues that are only relevant to the particular time, space, and communities. Real world changes, Marshall and Rossman claim, make replicability irrelevant to qualitative studies (p. 254). For another, pilot studies provide a venue to build trust and respect of participants (Creswell, p. 88) and add to the practicability of the research.

Qualitative researchers might also clarify potential ethical issues as a way to validate their research. Among various valuable suggestions in both Marshall and Rossman (2011) and Creswell (2009), I find the non-intrusion principle highly valuable. It is advised that researchers should leave a research site without disrupting its natural order (Marshall & Rossman, p. 267; Creswell, p. 90). Another way researchers can exercise ethical practices is to view participants as co-researchers (Marshall & Rossman, p. 267). Researchers may promote reciprocity by avoiding coercion in recruitment, interviewing, and interpretation of data (Creswell, p. 90), refraining from deception in the purpose (p. 89), using unbiased language in the writing (p. 92), and checking accuracy of interpretation and showing the final presentation with participants (p. 91). These practices not only ensure benefits for participants but also increase the validity of the research. 

Response 

4/15

The research roundtable this week offered me a great opportunity to work with colleagues and reflect on my own pilot study. To start with, the physical setting of the roundtable was perfect. The casual place, the background noise, the actual roundtable, and the beer, all contributed to an atmosphere that was conducive to unconstrained conversation between fellow student researchers. For me, I was not afraid of talking about something that was in progress and ask what might have been stupid questions. While normally I would hesitate in making suggestion, the setting sort of encouraged me to “offer advice.” Four people in a group made it just as cozy and comfortable to share something that is very, very rough in nature. Yet, I find the casual setting and event very beneficial for student researchers and research in progress. Everyone in the group was ready to be supportive by listening to the concerns and trying to answer questions. I believe I wouldn’t have said the same thing should the event take place in a conference room.

I was able to share some questions I had about my pilot study and the research design. I had always wanted to have someone audit my process of participant recruitment, which I found to be very hard. In my pilot study, I had planned to do an experimental study on teachers’ commentary on second language writing. I planned to have 2 composition teachers to respond to student writing in way to highlight cultural, rhetorical, and larger writing issues and 2 other composition teachers to emphasize the language issues, including grammar, vocab, sentence, and mechanics. Accordingly, I planned to have 2 second language students to evaluate one type of teachers’ commentary and 2 other second language students to evaluate the other type. Meant as an experiment, my original design needed only 8 participants. But the design involved a couple of issues. First, I will need to recruit my colleagues who teach college writing. I wanted to find out if it seemed like an ethical issue for my research colleagues. Second, I was not quite sure about the experiment. Would it represent the actual practices of composition teachers? My colleagues’ response will help me answer the question. Third, I was not certain if second language students may respond to teachers’ comments in similar patterns. My colleagues experience might offer my insights. While I had talked to my second language friends and found that they thought about teachers’ commentary differently, I would benefit from perspectives of colleagues with teaching experiences.

I also needed feedback from fellow student researchers on my revised design and recruitment process, which I carried out prior to the roundtable. In the revised design, I created an artificial essay with multiple language and writing issues. I wanted only student participants who were expected to evaluate each of a list of 23 comments from teachers and answer some extended questions. The design changed the experimental nature of the project to a qualitative study. I received really valuable feedback from colleagues, who pointed out that the artificial sample and the out-of-classroom context might have weakened the validity of the design. One of colleagues also offered a similar setting in the Writing Center. Writing tutors here at Kent State are trained to give different types of comments to students who have different needs. This case made me think that even with second language students, I cannot assume that they have the same concerns about their writing and expect the same from teacher’s commentary. My initial findings proved the case to be consistent with second language students.

While I receive value feedback on my own design, I also benefit from thinking about various issues in colleagues’ research. Our group touched on such important issues as entry into the research site, the contention between confidentiality and researcher’s positionality, and disciplinary concerns in research design. These issues provided an opportunity for me rethink about my design. I was led to ask other important questions about my project: How should I connect the research with the major disciplinary concerns? How does the project look like to intended journals? How should I incorporate the actual classroom practices to increase validity? And how can I recruit second language students efficiently and ethically?

Response

4/8

I find pilot studies very helpful in a couple of ways. First, before carrying out projects that involve a large number of participants, researchers, or data, it is valuable to test out how the project looks like in terms of logistics. Oftentimes, we face the question of how many participants to recruit and find no definite criteria for that decision. It is good to know from the pilot study whether we might be compromising validity of the design because of time, cost, or the labor involved in recruiting intended participants. I am curious if it is a good choice to opt for a small-scale, but different method for the current pilot study. Focus groups, for example, seem like a fitting method. But then, how is it relevant to the large-scale project we plan to conduct later? I wouldn’t have come to these questions if not to do a pilot first. Specific to my project, I feel like I have to limit the number of participants because of time. While I started out to randomize participants, I am now considering recruiting participants from what I see as stratified subgroups of intended participants. The pilot study offers me a different context to consider the same questions for a larger project. So, I wonder if there is an issue of reliability and validity in pilot studies.

It is also interesting to consider other validity issues involved in a pilot study. An important issue is whether or not we may include data from pilot studies in the actual, full-length project. There is the reliability issue again. Specific to my project, I find that I will need to adjust my questions to make them more specific and to cover more issues. The changes, though, will render both participants and data invalid for a later, actual project. I feel like I will see the pilot study as part of our rationale for a research design. The problems we encounter and the assumptions we prove may have a better place in the rationale for a research.

Pilot studies might often be regarded as irregular and gave rise to ethical issues. I was approached by a student researcher asking for participation in a pilot study on speech pathology when I visited Cleveland State University during the spring break. The person told me that because the research was a pilot study, no IRB approval was needed. I was not offered to sign an informed consent form nor was informed of anything about the purpose and nature of the research. I did not feel respected and was sort of rushed to the interview and the experiment (having my reading of minimal pairs of pronunciation recorded). It is not certain whether or not we need IRB approval for a pilot study. I don’t remember if van Teijlingen (2001) said so. But it is certain that participants need the same respect and protection in pilot studies as in actual, full-length studies.

The Chris McCandless case in the NYT article makes a valid point about the need of pilot studies as both bona fide researches and arguments for extended research. While many pilot studies involve only a small size of samples or participants, they are able to generate data that are meaningful. In case studies, focused groups, and other small-scale sampling methods, researchers might be able to create analytical categories and coding schemes that are valuable in themselves. Furthermore, pilot studies might be used to walk reviewers through the methodology, rationale, and design that are qualitative in nature (Marshall and Rossman, 2011, p. 242). For this purpose, the pilot study might be viewed as an argument for the proposal.

One thing that piqued me in Marshall and Rossman (2011, chapter 9, p. 236) is the sample budget. Much of the budget is distributed to salaries and payment to consultant and contracts. For one thing, the proportion of the budget conforms to the valuation of human labor. For another, the recognition of the intellectual work in the academic tradition in the States is a sure way to ward off possible corruption. Researchers in China have a hard time justifying a research budget because salary is often not a valid expenditure. But the denial of intellectual work leads to restrained financing and widespread embezzlement in research.

Response

3/11

Marshall and Rossman’s (2011) chapter 4, 5, 6, and 7 offer valuable suggestions on entry, sampling, cross-cultural settings, and exit, to name a few of the critical points in the research process. The importance of entry might be overlooked if researchers take it for granted that one can always make correction as he or she goes. What might be problematic with this approach to entry is that unplanned entry into a site might “contaminate” the site and make it no longer useful for the research. Should participants be asked to take a survey again, their prior exposure to the survey questions might influence their decisions and responses. In many cases, there is only one chance for researchers to enter a site to collect legitimate data. In sending recruitment emails, for example, researchers should be careful to provide sufficient information and consider potential ethical issues. The example provided on p. 102 includes necessary information for potential participants to decide whether or not they want to participate. As an example, the email includes an adequate introduction to the types and number of participants to be recruited, the sponsors of the project, and any benefits and harms the project might incur. Marshall and Rossman also advise researchers to phrase the recruitment email in a personal manner, as emails are personal and might be easily ignored and deleted (p. 101). Another issue with recruiting emails is that there is perhaps only one chance that recipients might be interested and agree to participate. The second time when a recipient reads a recruitment email, he or she might be either further driven away of coerced to participant. In either case, the recruitment process violates the ethical code of voluntary participation.

How participants are sampled is also an ethical issue, as well as one of validity and efficiency. To start with, the site to recruit participants and conduct face-to-face interviews and observation needs careful considerations. Public places, as Marshall and Rossman (2011) suggest, might be associated with various political implications (pp. 108-110). Classrooms or teachers’ lounges, for example, might impose pressure on the participant because other fellow students or teachers might be present. To implement the voluntary participation principle, researchers should recruit potential participants at a place where associated perception of the place will not affect potential participants’ decision-making. It is advisable that interaction between the researcher and participants take place in a physical setting that is agreed upon by both sides as comfortable and unconstrained. Moreover, the types of sampling are closely related to the purpose (p. 111) and participants of the research. Snowball sampling, for example, might have both procedural validity and efficiency. Participants who know each other might have already been in the same community in which the research is situated. The interpersonal connection between potential participants also helps researchers to recruit participants without unnecessary intrusion into the site. Another type, stratified sampling, might work well in qualitative researchers because participants and phenomena of qualitative research are often not as randomly distributed as those in quantitative research. The seemingly controlled sampling method is closer in meeting the natural research setting requirement than randomized sampling. Convenience might be tempting but may often incur problems. Recruiting participants from a researcher’s colleagues or students might be convenient but harms voluntary participation and credibility.

Much as negotiating entry and reciprocity is critical to building trust and respect from participants, the exit strategy is also important to a research. Marshall and Rossman (2011) warn researchers not to “grab the data and run” (p. 130). Exit is more than leaving with gratitude; it concerns the type and epistemology of research. To further involve the participant after collecting data and writing the paper means that researchers need to view participants not merely as “subjects” but as co-researchers. In the latter view, participants might continue to contribute to the research by working with researchers to check interpretations, representations, implications, and benefits. Exit is also the phase of research when the promised benefits are cashed in and the phase when the validity of research is further tested.

In cross-cultural settings, participants should be viewed as co-researchers in checking translation of their responses (Marshall and Rossman, 2011, p. 165) and culturally sensitive contents in the research. A researcher needs to consider providing sufficient linguistic and cultural support in translation and at the same time refrain from intrusion and misinterpretation.