Response

4/8

I find pilot studies very helpful in a couple of ways. First, before carrying out projects that involve a large number of participants, researchers, or data, it is valuable to test out how the project looks like in terms of logistics. Oftentimes, we face the question of how many participants to recruit and find no definite criteria for that decision. It is good to know from the pilot study whether we might be compromising validity of the design because of time, cost, or the labor involved in recruiting intended participants. I am curious if it is a good choice to opt for a small-scale, but different method for the current pilot study. Focus groups, for example, seem like a fitting method. But then, how is it relevant to the large-scale project we plan to conduct later? I wouldn’t have come to these questions if not to do a pilot first. Specific to my project, I feel like I have to limit the number of participants because of time. While I started out to randomize participants, I am now considering recruiting participants from what I see as stratified subgroups of intended participants. The pilot study offers me a different context to consider the same questions for a larger project. So, I wonder if there is an issue of reliability and validity in pilot studies.

It is also interesting to consider other validity issues involved in a pilot study. An important issue is whether or not we may include data from pilot studies in the actual, full-length project. There is the reliability issue again. Specific to my project, I find that I will need to adjust my questions to make them more specific and to cover more issues. The changes, though, will render both participants and data invalid for a later, actual project. I feel like I will see the pilot study as part of our rationale for a research design. The problems we encounter and the assumptions we prove may have a better place in the rationale for a research.

Pilot studies might often be regarded as irregular and gave rise to ethical issues. I was approached by a student researcher asking for participation in a pilot study on speech pathology when I visited Cleveland State University during the spring break. The person told me that because the research was a pilot study, no IRB approval was needed. I was not offered to sign an informed consent form nor was informed of anything about the purpose and nature of the research. I did not feel respected and was sort of rushed to the interview and the experiment (having my reading of minimal pairs of pronunciation recorded). It is not certain whether or not we need IRB approval for a pilot study. I don’t remember if van Teijlingen (2001) said so. But it is certain that participants need the same respect and protection in pilot studies as in actual, full-length studies.

The Chris McCandless case in the NYT article makes a valid point about the need of pilot studies as both bona fide researches and arguments for extended research. While many pilot studies involve only a small size of samples or participants, they are able to generate data that are meaningful. In case studies, focused groups, and other small-scale sampling methods, researchers might be able to create analytical categories and coding schemes that are valuable in themselves. Furthermore, pilot studies might be used to walk reviewers through the methodology, rationale, and design that are qualitative in nature (Marshall and Rossman, 2011, p. 242). For this purpose, the pilot study might be viewed as an argument for the proposal.

One thing that piqued me in Marshall and Rossman (2011, chapter 9, p. 236) is the sample budget. Much of the budget is distributed to salaries and payment to consultant and contracts. For one thing, the proportion of the budget conforms to the valuation of human labor. For another, the recognition of the intellectual work in the academic tradition in the States is a sure way to ward off possible corruption. Researchers in China have a hard time justifying a research budget because salary is often not a valid expenditure. But the denial of intellectual work leads to restrained financing and widespread embezzlement in research.