1.2 Research Methods Archives - Nailing Your Thesis

2020-02-212020-07-20

Traditional Theory Building and Validation in (Computer) Science

Many computer science degree programs do a lousy job at teaching science. A high school student, entering university, often has a good idea what science is about, based on their physics and chemistry classes. At least, it involves controlled experiments. At university, this is rarely picked up, and computer science students are given the idea that programming something novel constitutes science. With that idea, they are often bewildered when I teach them rigorous research methods, in particular if those originated in the social sciences (like qualitative interviews or hypothesis-testing surveys).

2019-08-132020-07-20

Anecdotal Evidence on the Method Wars

On a whim, I asked my Twitterverse (which includes a fair number of computer scientists) what they think about the following question:

When peer-reviewing somebody else’s work submitted for publication, what should you do if you find that the authors have a different belief than you about what can be known?

2018-12-162020-07-20

Alternative Definition of Theoretical Saturation in Qualitative Research

Theoretical Saturation: The mental state of a researcher wanting to finish up the work and go home for the holidays.

2018-08-302020-07-20

Evaluation of Theories vs. Validation of Hypotheses

Research should be presented with appropriate choice of words to the world. So it bugs me if researchers, maybe unknowingly, overreach and call the evaluation of a theory a validation thereof. I don’t think you can ever fully validate a theory, you can only validate individual hypotheses.

The following figure shows how I think key terms should be used.

Continue reading “Evaluation of Theories vs. Validation of Hypotheses”

2018-08-022020-07-20

Is Exploratory Data Analysis Bad?

Last weekend, I ventured into unchartered territory (for me) and attended the Berliner Methodentreffen, a research conference mostly frequented by social scientists. I participated in a workshop on mixed methods, where the presenter discussed different models of mixing methods with each other (“Methodenpluralität” in German).

She omitted one model that I thought is often used: First to perform exploratory data analysis to detect some interesting phenomenon and then to do some explanatory qualitative research to formulate hypotheses as to why this phenomenon is, was, or keeps happening.

During my question, the temperature in the room dropped noticeably and I was informed that such exploratory data analysis is unscientific and frowned upon. Confused I inquired some more why this is so, but did not get a clear answer.

Continue reading “Is Exploratory Data Analysis Bad?”

2018-08-012020-07-20

We May not Know What We are Doing…

From my excursion into qualitative research land (the aforementioned Berliner Methodentreffen) I took away some rather confusing impressions about the variety of what people consider science. I’m well aware of different philosophies of science (from positivism to radical constructivism) and their impact on research methodology (from controlled experiments to action research, ethnographies, etc.) I did not expect, however, for people to be so divided about fundamental assumptions about what constitutes good science.

One of the initial surprises for me was to learn that it is acceptable for a dissertation to apply only one method and for that method to only deliver descriptive results (and thereby not really make a contribution to theory). In computer science, it is difficult to publish solely theory development research (let alone purely descriptive results) without any theory validation attempt, even if only selective. The limits of what can be done in 3-5 Ph.D. student years are clear, but this shouldn’t lead anyone to lower expectations.

Continue reading “We May not Know What We are Doing…”

2016-03-232020-07-20

The ASA’s Statement on p-values: Context, Process, and Purpose

The American Statistical Associations’s answer to “statistical techniques for testing hypotheses…have more flaws than Facebook’s privacy policies.” is is now available for free.