When planning a publication strategy for a dissertation, invariably the question comes up where to submit your papers. Ph.D. students naturally are biased towards conferences, because if a paper gets accepted to a conference they get to travel to a (usually) nice place. I nip this bias in the bud right away: For a journal paper, every Ph.D. student gets a conference to attend for free. This lets us focus then on the economic value of a journal vs. a conference paper and how to best reap the benefits of hard research work. Here, I’m a contrarian (to most colleagues): I’m in favor of journals. It is also the economically smart choice for a Ph.D. student.
When teaching about modeling the world, I often talk about how concepts should be MECE, that is, mutually exclusive and completely exhaustive. I didn’t invent this acronym, I took it from Barbara Minto’s writings about structured reasoning.
I finally figured out the appropriate German translation, and, oh wonder, it is shorter than the English version. It is:
Überlappungsfrei und erschöpfend.
I just returned from the Berliner Methodentreffen. One of the initiatives that was most interesting to me is a new attempt at agreeing on and standardizing an open exchange format for qualitative data analysis projects between the different QDA tools. As of today, it is not possible to take your data from one vendor’s tool to another; you are locked-in to one product. The Rotterdam Exchange Format Initiative (REFI) is trying to change that using the budding QDA-XML format.
There are three common reasons for why such an exchange format (and hence the initiative) is important.
In our research, we often work with industry. In software engineering research, this is a no-brainer: Industry is, where there the research data is. That’s why we go there. For many research questions, we cannot create adequately, in a laboratory setting, a situation that lets us do our research.
Once a researcher realizes this, they need to decide on whether to charge the industry partner for the collaboration. Many researchers don’t, because sales is not exactly their strength. Also, many shy away from asking for money, because it is an additional hurdle to overcome, once an interested industry partner has been found.
I often discuss with my Ph.D. students how to structure their work and publish the results. There are many pitfalls. It gets more difficult, if we bring in other professors, who may have a different opinion on how to structure the work. Over time, I have found there are really only two main dimensions, though:
- Separate work into theory building and theory validation, and publish one or more papers on each topic
- Merge work on theory building and validation for an important aspect (hypothesis) into one and publish that
A researcher-friend recently complained to me that her research paper had been rejected, because the reviewers considered it “boring”. I recommended that she complain to the editor-in-chief of the journal, because in my book, “boring” is no acceptable reason to reject a paper. (“Trivial” may be, but that is a different issue.)
The reasoning behind rejecting a paper, because it is “boring”, is as follows: Research should be novel and provide new and perhaps even unintuitive insights. Results that are not surprising (in the mind of the reviewer, at least) are not novel and therefore not worthy of publication.
Traditional science has a clear idea of how research is to progress, rationally speaking: First you build a theory, for example, by observation, expert interviews, and the like, and then you generate hypotheses to test the theory. Over time, some theories will stand the test of time and will be considered valid.
Sadly, most software engineering research today, even the one published in top journals and conferences, often skips the theory building process and jumps straight to hypothesis testing. Vicky the Viking, from the accordingly named TV series of way back comes to my mind: Out of the blue, some genius idea pops into the researcher’s mind. This idea forms the hypothesis to be tested in research.
Researchers often use the term “qualitative research” to mean research without substantial empirical data, and use “quantitative research” to mean research with substantial empirical data. That doesn’t make sense to me, as most “qualitative researchers” will quickly point out, because qualitative research utilizes as much data in a structured way as it can. Everything else would not be research.
Some of my colleagues like to talk about how research that involves programming is “hard”, while research that involves human subjects is “soft”. Similarly, some colleagues like to call exploratory (qualitative) research “soft” and confirmatory (quantitative) research “hard”. Soft and hard are often used as synonyms for easy and difficult, and this is plain wrong.
Pretty much any research worth its salt is difficult in some way, and working with human subjects makes it even more difficult. I find research methods like qualitative surveys, involving interview analyses, for example, much harder than the statistical analysis of some data or some algorithm design. The reason is the lack of immediate feedback.
While you can (and should) put quality assurance measures in place for your interview analyses, ranging from basic member checking to complex forms of triangulation, it might take a long time until you learn whether what you did was any good. So you have to focus hard in your analysis without knowing whether you are on the right track. It doesn’t get more difficult than this.
A short Twitter thread:
Scientific research papers cite other research papers for related or prior knowledge they build on; they cite blog posts as primary material to work with in theory building; two very different things 1/4
A blog post needs to (a) properly use the scientific method and (b) be socially validated by peer review to be considered a scientific research article; for (b) it needs a traditional journal 2/4
The problem with research papers are for-profit publishers and the to-pay open access model; publishers are bottom feeding to increase revenue, hurting the reputation of science with low quality papers 3/4
Please don’t start another journal unless you have an answer that solves the conflict between for-profit publishing and academic publishing incentives @timoreilly 4/4