How to choose and read research papers
The quantity and scope of papers to read can be overwhelming.
There are several rules of thumb that we can use to guess about the
value of a paper:
All of these can provide fuzzy, noisy indicators of the value of a
paper to your effort.
- Where was it published?
- Something can be published in a workshop, a
conference, or a journal.
- Workshop papers consist often of incomplete ideas
or work-in-progress reports. Workshops tend to be
small, and sometimes just have authors and a select,
invited crowd who can give helpful advice. Some
workshops are nearly as strong as conferences, though,
an examples of which is the International Workshop
on Parsing Technologies (IWPT)
- Conference papers are where it's at: nearly everything
good has been published at a conference at some point.
IEEE and ACM conferences tend to be fairly good,
and conferences with a direction in their name might
not be so good; e.g., "Pan-Asian Conference on Multimedia".
- Journal articles are good and thorough, but it takes
a long time for a paper to be journalized. So while the
information is thoroughly discussed, it's not the newest
ideas any more. And journal papers tend to be fairly long.
Nearly every journal article is based on some conference
papers, so it's often better to find the original conference
papers to quickly digest the main ideas.
- Who are the authors?
- Some authors tend to do better work than other authors,
so it's worth paying attention to who wrote the good papers.
Also, by reading several papers by a single author, you can
get a sense for the sort of `baggage' that an author brings
along with him, and a sense for his approach to problems.
However, even `good' authors can start producing poor-quality
work, and unknown authors do produce good-quality work.
- When was it published?
- More-recent work (< 3 years old) often encapsulates
the significant ideas of the older, more significant papers.
But, of course, some older papers are valuable -- but you
don't want to spend time on an old, relatively-insignificant
paper. Before diving into an older paper, it's best to see that
it's cited several times in newer papers.
- How often is it cited?
- The relative importance of an older paper can be estimated by
the number of times that it's cited in works that are known
to be related to the effort.
- How and where is it cited?
- If paper X cites paper Y, and paper X is known to be
highly relevant, then evaluate how paper Y is cited:
- If Y is cited as part of a big group of bad or
vaguely-related ideas, then that indicates that Y might
not be very relevant.
- If Y is cited as something that's important, then
it likely is relevant.
- What is its title?
- Clearly, the title is going to give some idea about the
relevance of a paper.
Actually reading a paper is a serious time commitment.
The goal is to proceed only as far along this path as it takes
to derive the maximum value:
- Guess about its relevance by the frequency of its citation.
- Check the title, where and when it was published, and who
its authors are.
- Read the abstract and the first page; does their problem
approach make sense for your current effort?
- Read the section headings; what direction do they go
in their project? What are their main contributions?
- Look at the pictures: what do their graphs show?
- Skim the paper, looking for the main conclusions and
contributions. Avoid spending time on proofs, detailed
Everything before now can be done in ≤
- If the paper still seems valuable, read it carefully.