Journal Article Analysis

Journal Article Analysis

Page Table of Contents

About JAA

Journal Article Analysis (JAA) consists of reading journal articles and analyzing them.

You are responsible for identifying twelve parts of a journal article: title, main point, question, puzzle, debate, theory, hypotheses, research design, empirical analysis, policy implications, and contribution to the discipline, and future research.

Journal Articles vary in their organization and inclusion of these twelve parts.

Many articles explicitly describe all or most of these parts; however, other articles may not state a part, or may omit it entirely.

JAA PowerPoint Presentation

JAA YouTube Video Table of Contents

I recorded a YouTube Video that walks through the PowerPoint presentation above. In the recording, I offer some additional comments for many of the slides. Below is a Table of Contents that lets you jump to specific parts of the presentation.

  • 00:00 – Journal Article Analysis Walkthrough
  • 00:18 – About
  • 00:56 – JAA Module
  • 01:09 – How does the JAA fit in with the entire Course?
  • 01:18 – Visual Representation of Categories and Assessments
  • 01:44 – Gantt Chart of Assessments during 6-week session
  • 02:05 – Gantt Chart of Assessments during 8-week session
  • 02:25 – Gantt Chart of Assessments during 16-week session
  • 02:49 – 12 Parts of a Journal Article Analysis
  • 03:18 – JAA #1
  • 03:49 – JAA #2
  • 03:57 – JAA #3
  • 04:47 – Handwritten Annotating
  • 05:34 – Electronic Annotating
  • 06:12 – Getting Ahead
  • 06:45 – iPoliSci Workshops for JAA
  • 07:25 – Abbreviated12 Parts
  • 07:40 – 1 – Identify Title
  • 08:00 – 2 – Identify Main Point
  • 09:16 – 3 – Identify Question
  • 10:18 – 4 – Identify Puzzle
  • 11:09 – Journals are Pieces of the Puzzle
  • 12:14 – 5 – Identify Debate
  • 13:34 – 6 – Identify Theory
  • 14:08 – Theory in Detail
  • 15:36 – 7 – Identify Hypotheses
  • 17:13 – 8 – Write Research Design
  • 18:15 – 8 – Write Research Design
  • 20:24 – Examples of Research Designs
  • 22:47 – 9 – Identify Empirical Analysis
  • 23:50 – 10 – Identify Policy Implications
  • 25:50 – 11 – Identify Contribution to the Discipline
  • 26:38 – 12 – Identify Future Research
  • 27:33 – Strategies and Tips for Successfully completing the JAAs
  • 27:43 – Sticky Note Strategy
  • 28:52 – Accepting Ambiguity
  • 30:24 – Not Letting Perfect be the Enemy of the Good
  • 30:57 – Submit your JAA, and Let’s Discuss

Further Details for Writing out a Research Design

Writing out a Research Design is worth 19/100 points of a JAA. To put the weight of Research Design into perspective, writing it out is worth about 1% of your overall course grade.

Students can struggle with Research Design, mostly because it’s a newer concept. However, after you introduce yourself to the topic, it will become a powerful tool for analyzing a core element of any political science journal article.

Definition of Research Design

Let’s start by recalling the definition from the “Typical Anatomy of a Journal Article in Political Science” page: The Research Design is how the author compares the effect of the explanatory variable (X) on the outcome variable (O) in a group (G) or set of groups.

Further Description of Research Design

Next, let’s recall the description of Research Design in the “Details of Analyzing Journal Articles” page: The Research Design is how the author compares the effect of the explanatory variable (X) on the outcome variable (O) in a group (G) or set of groups.

  • Some political scientists use notation to denote research design. Below are 4 common examples, and 2 complex examples:
    • Example 1: G O. This is a single group, observation only.
    • Example 2: G X O. This is a single group, treatment then observation.
    • Example 3: G O X O. This is a single group, observation before treatment, the treatment, then observation after treatment
    • Example 4: G X O and G _ O. This is a two-group design. Group 1 receives them treatment, then is observed. Group 2 does not receive the treatment, then observed.
    • Example 5: G O X O and G O _ O. This a two-group design. Group 1 and Group 2 are observed, then Group 1 receives the treatment while Group 2 does not receive the treatment. Finally, both Groups are observed again.
    • Example 6: G O X O _ O and G O _ O X O. This is a two-group design, known as a switching replications design. Group 1 and Group 2 are observed, then Group 1 receives the treatment, while Group 2 does not receive the treatment. Then both Groups are observed. Next, Group 1 does not re-receive the treatment, and Group 2 receives the treatment for the first time. Then both groups are observed again.

Why You Have a Hard Time Finding the Research Design in an Article

Research design notation is not common in political science journal articles.

Part of the reason is that political scientists do not agree on a conceptual, let alone an operational, definition of research design.

This lack of agreement is a function of the diversity of graduate training and experience that political scientists experience.

I struggled with this concept for years during my Ph.D. program at UC Merced. The struggle lead me to engage many books and articles on the concept.

Research design can be broadly defined or narrowly defined. The broad definition essentially encompasses the entire research process of idea, theory, hypothesis, empirical analysis, and conclusions. A narrow definition, which is the one I use, is simply how groups are compared.

Walkthrough of a Two-Group, Pre-test and Post-test Research Design

  • Recall example 5: G O X O and G O _ O. This a two-group design. Group 1 and Group 2 are observed, then Group 1 receives the treatment while Group 2 does not receive the treatment. Finally, both Groups are observed again. We can rewrite this explanation as the following:
G O X O
G O _ O
  • We read this notation from left to right, and line by line. For example, G O X O would be read as “Group 1, outcome variable before and after the treatment.” And G O _ O would be read as “Group 2, outcome variable before and after the absence of the treatment.”
  • We can take the notation used above, and add subscripts to differentiate between the G and O symbols:
G1 O1b X O1a
G2 O2b _ O2a
  • Before we continue on, let’s define each of these symbols:
    • G1 is Group 1
    • O1b is value of outcome variable (aka Y variable, or Dependent Variable, from your Theory) before treatment (aka X variable, or Independent Variable, from your Theory)
      • This is also called the “pre-test” of Group 1
    • X is the Treatment (aka Independent Variable)
    • O1a is value of outcome variable after Treatment
      • This is also called the “post-test” of Group 1
    • G2 is Group 2
    • O2b is value of outcome variable for Group 2 before Treatment
      • This is also called the “pre-test” of Group 2
    • _ is the underscore symbol and represents the absence of the Treatment
    • O2a is value of outcome variable for Group 2 after Treatment (or Absence of Treatment in this example)
      • This is also called the “post-test” of Group 2

Comparisons generated from a Two-Group, Pre-test and Post-test Research Design

  • The two-Group, pre-test and post-test research design allows for four unique comparisons of the values of the outcome variable:
    1. Within Group 1 Pre-test and Post-test: O1b compared to O1a
      • This is represented by the green bracket in the image below
    2. Within Group 2 Pre-test and Post-test: O2b compared to O2a
      • This is represented by the blue bracket in the image below
    3. Between Group 1 and Group 2 Pre-tests: O1b compared to O2b
      • This is represented by the red bracket in the image below
    4. Between Group 1 and Group 2 Post-tests: O1a compared to O2a
      • This is represented by the yellow bracket in the image below
  • This comparisons can be visually represented below:
Two-group pre-test post-test research design

Why you would want to make Comparison #1

You want to make comparison #1 because you want to see what effect, if any, the Treatment (represented by the symbol X, and also called the Independent Variable from your theory) had on the outcome variable (represented by the symbol O, also called the Dependent Variable from your theory) for Group 1.

Why you would want to make Comparison #2

You want to make comparison #2 because you want to see if the Absence of the Treatment (represented by the underscore _ symbol). Given the absence of X, you would expect to see no difference between the before and after values of the outcome variable for Group 2.

Why you would want to make Comparison #3

You want to make comparison #3 because you would expect, assuming both groups were randomly selected, that the values of the outcome variable for Group 1 and Group 2 would be the same before Group 1 experiences the treatment and Group 2 has no treatment.

Why you would want to make Comparison #4

You want to make comparison #4 because you would expect that the value of O1B would be different from the value of O2B since Group 1 received X and Group 2 did not receive any X.

Application of a Two-Group, Pre-test and Post-test Research Design

  • Let’s bring back our simple representation of the two-group, pre-test and post-test research design:
G O X O
G O _ O
  • At this point, you should be telling yourself “Wow, I didn’t realize a few symbols could mean so much to a political scientist!” Yes, they can. And this is part of the reason I ask you write out a research design for each of the journal article you analyze as part of the JAA.
  • Let’s apply this research design to two examples: an everyday life example and a political science example

Everyday Life Example

Imagine there are two neighborhoods: G1 and G2. Let’s assume they are the same in every respect, except receiving or not receiving X.

Let’s say X are king-sized Reese’s Peanut Butter Cup candies handed out during Halloween. In other words, the kids in G1 receive king-size candies, while the kids in G2 does not receive such candies. Consider, and answer, the following four questions:

  • What would be the energy level of the kids before X or _?
  • What would be the energy level for G1 kids before versus after X?
  • What would be the energy level for G2 kids before versus after _?
  • Which group of kids would have more energy after X or _: G1 or G2?

Political Science Example

Katerina Linos, Laura Jakli, and Melissa Carlson wrote an article titled “Fundraising for Stigmatized Groups: A Text Message Donation Experiment” which was published in the The American Political Science Review.

Abstract: As government welfare programming contracts and NGOs increasingly assume core aid functions, they must address a long-standing challenge—that people in need often belong to stigmatized groups. To study other-regarding behavior, we fielded an experiment through a text-to-give campaign in Greece. Donations did not increase with an appeal to the in-group (Greek child) relative to a control (child), but they were halved with reference to a stigmatized out-group (Roma child). An appeal to fundamental rights, a common advocacy strategy, did not reduce the generosity gap. Donations to all groups were lower near Roma communities and declined disproportionately for the Roma appeal. Qualitative research in 12 communities complements our experiment. We conclude that NGO fundraising strategies that narrowly emphasize either in-groups or out-groups, or fundamental rights language, may not be as effective as broader appeals, and we discuss implications for public goods provision in an era of growing nationalism.

Using only the Abstract above, can you write out a two-group pre-test post-test research design? To do so, you need to answer the following four questions:

  • Who is G1?
  • Who is G2?
  • What is O?
  • What is X?

Want to Learn More about Research Design?

If you want to learn more about research design, read Chapter 6 of Introduction to Political Science Research Methods. This Open Education Resource Textbook was co-authored by me and freely available to you.