Teaching article reading

Today my research students and I did a fun activity. We’ve found several papers that are connected to the work we’re doing, and I wanted to sit with my students to pore over them to see where the gaps were for us to work. In the process, we had a great conversation about how experts and novices approach journal articles. I wanted to get the gist of the conversation down to see if others have any other good insights for this.

On twitter I posted this, which I think captures the essence of the conversation:

4 me: title, abstract, figures, variables, conclusion, finally the details. For them: title, abstract, 1st para, 2nd para, etc. very diff

— Andy Rundquist (@arundquist) June 19, 2013

(really I just put that to try to get a few more of you to see the power of twitter, sorry).

So, what did I do? First, I made sure that I hadn’t looked at the papers before we sat down together. They found them, based on a forward search (using the awesome Google Scholar), and a few of them had read a few of them (wow, that’s a really unclear sentence fragment). But I wanted to show them how I would approach them, so, even though they found them the day before, I didn’t really look at them other than the titles.

We sat around a table, each of us with paper copies, along with our awesome Livescribe Sky pen taking notes and recording the conversation, which sync-ed immediately to Evernote (I’m really loving that). We decided what order we’d do them in and we jumped in. I decided to just talk out loud about how I would go about reading the first one. That took an hour! After that I asked them to use a similar process on the other three papers.

Here’s how that hour went (in general terms):

1. I analyzed the title to see what the authors were proud of. There was a funny pun-like title, but not much other than that to show that it was connected to our work.
2. I looked at the abstract with a similar eye. It was interesting that they talked about how the system was interesting because the final setting was calculable by minimizing the virtual work done by the tension. I told my students that that just meant that it obeyed Newton’s laws. What I went on to explain was that they used a great tool, one they were good at, but, ultimately, just one of many tools that could do the same thing.
3. I flipped to the first figure, happy to see that it was a diagram of the main experimental setup. We analyzed the variable names, and I asked them to sketch various s vs z curves (s was the length of the hanging rope, z was the height variable) because I noticed a nearby equation that asked for ds/dz. There was one angle shown, so I asked them to figure out how that angle changed over the length of the whole curve. I told them that having a good handle on the functions would let them image where the math might be going.
4. I flipped to the next few figures, which were comparisons of simulations to experimental data. We noted how similar they were to our our data, but we were mystified as to why they didn’t do an overlay. I told them it was probably because the overlay wouldn’t have been as good as they claimed in the text, but later we realized that it would have been nearly perfect if you compared the data to the full-blown theory. The problem was that they wanted to brag about how good an approximate (read: easy) theory’s prediction was. Yes, they looked similar, but an overlay to that theory would have been pretty crappy.
5. I scanned the text to find the first mentions of the various data/theory figures to see what they said beyond the figure captions. I told my students that we’d be able to see what they were proud of there, rather than the figure captions.
6. I jumped to the concluding paragraph to see what the authors thought was the big deal.

All along we had conversations about why I was doing what I was doing. One thing that came up was that we weren’t really poring over their math. I told them that the big mistake I used to make was to check author’s algebra all the time. I told them that it’s mostly a fruitless endeavor, talking up the power of peer review. I said that if we were to go through the derivation, then we’d be forced to go down the path the authors wanted us to regarding their data. But what if we wanted to go down a different path? The derivation will be very useful to us moving forward, but my approach above helped us figure out what we really cared about in the paper. Now we know which math portions are going to be helpful to us.

I was shocked to see that an hour had gone by, but I think it was really a great activity. We ended with a short, half-page section of notes about what the paper did for us, and I asked them to do something similar for the other three. I plan to check in with them about that tomorrow. We’ll see!

So what do you do in situations like this? How do you read journal articles?

Some sample comment threads for you:

• You have to check the math! You can’t just swallow it whole!
• You should have read it first and spoon fed it to them. They’d get it better rather than throwing them in the deep end.
• Can you post the pencast of the session?
• I was one of the students, and I thought it really sucked. It was one of those rare gorgeous Minnesota summer days and we had to waste it in the basement of the science building.
• I was one of the students, and I thought it was great! Andy just spoon fed us the useful stuff and I zoned out the whole “let’s learn how to read” stuff.