Today my research students and I did a fun activity. We’ve found several papers that are connected to the work we’re doing, and I wanted to sit with my students to pore over them to see where the gaps were for us to work. In the process, we had a great conversation about how experts and novices approach journal articles. I wanted to get the gist of the conversation down to see if others have any other good insights for this.
On twitter I posted this, which I think captures the essence of the conversation:
(really I just put that to try to get a few more of you to see the power of twitter, sorry).
So, what did I do? First, I made sure that I hadn’t looked at the papers before we sat down together. They found them, based on a forward search (using the awesome Google Scholar), and a few of them had read a few of them (wow, that’s a really unclear sentence fragment). But I wanted to show them how I would approach them, so, even though they found them the day before, I didn’t really look at them other than the titles.
We sat around a table, each of us with paper copies, along with our awesome Livescribe Sky pen taking notes and recording the conversation, which sync-ed immediately to Evernote (I’m really loving that). We decided what order we’d do them in and we jumped in. I decided to just talk out loud about how I would go about reading the first one. That took an hour! After that I asked them to use a similar process on the other three papers.
Here’s how that hour went (in general terms):
- I analyzed the title to see what the authors were proud of. There was a funny pun-like title, but not much other than that to show that it was connected to our work.
- I looked at the abstract with a similar eye. It was interesting that they talked about how the system was interesting because the final setting was calculable by minimizing the virtual work done by the tension. I told my students that that just meant that it obeyed Newton’s laws. What I went on to explain was that they used a great tool, one they were good at, but, ultimately, just one of many tools that could do the same thing.
- I flipped to the first figure, happy to see that it was a diagram of the main experimental setup. We analyzed the variable names, and I asked them to sketch various s vs z curves (s was the length of the hanging rope, z was the height variable) because I noticed a nearby equation that asked for ds/dz. There was one angle shown, so I asked them to figure out how that angle changed over the length of the whole curve. I told them that having a good handle on the functions would let them image where the math might be going.
- I flipped to the next few figures, which were comparisons of simulations to experimental data. We noted how similar they were to our our data, but we were mystified as to why they didn’t do an overlay. I told them it was probably because the overlay wouldn’t have been as good as they claimed in the text, but later we realized that it would have been nearly perfect if you compared the data to the full-blown theory. The problem was that they wanted to brag about how good an approximate (read: easy) theory’s prediction was. Yes, they looked similar, but an overlay to that theory would have been pretty crappy.
- I scanned the text to find the first mentions of the various data/theory figures to see what they said beyond the figure captions. I told my students that we’d be able to see what they were proud of there, rather than the figure captions.
- I jumped to the concluding paragraph to see what the authors thought was the big deal.
All along we had conversations about why I was doing what I was doing. One thing that came up was that we weren’t really poring over their math. I told them that the big mistake I used to make was to check author’s algebra all the time. I told them that it’s mostly a fruitless endeavor, talking up the power of peer review. I said that if we were to go through the derivation, then we’d be forced to go down the path the authors wanted us to regarding their data. But what if we wanted to go down a different path? The derivation will be very useful to us moving forward, but my approach above helped us figure out what we really cared about in the paper. Now we know which math portions are going to be helpful to us.
I was shocked to see that an hour had gone by, but I think it was really a great activity. We ended with a short, half-page section of notes about what the paper did for us, and I asked them to do something similar for the other three. I plan to check in with them about that tomorrow. We’ll see!
So what do you do in situations like this? How do you read journal articles?
Some sample comment threads for you:
- You have to check the math! You can’t just swallow it whole!
- You should have read it first and spoon fed it to them. They’d get it better rather than throwing them in the deep end.
- Can you post the pencast of the session?
- I was one of the students, and I thought it really sucked. It was one of those rare gorgeous Minnesota summer days and we had to waste it in the basement of the science building.
- I was one of the students, and I thought it was great! Andy just spoon fed us the useful stuff and I zoned out the whole “let’s learn how to read” stuff.
I find that my approach depends on my purpose for reading the paper. If I am trying to see if it is relevant, I follow an approach that is similar to yours. If I’m pretty certain it is something that is relevant based on reading the abstract or some external factor I tend to read it linearly, but gloss over the intro and theory details. Depending on my familiarity with the topic I may or may not dig into the intro and theory.
what would then cause you to go back to the theory section in detail?
I will have to pay attention when I’m reading some papers in the near future and see if I can flesh out my current strategy a bit more. Of course now I’m tainted with your approach so things may never be the same again.
I was in fact one of those students, though I cannot say I find myself backing either of Andy’s last two sample thread buds.
Maybe it was partly due to the quad-shot Hazelnut Macchiato amping my synapses, but I rather enjoying tearing into sources 3 and 4 on my own in the style you described above. Varying slightly, I checked out the sources they used primarily to know when they were citing articles I knew. Then I followed Andy’s formula for paper dissection. I’ll admit in analyzing #4, I did have to dip into their math to find out (to the best of my knowledge) what the heck epsilon represented in figure two. [I’m sure this means a lot to anyone outside the five of us]
I think Andy described the greatest benefit from this m.o. when he mentioned how by NOT directly following their article cover to cover you retain a level of skepticism.
It was fun working with Marten and the rest this morning going over “paper number 4.” It took a while to figure out what was even being plotted, but now it seems we have a good plan of attack for the next few days.
I read differently.
I usually read to gain knowledge, not to prove or disprove.
It seems we agree. How do you read an article to best gain knowledge?
I read the introductory paragraph. Then I read first and last sentences. If it is a GOOD article, I will slow down for some comprehension.
If it is a bad article, badly written, I will usually stop after a couple of paragraphs and look for a better article.
My one insight is that analyzing the title in terms of authorial pride and intentionality is not always legitimate. Sometimes the editors change (or insist on changes to) the title in ways that may not reflect the actual core values of the authors.
– Elizabeth (@cheesemonkeysf)
That’s an interesting point. I guess that’s never happened to me. Can you give me an example?