JOHN ROBERT ("ROB") WARREN
  • Home
  • Research
    • Recent Publications
    • One Thing I learned...
    • Publication Trends
  • Teaching
    • Basic Social Statistics (SOC 3811) - Fall 2020 >
      • Lecture & Lab Materials
      • Problem Sets
      • STATA Exercises
      • Help with Math
    • Getting Your Paper Published (SOC 8090) - Spring 2020
    • Population Studies Research Practicum (SOC 4881) - Spring 2019 >
      • Course Readings
      • Individual Assignments
      • Group Assignments
      • Lecture Materials
    • Education & Society (SOC 3452) - Fall 2018 >
      • Week 1
      • Week 2
      • Week 3
      • Week 4
      • Week 5
      • Week 6
      • Week 7
      • Week 8
      • Week 9
      • Week 10
      • Week 11
      • Week 12
      • Week 13
      • Week 14
      • Week 15
  • Fun
    • Me at The Moth
    • Running Routes

One Thing I Learned By Editing Sociology of Education

3/8/2016

15 Comments

 
Picture
[From the August 3, 2016 issue of the newsletter of the American Sociological Association’s Section on Sociology of Education]
 
I very recently finished my service as editor of Sociology of Education (SOE)1—the global flagship journal of this subfield, one of the best journals in both sociology and education research. In that capacity, I read nearly 800 of your new and revised manuscripts—roughly one every weekday for three years—but had room to accept for publication only about 50 them.
 
Eric Grodsky asked me to share with you my sense of the subfield after reading all of those manuscripts. I am going to give you my honest assessment because I care deeply about education research.  I believe that as a subfield we are capable of making meaningful contributions to the scientific understanding of what education is, how it works, for whom it works and for whom it does not work, and what our society can do to improve it.
 
In short, I do not believe we are living up to our collective potential.
 
When I started as editor, I worried that the biggest challenge would be this: How will I choose from among all the great submissions? How will I select just the four or five best articles to publish in each quarterly issue?  That worry was based on my sense that a high percentage of submissions would be—or could, with modest revision, become—publishable.
 
In fact, I consistently had exactly the opposite problem. For each of the 12 issues I published, I struggled to find four high-quality publishable manuscripts. During my first year as editor, SOE had an extra page allocation from ASA;2 I did not fill all the extra pages because there were not enough acceptable papers. In all three years, I pressed authors to write more concisely so that I could fit a fifth article in each issue; only once did I succeed in finding five publishable papers.3 I am very proud of the articles that appeared in SOE during my three years as editor—and of their topical, methodological, and theoretical diversity—but I leave the journal deeply dismayed at the state of the subfield. I am somewhat dismayed that there were so few high-quality papers.  I am more dismayed that there were so many lesser-quality ones.
 
Most of the papers that I read had one or both of two basic problems:
 
First, a large percentage of papers had fundamental research design flaws. Basic methodological problems—of the sort that ought to earn a graduate student a B- in their first-year research methods course—were fairly common.4 (More surprising to me, by the way, was how frequently reviewers seemed not to notice such problems.) I’m not talking here about trivial errors or minor weaknesses in research designs; no research is perfect. I’m talking about problems that undermined the author’s basic conclusions. Some of these problems were fixable, but many were not.
 
Second, and more surprising to me: Most papers simply lacked a soul—a compelling and well-articulated reason to exist. The world (including the world of education) faces an extraordinary number of problems, challenges, dilemmas, and even mysteries.  Yet most papers failed to make a good case for why they were necessary. Many analyses were not well motivated or informed by existing theory, evidence, or debates. Many authors took for granted that readers would see the importance of their chosen topic, and failed to connect their work to related issues, ideas, or discussions. Over and over again, I kept asking myself (and reviewers also often asked): So what?
 
I gradually came to understand that (1) many authors just hadn’t yet fully thought through the “so what?” questions and (2) many authors were submitting papers long before they had fully worked through crucial issues related to research design, quality of evidence, and coherence of argument. They didn’t do a great job of motivating their questions because they weren’t yet fully sure how their work fit in the larger scheme of things.  They hadn’t thought through the “so what?” of their findings because they hadn’t had time to fully make sense of them.  They made assumptions or mistakes in their research design and analyses—just like everyone does in the early iterations of a project and paper—but they submitted their papers anyway. 
 
I think there are two related explanations for why the vast majority of SOE submissions over the past three years were simply not all that great and did not live up to their potential.  First, junior scholars are under far too much pressure to quickly publish lots and lots of papers. The problem is that it takes time and thought and trial and error to generate a really compelling research question and to deeply understand how to articulate the importance of that question. Then, it takes more time and thought and trial and error to come to a really convincing answer to that question, to fully develop the argument or story, and to write a manuscript that does justice to the value of the research. Unfortunately, junior scholars do not have enough time or enough room to think or make missteps. Graduate students must publish frequently—while still in graduate school—to complete for fellowships and professional jobs. Assistant professors must publish early and often to get tenure; associate professors must do the same to “make full.”  In each case, the pressure is to work quickly and efficiently. They have to get papers out for review as soon as possible, even if they really haven’t had the time to think them through or sharpen their argument or evidence. Quantity matters most, not quality. The incentive is to publish a dozen “just OK” papers rather than to publish four or five well-developed and higher-quality papers.  The collective result for the subfield: a mass of underdeveloped papers that often have some potential but really aren’t acceptable for publication in high-quality journals5.  These are papers that exist for the sake of being papers because the author is under pressure to write lots of papers.  I sincerely empathize with those authors—I once was one, and I socialize and work closely with many of them. Senior scholars in the subfield bear the responsibility for this state of affairs; they are in a position to change it. There simply has to be a better way to run a profession.
 
Second, junior scholars are not as well trained, mentored, or supported as they need to be. Partly, this goes back to the pressures to publish early and often. But partly, this is the collective fault of graduate training programs and junior faculty mentoring programs (and thus, again, of the senior scholars who lead those programs). Collectively, we simply need to do a better job of training, mentoring, and supporting junior scholars. Standards and expectations need to be higher, but junior scholars need to be given the resources—including time and mentoring—required to meet them. The subfield needs to be more innovative and energetic in designing creative ways to improve training and scholarship.
 
The modal SOE submission during my three years as editor was written by a junior scholar, had non-trivial methodological problems, and lacked a compelling theoretical or substantive rationale for having been written. This is despite the fact that the authors were well-intentioned, were extremely bright, and worked hard at their craft. Virtually every paper had some kernel of real potential in one way or another—some creative idea, some new insight, some thoughtful new approach—but far too often that potential was not realized. All of this represents, to my mind, a waste of resources—including authors’ time and energy and frequently tax-payers’ money. And, it doesn’t help any of us move closer to addressing the problems, challenges, dilemmas, and mysteries that we all face in education and our broader society.
 
Nobody asked for my views on how to fix these problems, but here they are anyway. First, sociology departments, education schools and their kin need to de-emphasize the quantity of published papers in making hiring and promotion decisions. Second, those units and fields need to primarily reward the quality of publications. Third, we all need to be more energetic, more creative, and more collective-minded in training, mentoring, and supporting junior scholars.  To be clear: The onus of implementing these badly-needed changes falls on senior scholars.  Junior scholars are appropriately and understandably focused on their own careers.  It falls on senior scholars to transform incentive structures—rewarding quality over quantity—and to provide junior scholars with the time and support they need to do better research and write better papers.
 
ENDNOTES
 
1 Although my name is on the masthead as editor for the rest of 2016, incoming editor Linda Renzulli took over the day-to-day operations of the journal on July 1.
2 That’s right: In the 21st century ASA still thinks in terms of printed page allocations and mailing costs.
3 One possibility is that I was too harsh a judge of submissions.  Maybe, but: About one in seven authors of new submissions was invited to revise and resubmit their manuscripts, most revised papers were subsequently accepted for publication, and I did ultimately publish at least four papers per issue.
4 The most common questions I wished authors had asked themselves before they submitted their papers included: Is this the best group of people to study in order to achieve my research objectives?  How should I measure key concepts, and what are the relative strengths and weaknesses of those measures?  Does the evidence I have presented speak directly to my questions or hypotheses?  Does that evidence adequately support my arguments, claims, and conclusions?  Am I making causal claims, and if I so does my evidence support such claims? If I am comparing multiple groups, have I actually compared those groups? 
5 This is not to say that as editor I expected papers to be acceptable and publishable right out of the starting gate.  I invited the authors of about 13% of new manuscripts to revise and resubmit their papers based on reviewers’ and my own comments.  I know that a big part of the job of a journal editor is to identify promising papers and help them become higher-quality published articles.  Like I said, I’m very proud of the papers I published.
 

15 Comments
Matt
3/8/2016 09:14:18 pm

Thanks for your words. I only hope that programs and senior faculty heed them.

Reply
Steve
4/8/2016 11:53:44 am

Great essay!

Reply
Julie Kmec link
4/8/2016 01:10:20 pm

Great post, Rob. This problem may only get worse as we see hiring freezes, shrinking faculty sizes, larger classes, more pressure to get grant money. I wonder if the recent rise in the # of sociology journals has also mattered, giving us more options to send out mediocre work.

Reply
Jim
6/8/2016 11:45:27 am

This largely reflects with my experience as a graduate student 10-15 years ago. Also, in my dep't, near complete lack of financial support for graduate student research support led to semester-by-semester funding hunts that defeated efforts to focus on developing good research projects. Students that did well seemed to have ample external financial support to get them through summers or even years where they could just focus on their research.

Reply
Jo VanEvery link
10/8/2016 05:18:52 am

A related problem is that far too many scholars see publication in a high quality journal primarily in terms of what it does for the CV rather than as a means to communicate research results with other scholars in the field and participate in a conversation that will advance the field as a whole. This strikes me as particularly relevant to the "So What?" question. The writer is not sufficiently concerned with the reader.

Reply
Paul link
5/10/2016 08:14:09 am

We all need to know that we're important in other people's lives. We all want to be noticed and valued, especially if we've done a job that is noteworthy.

Reply
Nico
15/12/2016 01:01:39 pm

This is precisely why I'm applying to Minnesota. As an incoming graduate student, a mentor with skeptical eyes toward that which is and a willingness to improve it is worth far more than any other factors.

Reply
David
12/1/2018 07:21:54 am

Yes, this assessment of sociology of education is correct. I have reviewed for the journal countless times and faced the same struggles Rob mentioned. Pointless, sloppy work without a compelling word to say about large issues was the norm. But there is now a larger problem. Sociology of Education now is rejecting at the editorial level, without review, papers that seek to development macro-theoretical frameworks that would help guide meaningful research. This is a dreadful dereliction of duty to the discipline.

Reply
simon link
7/5/2019 12:22:36 pm

Thanks for sharing this post,
is very helpful article.

Reply
Brittany link
23/12/2020 08:35:44 am

Hi!

Reply
Dennis Robinson link
4/11/2022 12:05:42 pm

Say career certain half difficult direction economic. Say human seem whole Republican skin.

Reply
Sergio Munoz link
9/11/2022 06:21:54 pm

The security weight this page election recognize. Place type site run fine miss. Your recently born.
For resource old seek now.

Reply
Edward Howard link
12/11/2022 11:30:06 am

Number seem light mother whom before three. Than sing car gun upon political.

Reply
Sean Wyatt link
12/11/2022 06:37:22 pm

Finally material we what real inside strategy.
Whatever scientist option history term. Business reduce off collection college set time.

Reply
Adam Martinez link
13/11/2022 11:28:56 am

Manager east our future what. Little dream if. Up goal owner state around military community.
Friend wish edge coach. Sometimes everybody source pick worry daughter. Seek under music.

Reply



Leave a Reply.

    Rob Warren

    University of Minnesota
    (Outgoing) Editor of Sociology of Education

    Picture

    Archives

    August 2016

    Categories

    All

    RSS Feed

  • Home
  • Research
    • Recent Publications
    • One Thing I learned...
    • Publication Trends
  • Teaching
    • Basic Social Statistics (SOC 3811) - Fall 2020 >
      • Lecture & Lab Materials
      • Problem Sets
      • STATA Exercises
      • Help with Math
    • Getting Your Paper Published (SOC 8090) - Spring 2020
    • Population Studies Research Practicum (SOC 4881) - Spring 2019 >
      • Course Readings
      • Individual Assignments
      • Group Assignments
      • Lecture Materials
    • Education & Society (SOC 3452) - Fall 2018 >
      • Week 1
      • Week 2
      • Week 3
      • Week 4
      • Week 5
      • Week 6
      • Week 7
      • Week 8
      • Week 9
      • Week 10
      • Week 11
      • Week 12
      • Week 13
      • Week 14
      • Week 15
  • Fun
    • Me at The Moth
    • Running Routes