Academics working today are laboring in a rapidly changing landscape of scholarly communication.

When acclaimed Internet researcher danah boyd published her recent book, “It’s Complicated,” about the social lives of networked teens with the highly reputable academic house Yale University Press, she also put a free PDF of the book up on her own website.  She wrote this about that decision:

“…I didn’t publicize this when I did so. For those who are curious as to why, I want to explain. And I want you to understand the various issues at play for me as an author and a youth advocate.

I didn’t write this book to make money. I wrote this book to reach as wide of an audience as I possibly could. This desire to get as many people as engaged as possible drove every decision I made throughout this process. One of the things that drew me to Yale was their willingness to let me put a freely downloadable CC-licensed copy of the book online on the day the book came out. I knew that trade presses wouldn’t let a first time author pull that one off. …But what I started to realize is that when people purchase the book, they signal to outside folks that the book is important. This is one of the reasons that I asked people who value this book to buy it. Your purchasing decisions help me signal to the powers that be that this book is important, that the message in the book is valuable.” (emphasis in the original)

It’s an important and worthwhile book, and you should buy it and/or download it, depending on what you can manage. What I so appreciate about what she’s done here is to find a way to thread the very thin needle of open access and a prominent, scholarly book.

Elsewhere in that post, she describes her experience with the machinery of publishing, and it goes like this:

“If you haven’t published a book before, it’s pretty unbelievable to see all of the machinery that goes into getting the book out once the book exists in physical form. News organizations want to promote books that will be influential or spark a conversation, but they are also anxious about having their stories usurped by others. Booksellers make risky decisions about how many copies they think they can sell ahead of time and order accordingly. (And then there’s the world of paying for placement which I simply didn’t do.) Booksellers’ orders – as well as actual presales – are influential in shaping the future of a book, just like first weekend movie sales matter. For example, these sales influence bestseller and recommendation lists. These lists are key to getting broader audiences’ attention (and for getting the attention of certain highly influential journalistic enterprises). And, as an author trying to get a message out, I realized that I needed to engage with this ecosystem and I needed all of these actors to believe in my book.”

Her experience with publishing is quite different from the traditional academic’s experience, but then that might be expected as danah boyd is not a traditional academic.  If you’re not familiar, danah boyd is something of a celebrity among folks who study the Internet, works as a Principle Researcher at Microsoft, and is starting her own research shop called Data & Society. Her work is also on two areas  — the Internet and teenagers — that has wide public appeal.

The reality for most traditional academics is that they produce “Long, complex monographs are expensive to produce yet sell only 150 to 300 copies.”

The news is even worse for academic papers published in traditional journals. A study at Indiana University found that:

“as many as 50% of papers are never read by anyone other than their authors, referees and journal editors.” That same study concluded that “some 90% of papers that have been published in academic journals are never cited.”

This is a certain kind of impact, to be sure, if who you are trying to have an impact on is an elite group of specialists in your field.  But this model of publishing is never going to have much of a wider reach.

As Anthony DiMaggio, writing for CounterPunch, notes about his own field of Political Science, that it is dominated by “over-specialization and obscurity” with scholars who carve out “extremely narrow niches” that have “no practical utility.”  DiMaggio minces no words as he calls out social science academics broadly for a lack of relevance and what he deems as cowardice:

“Lack of relevance to the political world doesn’t make one’s research interesting or worthwhile, but this message falls on deaf ears in insulated places like high ed social science departments.  A main reason for scholars’ contempt for political advocacy is cowardice.  The vast majority of scholars have been socialized their entire lives to believe they must always remain ‘objective,’ and that to take a position on an issue would be heretical.  Most scholars operate according to a pack mentality – fearful of engaging in unconventional behavior.  By producing useful real world research, one is challenging the sacred rules governing ‘objective’ social science that celebrate esoteric research agendas. To step outside that mold would be to endanger one’s prestige, and risk that one will be seen as unprofessional in colleagues’ minds. Such pressures ensure that academics remain part of the problem, not the solution. They fail by design to challenge the political and economic power status quo and injustices that occur around them.”

There’s something to what DiMaggio says here, but I don’t know if it’s cowardice as much as institutional reward structures.  Or, perhaps those are two sides of the same coin.

The legacy model of scholarly communication values writing obscure books and papers for tiny audiences makes sense within a certain kind of reward structure. Within legacy academia, the people that sit on hiring, tenure and promotion committees still place value on at things like ‘impact factor’ of little-read journals and the fading prestige of boutique publishers with minuscule runs.

However, the appearance of digitally fluent, hybrid scholars – like danah boyd – who are more interested in reach and impact on a broader public, point to a new kind of reward structure, one that values influence beyond a small group of specialists.

The real challenge, I think, comes when a researcher that doesn’t have the star-power or following of a danah boyd wants to write about something that’s much less appealing than what teenagers are doing on the Internet.  What kind of broad reach or impact can a relatively unknown scholar writing about a topic that’s unpopular expect to have? This remains an open question in this changing landscape of scholarly communication, but it seems to me that the Internet offers a set of opportunities to reach beyond the conventional audiences for academic research.

Still, even when academics use social media there’s little to indicate they are doing so in order to reach a broad, general audience. Indeed, we know from recent research that even when academics use social media, such as blogging, they mostly don’t do this to engage with a broader public. In a recent study of 100 academics blogs, researchers found that most academics are blogging for professionals peers, rather than for the public in any general sense: 73% of the blogs analyzed were geared toward other academics, while just 38% were designed for general readers.

I can’t help but wonder how different academic research would look if we were guided by danah boyd’s goal: “I wrote this book to reach as wide of an audience as I possibly could.”  

The counter to this, of course, and one that I often hear in talks I give about this work, is something along the lines of: “well, small publishers and journals are providing a valuable service for getting academic work published that wouldn’t ever be interesting to a wide, public audience. This work is often too complex, theoretical, esoteric, important, too politically unpopular for a wide audience, so we must rely on the obscure publishing options to keep doing what we do as academics.”

There is something to this argument.  For example, I write about racism – a thoroughly unpopular topic in the US.  My academic books have done ok, but they will never be as popular as the work that danah boyd does.  It’s also the case that academic presses have published books of mine that probably would not have been picked up by trade presses for a general, public audience. Still, what I also know to be true is that the work I do on racism has gotten a much bigger following from my various social media outlets than it has from the books and articles I’ve published.

The skepticism about “reach” for academic work is built on a misconception that there won’t be an audience for that work. In fact, I think there are multiple audiences, varied publics and a wide citizenry that’s really interested in more substantive contributions about the state of the world than they’re currently getting.  And, I think academics can step up and make a contribution, if we’ll begin to re-think what scholarly communication is now.

As we come to the end of the meta-MOOC #FutureEd conversations prompted by Cathy Davidson and our lunchtime interlocutors here at CUNY, I’m pondering how we measure the impact of such a course, and more broadly, the impact of the work we do as academics.

In a recent post at The Chronicle of Higher Ed, Douglas Howard considers the unfinished feeling of so much teaching in general:

Did Jim, who talked about becoming a therapist, go on to graduate school in psychology? Did Jessica, who argued so passionately in class against the death penalty, make it as a lawyer? We fill in the blanks about them based upon what we know (or think we know), and tell ourselves that their stories ended the way that we hoped.

Teaching, in this regard, is the great open-ended narrative, the romantic fragment, the perpetually unfinished symphony.

The fact is, we almost never know if a course we taught, or a book we wrote, or an article we labored over, has any impact on anyone else.

Mark McGuire has a thoughtful post at HASTAC about the difficulty in rating transformational experiences specifically in the context of a MOOC, such as the one on #FutureEd:

We rate hotels, music, live performances, movies, etc., so that others are able to make an informed decision about how to best invest their time and money. Rating and reviewing MOOCs seems like a sensible thing to do for similar reasons, and it would not be surprising to see such a practice develop. However, unlike a hotel or restaurant franchise, a living, changing, organic learning experience cannot be packaged, replicated, and sold to consumers who are looking for a satisfying (and predictable) product or service. It can’t offer the same experience to the same person twice, and one person’s experience may not be a good indicator of the experience another person will have. A transformational learning experience is like a good pot luck dinner party — you might have had the time of your life, but it can never be repeated.

I like the dinner party metaphor for teaching much better than the “unfinished symphony,” perhaps because I’m much more likely to attempt a dinner party than a symphony.  I do think that there is a kind of alchemy in teaching, and good dinner parties, that makes it easy to assemble the same people, elements and conditions but difficult to replicate magic when it happens.

It’s also difficult to think about how one might measure any of this. Unfortunately, to my way of thinking, the word “measure” has become synonymous with “quantify.”  When we think of measurement exclusively in terms of quantification and counting, we lose a great deal of the story of impact.

I want to suggest an alternative way of thinking about impact that includes qualitative measures. In the chart below, you’ll see a way of conceptualizing “transactional” measures – quantifiable things we can count, alongside “transformational” measure – qualitative measures, that are more difficult to count but represent more lasting change.

(Chart content from Bolder Advocacy, h/t gabriel sayegh.
Chart design by Emily Sherwood)

In this schema, transactional metrics include both traditional (or “legacy”) academic measures of impact such as citations, along with alternative measures (or “altmetrics”), such downloads, or mentions on social media. Quantitative, transactional measures of impact can also include lasting social change, such as changes to public policy.  One of the things I like about this conceptualization is that it illustrates the incremental change that altmetrics represent. In other words, altmetrics are just another way of counting things – downloads and social media mentions, rather than citations – but it’s still just counting things. Counting and quantification can tell us somethings, but it doesn’t tell the whole story.

On the “transformational” side are those things that it’s difficult, perhaps even impossible to measure, but that are so crucial to doing work that has a lasting impact. These include identifying allies, building relationships, establishing collaborations, and co-creating projects. Ultimately, transformational work is about changing lives, changing the broader cultural narrative, and changing society in ways that make it more just and democratic for all. These kinds of transformations demand a different kind of metric, one that relies primarily storytelling.

How might this work in academia? Well, to some extent, it already does. 

To take the example of teaching, you may have gotten the advice – as I did – “save everything” for your tenure file. This advice often goes something like, “everytime a student sends you a thank you card, or writes you an email, or says, ‘this class made all the difference for me’ save that for your tenure file.” That’s part of how we ‘finish the symphony,’ to borrow Doug Howard’s metaphor, we get notes from students, we compile those into a narrative about our teaching. It’s impartial, to be sure, but it’s something. The comments that students add to teaching evaluations are another place we see that impact in narrative form, although these are so skewed by the context of actually sitting in the class that it misses the longer term impact of how that course may have changed someone.

For the diminishing few of us on the other side of the tenure-hurdle, think about those letters of recommendation we write for junior scholars.  Whether we’re writing for someone to get hired, promoted, or granted tenure, what we’re doing when we craft those letters is creating a narrative about the candidate’s impact on their corner of the academic world so far. Of course, we augment that with quantitative data, “this many articles over this span of time,” and “these numbers in teaching evaluations.”

The fact is, we already combine transactional and transformational metrics in academia in the way that we do peer evaluations. What we need to consider in academia is expanding how we think about ‘impact’ and realize the way that we already use both quantitative and qualitative measures to evaluate and assess the impact of our work.

More than this, we need to reclaim storytelling and narrative, augmented by the affordances of digital media, to tell stories of impact that make a difference.

Our new series on scholarly communication continues with a look at the idea of “scholarly impact,” a topic we’ll feature regularly.  The central issue at hand: how do we measure the value of scholarly work in a meaningful way?

In today’s Chronicle of Higher Education, Aisha Labi writes about the resistance among researchers in the UK to having the impact of their work measured.  As Labi describes it:

The fundamental idea is relatively uncontroversial: As government spending in Britain has become more constrained, public investment in research must be shown to have value outside academe.

But calculating research’s broader value is a challenge—and a growing number of academics find themselves arguing that the requirements are unduly burdensome and do little to achieve their stated goals.

In the context of the UK, there is something called the Research Excellence Framework (REF), which requires that the impact of research by university departments accounts for 20% of the formula for financing them.

The resistance among UK academics, like Professor Philip Moriarty quoted in the article, is that while he sees his research in nanotechnology having a broad benefit to society, a focus on impact is a “perversion of the scientific method,” one that emphasizes “near-market” research, designed to generate a speedy economic return for taxpayers, he says.

I agree. If the definition of impact of scholarly work is going to be defined in business-school terms about ROI for taxpayers, then that’s not only a “perversion,” it’s a recipe for disaster for higher education as a long-term endeavor.

Resistance though there may be, some in the UK see value in the discussion of impact. The good folks at the LSE Impact of Social Sciences project are involved in a multi-year project to demonstrate how academic research in the social sciences achieves public policy impacts, contributes to economic prosperity and informs public understanding of policy issues and economic and social changes.

In the US, there’s a much different landscape of higher ed and impact is not (yet) tied to research funding in as systematic a way as in the UK with the REF.  This doesn’t mean that academics in the US are uninterested in the impact of scholarly work, quite the contrary. There’s a long history of attempts to measure impact here.

How many inches? 

The idea of measuring impact within scholarly disciplines has for most of the last century relied on counting the number of citations within peer-reviewed journals. For example, an individual scholar’s listing in the Social Sciences Citation Index (SSCI), which compiles number of citations in journals, has been a frequently consulted resource in tenure and promotion cases.

(Image source)

I know of stories in the olden days of analog when tenure and promotion committees would graduate student assistants to the library with an actual ruler in order to measure the number of inches a prospective candidate had in the SSCI.  Sometimes a ruler is just a ruler, but is this really the best we can do in measuring the impact of our work?

Alternative Measures, or “Altmetrics” 

Recently, attention in higher education has turned to new ways of measuring scholarly impact by incorporating the use of social media. The idea behind alternative measures, or “altmetrics,” is that the traditional metrics of citations in peer-reviewed journals (a la the (SSCI) should be joined by new measures, like number of page views, downloads, “likes” and re-tweets on social media. These have spawned a new generation of tools to automate the collection of this data into one platform or indicator (e.g., Almetrics,FigShare, PlumAnalytics, ImpactStory).

However, altmetrics are not yet widely used forms of measurement within academia by things like hiring committees or tenure and promotion committees. In fact, people in higher education don’t yet know what to make of these alternative measures and are actively working on how to resolve these issues. I, personally, sit on at least 3 committees within my institution and 1 committee in a professional association, that are all trying to come up with equitable, reasonable and widely understandable ways to measure scholarly work in the digital era.

Upworthy is Not the Same as Peer-Review

Many scholars express concern about the turn to social media as a measure of impact because of the kinds of information that often gets rewarded in an economy of “likes.”  We might call this the “upworthy” problem. If you’ve ever seen this site, or been lulled into clicking on something there, there’s a kind of relentless cheeriness and warm, touchy-feelingness to all things shared on the site. People who want to endorse content there indicate that it is “upworthy,” meaning worth moving “up” on your pile of things to read and do online. But it’s hard to imagine most of the research I’m familiar with and admire ever getting a vote as “upworthy.”

In a recent piece for the Chronicle of Higher Education, Jill Lepore turned a critical eye to the problem of using social media as an indicator of scholarly impact. Lepore writes:

“…when publicity, for its own sake, is taken as a measure of worth, then attention replaces citation as the author’s compensation. One trouble here is: Peer review may reward opacity, but a search engine rewards nothing but outrageousness.”

Lepore is right to challenge us to think about what it is that search engines reward.  And, I think it’s also critically important to reconsider the value we put in publishing writing that is opaque in journals locked behind paywalls with tiny audiences.

I think where Lepore, and lots of others, get the importance of social media for scholarly work wrong is when they assume that it’s all just “publicity” or self-promotion.

The reality is that many scholars are using social media so that they can have an impact on the world, not just on their peers in academe.

There’s a Whole World Beyond Academe

Many disciplines have traditions of talking about “impact,” but it’s usually in a negative context. I’ll pick on a couple of disciplines that I spend some time in. So, for example, sociologists are very accustomed to pointing out the negative impacts of social structures and policies on inequality. Scholars in public health and demography are all about measuring the impact of lots of things on “mortality rates” – a serious measure of impact if ever there was one.  And, most social scientists of all stripes are perfectly fine with tracing worsening measures of inequality to changes in policy.

Yet, scholars are much less clear, timid even, on how scholarship might affect those laws and social policy.

This is especially ironic given what we know about what scholars want. According to a recent survey 92% of social science scholars said they wanted “more connection to policymakers.” 

Are academics just not capable of thinking about their own impact on the world? I think we are because, well, because we’re human.

The Desire to Have an Impact is a Deep, Human Need

We all, as human beings, want to know that our life matters, that we had an impact. Many scholars, but certainly not all, want to know that the work they spend so much time, training, money and effort into matters in some way beyond the small circle of experts in their chosen field.

Martin Rees, an emeritus professor of cosmology and astrophysics at the University of Cambridge and one of Britain’s most noted scholars, quoted in the Chronicle piece mentioned at the top, says:

“Almost all scientists want their work to have an impact beyond academia, either commercial, societal, or broadly cultural, and are delighted when this happens. But they realize, as many administrators and politicians do not, that such successes cannot be planned for and are often best achieved by curiosity-­motivated research.”

I think Professor Rees is right that most academics want their work to have an impact beyond academia. I don’t know that I agree with him that there’s no planning for it (more about that another time). The desire for our work to matter is a class-bound one, to be sure, in ways that may not be obvious. Sanitation workers have possibly the most important jobs from a public health perspective (much more important than doctors); they can certainly take comfort in the impact their job is having on promoting the health of large populations of people. It’s harder for academics, who trade in ideas, to point to the impact of our work, but I think that the desire is an existential one.

In many ways, the classic Frank Capra film, “Its’ a Wonderful Life,” (1946) is a film about impact. As you may recall, Jimmy Stewart’s character, George Bailey, on the verge of suicide, is given the gift of seeing what the world would have been like without him in it. A guardian angel, Clarence, replays key events in his life and then runs the reel of what unfolded because he wasn’t there. “Your brother died, George, because you weren’t there to save him when he fell in the ice,” Clarence explains. As he sees more and more of this alternative reality without him in it, George Bailey begs, “I want to live again,” and his wish is granted.

(Image source)

The moral of the film, of course, is that we all have a much greater impact than we realize on the lives of others.  But it seems to be lesson that is lost on scholars and academics. Perhaps we are too practiced in the art of cynicism and critique to imagine that our research could have an impact.

The place where most academics I know are much clearer about their impact on the world is in the classroom. Academics will joke amongst ourselves about “shaping young minds,” but the joke reveals a truth we hold close: that what we do there matters. It can change lives. In many instances, we are academics because there was a scholar once, somewhere, who changed our lives, and then all we wanted to do was that… talk about ideas in ways that changed peoples’ lives. How do you measure a life? In cups of coffee, or in lectures given, in semesters taught.

Can We Work for Justice, Measure Impact, and Resist? 

There are many reasonable arguments on the side of those who want to “resist metrification,” as my colleague Joan Greenbaum puts it. Governmental, institutional attempts to link research output to business profits are, to my way of thinking, wrong-headed and doomed to fail. We should, and must, resist efforts to use any form of measurement to surveil and discipline faculty in the service of economic gain.

But, I don’t think that’s the moment we’re in right now in the US.  

I think that the moment we’re in is one in which academics are beginning to work in new, digitally augmented ways, and most institutions of higher education have no clue how to assess that work or evaluate the impact of a scholar who is up for tenure or promotion with a mostly digital portfolio.

I also think the moment we’re in is one of appalling economic inequality, and many, many academics I know want to join their work to the struggle to reduce that inequality.  Mostly, they don’t know how to go about doing that. And, if they do go about doing that, they wonder: how will this work “count” for me when it comes time for hiring, tenure or promotion? 

I think the moment we’re living in requires us to come up with innovative new ways to measure impact that take into account more kinds of work, including work for the public good. This is not as radical an idea as it sounds. I think we do this in some ways already. 

When I write a tenure letter for someone to get promoted, I’m crafting a story about their impact on the world as a scholar, a teacher, and a member of a community.  It’s very often the case that I will write about scholar up for tenure something like: “This scholar has made a profound impact on her/his community through their work engaging local residents about the topic of her/his research…” and then go on to detail the forms this impact has taken.  Quite simply, a tenure letter is a way of crafting a story about impact.

We’re left with many questions about scholarly impact in the digital era. Most pressing for me is this rather grand question: How do you measure an idea that takes hold and changes peoples’ lives, changes public policy, and changes the way knowledge is created and shared? 

I don’t think we know the answers to this question yet. We are still way before the beginning in understanding how to measure impact in ways that are meaningful.