Some hard facts about science, money and evaluations

The most fun experiments are those that generate a lot of noise and heat. So why not ask a scientist about her latest grant application? Or about the reports she has been asked to write describing the results of her research? These simple questions will likely result in spectacular fulminations at the evil minions of government agencies, the incompetent idiots that call themselves university leaders, or the ill-will of society and the universe in general.

Scientists want to get on with their work. They hate the bureaucratic drudgery of writing grant applications and reports. Who can blame them? Every day spent on grant writing is a day less for real work. The worst part is that most grant applications fail. For example, the Swedish governmental science grants agency VR received a total of 222 applications in the field of natural sciences and technology in the round of 2016. The results were announced 23 Feb 2017. A total of 18 (eighteen) applications were successful. That’s 8%. The rest, 92 %, were written in vain. From discussions in the media, it is clear that this is a general phenomenon in most of the world. Who can say that scientists should not hate this system?

Although similar hard numbers are difficult to find for the (non)effects of report writing, most scientists are convinced that an overwhelming part of the effort that goes into compiling reports is wasted. Who actually reads the report? Do the conclusions of the report actually influence any decisions? There has recently been a round of reporting at the outfit where I work (Science for Life Laboratory in Stockholm). The purpose was to look at the scientific impact of the service facilities of SciLifeLab. Anecdotal evidence suggests that the subsequent allocation of money did not bear any comprehensible relation to the data of the report. That is in any case what I hear from several persons involved. If that is what those affected believe, then the reporting exercise seems to have generated one tangible result: demoralization.

There is a problem, clearly. But it is worse than that. There is also a fundamental confusion in the discussion among scientists about this very problem.

Scientists typically argue as if it is the evaluation of grants and scientific work that is the root evil. If we got rid of grant applications and reporting, i.e. stopped doing evaluations, and just gave the scientists the money they need to get on with their work, everything would be fine. An example of this type of reasoning can be found in the academic paper Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition by Marc A. Edwards and Siddhartha Roy (Environmental Engineering Science. January 2017, 34(1): 51-61. doi:10.1089/ees.2016.0223). The paper makes many good points, for instance that any metric for evaluation is bound to become corrupted, reiterating the well-known Goodhart’s Law which states: ”When a measure becomes a target, it ceases to be a good measure.” The paper also makes the valid point that the problem of irreproducible scientific publications may be caused, at least in part, by the incentives created by using bibliometric analysis as selection criteria. The conclusions are mostly about trying to find better incentive schemes, which is pretty lame given the reference to Goodhart’s law. There is one single statement about funding levels being beyond the control of scientists (well, duh!). The paper does not say it explicitly, but as in so many other similar articles, there is a strong undercurrent that if just the politicians understood the problem more resources would be allocated to science.

At best, this is wishful thinking. It is not so easy to understand how intelligent persons, which most scientists are, can truly think that this is a serious suggestion. It is as if scientists are unable to dispassionately analyse their own situation and its inherent constraints in the same detached way as they view the subject of their own research. When studying, let’s say, bacteria, it is a given that bacteria are constrained by the fundamental laws of thermodynamics, conservation of mass, evolution, and so on. So if a bacterium divides and grows, it requires energy, substrates and a certain environment. Anyone putting forth theories about the behavior of bacteria that violate these constraints will be rightfully laughed out of the seminar room.

So let me spell out some very simple, almost trivial, facts about science and resources.

First point: The amount of money allocated to science is finite. At any given moment, or in any budget put before parliament (national, regional, community), there is only so much money to be divided between all sectors, from the military, police, social services, medical care, culture, and so on. Science and higher education is one of these sectors. There will never be enough money to make everyone happy. Ever.

An interesting argument has been advanced that since money allocated to science can be viewed as an investment (assuming that economically beneficial results are obtained eventually), it must be viewed in a special way, such that more money should be given to science. This is rubbish.

At time t0, there is a certain amount of resources R to be allocated. We (the state) give a certain fraction f0 to science. Assume that we can then expect to obtain an amount of resources S at some later time t1, where S is hopefully larger than R. If the same fraction f0 is applied at that time t1, then science will of course obtain more in terms of absolute resources. But, please note, this is at time t1, not t0. I apologize for stating the obvious, but those increased resources S are not available at time t0.

The idea that science funding is an investment is not, in itself, a conclusive argument that more money should be spent on science. There are always other investment alternatives. Maybe building a bridge would be better. Or decreasing the number of pupils per teacher in primary school. Or something else. Science is in no way special in this regard. Uttering the word ”investment” does not produce extra resources magically at time t0.

As an aside: the investment argument is often used in combination with the idea that we (the state) should borrow the money, since the results of an investment will pay the interest for the loan and more. To which one may give a very brief response: Greece.

Second point: The fraction of money allocated to science cannot reasonably be expected to increase significantly. This just will not happen unless scientists can convince others (voters, taxpayers, politicians) that this is a sensible policy. There are no hidden treasures. Without good arguments, no larger fraction will be allocated. Good arguments require some kind of reasoning about the impact of science, either in terms of its general place in our civilisation where truth and knowledge is valued highly, or that science produces more tangible economic, medical or other good consequences. For this, evaluations of some kind of are needed.

In this connection, I would like to point out that scientists need to consider if their arguments are effective. The above mentioned academic paper describes a gloomy state of affairs in science, and blames it on hypercompetition and lack of funding. What the taxpayer hears on being told this is: ”The research system is broken. We produce lots of questionable papers. Give us more money.” Can we expect the taxpayer to say: ”But of course, more money is the obvious solution.” No, I don’t think so.

Third point: More people want to become scientists than it is possible to accommodate. There is competition for the available research positions. This is a good thing! Imagine if being a scientist was not considered attractive. That would really signal a crisis in our type of civilization. So, in order to decide who is going to become a scientist and who is not, some kind of selection is required.

Fourth point: The idea that a scientist should be given ample resources as part of her position is very attractive. In fact, this kind of arrangement does exist. There are research institutes that provide their scientists with fairly substantial amounts of resources. There are also special positions at some universities for young brilliant scientists who get grants as part of the position. These are almost always limited in time, the idea being that the scientist should be able to attract other grant money after this start-up period.

But the essential point with these arrangements is that they are invariably elitist. The selection of which scientists get such positions is very strictly meritocratic, i.e. the result of a very tough evaluation. It is unavoidable. There simply are not enough resources for every aspiring or current scientist to get this kind of deal.

Fifth point: Modifying or totally redesigning the allocation mechanism cannot change the fundamental issue of resource scarcity. If there is not enough money to award in grants, then there is not enough money, full stop. There is no allocation scheme that can change that.

Recently, Shahar Avin has made a very intriguing proposal, namely that we should allocate research funds by lottery. From the set of reasonable research proposals, simply select randomly which to fund. Then the evaluation will just amount to weeding out the worst proposals, keep the rest without bothering to pick the best, which is hopeless anyway, and let Lady Fortuna decide. I think this definitely is worth considering.

But please note that Avin’s idea solves fewer problems than it seems. Avin begins his article by giving an example of a research programme which yielded the drug propranolol in the 1940s, but which would have been rejected using today’s system because it looked far-fetched. Perhaps. But with Avin’s system, it would probably have had the bad luck to lose the lottery, since most proposals would lose. So what is gained from the system perspective? Avin also points out that with the low success rate of grant proposals (8% in my example above), there is a lot of waste in writing grant proposals that eventually fail. Again, this is not changed by moving to a lottery system. The same 8% (or whatever) of grant proposals will succeed, just a different 8%. The waste in grant proposal writing will be exactly the same.

Things are, at the end of the day, very simple: Since resources are restricted, there has to be a selection at some point. Either the number of scientists is kept low, in which case each scientist can get more money, maybe without having to write grant applications. Or more people are allowed to become scientists, in which case they will have to share the same amount of resources by some allocation mechanism. The result is, unavoidably, that the average scientist will get less than in the former scenario, since more scientists compete for the same resources.

We need to discuss the problems of science such as bad incentives, shoddy publications, me-too research, and so on. This is very important. But it is futile to discuss these issues as if the solution was to just give researchers more money. That is not going to happen.

So, when discussing mechanisms for resource allocation, please take into account the fundamental fact that resources are finite, and the political fact that research as a sector in society is very unlikely to get a larger fraction of the available resources than today.

If your idea does not allow for these facts, then ask yourself: What is the point? A proposed solution that tries to avoid reality is not a serious proposal. It is a waste of time.

3 reaktioner på ”Some hard facts about science, money and evaluations

  1. This was a good read, but i think it misses on a more fundamental point, which is also relevant to the random allocation of grants scheme. Specifically, the question of ”are scientist fungible” and if not, is the difference big enough to warrant the matching cost?
    I see this issue being generally ignored and both a yes and no answer given at the same time. On the one hand, we get peer reviewed (implying all scientist are equal) and on the other hand we talk about geniuses and people taking the field forward and all that jazz.
    Ultimately, if the answer is yes and we think that any given qualified scientist would be as good as any other at tackling a question, then random lot sounds to me like a perfect solution. I could go into refuting some of the issues that Per brings up, but i think most scientist would disagree that they’re fungible.
    Thus, let’s go with a no answer: given the same resources one person would be better than others at pursuing a specific research question. This however doesn’t disqualify the distribution by lot option, since it could well be that the difference is minor and the matching cost (that of matching the money with the most capable person) is higher than the benefit. And i think this is exactly the situation we find ourselves in (at leas in biology, i’m ignorant of most other fields). So, instead of trying to fix or come up with a scheme that would identify the ”better scientist”, let’s accept that while these exist, they are only marginally better.

    1. Well, I don’t think scientists are fungible, and I don’t think my arguments above assume that. About peer review: I don’t think that system assumes fungibility (is that a word?), just that a reviewer can pronounce on the brilliance, or not, of another scientist’s idea. That may be possible without all parties being equally good scientists.
      One of my arguments is the sheer difficulty of ranking grant proposals (or for that matter, the brilliance of individual scientists). One reason is that proposals (persons) cannot be naturally represented in a one-dimensional space. Another is that even extremely bright scientists can be totally wrong about the feasibility of some approach (person).
      I do not think lottery should be used for all grant processes. But maybe there is a place for one such grant round among several others?

  2. rolandnwp

    Good read. I think the crux of the matter is that (1) evaluating research properly takes a lot of time, and (2) the number of scientists (and therefore the number of applications) have increased rapidly, so that we simply don’t have that time anymore. The current ”peer review” system doesn’t scale up because of the demographics: the reviewers are generally not peers, but more experienced and/or successful scientists (of which there are few), while the applicants are the newcomers and/or the less-successful (of which there are many). This has led to the bibliometrics system as a ”quick fix” that lets funding agencies weed out applicants using some simple algorithm (say, total citations or h-index), to keep the numbers manageable.

    Of course, I agree that some selection scheme is needed. Random lottery seems like a baseline method, but I think we should be able to do better. I absolutely don’t think that all scientists are equal; like in all pursuits, there is a wide range of abilities, there is genius and mediocrity. I don’t know what the best scoring system is (considering Goodhart’s law), but I think that in any workable approach, we will need to allocate much more time to review. It should be a part of the job description of all scientists to allocate a significant fraction of their time to reviewing other’s work. A complex paper represents many man-years of work, and it should be allocated at least a few weeks of review time — now it’s more like a few hours. Reviewing should be paid time, and it should be merit-generating in itself. With no resources allocated to reviewing (it’s more or less considered extracurricular work), it’s virtually guaranteed that any evaluation system will be superficial and easy to ”game”.

Kommentera

Fyll i dina uppgifter nedan eller klicka på en ikon för att logga in:

WordPress.com Logo

Du kommenterar med ditt WordPress.com-konto. Logga ut / Ändra )

Twitter-bild

Du kommenterar med ditt Twitter-konto. Logga ut / Ändra )

Facebook-foto

Du kommenterar med ditt Facebook-konto. Logga ut / Ändra )

Google+ photo

Du kommenterar med ditt Google+-konto. Logga ut / Ändra )

Ansluter till %s