Good Marks for FWF’s Decision-Making Procedure in Long-Term Study by University of Zurich

In a long-term study, independent experts at the University of Zurich scrutinised more than 10,000 funding decisions made by the Austrian Science Fund (FWF). They found that the quality of proposals has continuously increased, that there is a fair distribution of funding odds across all disciplines, and that age and gender have no effect on the probability of receiving funding. Minimal distorting effects were found only for interdisciplinary research projects.
June 9, 2022

The FWF Board makes more than 3,300 funding decisions every year. From submission to review of proposals, the FWF seeks to ensure fair and independent decision-making procedures for all researchers, based on the principle of international peer review. In order to measure the quality of its decision-making procedures, the FWF began systematically assessing them for bias in favour of or against certain categories, such as applicants’ gender, age or original discipline, more than12 years ago. Based on the data obtained, a number of studies were subsequently carried out by international experts. 

This process of analysis was recently resumed for the years 2010-2019 by Rüdiger Mutz and Hans-Dieter Daniel (University of Zurich) in cooperation with the FWF. The initial findings have now been published in the study Scientific analysis of data on proposals and the decision-making procedure of the FWF with particular focus on the programme “Stand-Alone Projects” in the years 2010-2019. 

The new study is based on data for 50 sessions of the FWF Board, 10,871 grant applications, 23,646 international reviews, 1,582 final reports and 1,317 final report reviews. The study assessed only proposals for the programme “Stand-Alone Projects”.

The main findings of the study were the following:

  • The numerical ratings given by international reviewers have risen. This indicates that the scientific quality of the proposals has improved accordingly.
  • The homogeneity and/or heterogeneity of reviews in terms of ratings of proposals falls in a range of the sort that can also be observed in reviews for other funding organisations or scientific journals. Moreover, no significant differences were found between the main disciplines of the FWF.
  • The odds of approval did not differ significantly by applicants’ gender, age or discipline, nor by the sessions of the FWF Board in which the proposals were discussed. A preference for male and older applicants (> 41 years) was found, though, in the external international peer review. However, the FWF’s decision-making procedures compensate for this, e.g. by awarding bonuses to certain groups of applicants (including junior researchers and applications as grant-salaried principal investigators).
  • Interdisciplinary proposals were at a disadvantage – although a very slight one – in terms of odds of approval, as compared to non-interdisciplinary proposals.

Evaluating grant applications without regard to the applicant’s position or academic degree is one of the values espoused by the FWF. The FWF is therefore gratified to note that this standard holds up under empirical scrutiny.

Further detailed analyses based on the data underlying the study will be published in relevant scientific journals in the coming months and years.