Evaluations and Quality Assurance

Evaluation is a key quality assurance tool for institutions like the FWF, which have to be accountable to various stakeholders on a number of different levels. These stakeholders include the scientific community as the recipients of FWF funding, the ministries and the government who provide the FWF with the funds it needs for its work, and finally the general public.

The FWF has developed a set of the highest evaluation and decision-making standards and reviews them regularly. In cooperation with ministries, funding institutions, and as a member of the Austrian Platform for Research and Technology Policy Evaluation (fteval), the FWF is helping further establish these standards in Austria's research funding culture. A detailed description of the underlying principles can be found online. The FWF both uses and commissions evaluations, as well as being subject to them itself.

Peer review of research proposals

The custom-tailored review of research proposals is key to the FWF’s work. It represents the basis for quality assurance in all the FWF’s funding programs. International peer review is used for this purpose: Experts in the relevant field (researchers, i.e. “peers” working in fields related to the respective proposal or applicant) write reviews that form the basis for the decision-making process (more information on the decision-making process). The FWF does not have a fixed pool of reviewers; they are selected on a subject-specific basis for each proposal. Internationality is a core element of the FWF’s peer review process: Only reviewers from abroad are contacted. One third of them are from non-German-speaking countries and another third from overseas; in addition, the FWF strives to obtain more than 30% of its reviews from women researchers. The majority of reviews are obtained in writing, but for some complex cooperative programs, awards, or the excellence initiative excellent=austria, review panels or juries are also used, where experts interact with the applicants in hearings and/or use the reviews as a basis for decision-making.

In all stages of the procedure the FWF relies on comprehensive measures to avoid conflicts of interest and to ensure that all operations are checked (using the “many eyes” principle). This applies to all stakeholders: Reviewers, jury members, Scientific Consultants, and FWF staff members.

Review of completed projects

Completed projects were subject to peer review from 2003 to 2023; starting in 2024, this will be replaced with a monitoring system that will be used for internal and external evaluations and provide the public with convenient access to project findings.

This final review gives insights on how the project progressed and its success, and provides materials for further science communication. Key data on project results are published on the FWF Research Radar page. The FWF Research Radar documents scientific publications and conference proceedings, career trajectories of project participants (especially junior researchers), special awards and prizes in connection with the project, application-oriented results such as patent applications and licenses, and science communication measures. The FWF is placing increasing emphasis on the use of open access platforms when publishing research results.

All of these data are essential for internal and external evaluations and for illustrating the impact of the FWF’s research funding, both nationally and internationally.

Program evaluations commissioned by the FWF

The FWF's funding programs are subjected to review on a regular basis; reviews look at the programs’ design, processes, and impact. This creates a solid foundation for testing the efficiency and impact of each funding activity, and allows the FWF to analyze them in comparison with other national and international programs. The Standards of the Austrian Platform for Research and Technology Policy Evaluation apply.

Evaluations are generally only assigned to independent, professional evaluators (or institutions) with relevant experience and expertise.

Depending on the size and duration of the program, one of four evaluation methods is usually applied:

  • Program evaluation: Evaluation of a single program after a certain period of time (by external experts)
  • Portfolio evaluation: Evaluation of similar programs or the entire program portfolio (by external experts)
  • On-going evaluation: Evaluations of application and decision-making procedures for new programs (by external experts)
  • Key performance indicator monitoring: Collection and analysis of key input and output indicators for smaller FWF programs (usually by the FWF itself)

All evaluations and the underlying materials/data are published (usually together with a statement by the FWF) on the repository Zenodo, the fteval repository, and the FWF website under a CC BY license and are freely available.

Evaluating the FWF – the FWF as the subject of evaluations

Research policy dictates that funding institutions like the FWF must also be subject to regular evaluations, which review their operation, program portfolios, and their position in the context of national and international research innovation. Evaluations and analyses such as these depend on project monitoring and program evaluations.

In addition, the FWF has initiated a series of (self-)evaluation studies that analyze decision-making procedures, research output, and other topics at regular intervals. You can find the results online here: Zenodo

According to the Austrian Research and Technology Funding Act (FTFG), the FWF’s long-term planning and its funding agreement with the federal ministry represent central pillars of the FWF’s performance evaluation, making it accountable to the public.