PrEval-Plattform

The methodological approach below refers to the nationwide survey from 2020/2021. The new approach will be outlined after the new survey phase and the update of the PrEval Platform (fall 2023).

Methodological approach

To survey evaluation capacities in Germany, the Institute for Interdisciplinary Research on Conflict and Violence (IKG) conducted a nationwide telephone survey with evaluation actors focusing on three underlying questions:

Who conducts evaluations (e.g., university or non-university institutions, for-profit companies, etc.)?
How are these evaluations conducted?
Which methods are used, which criteria, approaches and forms of evaluation? But also what resources are available for carrying out evaluations?
What are the objects of evaluations?
Projects and measures, programs, whole organizations or institutions?

The first step was to identify evaluation actors along federal and state programs down to the municipal level. For this purpose, extensive online and offline research was conducted but also important gatekeepers (such as Federal State Democracy Centres (Landes-Demokratiezentren)) and other stakeholders/sources were included in the identification phase. Additional actors could be identified through the input from the PrEval sub-project »Monitoring of Evaluation and Quality Assurance Needs« in cooperation with the »Kompetenznetzwerk Islamistischer Extremismus (KN:IX)« and the »Kompetenznetzwerk Rechtsextremismusprävention (KompRex)«. With the help of the MAPEX survey data, further self-evaluating prevention actors could be identified and included in the PrEval survey.

Selection criteria

External and self-evaluation actors were selected by the following criteria:

External evaluation

The aim was to include an exhaustive dataset of external evaluators in the field of extremism prevention with interfaces to violence prevention and political education in Germany.
The survey did not include self-employed individuals, freelancers, or consulting firms that evaluate business processes in the manner of a management consultancy.

A total of 90 institutions could be identified that met the criteria mentioned above.

Self-evaluation

Selected were:

Carriers based on feedback from the KN:IX and KompRex surveys (see above).
Institutions included in the MAPEX data set,
- which had indicated that they had already conducted self-evaluations and were active in the field of Islamist extremism, right-wing extremism or left-wing extremism.
- In order to achieve a broad (geographical) coverage, the MAPEX data was differentiated on basis of range/regions where projects/carriers are active (this includes online activity, Germany-wide, nationwide, municipal or city/district activity)
Individual projects or measures were not taken into account unless they acted as an competence or advice center .
Institutions that stated in the interviews that they carried out external evaluations internally were included in the self-evaluation dataset. The different operational areas could not be clearly distinguished in these cases (i.e. there was an overlap between evaluation and project work among the involved personnel). So they could be seen more likely as self-evaluation actors.

Due to the short duration of the PrEval project, it was not possible to draw a random sample across the entire carrier landscape in Germany. By using the three sources mentioned above it was already clear that these selected evaluation actors should have already carried out self-evaluations or an internal evaluations.

In total, 183 institutions could be identified in the field of self-evaluation.

Data basis

Of the total of 273 external and self-evaluating institutions surveyed,

104 institutions (38%) could be won for an interview, 37 (36%) of them in the field of external evaluation and 67 (64%) in the field of self-evaluation.
77 institutions (28%) stated that no (self-)evaluations had been carried out in their actual sense, no evaluations had been carried out in the areas relevant for PrEval, or that no evaluations had been carried out at all.
For 48 institutions (18%) no contact could be made, the institution no longer existed, an already scheduled interview could no longer be realized or other reasons prevented an interview from taking place.
For 44 facilities (16%) participation in an interview was not possible due to a lack of time or resources or an interview was explicitly refused.

Survey and field phase

The telephone interviews were conducted from September 2020 to June 2021 with some catch-up interviews if an interview date could not be realized during the main survey period. The survey phase was preceded by a standard pretest to check the questionnaire for comprehensibility and statistical suitability and then to revise it for the main survey phase.

The interviews were conducted by the project staff of the IKG. On average, five contact attempts were necessary before an interview could be conducted. The mean interview duration was 59 minutes, with a maximum of 166 and a minimum of 25 minutes. The long average interview duration also underscores the need for discussion and exchange that emanated from both the executing agencies and the evaluating institutions. The focus of the survey was on

evaluation actors,
their clients,
applied evaluation methods, approaches and forms,
qualifications of the evaluators
success criteria and goals of evaluations,
evaluated work areas and prevention levels,
challenges, evaluation and research needs from the evaluators’ perspective,
potentials for improvement in the cooperation with funding agencies and evaluated institutions and projects,
but also the positive developments over the last years.

A total of 30 different categories were queried.