SPOCs & surveys: A novel way of conducting comparative research into criminal justice systems

Abstract

Background

Criminal justice researchers are increasingly confronted with research questions that require them to use ‘foreign information’ in their analyses. Though it is generally accepted that working with foreign information is challenging, several authors pointed to the lack of serious recent work on comparative methodology. Many researchers therefore avoid comparative research.

Method

Based on a literature review and practical experience, the challenges of comparative criminal justice research are uncovered, the strenghts and weaknesses of existing methods identified and a novel method developped.

Result

The so-called ‘SPOCs & survey’-method consists of nominating a SPOC (short for a single point of contact) for the countries to be included in the comparative analysis and asking them to fill out a multiple choice based survey, complemented with free text fields and the option to request documents to be attached. To assess the merit of the method, the paper elaborates (1) on the different degrees to which the method can be integrated in various types of research and (2) the extent to which the method is better tailored than existing methods to successfully overcome the challenges of comparative research referred to in literature.

Conclusion

Though the method depends on the access to knowledgeable SPOCs, the weakness is counterbalanced by the strengths of its swiftness, reliability and validity.

Introduction: evidence based policy making

Evidence based policy making has been a buzz word for quite some time now and it has been argued elsewhere that the increased importance of evidence based policy making should be applauded (De Bondt, 2014). It is generally accepted that policy decisions based on systematically gathered and analysed evidence are more likely to produce better outcomes when compared to policy decisions based on the heat of the moment (Sutcliffe & Court, 2005). Putting the best available evidence at the very heart of policy development helps policy makers in making well informed decisions (Davies, Nutley, & Smith, 2000; Lee, 2004; Welsh & Farrington, 2007).

In light thereof, the question arises which evidence should be used to support policy decisions. In today’s reality, it makes sense that criminal policy making is not solely based on evidence related to national experiences with preventing and fighting crime. Rather, criminal policy making should also take account of evidence related to foreign experiences with preventing and fighting crime. Fortunately, policy makers are showing an increased interest in foreign experiences (Örücü, 2007, p. 45) both at national and international level^a. This interest is mirrored in criminal justice research. Hence, criminal justice researchers are increasingly confronted with research questions requiring the use of ‘foreign information’ in their analyses. That foreign information can relate to technical legal provisions, to case law or to empirical data gathered in the context of an analysis conducted from the perspective of another country. A first possible situation where foreign information will need to be used, is research looking into cross-border crime phenomena. In those kinds of research it is almost self-evident that a criminal justice researcher has to take account of legislation, case law, information on the allocation of resources and/or crime related statistical information. However, the need to look into foreign information can also arise in research looking into mere national crime phenomena. Even in those kinds of research, knowledge of foreign systems has a merit in provoking and supporting criticism and from that improvement of the own system (Örücü, 2007, p. 46). In addition to this, in recent years, criminal justice researchers are increasingly solicited to conduct research called for by international cooperation structures and organizations such as the Benelux, the European Union, the Council of Europe and the United Nations. Those international criminal policy makers too recognise the need for evidence based policy making. Their underlying goals can vary from striving to unify certain parts of the national legislation and practices (Delmas-Marty, 2004, p. 253) to facilitating and supporting cooperation. Reference can be made to calls for research to evaluate the extent to which they have been successful in their attempt to harmonise national legislation, to align practices or to support and facilitate the fight against crime.

It is generally accepted that working with foreign information is challenging for most researchers. The top-three challenges referred to in literature are language barriers, limited accessibility of up to date information and the risk of misinterpreting information due to an inadequate understanding of the specificities of the foreign criminal justice system (Lomio et al., 2011, p. 58; Monateri, 2012, p. 9; Nelken, 2007, p. 139; Örücü, 2007, p. 49; Rabel, 1947; von Bar, 2004, p. 132). Before including foreign information into their analyses, researchers need to have a comprehensive and in depth understanding of the criminal justice system the information relates to (Cotterrell, 2007). Only in doing so, they will be able to correctly interpret the available criminal justice data.

The growing academic interest in comparative research is said to be apparent from the rise of new journals on comparative research (Örücü, 2007, p. 44) and the seemingly renewed debate on methodology in comparative research more generally (Neergaard, Nielsen, & Roseberry, 2011, p. 7). However, in spite of that growing academic interest, the critiques formulated regarding the insufficient attention for the methodology of doing comparative research (Howard et al., 2000, p. 142) are still valid. It was and can still be argued that though much has been written on the outcome of comparative research, only little has been written on the way of doing it. The founders of comparative research were more pragmatically than methodologically interested (Gordley 2012, p. 107). Most substantive comparative works do not make their methodological choices sufficiently clear (Adams & Bomhoff, 2012, p. 1). In light thereof, several authors pointed to the lack of serious recent work on comparative methodology (Samuel, 2004, p. 35), regretting that true questions of methodology hardly ever rise (von Bar, 2004, p. 130) and complaining that attempts to develop even a moderately sophisticated method of comparison is exceedingly rare in comparative studies. Many projects are said to adopt an anything goes attitude of methodological questions (Adams & Bomhoff, 2012, p. 1). Therefore, the role and method of comparative research remain controversial (Andenas & Fairgrieve, 2012, p. 28).

Because there is still significant uncertainty about the best way to conduct comparative research, this paper will detail a novel way to conduct comparative research into criminal justice systems. Based on a literature review and experience with the method in the past, the strengths and weaknesses will be elaborated on. First, the main characteristics of the method will be briefly introduced and compared to existing methods. Second, the merit of the method will be elaborated on in light of the degree to which it can be integrated into research and in light of the extent to which it is tailored to overcome the main difficulties of doing comparative research identified in literature.

The method in brief: SPOCs & MC-Surveys

SPOCs is short for Single Points of Contact and MC-Surveys is short for Multiple Choice based surveys. This section will briefly elaborate on the main characteristics of the method. In sum, the method consists of working with a single point of contact in each of the countries included into the comparative analysis. The SPOC is asked to ensure that the multiple choice based survey is duly filled out, either by him or herself or with the team he/she has decided to compose. It can already be added here that the surveys are not exclusively composed of multiple choice questions. It will be elaborated on below that the surveys are only predominantly multiple choice based and therefore the surveys also contain a number of free text fields and can even include the requirement to upload or attach documents (e.g. original or translated versions of legislation).

The method of working with SPOCs and MC-Surveys is designed to be a simple hands-on method to get what is called ‘a targeted descriptive and informative picture of the current situation in another state’ (Westmarland, 2011, p. 51). That factual picture will allow the researcher to collect information and/or to assess which information is available, where to find it and the extent to which it can be used to create an evidence base in relation to a specific research question.

SPOCs: single points of contact – the next step in the evolution

The first element to the method is the use of Single Points of Contact in the countries included into the comparative analysis. Before going into the criteria and best practices for the selection of the Single Points of Contact, the novelty of working with Single Points of Contact will be contextualised in light of the currently most common ways of conducting comparative analyses, being the ‘individual & desk top’-method and the ‘team & report’-method.

The ‘individual & desk top’-method is the method most commonly used for comparative research. Traditionally (mirroring any other type of legal research) comparative research into criminal justice systems is done individually. An individual researcher looks into foreign criminal justice systems and foreign information sources attempting to gain a sufficiently in depth understanding of the foreign legal system to the extent needed for the analysis. Based on its main characteristics, this method will be referred to as the ‘individual & desk top’-method.

The emergence of the ‘team & report’-method symbolises a first step in the evolution from individual research to team efforts. Others too have argued that, despite the fact that it may be contrary to traditional individual work, team work is highly recommended if not necessary. In light thereof von Bar has argued that we are living in a transitional period and the attitude as if research needs to be individual work, needs to be changed (2004, p. 132). As Watson puts it “an understanding of a foreign legal system cannot be obtained simply by heaping up nuggets of information. […] Even an exceptionally skilled comparatist can make mistakes” (2010, p. 209). The by far most reliable way to gain a comprehensive and in depth understanding of a foreign criminal justice system is by calling upon a foreign colleague. The shift from individual towards team work has resulted in a more modern way of conducting comparative research. Today, much of the academic comparative work is based on country reports drawn up by the members of a comparative research group. Having agreed on the chapter build-up, a group of academics draft individual country reports, summarising the main lines of their criminal justice system in relation to a specific topic. Those country reports form the basis for the comparative work. Based on its main characteristics, this method will be referred to as the ‘team & report’-method.

The innovative ‘SPOCs & survey’-method elaborated on in this paper aims to take a next step in this evolution. It is a plea to use team efforts more often. Though it is used in general legal comparative research, it is not often used in policy driven criminal justice research. Additionally, it is a plea to be more flexible and open minded when approaching and selecting foreign colleagues. That flexibility and open mindedness is reflected in two novel criteria to select SPOCs: (1) consider academics, practitioners and policy makers and (2) compose a multi-disciplinary team.

First, whereas comparative research teams are usually composed of academics only, there is no need to require the foreign colleague to be an academic. Depending on the topic of the research and the particularities of the questions asked, a practitioner may be better placed to ensure that the difference between the law in the books and the law in practice is duly taken into account^b. Along the same line of argumentation, it may very well be that a policy maker is best placed to answer the questions. Therefore, a first novelty of the method relates to the use of academics, practitioners and policy makers.

Second, it does not need to be a problem that the profiles of the SPOCs contributing to a specific comparative research study vary. Given that the method is used to obtain ‘a targeted descriptive and informative picture of the current situation in another state’ (supra), a SPOC will inform the researcher of the facts of its criminal justice system^c. That information will be exactly the same, regardless of who provides the information. A SPOC is not so much a ‘respondent’ sharing his/her opinion with the researcher, but more an ‘informant’ (Brinkman, 2000, p. 28).Therefore, there is nothing to argue against the profile of the SPOC varying between countries. Not the profile but the knowledgeability of the SPOC ought to be decisive^d. Therefore, a second novelty of the method relates to the use of a multi-disciplinary SPOC-network.

Furthermore, two best practices can be deduced from past experience with the method: (1) nominate a SPOCs per subsystem and (2) use existing networks to identify possible SPOCs.

First, depending on the topic of the research, it may be needed to have multiple SPOCs per country included into the research. Especially for criminal justice research, it is not uncommon for information to differ according to regions within one country. Reference can be made to e.g. the differences between England and Wales on the one hand and Scotland on the other hand (Westmarland 2011, p. 59). As a result, it will often not be possible to provide one single answer for the entirety of the United Kingdom. To avoid complexities both for filling out as well as for analysing the survey, it is best to anticipate to the differences and nominate one SPOC for England and Wales and one SPOC for Scotland. Not all researchers will have sufficient prior knowledge of the foreign criminal justice systems to anticipate to this. It is therefore advised to explicitly ask each of the SPOCs approached to assess the need to differentiate between subsystems in light of the specificities of their criminal justice system.

Second, the possibility to work with this method is dependent on access to a suitable SPOC. Most research institutes have their own networks. In addition thereto, various official networks exist. Depending on the research topic, networks such as (1) the European Criminal Law Academic Network (ECLAN)^e, (2) the Eurojust national members^f, (3) the Europol Liaison Officers (ELOs)^g, (4) the Genocide Network^h, (5) the European Judges and Prosecutors Association (EJPA)ⁱ, (6) the European Criminal Bar Association^j, (7) the European Lawyer Database^k or (8) the European Organisation of Prison and Correctional Services (EuroPris)^l could be used as a starting point. To ensure the participation of sufficiently knowledgeable SPOCs, it is advised to remunerate the work of the SPOC via an expert fee.

Surveys: predominantly multiple choice based

The second element to the method is the use of surveys. In general, surveys are said to become increasingly popular, because of their efficiency (Bachman & Schutt, 2012, p. 164) both in filling out and in processing and generating results. In spite of their popularity, surveys are rarely used in comparative research. Comparative research is predominantly conducted using either the ‘individual & desk top’-method or the ‘team & report’-method mentioned above. When questionnaires are used, they are mainly based on open questions requiring to provide a summary overview of their criminal justice system in the form of an essay type country report.

The innovative ‘SPOCs & survey’-method works with (1) multiple choice based surveys, (2) validated by a pilot group and (3) complemented with a free text fields (and possibly upload modules) to acquire targeted country specific input.

Type of questions and answering categories

The ‘SPOCs & survey’-method is designed to be a way to swiftly obtain ‘a targeted descriptive and informative picture of the current situation in another state’. That objective is reflected in the type of questions and answering categories included in the survey.

Questions are either formulated to be yes/no questions or a question is followed by a list of predefined answering categories for the SPOC to choose from. An example of a simple yes/no question could be:

Was a change in legislation required to implement Art. X from EU Directive Y?

• ○ Yes

• ○ No

An example of a question with a limited number of answering categories could be:

How long can a person be arrested in your country?

(i.e. without being apprehended by a judge)

• ○ A: 12 h

• ○ B: 24 h

• ○ C: 36 h

• ○ D: 48 h

• ○ E: 60 h

• ○ F: 72 h

• ○ G: other [please insert duration]

This last type of questions illustrates that a large number of answering categories can be provided. Including an ‘other’-category anticipates to the possibility that the legislation of a country provides for yet another duration. Even though an ‘other’-category is an effective fall back option if the correct situation is not reflected in any of the answering categories, it is still advised to provide all plausible options drawn from prior knowledge. For quick processing of the answers it is best to try and avoid that SPOCs need to use the ‘other’-category. Moreover, it could also be a valuable finding that one or more answering categories do not exist in any of the countries included in the comparative analysis.

These kinds of simple and targeted multiple choice questions are preferred over open questions. Answering open questions is demanding for the SPOC, who needs to consider which aspects of his/her criminal justice system would be appropriate to elaborate on and draft his/her answer in a comprehensible way. Additionally, from a mere academic perspective, there is little intellectually challenging about giving a summary account of the specificity of one’s own criminal justice system. Ticking a number of boxes in a survey comprised of closed questions is less time-consuming. Especially given the so-called questionnaire fatigue (Semmens, 2011, p. 63) that can be noticed amongst a lot of potential SPOCs, being able to state that the survey will have an easy multiple choice based character, may increase the willingness to participate.

Even though the method is essentially designed to swiftly obtain factual information about other criminal justice systems, questions aimed to obtain the personal opinion of the person filling out the survey could also be included. In that case, the SPOC-informant becomes a SPOC-respondent. When including questions aimed to obtain a personal opinion, it should be kept in mind, that the individual opinion of the single respondent is not representative for the opinion of academics/practitioners/… of that country, unless of course the SPOC-respondent is an official representative of a governmental actor or professional organisation requested to indicate what the official position would be. From that perspective questions looking into e.g. the necessity of taking legislative action in relation to a specific problem, could also be included in the survey.

Validate questions in a pilot group

Depending on the nature and complexity of the questions and answering categories, it may be advised to test and where necessary revise the questions (Bachman & Schutt, 2012, p. 167; Wilson & Sapsford, 2006, p. 103) before sending out the surveys to all SPOCs. When seeking to validate the questions, it is advised to set up a small pilot group amongst the SPOCs. The question arises how to compose the pilot group. Based on a literature review and practical experience, it is advised to aim for a multi-family and where possible a multi-disciplinary group.

First, multi-family refers to the legal families in which the criminal justice systems are divided. Traditionally, criminal justice systems are divided into families based on their common roots and (presumed) similarity (Monateri, 2012, p. 9). Distinction is made between the Romanistic legal family (France, Benelux Italy, Spain and Portugal), the German legal family (Germany, Austria, Croatia, Switzerland, Greece), the Anglo American legal family (England & Whales, Northern Ireland, Scotland) and the Nordic legal family (Denmark, Finland, Iceland, Norway, Sweden) (Lomio et al., 2011, p. 32). The division of the criminal justice systems into legal families is not free from critique. It is argued in literature that because of the legal transplants and the softening of the once sharp differences between common law and civil law (Monateri, 2012, p. 9), the division in legal families has somewhat lost its merit (Devroe, 2010, p. 46). That is why the division into legal families is sometimes even referred to as the legal family ‘trap’ (Glenn, 2006, p. 437) rather than a helpful legal family ‘tool’. Though concurring with the argument that it is becoming increasingly difficult to rely on the division into legal families, practical experience comes to testify that it can still be beneficial to include SPOCs of countries (that were once) a member of different families into the pilot group, even if it is only for political/policy reasons.

Second, multi-disciplinary refers to the possible diversity in the background of the SPOCs included in the study. SPOCs with an academic, practical or policy background review the draft survey from a different perspective. Practical experience comes to testify that this diversity can be enriching when reviewing the formulation of either the questions or the answer categories and can even lead to the inclusion of additional questions and/or answering categories.

Though validating the questions is an important step in the preparation of the survey, it should be kept in mind that the formulation of questions and/or answering categories can never fully grasp all specificities of the different legal systems. If the pilot phase would be able to have that result, sending out the survey to be completed by all SPOCs would not be necessary anymore. Additionally, the account of the diversity in the criminal justice systems of countries included in the comparative analysis should only be as detailed as needed in light of the underlying research question(s). Some of the obscure differences or specificities are not relevant for the research question. Furthermore, the remaining relevant specificities of the criminal justice systems included in the analysis, will surface via the free text fields included (infra).

Free text fields and attached documents

Working with a multiple choice based questionnaire is intended to speed up the filling out of the questionnaire, improve the cross-country comparability of the results and facilitate the processing of the input. However, for some questions, it will not be possible to fully grasp all the specificities of the different criminal justice systems into the answering categories. In literature this is described as the most important risk of multiple choice based surveys (Bachman & Schutt, 2012, p. 171; Brinkman, 2000, p. 68). Indisputably, the main and important down side of working with multiple choice based surveys is the loss of the fundamental benefit of an open question that allows the respondent to freely and unlimitedly elaborate on each specificity of its criminal justice system and go into every exception.

To accommodate this limitation inherent to multiple choice-based surveys, free text fields are added to every answering category, allowing the SPOC, where necessary, to further elaborate on its specific national situation. In doing so, the open part of the question is kept to a minimum and the input from the SPOCs will be very targeted and contain exactly what is needed to understand the specificity of their system. The combination of response categories and free text fields has proven to be an incentive for SPOCs to either elaborate on a specificity of a national criminal justice system or refine the response category to fit in perfectly with the national criminal justice system.

Depending on the specific needs of the research, the surveys can also be complemented with a requirement to where relevant or necessary attach full text documents containing e.g. legislation, case law, statistical data or policy documents. In doing so, the researcher has easy access to up-to-date original source material, a selection of which can be identified as requiring translation. This approach will avoid unnecessary translation costs of outdated or irrelevant documents.

The merit of the method

To assess the merit of the ‘SPOCs & survey’-method (in light of the ‘individual & desk top’- and ‘team & report’-methods), the following paragraphs will elaborate (1) on the different degrees to which the method can be implemented into research and (2) the extent to which the method is better tailored than existing methods to successfully overcome the difficulties of comparative research referred to in literature.

Degree of implementation into research

The method can be implemented into research in different ways. It can be used (1) somewhat reservedly solely to support methodological choices, (2) to obtain detailed and up-to-date factual information or (3) more extensively to collect not only factual information but also personal opinions from the respondents. The examples elaborated on below are merely indicative and are not intended to be exhaustive. Because experience with the method is predominantly linked to EU studies, the examples relate to comparative analyses covering the 28 EU Member States. However, the method may just as well be used to include other countries into the comparison.

Supporting methodological choices: identifying relevant countries

A study aimed at preparing the revision of the position of the victim in the national criminal procedure would benefit from a comparative legal analysis of the position of victims in other criminal justice systems. A such analysis would reveal differences, foreign experiences and best practices. The selection of countries that are relevant and interesting to include in the comparative analysis is challenging. It requires basic prior knowledge of other criminal justice systems which the researcher may not have. In this situation, the ‘SPOCs & survey’-method can be used solely to identify the relevant countries to be included in a comparative analysis that will be conducted in a later stage using the researcher’s method of choice.

To be able to select the relevant countries, knowledge is required about the current position of the victim in other criminal justice systems. Based on a literature review the researcher should be able to draw up a list of options regardless of which options feature in which criminal justice system. Whether the options (still) exist in any country and to what extent the options appear in a combined form, is not relevant when building a first draft questionnaire. The review conducted by the pilot group elaborated on above will help identify whether the researcher has missed possible options. The result of that combined effort will lead to a survey that will look similar to the following example:

To what extent can a victim start criminal proceedings? (more options are possible)

• □ A victim can bring a case immediately to court to be tried by a court judge

• □ A victim can start a compulsory investigation which will lead to the case being tried by a court judge if the national criteria are met (e.g. sufficient indications of guild) and which excludes the possibility to drop the case, e.g. for policy reasons

• □ A victim can file a complaint and appeal if no investigation is started

• □ A victim can file a complaint and appeal if after an investigation the case is dropped, e.g. for policy reasons

• □ A victim can only file a complaint and has no other role to play in the criminal procedure

• □ A victim has another way to (influence the) start of criminal proceedings

A such survey is an effective and efficient way of mapping what the position of a victim looks like in other countries. Further details e.g. on who qualifies as a victim, how options are combined, whether all options are available to all victims of all crimes, what the rationale behind the choice is can either be subject to similar multiple choice questions, to open questions with free text fields or to follow-up comparative ‘individual & desk top’-research.

When specifically interested in changes in legislation in the countries included in the comparative analysis, the survey can be made more informative through a simple change in the way the question and answer categories are presented. Instead of using single tick boxes linked to each of the answering categories, a distinction can be made between:

• - never had this option and it has never been considered

• - this option has been considered but was never adopted

• - this option has been adopted after in depth consideration

• - this option existed in the past, but was abolished

• - this option exists and has never been subject to thorough debate

In doing so, this simple add-on will result in more detail in the results and can be of importance to select which countries should be included in a more in depth comparative analysis.

This example is based on a recent study into the hurdles in the Belgian criminal procedure with a view to drafting a new criminal procedural code^m. As part of the study, it was agreed with the contracting authority to include a limited comparative analysis. The selection of the countries to be included in that comparative analysis was based on a multiple choice based survey sent to academics and legal practitioners in all other 27 EU Member States. The design of the survey was not identical but similar to what was described above. Used in this way, the method is a tool to support the methodological choice of the countries to be included in a comparative legal analysis.

Obtaining factual information: gap analysis and policy evaluation

Building on the evidence based policy idea, the European Parliament and the Council of the European Union need comparable information from all 28 EU Member States to support their legal and policy decisions and to conduct a posteriori policy evaluations. In light thereof it is valid to look into the extent to which comparable information is available. The lack thereof would point to a gap in the policy line of the European institutions.

Reference can be made to a study aimed at assessing the extent to which statistical information on crime trends of different EU countries can be compared. Possible problems with the cross-country comparability of crime statistical information are linked to differences in the national criminal justice systems. Reference is made to differences in the recording practices (Carr-Hill, 1979), differences in the counting units (Aebi, 2008) and not in the least differences in the scope of the underlying offences (Harrendorf, 2012; Jehle, 2012). As a result, collecting comparable data is continuously considered to be an almost impossible task (Collmann, 1973; Kommer 1995; Gratia, 1995; Farrington et al.,2004; Vettori, 2006; Robert, 2009; Savona et al., 2005). Aimed at demonstrating the short coming of the EU institutions as legislators and criminal policy makers for not ensuring the availability of the statistical information needed to provide the evidence base for (the evaluation of) their own legislating and criminal policy making, problems related to any of the listed differences would suffice.

The ‘SPOCs & survey’-method is perfect to conduct the necessary comparative analysis. The national statistical authorities can act as SPOCs and the questions can be presented in a simple multiple choice based survey. With a view to demonstrating problems relating to the scope of the underlying offences, the SPOCs can be asked to indicate whether or not their national information architecture would allow the production of the required statistical information. A distinction can even be made between:

• - A: are producing statistics related to this part of the offence;

• - B: are currently not producing statistics related to this part of the offence, but could produce statistics in the future, without having to change the national information architecture; and

• - C: are currently not producing statistics related to this part of the offence, and could not produce statistics in the future without having to change the national information architecture.

This example is based on a study into the availability of comparable crime statistical information for the 10 EU priority offences. It was argued that the EU as a legislator and policy maker is responsible to ensure that statistical information is available to be used as an evidence base for future legal and policy initiatives. The inability to provide comparable statistical information that was apparent from the study, points to an important gap in the EU criminal policy line and thus a short coming of the EU legislator and criminal policy maker (De Bondt, 2014).

Obtaining factual information and opinions: needs and feasibility studies

The ‘SPOCs & survey’-method can be used in a more extensive way. As described above, mere informative questions aimed at attaining ‘a targeted descriptive and informative picture of the current situation in another state’, can be combined with questions seeking the opinion of the individual filling out the survey. That opinion can be formulated either in his/her personal capacity or as an official representative of a governmental actor or professional organisation. In doing so, the method can be used to conduct needs and feasibility studies.

Both at the level of the Council of Europe as well as at the level of the European Union, legal instruments exist governing the transfer of prisoners from one country to another. For long, the need is felt to complement the physical transfer of prisoners with the exchange of more detailed information about that prisoner. It is unclear however what information is exactly needed and whether exchange of information is feasible in light of practical and legal obstacles. Recently, the discussion gained momentum in light of the implementation of a new EU instrument. Because the new EU instrument links transfer to ‘furthering social rehabilitation of prisoners’, the question arises whether this requirement has an impact on the prisons’ information need. In essence, the research combines two questions: (1) what information is available for exchange with other prisons (factual information) and (2) what information needs to be exchanged with other prisons to be able to comply with the requirement to further the social rehabilitation of the transferred person (opinion). The second set of information types does not (necessarily) match the first set of information types because available information may not be needed or vice-versa needed information may not be available.

A basic use of the method to obtain only factual information would entail the compilation of a list of information categories for which the SPOC is to indicate whether or not the information is available to be exchanged with other prisons. Based on that selection, the researcher can make his/her own assessment – based on his/her own conceptualisation of what social rehabilitation should entail – of whether the information needed to comply with the requirement to ‘further the social rehabilitation of the transferred person’ is available for transfer.

When using the method in a more extensive way, the survey could also be used to validate the researchers own conceptualisation of what social rehabilitation should entail and test to what extent the researchers views match the views of the respondents. To that end the SPOCs would also be asked to indicate which information categories they consider ‘needed in the context of social rehabilitation’ based on the practice in their own prison environment. The outcome of the analysis of that question is not representative of the opinions in each of the 28 Member States, but has the value of 28 individual practitioners views.

This example is based on an ongoing study into the need to complement the 2008 EU Framework Decision on the mutual recognition of custodial sentences with a mechanism to exchange information on the prisoners involved. The method was used to collect information and opinions from prison directors within the 28 EU Member States. The national central prison authorities were used as SPOCs acting as a gateway between the research team and different types of prisons in the Member Statesⁿ. Results of the study will be published by the end of 2015.

Overcoming difficulties

A second way to evaluate the merit of the ‘SPOCs & survey’-method, is by assessing the extent to which it is able to overcome the hurdles of comparative research identified and elaborated on in literature. Comparative methodology has developed mainly via trial and error (Devroe, 2010, p. 38) and is anything but inviting or encouraging. Comparative methodology is presented as an accumulation of almost undoable tasks (Lomio, Spang-Hanssen, & Wilson, 2011, p. 36). The main difficulties identified relate to (1) language, (2) access to up to date information, (3) correct interpretation and contextualisation of foreign information, (4) comparability of information on foreign systems and (5) the time consuming nature of collecting and processing information. To assess the merit of the method, it will be compared to traditional ‘individual & desk top’-method and the more modern ‘team & report’-method.

Language

The language hurdle is generally considered to be the biggest hurdle to take in comparative research (Nelken, 2007, p. 139; von Bar, 2004, p. 132). The evolution from traditional individual research to team work is a significant step to overcome this hurdle. Even though it could be possible to have the necessary information on a foreign criminal justice system translated into a language known to the researcher, it is often argued that comparative research should as much as possible rely on original primary source information (Devroe, 2010, p. 48). The members of a team of comparative researchers producing country reports or the members of the SPOC network in the method evaluated would not have a problem with the language of the source material. From that perspective, there is no difference between the more modern ‘team & report’-method and the innovative ‘SPOCs & survey’-method.

However, it should be underlined, that language should continue to receive special attention. Relying on colleagues mastering the language of the source material, does not do away with the fact that there will always be conceptual differences, i.e. differences in meaning even if similar labels are used. In light thereof Howard et al. have referred to the “endemic problem confronting comparative criminology”, being the enormous diversity in the way different cultures and nation-states define crime, justice and other relevant concepts (2000, p. 166). Comparatists repeatedly put out warning signs about homonyms and faux amis (Devroe, 2010, p. 51; Van Hoecke, 2004, p. 174). Examples are legio and reference can for example be made to the ongoing discussion on the conceptualisation of organised crime^o and the resulting difficulties in cross-nationally comparing data about that phenomenon (Calderoni, 2008; Symeanidou-Kastanidou, 2008; van Dijck, 2007).

The innovative ‘SPOCs & survey’-method has two built-in ways to further accommodate possible language problems. First, inherent to the multiple choice based character of the survey, is the detailed formulation of questions and answering categories. The review of the draft survey by the members of the pilot group, is a first way of picking up possible language problems. Consulting SPOCs active in criminal justice systems from different ‘legal families’ is a way to detect problems in an early stage. Second, respondents can be encouraged or even required to use the free text fields complementing the answering categories to provide a definition of a concept used in either the question or the answer. In doing so, possible differences will become clear.

In doing so, researchers and respondents are constantly triggered to be very attentive to possible language issues. Language issues are likely to be dealt with in a more consciously fashion using the ‘SPOCs & survey’-method compared to the ‘team & report’-method even though there too, language issues may surface during the preparatory discussions.

Access to up to date information

The more traditional technique of comparative legal methodology consisting of individual comparative desk top research requires a researcher to find applicable and up to date source material. The accessibility of up to date information is listed as a second major hurdle to comparative research (Örücü, 2007, p. 49). Recognising the need to support researchers conducting comparative legal analyses, a large number of handbooks have been written with overviews of how to find up to date foreign sources of law (Lomio et al., 2011, p. 58). Though recognising the merit of those overviews, the ease of having a foreign colleague to help find the relevant source material can never be matched by even the most elaborate handbook. From that perspective comparative methods that actively involve foreign colleagues are equally good in overcoming this hurdle. There is no difference between the modern ‘team & report’-method and the innovative ‘SPOCs & survey’-method.

Correct interpretation and contextualisation

The challenge to correctly interpret and contextualise information on foreign criminal justice systems is widely commented on in literature. It is repeatedly argued that one cannot properly grasp how rules function without situating them into their legal, economic and cultural context; it is argued that there is little use of comparing paper institutions, or rules and doctrines merely as they appear in books (Rabel, 1947; Van Hoecke, 2004, p. 167; Westmarland, 2011, p. 59). Whereas traditional comparative research is said to have failed because it focussed too much on a ‘law as rules’-approach, and gave insufficient attention to the context (Örücü, 2007, p. 45), more recently culture and diversity have become a central concern of comparative research (Monateri, 2012, p. 9). Here too, comparative methods that actively involve the participation of foreign colleagues are equally good in overcoming this hurdle. There is no difference between the modern ‘team & report’-method and the innovative ‘SPOCs & survey’-method.

Comparability of information

With respect to the comparability of information however, there is a clear added value of using the innovative ‘SPOCs & survey’-method, when compared to the ‘team & report’-method.

Working with open questions with a view to drawing up a country report has the benefit that respondents both have the liberty and are challenged to formulate their own reply (Brinkman, 2000, p. 68). Respondents will use their discretion to provide a summary of their own criminal justice system detailing a selected number of specificities. In methodological literature it is emphasised that particularly for research that intends to identify a personal opinion of a respondent, without steering the person into the one or the other direction, working with open questions is the best choice. However, when conducting comparative research looking into the differences of the criminal justice systems, a researcher is not looking for personal opinions, but looking for a similar and detailed factual account of the situation in each of the countries included in the study. From experience, it becomes clear that open questions and essay responses have significant down sides. Having an open question such as “What is the position of a victim in your criminal justice system?” will result in a detailed account thereof for each of the consulted countries. However, chances are high that each of the SPOCs will have selected those aspects from the criminal justice system he/she deemed most interesting or relevant. As a result, it is not guaranteed that based on their input the researcher will have sufficient information for all consulted countries to be able to conduct a proper comparison. Some SPOCs may have focussed on the specificities of ‘who qualifies as a victim’, other SPOCs may have focussed on the specificities of the legal position of the victim and the ‘participation during the criminal proceedings’, whereas yet other SPOCs may have focussed on the specificities of ‘the support a victim can benefit from’, either or not including financial support and compensation. Colleagues working with open question based-templates to draft country reports indicate that keeping all national experts in line is the most challenging part of the project. The validity and comparability of the efforts of all SPOCs are not fully guaranteed. This problem is accommodated in the ‘SPOCs & survey’-method. By their very nature, closed questions leave no doubt as to the specific topics intended to be addressed. Closed questions are more targeted which reduces the ambiguity in relation to the objective of the survey and the information to be included (Bachman & Schutt, 2012, p. 168).

It should however be recalled that the surveys in this innovative method are only ‘predominantly’ multiple choice based and include free text fields in which respondents are able, encouraged or even required to elaborate on the specificities of their own criminal justice system. In doing so, one of the hurdles identified in literature is retained in the method. However, the significance of the hurdle is reduced in two ways. First, the extent to which the researcher is confronted with input via free text fields is significantly reduced. Second, the content of the free text fields is more targeted focussing on a small aspect of the criminal justice system. In doing so, the likeliness of the information included in the free text field being relevant to the researcher is significantly increased. The added value of (if not the need for) allowing the respondents to elaborate on the specificities of their criminal justice systems clearly outweighs the benefit of banning free text fields altogether.

Collecting and processing information

The time needed to collect and process information in the context of a comparative analysis is listed as a hurdle. Though drafting a survey can be relatively time consuming, especially taking the pilot phase into account, this way of working is still far less time consuming than conducting the in depth comparative analysis as an individual researcher. More importantly, collecting and processing information through a multiple choice based survey is less time consuming when compared to collecting and processing information through country reports. Completing a country report is generally more time consuming than completing a multiple choice based survey (Wilson & Sapsford, 2006, p. 101). Processing the information of a country report and producing a comparative analysis is far more time consuming than processing the information included in a multiple choice based surveys. Processing surveys completed in an electronic format can be even largely automated.

Finally, the question arises whether multiple choice based surveys should best be web based or distributed in an offline electronic format. Web based surveys are becoming more and more popular (Bachman & Schutt, 2012, p. 184; Wilson & Sapsford, 2006, p. 93). Notwithstanding the increasing popularity, own experiences with web based surveys have shown that SPOCs are not so keen on working online^p. The need for internet access (which is e.g. not self-evident for law enforcement authorities in some Member States), the struggle with the use of login and password information (especially when the SPOC invites more people to contribute), the relative complexity of continuing the survey at a later time (e.g. when some details of the criminal justice system need to be checked or internally discussed before being able to tick the right box) and the difficulty in having an own copy of the replies are the most common arguments raised against the use of web based surveys. Based on that experience, the choice was made to no longer use web based surveys when targeting that specific audience. In recent studies pdf surveys that can be emailed back and forth between the researchers and the SPOCs have been used^q. Surveys can be easily circulated amongst colleagues, accessed without login information, printed from the very start (with a view to filling it out offline first) to the very end (with a view to keeping a personal copy), and surveys are easily saved and reopened to continue working on it^r.

Conclusion

Whereas a lot of methodological thinking is criticized for remaining merely theoretical, omitting any sustained testing in practice (Adams & Bomhoff, 2012, p. 1), the ‘SPOCs & survey’-method detailed in this paper has been successfully used in past research projects. The method consists of working with a single point of contact in each of the countries included in the comparative analysis. SPOCs are asked to fill out a predominantly multiple choice based survey, complemented with free text fields and where relevant the request to attach documents. Some of the SPOCs will be invited to become a member of the pilot group to review the survey before it is send out to all participating SPOCs. The SPOCs either work alone or compose a team around them.

The main weakness of the method is the need for participation. The method is dependent on identifying the right SPOCs and convincing them to cooperate and provide the information needed. The reliability of the comparison is dependent on the accurateness of the information provided and thus the knowledgeability of the SPOC. To avoid significant problems it is advised that SPOCs are being paid, which can make the method costly.

That weakness is counterbalanced by the strengths of the method: (1) swiftness, (2) reliability and (3) validity. First, once a network is build, identification and nomination of a SPOC can go relatively fast. Furthermore, using a survey is a very swift way of gathering information. Notwithstanding the fact that the drafting of the survey takes a lot of work, including an intensive literature review and pilot phase, the amount of time that needs to be invested does not compare to the amount of time that would need to be invested in individual research into the criminal justice systems for each of the countries involved. Furthermore, the actual collecting and processing of the information is less time consuming compared to other methods used. Second, because information is provided by SPOCs who are knowledgeable about their own criminal justice system and the specificities of the topic subject to analysis, the reliability of the information is considerably higher when compared to the reliability of the information compiled by a researcher that is unfamiliar with the foreign criminal justice system. The method entails a number of guarantees to overcome language barriers and is an effective way to guarantee that up to date information is correctly interpreted and contextualised. Third, the validity of the outcome of the research based on the ‘SPOCs & survey’-method is higher compared to the outcome of research based on the ‘individual & desk top’-method or the ‘team & report’-method. That benefit is largely due to the specificity of the questions and answer categories in the multiple choice survey and the targeted character of the information provided via the free text fields and attached documents. Problems with incompleteness or incomparability of the information the ‘team & report’-method suffers from, are successfully overcome.

Provided the researcher has access to knowledgeable SPOCs, the ‘SPOCs & survey’-method is an easy way to obtain ‘a targeted descriptive and informative picture of the current situation in another state’. It was successfully used in various degrees from somewhat reservedly solely to support methodological choices over obtaining detailed and up-to-date factual information to more extensively collect also personal opinions from the respondents.

Further reading

M. Adams & J. Bomhoff ( 2012 ), Practice and Theory in Comparative Law. Cambridge Cambridge University Press.

Too often, explicit methodological discussions in comparative law remain limited to the level of pure theory, neglecting to test out critiques and recommendations on concrete issues. This book bridges this gap between theory and practice in comparative legal studies. Essays by both established and younger comparative lawyers reflect on the methodological challenges arising in their own work and in work in their area. Taken together, they offer clear recommendations for, and critical reflection on, a wide range of innovative comparative research projects.

P. G. Monateri ( 2012 ), Methods of Comparative Law. Cheltenham: Edward Elgar Publishing Limited

Monateri goes into the development of the method of doing comparative research. He brought together a wide range of authors who provide contributions exploring theoretical and empirical issues in comparative law.

M. Reimann & R. Zimmermann ( 2008 ), The Oxford Handbook of Comparative Law. Oxford: Oxford University Press.

The Oxford Handbook of Comparative Law summarizes and evaluates comparative law as an upcoming discipline. Especially Section II on the major approaches to comparative law - its methods, goals, and its relationship with other fields, such as legal history, economics, and linguistics and the comparative criminal law chapter in Section III are particularly interesting.

Endnotes

^aCriminal policy making is no longer exclusively situated at national level. As a result of cooperation initiatives such as the Benelux cooperation, the European Union and the Council of Europe, criminal policy is also made at ‘international’ level. The need to have an evidence base for criminal policy making was first recognised at national level. Today, that need is also recognised at international level.

^bIn the 2011 study on the material detention conditions in the Member States the SPOC network was made up of practitioners (Vermeulen et al., 2011). The aim of the study was to get an overview of the practical situation regarding the detention conditions in each of the member state. Because of the importance to get a practitioners’ view, all SPOCs were practitioners.

^cIn the 2010 study on the future of judicial cooperation in criminal matters in the EU a mix of academics, policy makers and practitioners were used as SPOCs (Vermeulen, De Bondt, & Ryckman, 2012b). The aim of the study was to get input regarding both legal, practical and policy issues that would arise in relation to a set of future policy options prepared by the researchers. To ensure the inclusion of reflections from all actors, the choice was made to include ‘representatives’ of all actors in the SPOC network and in doing so in the pilot discussions on the drafting of the survey.

^dKnowledgeability can refer to the own knowledge to complete the survey, or the knowledge and access to another person to complete the survey.

^e http://www.eclan.eu/Default.aspx.

^f http://eurojust.europa.eu/about/structure/college/Pages/national-members.aspx. In the 2009 study on the laws of evidence in the Member States, the Eurojust college was used, composed of the Eurojust national members as a starting point (Vermeulen, De Bondt, & Van Damme, 2010).

^g https://www.europol.europa.eu/content/page/our-people-19.

^hEuropean network of contact points in respect of persons responsible for genocide, crimes against humanity and war crimes was set up by Council Decisions 2002/494/JHA of 13 June 2002 and 2003/335/JHA of 8 May 2003 (Genocide Network). http://www.eurojust.europa.eu/Practitioners/networks-and-fora/Pages/genocide-network.aspx.

ⁱ http://www.amue-ejpa.org/.

^j http://www.ecba.org/content/.

^k http://www.ccbe.eu/index.php?id=140&L=0.

^l http://www.europris.org/.

^mThe result of that study will be published as Traest, P., Vermeulen G. et al., (2015). Een praktijkgericht onderzoek naar de knelpunten in de huidige Belgische strafprocedure met het oog op het schrijven van een nieuwe strafprocedure, Antwerp-Apeldoorn-Portland: Maklu.

ⁿNot all Member States have one single information system used by their prisons. In most Member States, different ‘types’ of prisons can be identified, when using the information system as a variable. The SPOCs (i.e. Central Prison Authorities) were used to identify the prisons needed to gather the necessary information for the study. The selection of the prisons was done using – amongst others – that variable.

^oE.g. organized crime is sometimes used to mean the offences of ‘participating in a criminal organization’ whereas at other occasions it is meant to refer to the modus operandi of offences, be it or not a limited number of ‘serious’ offences.

^pIn the 2010 study on the future of judicial cooperation in criminal matters in the EU, web based surveys were used (Vermeulen, De Bondt, et al., 2012b).

^qIn the 2012 study mapping the diversity in the criminal liability for legal persons in the EU, as well as in the 2012 study on mapping the diversity in disqualifications as sanction measures in the EU, emailed pdf based surveys were used (Vermeulen, De Bondt, & Ryckman, 2012a; Vermeulen, De Bondt, Ryckman, & Persak, 2012).

^rIn practice it is important to take due account of the fact that some respondents may be working with older software versions requiring some adjustments in the compatibility settings (e.g. when using Adobe Professional X or up, simply saving the document in an extensive format, enabling the extra options will allow also users with older Adobe Reader versions to complete the survey).