1. Introduction

The experimental method is deeply rooted in history. Francis Bacon (1561-1626) and Robert Boyle (1627-1691), two well-known early advocates of experimental history, defined experimentalism as the collection and description of existing factual knowledge developed in the arts and crafts (Klein, 2005). Over the centuries, the experimental method expanded to multiple sciences, including law and policy (McDermott, 2002). Experimental policies, laws, and regulations were originally adopted in 17th century France and later in parts of the former British Empire (Ranchordás, 2014). These historical examples of experimental laws and policies were not designed to promote fact-based lawmaking but rather to accommodate national policy goals to local needs, budgets and customs (Crouzatier-Durand, 2003; Ranchordás, 2013). The experimental method in law and policy emerged thus to channel power and promote self-governance rather than to improve the quality of regulations. In 1932, Justice Brandeis coined the metaphor ‘states-as-laboratories’, suggesting that the use of experiments for self-governance purposes could also be combined with the advancement of new policies ‘without risk to the rest of the country’ (New State Ice Co. v. Liebmann). More recently, experiments with laws and policies have been more directly associated with the promotion of technological innovation (Cortez, 2013, Crootof & Ard, 2021; Ranchordás, 2015), the development of novel competition law and market regulation enforcement techniques (Svetiev, 2020), the construction of alternative scenarios that help institutions design anticipatory policy for uncertain futures (Campbell-Verduyn & Huetten, 2022; de Goede, 2020), and advance evidence-based lawmaking (Van Gestel & de Poorter, 2016). The modern development of experimental laws, policies and other experimental legal regimes such as regulatory sandboxes occurred against the backdrop of growing scholarship on the principle of effectiveness (Mousmouti, 2012), evidence-based lawmaking (Van Gestel & de Poorter, 2016), temporary legislation (Gersen, 2007), and regulation and innovation, particularly in the financial sector (Allen, 2019)
Experimental legal regimes, a broad term employed in this article to refer to all forms of experiments with laws and regulations, normally take the form of a temporary derogation from general rules (Heldeweg, 2017). It entails that a different legal regime will be applied to a part of the country for a limited period in order to test its effectiveness (Mousmouti, 2012). For example, in the case of experiments with road traffic legislation, an experimental legal regime could establish that in a set number of highways and for a determined period, drivers could be allowed to exceed the maximum speed limit by 20 km/h during the weekend. Lawmakers could test whether this experimental measure has a positive or negative impact on traffic flow and air quality. These results would then be compared with other roads where drivers were required to comply with the maximum speed limit.
Regulatory sandboxes are more recent and more specific types of experimental legal regimes which encompass a close collaboration between public and private actors (Lim & Low, 2019). Regulatory sandboxes create a safe testbed for innovation either by allowing for the temporary application of a different regulatory regime to a small group of firms or by offering compliance guidance (Allen, 2019). For example, in the FinTech sector, the group of firms admitted to a regulatory sandbox can be given the temporary opportunity to test new financial products without fully complying with regulations that could otherwise make these innovations impossible or more burdensome (Zetzsche et al., 2017). The financial regulator which organizes the sandbox will oversee the experiment, evaluate it and then assess the results of the sandbox (e.g. whether the novel product can be developed in compliance with general rules, whether the rules that were set aside truly were necessary to protect the consumer).
Despite their potential to advance regulatory and economic innovation, experimental legal regimes have been criticized because of their loose methodology, casuistic nature and the limited validity of their results (Conseil d’Etat, 2019; Omarova, 2020; Philipsen, Stamhuis & de Jong, 2021). In addition, existing scholarship has unveiled many other flaws of experimental legal regimes, including their frequent politicization, their premature termination, dissatisfactory evaluations, and the general absence of methodological preoccupations (Ranchordás, 2014).
I argue that the methodological deficiencies of experimental legal regimes are problematic for three reasons: First, the limited internal validity of experimental regulations directly affects the learning value of the experiment while external validity problems negatively affect the generalization of results. Second, the use of an experiment in lawmaking may convey a false appearance of objectivity. This scientific façade is likely to be captured by the political rationale and subvert the original goals of experimental regulations which include the promotion of evidence-based, iterative, and innovation-friendly regulatory responses (Fosch-Villaronga & Heldeweg, 2018; Samaha, 2012; Van Gestel & Van Dijck, 2011). Third, the legality of experimental legal regimes is intrinsically connected to their methodology (Keyaerts, 2013) as there is significant overlap between the principles of good science and the principles of good rulemaking. Experimental regulations without a transparent, proportionate and systematic design and implementation can be thus critiqued from both legal and scientific perspectives. This last aspect has contributed to the reluctance of courts and legal scholars to accept the broad adoption of experimental legal regimes (Ranchordás, 2014).
Indeed, legal scholars and courts throughout Europe (e.g. in France, Germany, The Netherlands) were long critical of the constitutional admissibility of experimental laws and regulations (Crouzatier-Durand, 2003; Horn, 1989; Ranchordás, 2014; Interdepartementaal wetgevingsberaad inzake experimenteerbepalingen, 2000) as well as the legal resilience of regulatory sandboxes (Philipsen, Stamhuis & de Jong, 2021). Nowadays, the admissibility of experimental legal regimes is consensually accepted, provided that they are based on clear legislative mandates (Ranchordás, 2014). National variations aside, central legal requirements tend to include a legislative basis with the objectives of the experiment, its duration, and evaluation criteria (Ranchordás, 2013, 2014); and the obligation to comply with regulatory quality criteria as well as other legal principles (transparency, proportionality, legal certainty). Even though these legal requirements have not been systematically implemented, little attention has been devoted to the connection between the methodological and legal framework of experimental legal regimes.
This article explores the relationship between the legal and methodological frameworks of experimental legal regimes. Given the resemblance between experimental legal regimes and field experiments, I argue that regulators should devote more attention to the design of these experiments and improve their methodology by drawing inspiration from existing social science scholarship on field experiments and adapting it to law and policy (Druckman et al., 2006). This article, though not comparative in its methodological approach, focuses on recent doctrinal and empirical studies on experimental legal regimes which were conducted in the United Kingdom, France, The Netherlands, and, to a more limited extent, in the EU. These European jurisdictions were selected because of their growing interest in experimental laws and regulations.
This article is organized as follows. Section 1 provides brief background information on experimental regulations and regulatory sandboxes and their aimed regulatory functions. Section 2 offers an overview of the central features of the experimental method. This information is important to understand the methodological shortcomings of experimental legal regimes. Section 3 discusses common design deficiencies of experimental regulations and regulatory sandboxes and it explains why these methodological deficiencies also have important legal implications. Section 4 advances a legal framework comprising of methodological and legal suggestions to help advance the validity of experimental legal regimes.

2. Background Information

There is no widely accepted definition of ‘experimental regulation’ or ‘experimental legal regime’. Instead, this definition is established by national and constitutional frameworks which, for example, in the case of France, determine the types of experimental legal regimes that national and local legislators may use and within what limits (Conseil d’Etat, 2019). I define ‘experimental legal regime’ as a legislative or regulatory instrument with a temporary nature, with limited geographic and/or subject application which is designed to test a new policy or legal solution.

2.1 Experimental Legal Regimes

Over the last two decades, lawmakers and regulators have become increasingly interested in the adoption of experimental, flexible, temporary, and agile regulatory instruments (Ahern, 2020; Bar-Siman-Tov, 2018; European Council, 2020). In 2018, there were more than 50 statutes in The Netherlands allowing regulators to adopt experimental regulations (Cnossen & Van der Laan, 2018). In France, the list of experimental legal regimes was longer and it spanned a wide range of policy areas from urban planning to sports (Conseil d’ Etat, 2019).
At EU level, the number of experimental EU regulations and directives remains limited (e.g. Council Directive 1999/85/EC on the possibility to apply a reduced VAT rate on labour-intensive services on an experimental basis) but the literature has often referred to the importance of EU experimentalist governance (Sabel & Zeitlin, 2008) and different theories of pluralism in EU law (Kumm, 2012). Experimentalist governance is regarded as an important theory of transnational governance that allows a broadly agreed set of framework goals to be implemented in a diverse and multilevel context (de Búrca, 2017; de Búrca, Keohane & Sabel, 2013; Sabel & Dorf, 1998). In EU criminal law, the concepts of flexibility, differentiation, and experimentation have also been used to discuss the accommodation of difference between Member States (e.g. different emergency brake provisions as allowed by Arts. 82 and 83 TFEU) (Herlin-Karnell, 2013). In risk regulation, the EU regulation of new technologies has also been associated with the need for experimental approaches. This theory is excluded from the scope of this article since experimentalist governance is primarily focused on multilevel governance dynamics rather than on the legal and methodological aspects arising from regulatory derogation.
More recently, the European Commission in the EU Better Regulation toolbox (Toolbox 21 on Research and Innovation) and the Council of Europe have referred to the importance of employing experimental regulations more broadly at EU level in the context of the EU innovation policy and, more specifically, for the regulation of AI systems (European Council, 2020). The OECD has also expressed its interest in advancing experimental legal regimes for the regulation of blockchain, stating that this technology’s potential depends upon a ‘policy environment that supports innovation and experimentation while acknowledging and mitigating risks’ (OECD, 2019).
Experimental regulations can be employed in derogation of existing rules or in the context of devolution of powers (Heldeweg, 2017). In the first case, a statute allows regulators to derogate from one or multiple specific legislative dispositions on a temporary basis in order to test the effectiveness of alternative rules. Experimental regulations do not have a set duration as this is typically determined on a casuistic basis, depending on the specificities of the sector (Ranchordás, 2014). The conditions of the experiment, its scope of application and evaluation criteria are established on the legislative mandate upon which they are based. Additional guidance can be provided on secondary legislation or, for example in the case of The Netherlands, in the (Dutch) Guidelines for Regulation. To illustrate, Guidelines 2.41 and 2.42 contain specific guidance for legislative drafters on the type of information that each experimental clause (legislative mandate) should include (e.g. duration of experiment, rules from which derogation is possible, purpose of experiment). In federal and decentralized systems, legal experiments may also take place through the devolution of the power to experiment with state or local rules that suit their needs (Ranchordás, 2013).
Regulatory sandboxes have a more recent history than experimental regulations. They are regulatory instruments which allow for the safe testing of new products and services, the provision of bespoke guidance, comfort regulation or targeted guidance for better regulatory enforcement (Zetzsche et al., 2017). These instruments offer mainly regulatory relief to specific firms on a temporary and casuistic basis (Knight & Mitchell, 2020; Philipsen, Stamhuis & de Jong, 2021). Depending on their terms, they may set aside specific regulatory requirements for a small group of actors or only provide them with additional guidance on how to comply with existing rules. Regulatory sandboxes also operate as incubation models for start-ups, allowing innovation to be promoted while minimizing the risks for consumers and markets (Alaassar, Mention & Helge Aas, 2021).
Sandboxes were first mentioned in August 2014 in the context of the UK FinTech policy which was developed with the aim to elevate the United Kingdom to the global capital of FinTech. The first set of sandboxes was established by the UK Financial Conduct Authority and ran between 2015 and 2020. Regulatory sandboxes remain appealing to the financial sector within and beyond the United Kingdom. In the meanwhile, they have been embraced in other regulated sectors, including data protection, energy, health care, and telecommunications (Leckenby et al., 2021; Sherkow, 2021; Veseli et al., 2021).
Regulatory sandboxes are regulated at national level by the sector-specific regulators that establish them. These regulators define the terms and themes of the sandboxes for which private companies can apply for, the type of the sandbox and its regulatory relief (testing, bespoke guidance, comfort regulation). The literature has identified a number of characteristics that are commonly present in most regulatory sandboxes: (1) the focus is on the advancement of innovation and entrepreneurialism through regulatory relief; (2) the definition of specific entry rules including the need for regulatory relief, the applicants’ ability to participate in the sandbox and offer an innovative product or service; the limited scope of the sandbox; (3) the restricted extent of regulatory relief (for example, limited to predetermined regulatory requirements such as waiving certain fees or license payments); and (4) the beforehand determination of sandbox rules, including on circumstances that may determine the expulsion of sandbox participants (Martin & Balestra, 2019; Zetzsche et al., 2017). There are thus entry and exit requirements. After private companies (applicants) submit their proposals, the regulator that initiated the sandbox selects the best eligible candidates and defines the cohort of firms that will have access to the regulatory sandbox.
Thus far, existing sandboxes have had short lives. They were valid for periods of 6 to 12 months in the United Kingdom or, in France, up to 2 years. The entry criteria for applicants interested in joining a regulatory sandbox varied greatly depending on the regulator that established the sandbox. For example, entry criteria in the financial sector included the ability to demonstrate that the applicant ‘can promote genuine innovation’, the ability to generate customer benefit and show that the applicant has a genuine need to test innovation in the sandbox, the ability to demonstrate that the applicant is ready to test a new project, proof of sufficient resources. In UK FinTech regulatory sandboxes, the financial regulator has opened sandboxes not only to authorized financial firms but also to unauthorized firms seeking authorization and technology businesses. In the energy sector, the UK energy regulator (OFGem) admitted innovators that operated in the regulated energy market, including start-ups, new entrants, established sector players, businesses transferring in from other sectors, public or third sector bodies. The French Data Protection Regulator (CNIL) has recently established a regulatory sandbox which, without allowing for derogations from the General Data Protection Regulation, aims to help organizations develop innovative applications for privacy-by-design in the health care sector.
The expansion of sandboxes to different fields in the Member States has been welcomed with regulatory interest at EU level. An example of this interest is the inclusion of regulatory sandboxes in Title V (at the time of writing) of the EU AI Regulation Proposal (Artificial Intelligence Act) which was published in April 2021. In the AI Regulation Proposal, regulatory sandboxes are presented as ‘innovation-friendly measures’ that can be employed to assist SMEs in the development of innovative AI systems. The European Commission also refers to regulatory sandboxes in its Better Regulation toolbox, defining them as more ‘sophisticated experimentation clauses … framework[s] that allow innovations to be tested in a real-world environment subject to regulatory safeguards and support’. This is, nonetheless, a narrow and inaccurate approach to regulatory sandboxes. First, regulatory sandboxes are constructed as incubators with regulatory relief for a small number of selected innovations rather than for fully parallel regulatory systems. While experimental regulations change the legal status quo for a representative part of the population and they are based on a statutory authorization that is designed for this purpose, regulatory sandboxes are primarily enforcement policies that offer differentiated treatment to a limited number of firms (Philipsen, Stamhuis & de Jong, 2021). In some cases, no derogation of existing regulatory frameworks may take place as sandboxing can also consist in providing better or bespoke guidance to start-ups on how to comply with regulatory burdens (Bromberg, Godwin & Ramsay, 2017). Regulatory sandboxes are thus established in more limited settings than experimental laws and regulations, requiring a unique collaboration with private parties (often SMEs) that goes beyond the idea of elevating experimental regulations.

2.2 Functions of Experimental Legal Regimes

Experimental laws and regulations and regulatory sandboxes have been employed for a number of purposes. First, experimental regulations have been suggested as instruments that could potentially improve the quality of legislation (Van Gestel & Van Dijck, 2011). Experiments can help regulators gather information and thus evidence about the effectiveness of specific regulatory measures and their alternatives, creative room for regulatory and policy learning and contributing to the adoption of evidence-based legislation. Experimental legal regimes allow legislators to test new legislative measures on a part of the territory or with a group of citizens before they are implemented in the whole territory. They add to the value of other better regulation or regulatory quality instruments as they are placed between the ex ante instruments (e.g. regulatory impact assessments) and ex post evaluations. Experimental rules are thus a first step to more informed and often effective regulation.
Second, experimental regulations are also legitimacy and consensus-gathering instruments which create more room for political bargains (Buyse, Van Humbeeck & Van Nieuwenhove, 2018; Ranchordás, 2014). This reasoning has also been applied to justify the introduction of other temporary regulatory instruments; namely, sunset clauses, that is, dispositions that determine the expiration of a law on a specific date. Opponents to a specific law or provision within it will be more willing to pass it if there is a guarantee that the previously existing status quo will return after the sunset or if the experiment fails. Experimental legislation can be used to reach a compromise or to convince the opponents or sceptics of a certain rule as to its positive effects or absence of side-effects.
Third, experimental laws and regulations have been used to promote decentralization by allowing the implementation of legal experiments at local level, which enables decentralized units to experiment within their own powers and adapt national policies to local circumstances and needs. In France, experimental regulations have been implemented to further the autonomy of French local authorities (collectivités territoriales) (Art. 72 of the French Constitution).
Fourth, both experimental regulations and regulatory sandboxes are currently regarded by the literature and law and policymakers from different countries as important innovation-friendly measures (Buyse, Van Humbeeck & Van Nieuwenhove, 2018; Cortez, 2014; Council, 2020; Cuttino, 2021; Ranchordás, 2015). In November 2020, the Council of the European Union (‘the Council’) called on the European Commission to identify policy areas and regulations in which additional experimentation clauses could possibly help to foster innovation and advance regulation (Council, 2020).
Experimental regulations are attractive to both regulators and innovators because they create safe conditions to advance innovation, reduce regulatory burdens, improve the communication between firms and regulators and promote expedited regulatory decisions without putting consumers at risk (Buckley et al., 2019; Knight & Mitchell, 2020). The experimental character of these regulations and regulatory sandboxes is particularly interesting to start-ups in regulated markets (Alaassar, Mention & Helge Aas, 2021) as they combine the flexibility of market-based solutions with a robust and responsive regulatory and policy framework.
During the experimental period, regulators can gather more information on the effectiveness of these temporary rules, observe how technology is evolving, and update regulations taking into account potential novelties, side-effects of these regulations, or input from consumers and firms (Fosch-Villaronga & Heldeweg, 2018; Omarova, 2020). This experimental approach converts regulation into an iterative learning path, where uncertainty and change are regarded as opportunities to improve regulation by trying new rules and observing their effectiveness. More recently, regulatory sandboxes have also been used in The Netherlands to allow for a legal customization for innovation. This practice entails that financial companies can submit a request for regulatory relief through the customization of specific rules or their enforcement for innovation to the Dutch financial regulators (AFM and DNB).
The legal literature has also presented regulatory sandboxes as flexible instruments that regulators can employ to meet policy goals and adapt their innovations in response to regulatory frameworks (Allen, 2019). Experimental regulations and regulatory sandboxes have been mentioned in recent studies on the innovation principle (Renda & Simonelli, 2019). This principle should help EU regulators consider the impact of new regulations on the advancement of innovation (Renda & Simonelli, 2019). This principle remains controversial in the EU context, as it has been proposed by the biochemical industry and it has been presented in scholarly work as a potential opponent of the precautionary principle (Garnett, van Calster & Reins, 2018). Beyond new technologies and market regulation, recent literature has also suggested the employment of regulatory sandboxes to promote responsible humanitarian innovation which is also characterized by uncertainty (Martin & Balestra, 2019).

3. Experimental Methods

Whenever we systematically manipulate our surroundings and seek to identify the causal connections and evaluate the effects of this manipulation, we engage in ‘experiments’ (Gray et al., 2007). This section offers a non-exhaustive overview of the central elements of the experimental method in social science research.

3.1 Causation

Social sciences are interested in the systematic study of institutions, behaviour, social actions, and gain insights into causal relationships (Teele, 2014). Causation has been mainly studied in social sciences through observational and experimental inquiries (Druckman et al., 2011). The aim to determine causation through experimental inquiry is the first element of interest of this section. A cause is an explanation for some characteristic, attitude or behaviour, including in the context of public policy and regulation (Check & Schutt, 2012). A causal effect occurs if the variation in the independent variable is followed by variation in the dependent variable, when all other things are equal. This last element is difficult to achieve outside laboratory conditions but it is possible to design research to create comparable conditions (Check & Schutt, 2012).
Experimental research is ultimately defined as an investigation where the experimenter manipulates one or more variables under carefully controlled conditions (Gray, 2007). ‘True experiments’ and laboratory experiments with perfectly hermetic conditions are unusual in social sciences. Social scientists do not rely as much on laboratory experiments as natural scientists do because the phenomena they study are not easily manipulated under artificial conditions. Field experiments, quasi-field experiments, and demonstration experiments are thus preferred. Also, quasi-experiments, that is, studies that only have some of the features of the experimental design but do not fulfil all the internal validity requirements, have proven to be helpful in evaluation research studies (Bryman, 2012).

3.2 Validity

Threats to the internal or external validity of research can undermine experimental research. Internal validity refers to the ways in which the conduct or process of experimentation itself may affect the results obtained (Gray et al., 2007). Internal validity is essential to establish whether the treatment that is applied is indeed able to produce a certain effect. Threats to internal validity exist when the scientist’s ability to assess the relationship between the independent and dependent variables is blurred (Gray et al., 2007). External validity refers to difficulties in generalizing the findings of experimental research. This critique is often made to laboratory experiments due to the artificial control of the environment in which the testing takes place and the impossibility of reproducing it in the real world.

3.3 Randomization

The classical experimental design is also known as randomized experiments or randomized trials. Randomization allows for unbiased comparisons as random assignment creates two groups that, apart from the different experimental treatment they are will be administered, only differ from each other by chance (Gerber, 2011). In this context, the experimental inquiry starts with the need to test different ideas, supported by theoretical assumptions, the definition of the independent (proposed causal variable) and the dependent (proposed effect) variables, and ultimately reduced to a hypothesis which will then be tested on a treatment group and a control group. These two groups will be established and it is this that forms the experimental manipulation and therefore the independent variable (Bryman, 2012). The treatment group will be given a different input from the latter. The treatment group will be confronted with the measure under test (independent variable) while the control group will not. The analogy with medicine is perhaps the least complex one in terms of terminological understanding: the treatment group is administered the medicine under trial while the control group will receive a placebo.
One of the central aspects of the experimental procedure is the definition of the two groups, the recruitment of subjects and their random assignment. Through the isolation or minimization of exogenous factors, researchers can determine causality when outcomes vary across the treatment and control groups (Teele, 2014). Randomization in itself also produces similar effects to controlled environments: While the control and treatment groups will not be exactly identical, the law of large numbers will guarantee that as the size of the group increases, there will be a higher probability of convergence. This will mean that the distribution of both groups with regard to every feature − except for the treatment under experiment − are increasingly likely to be the same (Abramowitz, Ayres & Listokin, 2011; Ayres, 2008). Randomized experiments have become particularly popular in development economics research. Thanks to the close collaboration between researchers and implementers, they allow the estimation of parameters that would not otherwise be possible to estimate (Banerjee & Duflo, 2009).

3.4 Validity

Field experiments − randomized control trials − allow social scientists not only to establish causality but also to test the magnitude of treatment effects in real-life conditions (John, 2017). They combine the internal validity of randomized experiments with increased external validity, or generalizability, obtained through conducting an experiment in real-world circumstances (Gerber, 2011). Over the last decades, field experiments have proven to be particularly interesting to economists and political scientists who have sought to gain the advantages of internal validity under experimental controls while limiting the challenges of external validity (Druckman et al., 2011; Teele, 2014). Experiments in the field allow policymakers to assess the effect of changing or implementing new institutions on a small scale before fully implementing a project which could have important costs and consequences for society as a whole (Carpenter & Harrison, 2005). It is thus not only a method to establish causality but also to gather information.
The realism offered by field experiments comes nonetheless at the expense of the loss of control. Field experiments, even when well-designed, have limitations (Cuttino, 2021). In field experiments, it is more difficult to control stimuli the treatment group is exposed to, there are more recruitment challenges, and there is the risk that recruited subjects cannot or do not want to participate after they have been recruited (Druckman et al., 2006, 2011). Many of the objections and concerns raised against field experiments are common to any form of microevaluation, experimental or not (Banerjee & Duflo, 2009). These objections have been addressed by social scientists through enhanced caution with overinterpreting results, the organization of replication studies and the definition of best practices when designing field experiments (Banerjee & Duflo, 2009).

4. Methodological and Legal Shortcomings

In 2019, the French Council of State conducted a thorough research into national and local experimental legal regimes, concluding that most experiments conducted in the previous two decades were characterized by a number of methodological shortcomings (Conseil d’Etat, 2019). These shortcomings included (i) the definition of contradictory objectives; (ii) ill-defined association of actors and the public concerned by the experiment; (iii) inaccurate construction of the sample; (iv) generalization of the measure tested before its evaluation (Conseil d’Etat, 2019). These methodological shortcomings weakened the results of the underlying legal experiments and could potentially mislead public decision-makers. The French Council of State also found that certain legislative actions were incorrectly portrayed as experiments or were used not to gather evidence but rather to support decisions that had already been taken (Conseil d’Etat, 2019). In some cases, authorities decided to implement the measure before the experiment came to an end. The French Council of State recommended greater care in the design of the experimental legal regimes; in the definition of their duration, the definition of its objectives, and the establishment of their evaluation criteria. This section draws upon these methodological shortcomings (Conseil d’Etat, 2019) to discuss the relationship between methodological shortcomings and the failure to comply with the legal frameworks applicable to experimental regulations and regulatory sandboxes.
At the resemblance of other permanent and non-experimental laws and regulations, experimental legal regimes must comply with the general principles of national and EU law and follow general rules on good legislative and regulatory drafting (Baldwin, 2010; Xanthaki, 2014). Some decades ago, the scholarly literature as well as courts in The Netherlands, France, Germany and EU (Horn, 1989; Philipsen, Stamhuis & de Jong, 2021; Ranchordás, 2014) were critical of the constitutionality and legality of experimental legal regimes. However, nowadays, there is greater consensus that if experimental legal regimes are ‘well-designed’, these instruments will be deemed as legal. Nevertheless, there is little guidance or research into the meaning of ‘the good design’ of experimental legal regimes.
Legal principles, regulatory and methodological requirements serve different purposes (rule of law, rationality of regulation, and soundness of research, respectively). However, I argue in this section that all these requirements establish that experimental legal regimes should be designed in a transparent, accountable, proportionate and predictable way. As this section explains, there are important connections between the legal framework applicable to experimental legal regimes and their methodological design. This section presents first the general elements that in the countries under analysis, are part of the legal frameworks applicable to experimental legal regimes. I then explain how experimental legal regimes fail to comply with the law when their methodology is tainted with deficiencies.

4.1 Legal Requirements

This subsection offers a brief glimpse of some of the key legal requirements applicable to experimental legal regimes. National legislation, regulatory guidance and judicial interpretation of rules may differ and impose additional or additional legal requirements.

4.1.1 Principle of Legality

Experimental legal regimes should abide by the principle of legality, which means that experimental regulations and regulatory sandboxes (at the resemblance of other forms of delegation or devolution) should be based on an explicit statutory authorization (statutory experimental clauses) which sets the limits of the power to experiment (Heldeweg, 2017; Ranchordás, 2014). An explicit statutory authorization is not required when the regulator in question already has the power to implement a law in different ways and the power to experiment is implicit and not contrary to conferred regulatory powers. Before the 2003 amendment of the French Constitution, the French Council of State (‘Conseil d’État’) and the French Constitutional Council (‘Conseil Constitutionnel’) were asked on several occasions to analyse the legality of several experimental laws and establish a number of legality conditions which were later incorporated in Articles 37 and 72 of the French Constitution and embraced outside this country. According to these institutions, French experimental legal regimes should be limited in time; the legislator should define (by law) precisely the nature and the scope of the experiments, the cases in which they can be undertaken, the conditions, and the procedures according to which they must be the object of an evaluation leading to their maintenance, modification or abandonment (Conseil d’Etat, 2019). Nevertheless, the specific terms that are crucial for the methodological value and which determine how an experiment is operationalized (e.g. duration, condition, evaluation criteria) of an experiment are determined casuistically by law. Even though regulatory sandboxes are primarily enforcement policies, their legality can also be questioned as the statutory power to create enforcement differentiations tends to be based indirectly on the general regulatory powers of regulators (Philipsen, Stamhuis & de Jong, 2021). The AI Regulation Proposal is an exception as, at the time of writing, it already includes an explicit legal basis for regulatory sandboxes.

4.1.2 Legal Certainty

The principle of legal certainty entails that laws should be clear, foreseeable, coherent, determinate, and predictable so that citizens, when placed in a specific situation where they would like to exercise their rights, know what rules to follow (Popelier, 2008). The Dutch Council of State has expressed on numerous occasions its apprehension regarding the potential tension between experimental legislation and the principle of legal certainty (Interdepartementaal wetgevingsberaad inzake experimenteerbepalingen, 2000). Experimental legislation rarely amounts to a state of ‘bare uncertainty’ where individuals do not know what their rights are (Ranchordás, 2014). Moreover, in a fast-changing society, legal certainty is interpreted as a dynamic concept which cannot require that legislation foresees every single phenomena it applies to (Popelier, 2008). Nowadays, experimental legal regimes are thought to be compatible with legal certainty, if it is clear what regulators are exercising their powers to experiment and how, how long the experimental regime will last, how it will be terminated and evaluated, and if citizens can determine their own legal position in the context of an experimental legal regime (Philipsen, Stamhuis & de Jong, 2021; Ranchordás, 2014).

4.1.3 Equal Treatment

Experimental legal regimes introduce elements of differentiation between citizens and companies. Nevertheless, this differentiation is inherent to any experiment (AG Maduro, Arcelor Atlantique, Case C-127/07. EU:C:2008:728). Therefore, experimental legal regimes do not necessarily violate the principle of equal treatment as long as the experimental character of the law is clear, the terms of the experiment are well defined, the differentiation is necessary for the implementation of the experiment and proportionate considering its expected benefits. In 1993, the French Constitutional Council established that the principle of unequal treatment was not violated as long as

the nature and scope of these experiments [were precisely defined] as well the cases in which they [could] be undertaken, the conditions and procedures according to which they must be the subject of an evaluation leading to their maintenance, to their modification, their generalization or their abandonment. (Conseil d’Etat, 2019).

Since regulatory sandboxes are relatively novel instruments, the conditions for equal treatment have not been well investigated. In France, the financial regulator (AMF) has criticized the proliferation of the regulatory sandbox approach to regulation which distinguishes between historical actors fully subject to regulation; the new actors selected by the supervisor for a regulatory sandbox (and de facto subject to light regulations); and finally those also innovative market actors who were not selected by the regulator for the regulatory sandbox and thus are subject to the same rules as historical players, even though they offer services that are comparable to their competitors admitted to the ‘sandbox’.

4.1.4 Transparency and Accountability

Experimental legal regimes should be strictly guided by the general principles of transparency and public accountability. These principles require not only the publication of sufficient information about the legal framework and operationalization of the experiments but also their management. Since regulatory sandboxes apply more favourable conditions to a limited number of firms, they are also expected to be designed and monitored with enhanced transparency (Knight & Mitchell, 2020). Regulators should be able to guide firms, review their projects and correct their course if any risks emerge. Given the close collaboration between private firms and regulators in the context of regulatory sandboxes, there is the risk that public and private interests become blurred in these contexts (Philipsen, Stamhuis & de Jong, 2021). The casuistic nature of regulatory sandboxes also puts additional pressure on public accountability as it is difficult to control the exact benefits and shortcomings of the experimental regulatory relief and the advantage that is being provided to firms (Philipsen, Stamhuis & de Jong, 2021).

4.2 Methodology and Compliance With Legal Framework

Many of the mentioned legal requirements can only be fulfilled if sufficient attention is devoted to the design of experimental legal regimes. The violation of rules on how to conduct scientifically sound and objective experiments can entail a violation of the law for different reasons. First, the principles of good science overlap on several aspects with the principles of good lawmaking and good government (e.g. transparency, duty of care). While experimental legal regimes may at first sight restrict certain legal principles such as equal treatment, this restriction (or cost) will only be justified if the experiments yield certain benefits (e.g. evidence-based regulations following a successful experiment, complete information about the effects of a certain regulation) that exceed potential costs. If the experiment is poorly designed, it will most likely not produce these benefits and, therefore, those restrictions will not be justified and will be deemed disproportionate. To illustrate, if the experimental legal regime is only applied to a small and specific part of the population without any explanation of the reasons why this group was selected and why, there will be a violation of the equal treatment, proportionality and transparency requirements. At the same time, the experiment will have no external validity because the group in which the test occurred may not be representative of the whole population.
Experimental legal regimes showcase the connection between two important aspects of public policy which are often analysed in isolation: compliance with the rule of law and methodological design. Both experimental regulations and regulatory sandboxes must always be grounded in the rule of law and their design should be compliant with the principles of legality, legal certainty, equal treatment, proportionality and transparency. In order to assess whether this is the case, regulators must ensure that the design of experimental legal regimes fulfils several methodological requirements.
In The Netherlands, there are multiple examples of experimental legal regimes which were not adequately evaluated, did not have clear limits and objectives from the very beginning, or that were converted into permanent laws even though it was clear that the results of the experiment were negative (Ranchordás, 2014). The first example is the Dutch experiment with ‘dentist prices’ which was prematurely terminated. In 2012, an experimental law was introduced to introduce free market prices in dental hygiene which had been regulated until then. This law was by no means a school example of an experimental legal regime: The ‘experimental law’ was a temporary statute applied to the whole territory and there were thus no sample and control groups. Moreover, the experiment was prematurely terminated due to political pressure (De Nationale Ombudsman, 2015; van Loghum, 2012). The regulator’s preliminary evaluation (Nederlandse Zorgautoriteit) was disregarded, even though it was insufficient to understand whether the experiment would be successful after an initial trial period. The Dutch association of dentists submitted a complaint to the Dutch Ombudsman after unsuccessful court challenges, criticizing the lack of transparency and poor implementation of the experiment.
A second example of the methodological challenges of experimental legal regimes with legal implications is the Doctoral Education Experiment Decree. In The Netherlands, PhD students are university employees with a labour contract. This experiment derogates from this rule and converts PhD candidates into students with scholarships instead of labour contracts. This change could allegedly allow for the increase of the number of PhD vacancies, reduce costs, and encourage PhD students to design their own proposals rather than working on predetermined projects. Dutch universities could register for this experiment until mid-2016 but only the University of Groningen was interested in doing so. The experiment is set to end in August 2024 but its limited design, population and the lack of controlling elements affect both the internal and external validity of the experiment. After years of experimenting with a less favourable PhD-regime, it is unknown whether the availability of these scholarships indeed had a positive impact on the results or whether the latter had a different cause. Moreover, this experiment proved to be controversial from a legal perspective as PhD students from the only participating university feel unequally treated vis-à-vis their colleagues conducting doctoral research with a labour contract. It is also unclear whether this experiment is proportionate when weighing its costs against its benefits.
Regulatory sandboxes have also been critiqued for their casuistic design and non-representative cohorts which promoted a ‘race to the bottom’ and offer limited learning potential and applicability outside their original experimentation area (Allen, 2019; Omarova, 2020). The methodology of regulatory sandboxes is defined top-down by regulators and may vary not only from sector to sector but also from sandbox to sandbox. Since sandboxes are still fairly recent, there are no systematic studies comparing the methods used by regulators but it is clear that their design is determined on a case-by-case basis.
Experiments can help estimate the effectiveness of different laws and regulations but, as the literature (Abramowitz, Ayres & Listokin, 2011; Listokin, 2008) has explained, methodology is key. Lawmakers refer to ‘experiments’ too loosely, conferring a wrongful scientific sense to temporary laws or simply new measures for which there is no full political compromise at the time of their enactment. These loose experiments rarely comply with any clear set of methodological rules or the rules designed for scientific experiments such as random assignment of measures, the design of representative control and sample groups, and evaluation methods (Abramowitz, Ayres & Listokin, 2011; Dorf & Sabel, 1998). It is important to highlight that it should not be expected from legal experiments to fully emulate the scientific method. However, an appropriate level of randomization, a clear methodological framework could help ensure the objectivity, effectiveness, validity and legality of experimental legal regimes (Abramowitz, Ayres & Listokin, 2011).

5. A Legal and Methodological Framework for Experimental Legal Regimes

This section suggests a general legal and methodological framework for the design of experimental legal regimes. This framework does not aim to shape ‘perfect experiments’ in law and regulation. Instead, its central aim is to address the methodological shortcomings described in Section 3, ensure that experimental legal regimes can be evidence-based instruments and promote greater transparency, effectiveness assessment, rationality and objectivity within experimental legal regimes. It draws inspiration from the methods employed in field experiments (Banerjee & Duflo, 2009) without seeking to transpose every single element of it to the world of law and regulation. This section suggests that the adoption of general guidelines, best practices, and more uniform regulation for experimental legal regimes (including a minimum of methodological conditions) will not limit the room for experimentation (Cuttino, 2021). It will define it and increase its effectiveness and validity. Moreover, I do not argue that all the elements of an experimental legal regime should be set in stone through a general framework. Specific elements such as the duration and the individual evaluation criteria of an experimental legal regime can only be determined on a casuistic manner.
The proposed design framework could also promote uniformity in the design of experiments, including regulatory sandboxes. While regulatory sandboxes from the financial sector have inspired other sector-specific regulators who recently adopted these experiments, the existence of a general framework could establish minimum soundness requirements. Moreover, a general framework could be an additional tool for the judicial review of experimental legal regimes. For example, it could help courts assess whether individuals who fell outside the scope of the experimental legal regime were effectively disfavoured in a subjective and disproportionate manner or whether the differentiation introduced by the experiment was objective and inherent to the experimental method. The existence of clear and complete methodological elements for regulatory sandboxes can also contribute to their generalization and facilitate replication studies with other cohorts.

5.1 Design

The sound design of experimental legal regimes should entail that the objectives of the experimental legal regime or the hypothesis to be tested are defined from the very beginning of the experiment and if possible, precede it. This design framework should not require regulators to produce overly complex theoretical reports explaining on what grounds their hypothesis is based. A design framework for experimental regimes does not aim to replace other existing complex regulatory assessment frameworks such as regulatory impact assessments. It is instead a complementary checklist that can complement regulators’ better regulation toolbox when considering the use of experimental legal regimes. Nevertheless, there should be evidence that an experimental legal regime is objectively required and it is not merely the response to unfunded claims, political pressure or lobbying.
Moreover, the independent variable (cause) as well as the dependent variables should be clearly stated and explained in the design of the experiment. In a real-life setting, there is limited control over extraneous variables and also this should be acknowledged to guarantee that the results of the experiment are not overinterpreted. Moreover, the hypothesis should be formulated in comprehensive but easily perceptible terms so that all stakeholders are aware of what is being tested and how. The implementation of this requirement could be translated in the formulation of clear, well-defined, precise and manageable questions and study design components. The definition of the different components of the hypothesis under testing should also be accompanied by the determination of different possible options on how to conduct the intervention and a recommendation on the advised option. To illustrate, in the context of a regulatory sandbox, regulators may wish to test whether alleviating regulatory burdens (e.g. market entry licenses) on businesses will support the innovation process. A general design framework may offer guidance to regulators, elaborating on how to explicitly define the terms of regulatory relief and determine what options regulators could implement (e.g. temporary derogation from rules, assistance with compliance). Also, this framework can also offer guidance on how to determine the optimal duration of an experiment. In order to guarantee that an experimental intervention yields effective results, the duration of the experiment should be proportionate to the characteristics of the sector, the complexity of the experiment, and the willingness of firms and citizens to participate in short or long-lived experiments and the impact of this duration on their rights. In complex cases, pre- and post-testing moments can be added to the experimental design. Clear guidance on the design of the hypothesis, the intervention as well as a proportionate duration are likely to advance the protection of the principles of legality, legal certainty, equal treatment and proportionality in the context of the implementation of experimental legal regimes. In some countries and sectors, regulators may already consider these elements when reflecting upon the design of experimental regulations and regulatory sandboxes. The introduction of a general framework can guarantee that this occurs in a systematic way.

5.2 Treatment Group, Control Group and Random Assignment

The sound design of experimental regulations and regulatory sandboxes should encompass the definition of comparable groups as well as the random assignment of experimental legal measures within them. Random assignment will be difficult – but not impossible – to implement under real-life conditions in the context of experimental legal regimes. Moreover, from a legal perspective, random assignment requires that the conditions for random assignment are designed so as to ensure that the differentiated treatment that is imposed does not violate existing legal and constitutional frameworks.
Both regulatory sandboxes and experimental regulations should admit participants as much as possible through a controlled system of self-selection and voluntary participation. The selected groups should be representative of the population or sector. This ensures that the differentiated treatment is justified not only for the strict purpose of the experiment but also for external validity purposes. As it has been mentioned earlier, many experimental legal regimes compare the experimental measure to ‘past conditions’ but apply the new measures to the whole citizenry at once on a temporary basis. It is important to ensure that the treatment and the control elements take place at the same time for quasi-optimal comparison.
Other empirical measurements such as the development of performance indicators through descriptive statistics or focus groups could also help legislators assess the success or failure of different experimental regimes, including their desirability. It is, nonetheless, important that there is a clear term of comparison for any experiment and that the participants are representative of society or their sector.

5.3 Evaluation

The assessment of an experiment and its results should include clear guidelines for the evaluation of an experimental legal regime: evaluation criteria, type and moment of evaluation to be conducted, the profile or identification of the evaluator, data gathering methods and how to evaluate data, circumstances that justify terminating or extending the duration of an experiment. According to the French Council of State (Conseil d’État 2019), experimental legal regimes should be evaluated by either a general inspection, an independent council consisting of scientists and administrators, a specialized service, a ministry evaluation committee or an expert body. Once the evaluation indicators have been defined, the French Council of State (Conseil d’Etat, 2019) has underlined that it is important to ensure that the data to measure them is available throughout the experiment. This may require a specific data collection system, which should be planned before launch. Ideally, it should also be ensured that similar data will be available for a comparable sample or territory that remains outside the experiment. The data format must be uniform and usable. These datasets often include personal data, and are therefore subject to the legislation that protects them – namely the General Data Protection Regulation (GDPR) and other European and French sector-specific regulations.

6. Conclusion

In this article, I delved into the methodological shortcomings of experimental legal regimes and how this affects their legality. Experimental legal regimes are not the only regulatory instruments that suffer from methodological problems. Rather, this also affects permanent laws and regulations and this problem has been partially discussed by the literature on legislative methods, evidence-based lawmaking and regulatory quality (Mousmouti, 2018; Mosley & Gibson, 2017; Van Gestel & de Poorter, 2016). Indeed, the development of regulatory impact assessments and other regulation decision-making methodologies aims to ensure that regulation is justified by evidence-based procedures and is designed to be effective (Baldwin, 2010; Radaelli & de Francesco, 2010). Nevertheless, methodological deficiencies in experimental legal regimes raise greater concerns than permanent legislation in terms of legality, equal treatment, transparency, and proportionality because they introduce important differentiations between citizens (e.g. the regulatory relief provided by a sandbox means that the selected sandbox participants can develop their products with temporary disregard for otherwise required licenses). These differentiations are only legally justified when they serve the objective purpose of an experiment (e.g. to obtain evidence of the effectiveness of a new legal measure or approach). When these differentiations are no longer valid because their deficient methodology renders their results biased, then experimental legal regimes are affected by both methodological and legal shortcomings. This article has sought to address this problem and offer insights into the possible path to the greater objectification of experimental legal regimes. It does so by drawing a parallel between experimental legal regimes and field experiments.
The comparison between field experiments and experimental legal regimes is by no means perfect. Also, the aim of this article is not to encourage lawmakers to fully engage in field experiments. However, lessons can be learned from the scientific method. Both field experiments and experimental legal regimes are experiments with real-life circumstances. While this inevitably gives rise to ethical and practical concerns as well as the challenge of exercising control and determining causality, recent field experiments with social policies (Banerjee & Duflo, 2009) have demonstrated the possibility of conducting meaningful experiments. Despite the imperfect comparison, field experiments can inspire lawmakers and regulators to devote greater attention to the design of experimental legal regimes and confer greater objectivity to experiments in the legal realm. This will contribute not only to the validity of experimental legal regimes but also to citizens’ trust in their legality.
Experimental legal regimes do not aim to be the exact legislative or regulatory equivalents of clinical trials. They are instead acknowledgements that lawmaking takes place in a complex environment that is inherently characterized by uncertainties and permanent changes (Mousmouti, 2018). Future interdisciplinary scholarship can help refine this framework and with the support of empirical evidence, it will be possible to assess the broader value of regulatory sandboxes and experimental regulations in the advancement of evidence-based lawmaking.