August 2021
Download the Bias
Paper 
Access the Judicial Data 
Try the gender
classifier 
This project was funded in part by Emergent Ventures, the World Bank Research Support Budget, the World Bank Program on Data and Evidence for Justice Reform, the UC Berkeley Center for Effective Global Action, and the DFID Economic Development and Institutions program. Alison Campion, Rebecca Cai, Nikhitha Cheeti, Kritarth Jha, Romina Jafarian, Ornelie Manzambi, Chetana Sabnis, and Jonathan Tan provided excellent research assistance on this project.
Structural inequalities across groups defined by gender, religion, and ethnicity are seen in almost all societies. Governments often try to remedy these inequalities through policies, such as anti-discrimination statutes or affirmative action, which must then be enforced by the legal system. A challenging problem is that the legal system itself may have unequal representation. It remains an open question whether legal systems in developing countries are effective at pushing back against structural inequality or whether they serve to entrench it.
We studied ingroup bias in the Indian judiciary, where Muslims, women, and lower castes are substantially underrepresented. India is home to 195 million Muslims, and women represent 48% of the population, yet:
Does the gender and religious imbalance of the courts affect judicial outcomes? Using data on over five million court cases filed under India's criminal codes between 2010–2018 across the country, we examined whether defendants receive better judicial outcomes when their cases are heard by judges with the same gender, religious, or caste identity.
We collected data on the outcomes of close to the universe of criminal cases in India from 2010-2018 and took advantage of the fact that cases are effectively randomly assigned to judges. Whether a defendant sees a judge of the same gender/religion/caste as them is basically a coinflip--albeit a coin flip that is heavily stacked against women, Muslims, and lower castes due to their underrepresentation in the judiciary. We compare defendants charged under the same criminal act and section, in the same month, in the same district court; the only difference is that some are assigned to judges who match their identity on some dimension and some are not.
In the aggregate, we find no evidence of bias! Judges of different genders do not treat defendants differently according to defendant gender, nor do judges display favoritism on the basis of religion. We do not have caste identifiers in the data, so we proxy social group alignment by situations where the judge and defendant have the same last name. Naturally, this measure has limitations, which are discussed in the paper. On average, being assigned to a judge with the same last name as the defendant does not affect the acquittal rate. These null effects are seen both in terms of outcomes (i.e. acquittals and convictions) and in terms of process (i.e. speed of decision). The tables below show the relationship between judicial decisions and gender group (Table 1) and religious group (Table 2) of judge and defendant. The third row in each table shows the in-group bias in percentage points under a range of empirical specifications: a value of 0.002, for example, means matching the judge's identity raises the acquittal rate by 0.2% -- i.e. neither substantively nor statistically significantly.
Why is this surprising? Because a large number of studies have found substantial in-group bias among judges, using random assignment of judge designs very similar to ours. For instance:
The literature suggests that some contexts may make identity more salient than others. We explore a subset of cases where we might see this effect. First, we examine cases where the defendant and the victim of the crime have different identities; in these cases, the judge has an identity matching either the victim or the defendant, but not both. Second, we examine gender bias in criminal cases categorized as crimes against women, which are mostly sexual assaults and kidnappings. In both of these subset analyses, we continue to find a null bias.
In the third case, we examine whether religious bias is activated during Ramadan. Here, we find a weak effect-- Muslim defendants appear to have lower acquittal rates than non-Muslim defendants, only if it is during the month of Ramadan and they appear before a non-Muslim judge. Even in this very narrow subset of cases, the bias effect we find is very small relative to the literature.
Finally, we explore the effect of last name similarity in subgroups that have rare last names. When judge and defendant share a last name, but the last name is common, this may not be all that salient to the judge, because it represents shared membership only in a very large social group. If they share a rare last name, this indicates shared membership in a much smaller social group, which may raise the salience of that shared identity. To test this hypothesis, we run the shared last name estimation, limiting the sample to rare last names. Here, we find a substantial degree of in-group bias; sharing a rare last name with a judge raises your acquittal rate by three percentage points. This suggests that in-group bias is indeed activated when the group is very small. But the aggregate effect of this bias will remain small, because individuals with rare last names have by definition a low probability of being randomly assigned to a judge with the same last name.
In the paper, we discuss a number of reasons why the Indian judicial context does not appear to activate in-group bias in the majority of cases. Our findings on salience are suggestive. Most prior work on judicial bias is in the United States (focused on race) and Israel (focused on ethnicity); these may be contexts where group differences are extremely salient, particularly in the judicial system.
An alternative explanation is that the current consensus in the literature are influenced by publication bias. This occurs if papers that find surprising or provocative results are more likely to get finished and published than papers that find null effects. The graph below explains what publication bias is, and shows evidence that it does indeed play a factor in the literature on in-group judicial bias.
Note: This visualization is best viewed on Chrome or Firefox.
Table 3 shows a formal test of publication bias following Andrews and Kasy (2019). This test takes the existing papers, assumes that estimated treatment effects are normally distributed around the true value, and then calculates the relative likelihood that a statistically insignificant point estimate makes its way into a research paper, compared with a statistically significant point estimate. The 3rd column of the table tells us that a paper with a statistically insignificant estimate of in-group bias is only 3 percent as likely to be published — i.e. statistically significant estimates are 30 times more likely to be published in the literature.
For more information, please see our paper, In-group bias in the Indian judiciary: Evidence from 5 million criminal cases.
We have made all the data from this project public and open source: a dataset of 80 million cases in the Indian judicial system from 2010-2018 (5 million of which were criminal cases where we could identify judge and defendant religious or gender identity). To access the open data, click here.
To access the paper replication code and data, please visit the paper's public GitHub repo.
For a description of the public data with a small set of examples of potential use cases, see our medium post describing the data release.
You can upload your own CSV of (Indian) names to our trained gender classifier and it will give you its best guess at the gender, with a probability score. You can also download the code for the neural network algorithm and train it on your own data.
If you use these data, please cite our paper:
@Unpublished{ aabbcdgns2021bias,
author = {Ash, Elliot and Asher, Sam and Bhowmick, Aditi and Bhupatiraju, Sandeep and Chen, Daniel and Devi, Tanaya and Goessmann, Christoph, and Novosad, Paul and Siddiqi, Bilal},
note = {Working Paper},
title = {{In-group bias in the Indian judiciary: Evidence from 5 million criminal cases}},
year = {2021}
}
