Perhaps this should be renamed to "MA expungement analysis," since this is no longer a set of single standalone files without any outputs, though each file makes attempts at self-documentation. Make sure your code is actually well documented, add it to this folder, and then update this readme with a quick blurb below about what it does!
A few tables tables: currently used notebooks, and deprecated notebooks.
| Notebook | Description | Inputs | Outputs |
|---|---|---|---|
| MA_Data-1_Raw.ipynb Step 1 |
Anonymizes the MA prosecution data from Northwestern DA and Suffolk County and provides only fields required to answer our guiding questions. | All described in data/raw readme. Prosecution Northwestern DA 2014-2018 RAW DATA.xlsx (on Drive) data/Suffolk County Prosecution*.csv (raw data offline only) |
raw/nw.csv, raw/suff.csv |
| MA_Data-2_MergeCharges.ipynb Step 2 |
Gets all charges from the MA prosecution data from Northwestern DA and Suffolk County and maps expungement eligibility to each -- joins the Master Crime List ("raw/ExpungeCategories.csv") with the individual level data from Step 1. This results in output files with many duplicates and some missing values, since additional information is needed to determine eligibility, and must be updated in steps 3-5. | raw/nw.csv, raw/suff.csv, raw/ExpungeCategories.csv |
offense level file, with first stage of expungeability information: processed/prosecution_charges.csv, Individual level files, should be ignored: processed/prosecution_northwestern.csv, processed/prosecution_suffolk.csv |
| new_expungability_info_join_emily.R Step 3 |
Joins new expungeability info from Sana into data/processed/prosecution_charges.csv, since not all charges were successfully matched to expungeability eligibilities in the last step. | prosecution_charges.csv and Drive... /MA Prosecution Charges - Manual Identification of Chapter / Section / Expungability |
prosecution_charges.csv and missing_expungeability_07-21.csv |
| sex_murder_columns.R Step 4 |
Adds columns for inexpungeable sex/murder related charges to data/processed/prosecution_charges.csv, and sorts by chapter and section, to the best of our ability. | prosecution_charges.csv | prosecution_charges.csv |
| MA_Data_revised_joining.R Step 5 |
Adds more detailed information by which to join charges' expungeability info, aside from chapter/section. For example, repeat offense, or armed offense, involving elderly person, etc. | prosecution_charges.csv | prosecution_charges_detailed.csv |
| MA_Data-2_MergeCharges_alt Step 2 -- alternate or Step 6, depending on how you look at things. | MA_Data-2_MergeCharges.ipynb has individual level data, but has incorrect and/or missing expungement info because chapter & section do not uniquely identify expungability. Steps 2-5 add columns and details to determine eligibility for expungement. This step merges in one additional manually-created file (missing_expungeability_08-02). Results in a usable individual-level file with expungement info, ready for analysis. Ideally we would create an updated "Master Crime List" and update the full data flow from steps 2-6 (this is a work in progress). This is a crutch in the meantime. |
raw/nw.csv, raw/suff.csv, prosecution_charges_detailed.csv, and missing_expungeability_08-02.csv |
merged_suff.csv and merged_nw.csv |
| Middlesex_Clean.ipynb Equivalent of Steps 1-5, or MA_Data-2_MergeChartes_alt but for Middlesex. | Cleans Middlesex data, merges with expungement eligibility. | damion_database_2014-2019_6.30.csv, prosecution_charges_detailed.csv, and missing_expungeability_08-02.csv |
merged_ms.csv |
| sex_offenses_labelling.ipynb Step 6 |
Corrects the indicator of sex offense. sex_murder_columns.R used only text-based analysis; this removes false positives from that work and adds sex offenses based on chapter and section that were not caught by the text list. | merged_suff.csv, merged_nw.csv, merged_ms.csv | merged_suff.csv, merged_nw.csv, and merged_ms.csv |
| Notebook | Description | Inputs | Outputs |
|---|---|---|---|
| master_crime_list_mod.py A step towards a cleaner data flow | On 08/25/20 we agreed to make a new modified version of the Master Crime List (a.k.a. ExpungeCategories) where we will keep track of any extra criteria beyond chapter/section that are used to determine expungeability, and then this modified Master Crime List will be joined to the MA prosecution charges list by chapter, section, and extra criteria, rather than just chapter and section. This script simply re-runs the regex to identify chapter and section on the Master Crime List (originally in MA_Data-2_MergeCharges), after Emily manually added a new column for extra criteria. | ExpungeCategories_mod.csv | ExpungeCategories_mod.csv |
| Notebook | Description | Inputs | Outputs |
|---|---|---|---|
| How many expungable.ipynb | For Northwestern DA: Estimates the number of offenses which are eligible for expungement under a variety of conditions. Answer questions listed in issue #152 | merged_nw.csv | Creates the information for Northwestern tables |
| How many expungable-Middlesex.ipynb | For Middlesex DA: Estimates the number of offenses which are eligible for expungement under a variety of conditions. Answer questions listed in issue #152 | merged_ms.csv | Creates the information for Middlesex tables |
| Disposition exploration | Summary stats and exploration of the dispositions data in Middlesex, Northwestern, and Suffolk. | merged_ms, merged_nw, merged_suff | |
| Transfer to superior court | tl;dr: transfers do not appear to be a problem in the analysis of how many individuals have offenses eligible for expungement Uses the Northwestern DA data to analyze the impact of # people eligible of charges that are transferred from District to Superior Court. This occurs about 854 times. Removing these transfers we can identify makes absolutely no difference in how many individuals are eligible for exungement in the Northwestern DA data. With better identification of 'transfers', there may be a handful that would newly appear eligible; however, this is likely to be a very low number. |