This repo provides the source code & data of our paper "GreaseLM: Graph REASoning Enhanced Language Models".

<p align="center">

<img src="./figs/greaselm.png" width="600" title="GreaseLM model architecture" alt="">

</p>

- [Python](<https://www.python.org/>) == 3.8

- [PyTorch](<https://pytorch.org/get-started/locally/>) == 1.8.0

- [transformers](<https://github.com/huggingface/transformers/tree/v3.4.0>) == 3.4.0

- [torch-geometric](https://pytorch-geometric.readthedocs.io/) == 1.7.0

Run the following commands to create a conda environment (assuming CUDA 10.1):

conda create -y -n greaselm python=3.8

conda activate greaselm

pip install numpy==1.18.3 tqdm

pip install torch==1.8.0+cu101 torchvision -f https://download.pytorch.org/whl/torch_stable.html

pip install transformers==3.4.0 nltk spacy

pip install wandb

conda install -y -c conda-forge tensorboardx

conda install -y -c conda-forge tensorboard

pip install torch-scatter==2.0.7 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html

pip install torch-cluster==1.5.9 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html

pip install torch-sparse==0.6.9 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html

pip install torch-spline-conv==1.2.1 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html

pip install torch-geometric==1.7.0 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html

Download all the raw data -- ConceptNet, CommonsenseQA, OpenBookQA -- by

You can preprocess the raw data by running

You can specify the GPU you want to use in the beginning of the command `CUDA_VISIBLE_DEVICES=...`. The script will:

* Setup ConceptNet (e.g., extract English relations from ConceptNet, merge the original 42 relation types into 17 types)

* Convert the QA datasets into .jsonl files (e.g., stored in `data/csqa/statement/`)

* Identify all mentioned concepts in the questions and answers

* Extract subgraphs for each q-a pair

**TL;DR**. The preprocessing may take long; for your convenience, you can download all the processed data [here](https://drive.google.com/drive/folders/1T6B4nou5P3u-6jr0z6e3IkitO8fNVM6f?usp=sharing) into the top-level directory of this repo and run

The resulting file structure should look like this:

To train GreaseLM on CommonsenseQA, run

You can specify up to 2 GPUs you want to use in the beginning of the command `CUDA_VISIBLE_DEVICES=...`.

Similarly, to train GreaseLM on OpenbookQA, run

You can download a pretrained GreaseLM model on CommonsenseQA [here](https://drive.google.com/file/d/1QPwLZFA6AQ-pFfDR6TWLdBAvm3c_HOUr/view?usp=sharing), which achieves an IH-dev acc. of `79.0` and an IH-test acc. of `74.0`.

You can also download a pretrained GreaseLM model on OpenbookQA [here](https://drive.google.com/file/d/1-QqyiQuU9xlN20vwfIaqYQ_uJMP8d7Pv/view?usp=sharing), which achieves an test acc. of `84.8`.

To evaluate a pretrained GreaseLM model checkpoint on CommonsenseQA, run

Again you can specify up to 2 GPUs you want to use in the beginning of the command `CUDA_VISIBLE_DEVICES=...`.

SimilarlyTo evaluate a pretrained GreaseLM model checkpoint on OpenbookQA, run

- Convert your dataset to `{train,dev,test}.statement.jsonl` in .jsonl format (see `data/csqa/statement/train.statement.jsonl`)

- Create a directory in `data/{yourdataset}/` to store the .jsonl files

- Modify `preprocess.py` and perform subgraph extraction for your data

- Modify `utils/parser_utils.py` to support your own dataset

This repo is built upon the following work:

Many thanks to the authors and developers!

mkdir -p data/

mkdir -p data/cpnet/

wget -nc -P data/cpnet/ https://s3.amazonaws.com/conceptnet/downloads/2018/edges/conceptnet-assertions-5.6.0.csv.gz

cd data/cpnet/

yes n | gzip -d conceptnet-assertions-5.6.0.csv.gz

wget https://csr.s3-us-west-1.amazonaws.com/tzw.ent.npy

cd ../../

mkdir -p data/csqa/

wget -nc -P data/csqa/ https://s3.amazonaws.com/commensenseqa/train_rand_split.jsonl

wget -nc -P data/csqa/ https://s3.amazonaws.com/commensenseqa/dev_rand_split.jsonl

wget -nc -P data/csqa/ https://s3.amazonaws.com/commensenseqa/test_rand_split_no_answers.jsonl

mkdir -p data/csqa/grounded/

mkdir -p data/csqa/graph/

mkdir -p data/csqa/statement/

wget -nc -P data/obqa/ https://s3-us-west-2.amazonaws.com/ai2-website/data/OpenBookQA-V1-Sep2018.zip

yes n | unzip data/obqa/OpenBookQA-V1-Sep2018.zip -d data/obqa/

mkdir -p data/obqa/fairseq/official/

mkdir -p data/obqa/grounded/

mkdir -p data/obqa/graph/

mkdir -p data/obqa/statement/

export TOKENIZERS_PARALLELISM=true

dt=`date '+%Y%m%d_%H%M%S'`

dataset=$1

args=$@

ebs=32

mode=eval

echo "***** hyperparameters *****"

echo "dataset: $dataset"

echo "******************************"

run_name=eval__greaselm__${dataset}__${dt}

python3 -u greaselm.py \

--run_name ${run_name} \

--mode ${mode} -ebs ${ebs} --dataset ${dataset} \

$args