The repo is the source code for LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
–Chengwei Qin, Shafiq Joty
Accepted at 10th International Conference on Learning Representations (ICLR’22).
Setup
1. Download the code
git clone git@github.com:qcwthu/Lifelong-Fewshot-Language-Learning.git
cd Lifelong-Fewshot-Language-Learning
2. Install dependencies
2.1. Create conda environment
For NER & Classification:
conda create --name lfll_1 python=3.9.4
conda activate lfll_1
pip install gpustat
pip install tensorflow==2.5.0
pip install seqeval==1.2.2
pip install matplotlib==3.3.4
pip install numpy==1.19.5
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html
cd transformers; pip install .; cd ..
pip install fairscale==0.3.7
pip install datasets==1.11.0
pip install sentencepiece==0.1.95
For Summarization & Different task types:
conda create --name lfll_2 python=3.9.4
conda activate lfll_2
conda instal matplotlib=3.5.0
pip install scikit-learn==1.0.2
pip install gpustat
pip install tensorflow==2.5.0
pip install seqeval==1.2.2
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
cd transformers; pip install .; cd ..
pip install fairscale==0.4.4
pip install datasets==1.17.0
pip install sentencepiece==0.1.96
pip install more_itertools==8.12.0
pip install rouge-score==0.0.4
pip install rouge==1.0.1
2.2. Download LM-adapted T5 model
To run lifelong few-shot language learning with prompt tuning, you should download the LM-adapted T5 model from LM-Adapted: t5.1.1.lm100k. Then you should use convertmodel.py to convert Tensorflow checkpoint to Pytorch checkpoint.
mkdir lm_adapted_t5model
cd lm_adapted_t5model
Then you should follow gsutil_install to install gsutil.
After installing gsutil, run
gsutil -m cp \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/checkpoint" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model-info.txt" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.data-00000-of-00004" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.data-00001-of-00004" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.data-00002-of-00004" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.data-00003-of-00004" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.index" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/model.ckpt-1100000.meta" \
"gs://t5-data/pretrained_models/t5.1.1.lm100k.large/operative_config.gin" \
.
to download LM-adapted T5-large checkpoint T5-large TF ckpt. Finally,
cd ..
mkdir lm_adapted_t5model/torch_ckpt
python convertmodel.py
3. Run code
Remember to change lm_adapted_path and cache_path in script files to the right path on your server!!!
lm_adapted_path: the path of lm_adapted t5 model (Pytorch ckpt)
cache_path: the path of huggingface cache
3.1. NER:
cd NER
bash runall_1.sh
3.2. Classification
cd Classification
bash Classification.sh
3.3. Summarization
cd Summarization
bash Summarization.sh
3.4. Different task types
Without forward knowledge transfer:
cd DiffType/T5NoContinual
bash T5NoContinual.sh
With forward knowledge transfer:
cd DiffType/T5WithContinual
bash T5WithContinual.sh
Citation
If you find our paper or this project helps your research, please kindly consider citing our paper in your publication.
@inproceedings{
qin2022lfpt,
title={{LFPT}5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5},
author={Chengwei Qin and Shafiq Joty},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=HCRVf71PMF}
}