AutoTP is a project for SungKyun Kim designed to automatically convert a single GPU huggingface model to a tensor-parallel model.
To get started with AutoTP, you'll need to install the necessary dependencies. Follow these steps:
-
Clone the repository:
git clone https://github.com/yourusername/AutoTP.git cd AutoTP -
Install the package:
pip install . -
Install the required dependencies:
pip install -r requirements.txt
-
To run single GPU test, run the following command:
cd benchmarks python test_single.sh -
To run distributed GPU test, run the following command:
cd benchmarks python test_dist.sh