FilesExpand file tree

 main

/

fastapi_server.py

Copy path

Blame

More file actions

Blame

More file actions

Latest commit

 

History

History

History

37 lines (25 loc) · 653 Bytes

 main

/

fastapi_server.py

Copy path

Top

File metadata and controls

• Code
• Blame

37 lines (25 loc) · 653 Bytes

Raw

Copy raw file

Download raw file

Open symbols panel

Edit and raw actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

"""Example FastAPI server for llama.cpp.

To run this example:

```bash

pip install fastapi uvicorn sse-starlette

export MODEL=../models/7B/...

```

Then run:

```

uvicorn --factory llama_cpp.server.app:create_app --reload

```

or

```

python3 -m llama_cpp.server

```

Then visit http://localhost:8000/docs to see the interactive API docs.

To actually see the implementation of the server, see llama_cpp/server/app.py

"""

import os

import uvicorn

from llama_cpp.server.app import create_app

if __name__ == "__main__":

app = create_app()

uvicorn.run(

app, host=os.getenv("HOST", "localhost"), port=int(os.getenv("PORT", 8000))

)