TextSynth uses custom inference code to get faster inference on GPUs and CPUs. It has the following characteristics:
- Supports many Transformer variants (GPT-J, GPT-NeoX, GPT-Neo, OPT, Fairseq GPT, M2M100, CodeGen, GPT2) and Stable Diffusion.
- Integrated REST JSON API for text completion, translation and image generation.
- Very high performance for small and large batches thanks
to custom cuda kernels and efficient custom 8 bit and 4 bit
quantization (see
the benchmarks).
- Larger models work optimally on lower cost GPUs (e.g. RTX
3090, RTX A6000) thanks to efficient quantization.
- All is included in a single binary. No external dependencies (Python is not needed) so installation is very easy on most Linux distributions.
- Uses the LibNC
library for simple tensor manipulation using the C
language.
A CPU-only version
is freely available.
Benchmarks:
Performance using the GPT-Neox 20B model on a RTX A6000 Nvidia
GPU. For the speed measurement, 200 tokens are generated using a
batch size of 1:
| Precision |
LAMBADA (ppl) |
LAMBADA (acc) |
Max GPU memory (GB) |
Speed (tokens/s) |
| float16
| 3.66
| 72.6%
| 40.7
| 15
|
| 8 bits
| 3.66
| 72.6%
| 21.7
| 27
|
| 4 bits
| 3.71
| 72.0%
| 11.6
| 41
|
Performance using the Stable diffusion 1.4 model on a RTX A6000
Nvidia GPU. For the speed measurement, a single image is
generated using 50 timesteps and a batch size of 1:
| Precision |
Max GPU memory (GB) |
Generation time (s) |
| float16
| 2.8
| 1.90
|