TensorRT Laboratory

The TensorRT Laboratory (trtlab) is a general purpose set of tools to build customer inference applications
and services.

Triton is a professional grade production inference server.

This project is broken into 4 primary components:

• memory is based on foonathan/memory the memory module
  was designed to write custom allocators for both host and gpu memory. Several custom allocators are
  included.

• core contains host/cpu-side tools for common components such as thread pools, resource pool,
  and userspace threading based on boost fibers.

• cuda extends memory with a new memory_type for CUDA device memory. All custom allocators
  in memory can be used with device_memory, device_managed_memory or host_pinned_memory.

• nvrpc is an abstraction layer for building asynchronous microservices. The current implementation
  is based on grpc.

• tensorrt provides an opinionated runtime built on the TensorRT API.

Quickstart

The easiest way to manage the external NVIDIA dependencies is to leverage the containers hosted on
NGC. For bare metal installs, use the Dockerfile as a template for
which NVIDIA libraries to install.

docker build -t trtlab . 

For development purposes, the following set of commands first builds the base image, then
maps the source code on the host into a running container.

docker build -t trtlab:dev --target base .
docker run --rm -ti --gpus=all -v $PWD:/work --workdir=/work --net=host trtlab:dev bash

Copyright and License

This project is released under the BSD 3-clause license.

Issues and Contributing

• Please let us know by filing a new issue
• You can contribute by opening a pull request

Pull requests with changes of 10 lines or more will require a Contributor License Agreement.