{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Untitled0.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8k6IDrTDGZ5W"
      },
      "source": [
        "## Pytorch 추론시간 단축을 위한 정밀도 감소 - TensorRT / torch2trt library\n",
        "\n",
        "### 환경설정\n",
        "\n",
        "- [Nvidia developer portal](https://developer.nvidia.com/nvidia-tensorrt-7x-download) 회원가입\n",
        "- TensorRT 7.0.0.11 for Ubuntu 1804 and CUDA 10.0 DEB local repo package 다운로드 후 colab 업로드\n",
        "\n",
        "### TensorRT 7.0 을 colab 에 설치하는 방법"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ApojKcRcGf2j"
      },
      "source": [
        "!nvcc --version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZD-bw7YIlHL"
      },
      "source": [
        "### colab CUDA 10.1 삭제"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jJQrjrcfIm8u"
      },
      "source": [
        "# remove CUDA 10.1\n",
        "\n",
        "!sudo apt-get --purge remove cuda nvidia* libnvidia-*\n",
        "!sudo dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge\n",
        "!sudo apt-get remove cuda-*\n",
        "!sudo apt autoremove\n",
        "!sudo apt-get update"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tRhvY4fZIygS"
      },
      "source": [
        "### CUDA 10.0 설치 - 입력 세번 필요 (Y, 31, 1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "73tZNq2WI41u"
      },
      "source": [
        "# Installing CUDA 10.0\n",
        "\n",
        "!wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb\n",
        "!sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb\n",
        "!sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub\n",
        "!sudo apt-get update\n",
        "!wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb\n",
        "!sudo apt install -y ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb\n",
        "!sudo apt-get update\n",
        "\n",
        "# Install NVIDIA driver\n",
        "!sudo apt-get -y installnvidia-driver-418\n",
        "\n",
        "# Install development and runtime libraries (~4GB)\n",
        "!sudo apt-get install -y \\\n",
        "    cuda-10-0 \\\n",
        "    libcudnn7=7.6.2.24-1+cuda10.0  \\\n",
        "    libcudnn7-dev=7.6.2.24-1+cuda10.0 --allow-change-held-packages"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5HAS_SQ7LBgj"
      },
      "source": [
        "!nvcc --version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "emv-dfsgLEE9"
      },
      "source": [
        "### TensorRT 설치 - colab .deb 파일 경로 확인"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YI96JETCLJ3I"
      },
      "source": [
        "# install tensorrt\n",
        "!sudo dpkg -i \"/content/drive/My Drive/capstone1/CAN/nv-tensorrt-repo-ubuntu1804-cuda10.0-trt7.0.0.11-ga-20191216_1-1_amd64.deb\"\n",
        "!sudo apt-key add /var/nv-tensorrt-repo-cuda10.0-trt7.0.0.11-ga-20191216/7fa2af80.pub\n",
        "\n",
        "!sudo apt-get update\n",
        "\n",
        "!sudo apt-get install libnvinfer7=7.0.0-1+cuda10.0 libnvonnxparsers7=7.0.0-1+cuda10.0 libnvparsers7=7.0.0-1+cuda10.0 libnvinfer-plugin7=7.0.0-1+cuda10.0 libnvinfer-dev=7.0.0-1+cuda10.0 libnvonnxparsers-dev=7.0.0-1+cuda10.0 libnvparsers-dev=7.0.0-1+cuda10.0 libnvinfer-plugin-dev=7.0.0-1+cuda10.0 python-libnvinfer=7.0.0-1+cuda10.0 python3-libnvinfer=7.0.0-1+cuda10.0\n",
        "\n",
        "!sudo apt-mark hold libnvinfer7 libnvonnxparsers7 libnvparsers7 libnvinfer-plugin7 libnvinfer-dev libnvonnxparsers-dev libnvparsers-dev libnvinfer-plugin-dev python-libnvinfer python3-libnvinfer\n",
        "\n",
        "!sudo apt-get install tensorrt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DcpWP5snL2yy"
      },
      "source": [
        "### TensorRT 설치 확인"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uOW85nnxLdSB"
      },
      "source": [
        "!dpkg -l | grep TensorRT"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7xHrhZuNb0U"
      },
      "source": [
        "### 런타임 다시시작하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8EQ__AkjMH9G"
      },
      "source": [
        "### [torch2trt](https://github.com/NVIDIA-AI-IOT/torch2trt) 라이브러리 설치"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k7IztAyvMGlP"
      },
      "source": [
        "cd /content/drive/My\\ Drive/capstone1/CAN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qR0V215wMHXF"
      },
      "source": [
        "!pip3 install pycuda\n",
        "!git clone https://github.com/NVIDIA-AI-IOT/torch2trt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fG9UtL6sNAgM"
      },
      "source": [
        "cd /content/drive/My\\ Drive/capstone1/CAN/torch2trt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iMEc08ilNDW8"
      },
      "source": [
        "!python setup.py install"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LpK7xRFdNVbr"
      },
      "source": [
        "import tensorrt\n",
        "import torch2trt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1DDuNL4Npn-"
      },
      "source": [
        "### 추론시간 비교하기"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9WddajL5Pj6E"
      },
      "source": [
        "cd /content/drive/My\\ Drive/capstone1/CAN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PgKGWmSqNr4z"
      },
      "source": [
        "import utils\n",
        "import numpy as np\n",
        "import importlib\n",
        "importlib.reload(utils)\n",
        "\n",
        "utils.run_benchmark('./weights/fed_avg_50_0.9688.pth')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qh7_UaFsRykh"
      },
      "source": [
        "- torch: 464 msc / 96.87 % / 0.05931\n",
        "- trt: 354 msc / 96.87 % / 0.05931\n",
        "- trt float 16: 323 msc / 96.87 % / 0.05931\n",
        "- trt int8 strict: 401 msc / 96.86 % / 06405\n",
        "\n",
        "\n",
        "#### int8 양자화 실패한 이유 추측\n",
        "- input 이 0.0 ~ 1.0 이기 때문에 계산이 전부 float 연산\n",
        "- nvidia tensorRT  가 정확도가 너무 낮아질 경우 양자화 무시\n",
        "- 사용법 미숙, 모델이 너무 간단함\n"
      ]
    }
  ]
}