Setting up NVIDIA NIM for chat and reasoning

NVIDIA NIM lets CodeBot use language models through an OpenAI-compatible API. You can run a NIM container on a computer with a supported NVIDIA GPU, or connect CodeBot to NVIDIA's hosted API to try it without NVIDIA hardware of your own.

What You Need

For a local NIM deployment, you need:

  • A supported NVIDIA GPU with enough memory for the model you choose
  • A supported Linux system with Docker and the NVIDIA Container Toolkit
  • An NVIDIA account and the credentials required to download the NIM container and model

The exact hardware and software requirements vary by model. Check NVIDIA's NIM prerequisites and the support matrix for your chosen model before downloading it.

If you only want to test CodeBot's NVIDIA NIM support, you can instead use an NVIDIA-hosted model. That does not require a local NVIDIA GPU.

Short Version

For a local NIM:

  1. Follow NVIDIA's NIM quickstart for the model you want to run.
  2. Wait until the NIM readiness endpoint reports that the service is ready.
  3. Add a Local NVIDIA NIM account in CodeBot.
  4. Keep the default URL http://127.0.0.1:8000/v1 when NIM is running on the same computer.
  5. Leave the API key empty unless your NIM endpoint or gateway requires bearer-token authentication.

For a hosted test without local NVIDIA hardware:

  1. Choose a model in the NVIDIA API Catalog.
  2. Generate an NVIDIA API key for that model.
  3. Add a Local NVIDIA NIM account in CodeBot.
  4. Set the URL to https://integrate.api.nvidia.com/v1.
  5. Paste the NVIDIA API key into API Key / Bearer Token.

Run NVIDIA NIM Locally

NVIDIA distributes NIM as model-specific containers. Use the deployment command shown by NVIDIA for your chosen model rather than copying a command for a different model.

The standard NIM LLM setup exposes port 8000. Once the container has started, check its readiness:

curl http://127.0.0.1:8000/v1/health/ready

Then ask it which model it is serving:

curl http://127.0.0.1:8000/v1/models

The id returned by the models endpoint is the model name CodeBot will discover.

Add NVIDIA NIM to CodeBot

Once the NIM endpoint is ready:

  1. Open the application's settings.
  2. Open the CodeBot settings.
  3. Open Manage Accounts.
  4. Add a Local NVIDIA NIM account.
  5. Enter the NVIDIA NIM URL.
  6. If the endpoint requires authentication, enter its API key or bearer token.
  7. Click Add Account.

For a NIM running on the same computer, the URL is normally:

http://127.0.0.1:8000/v1

The /v1 suffix is required.

Local NIM itself does not normally require an API key for inference. The NGC credential used to download a container or model is separate from CodeBot's optional bearer-token field. Only put a token into CodeBot when the endpoint, proxy or service you are connecting to requires one.

Test Without NVIDIA Hardware

NVIDIA hosts selected NIM models in its API Catalog. These use the same OpenAI-compatible API shape and can be reached at:

https://integrate.api.nvidia.com/v1

Generate an API key from the selected model's page on build.nvidia.com, then use that key in CodeBot's API Key / Bearer Token field.

The hosted catalog is useful for testing, but its available models and usage limits can change. Check the model page for its current availability before relying on it for ongoing work.

Running NIM on Another Computer

You can run NIM on a Linux workstation or server and connect to it from CodeBot on another computer. Replace 127.0.0.1 with the server's hostname or local network address, for example:

http://192.168.1.25:8000/v1

Only expose an unauthenticated NIM endpoint on a trusted private network. For wider access, put it behind a properly secured gateway and enter that gateway's bearer token in the CodeBot account.

Troubleshooting

CodeBot Cannot Connect

Check that:

  • The NIM container is running.
  • http://127.0.0.1:8000/v1/health/ready reports that it is ready.
  • The CodeBot account URL includes /v1.
  • A remote host's firewall allows the configured port.

No Models Appear

Open the models endpoint directly:

curl http://127.0.0.1:8000/v1/models

If it returns no model, inspect the NIM container logs and confirm that the model finished downloading and loading. If it returns a model but CodeBot does not show it, verify that CodeBot is using the same host, port and /v1 path.

Unauthorized or Forbidden

For NVIDIA's hosted endpoint, make sure the NVIDIA API key is in CodeBot's API Key / Bearer Token field. For a local endpoint, leave the field empty unless you deliberately placed NIM behind an authenticated proxy or gateway.

See Also