Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeepSeek-Coder-V2-Lite stuck #3212

Open
m0wer opened this issue Sep 27, 2024 · 2 comments
Open

DeepSeek-Coder-V2-Lite stuck #3212

m0wer opened this issue Sep 27, 2024 · 2 comments
Labels
bug-unconfirmed

Comments

@m0wer
Copy link

m0wer commented Sep 27, 2024

Describe the bug
DeepSeek-Coder-V2-Lite is stuck at starting.

The web server does not even satrt after 10,000 s of “Starting“. Other models work fine. If within the container I kill the llama-server process and start it manually without --disable-log, it can work and the web server starts and provides completions.

If I then kill the manually started ollama-server process, the default one is able to spawn and be loaded to VRAM but it just doesn't reply to requests.

Information about your version

Happens with 0.16.1, 0.17.0 and 0.18.0-rc4.

Information about your GPU

NVIDIA GeForce RTX 3090

Additional context

Running a fresh instance with docker compose:

services:
  tabby:
    image: tabbyml/tabby:latest
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '5'
          memory: 30G
        reservations:
          devices:
            - driver: nvidia
              #count: all
              device_ids: ['1']
              capabilities:
                - gpu
    labels:
      traefik.enable: true
      traefik.http.routers.tabby.rule: Host(`domain`)
      traefik.http.routers.tabby.entrypoints: websecure
      traefik.http.services.tabby.loadbalancer.server.port: 8080
    volumes:
      - /data/tabby/data/:/data
    command: ["serve", "--model", "DeepSeek-Coder-V2-Lite", "--device", "cuda"]
    environment:
      RUST_LOG: debug

There's nothing relevant in the logs, just request logs but no errors.
@wsxiaoys
Copy link
Member

Hi - can you share the output of nvidia-smi when starting tabby? (or ollama with the model)

@m0wer
Copy link
Author

m0wer commented Sep 28, 2024

Sure!

Inside the container looks empty:

# docker exec -it tabby-tabby-1 nvidia-smi
Sat Sep 28 09:35:34 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:07:00.0 Off |                  N/A |
| 56%   58C    P2             129W / 280W |  14269MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Outside there are other unrelated stuff in the 2060 and the tabby processes in the 3090:

Sat Sep 28 11:36:14 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2060        On  | 00000000:06:00.0 Off |                  N/A |
| 34%   46C    P2              37W / 128W |   1819MiB /  6144MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  | 00000000:07:00.0 Off |                  N/A |
| 56%   58C    P2             129W / 280W |  14556MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    144128      C   frigate.detector.tensorrt                   246MiB |
|    0   N/A  N/A    144560      C   ffmpeg                                      182MiB |
|    0   N/A  N/A   1832724      C   ffmpeg                                      182MiB |
|    0   N/A  N/A   1900156      C   ffmpeg                                      156MiB |
|    0   N/A  N/A   1900946      C   ffmpeg                                      156MiB |
|    0   N/A  N/A   2774390      C   ffmpeg                                      182MiB |
|    0   N/A  N/A   2807270      C   ffmpeg                                      182MiB |
|    0   N/A  N/A   3208761      C   ffmpeg                                      182MiB |
|    0   N/A  N/A   3873962      C   ffmpeg                                      172MiB |
|    0   N/A  N/A   4073311      C   ffmpeg                                      172MiB |
|    1   N/A  N/A    154641      C   /opt/tabby/bin/llama-server                 284MiB |
|    1   N/A  N/A   2544258      C   ...unners/cuda_v12/ollama_llama_server    13442MiB |
|    1   N/A  N/A   4082688      C   /opt/tabby/bin/llama-server                 818MiB |
+---------------------------------------------------------------------------------------+

In the container logs with debugging enabled:

2024-09-28T09:37:05.162980Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.162988Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163051Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163058Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163106Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163111Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163164Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163170Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163229Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163237Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163303Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163311Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163370Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163375Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163427Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163433Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163489Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163495Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163550Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163557Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163617Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163624Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163690Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163698Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163760Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163767Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163831Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163840Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163897Z DEBUG reqwest::connect: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://127.0.0.1:30889/
2024-09-28T09:37:05.163902Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 127.0.0.1:30889
2024-09-28T09:37:05.163957Z DEBUG hyper_util::client::legacy::connect::http: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:634: connected to 127.0.0.1:30889

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug-unconfirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsxiaoys @m0wer