unable to generate response on multi gpu.

[ashunaveed]

while generating response, got following error: 2025-01-12 19:21:15,471 - INFO - main - Documents indexed successfully.
127.0.0.1 - - [12/Jan/2025 19:21:15] "POST /chat HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:21:21] "GET /new_session HTTP/1.1" 302 -
127.0.0.1 - - [12/Jan/2025 19:21:21] "GET /chat HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:21:21] "GET /static/css/styles.css HTTP/1.1" 304 -
127.0.0.1 - - [12/Jan/2025 19:21:23] "GET /switch_session/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b HTTP/1.1" 302 -
127.0.0.1 - - [12/Jan/2025 19:21:23] "GET /chat HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:21:23] "GET /static/css/styles.css HTTP/1.1" 304 -
127.0.0.1 - - [12/Jan/2025 19:21:28] "GET /get_indexed_files/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:21:32] "GET /switch_session/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b HTTP/1.1" 302 -
127.0.0.1 - - [12/Jan/2025 19:21:32] "GET /chat HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:21:32] "GET /static/css/styles.css HTTP/1.1" 304 -
127.0.0.1 - - [12/Jan/2025 19:21:47] "POST /rename_session HTTP/1.1" 200 -
2025-01-12 19:21:57,148 - INFO - models.retriever - Retrieving documents for query: what are the technical elegibility criteria for llp companies
2025-01-12 19:21:57,283 - INFO - models.retriever - Added image to list: images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_cc27c01785f684942697aea07701cfde.png
2025-01-12 19:21:57,388 - INFO - models.retriever - Added image to list: images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_ac356a1c131cf82f86e9c17138d8d27f.png
2025-01-12 19:21:57,498 - INFO - models.retriever - Added image to list: images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_0f80b130432a76157b9055a61a3411ad.png
2025-01-12 19:21:57,498 - INFO - models.retriever - Total 3 documents retrieved. Image paths: ['images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_cc27c01785f684942697aea07701cfde.png', 'images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_ac356a1c131cf82f86e9c17138d8d27f.png', 'images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_0f80b130432a76157b9055a61a3411ad.png']
2025-01-12 19:21:57,498 - INFO - main - Retrieved images: ['images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_cc27c01785f684942697aea07701cfde.png', 'images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_ac356a1c131cf82f86e9c17138d8d27f.png', 'images/2d76bc4f-cad3-42ee-b3b4-c269dbcba02b/retrieved_0f80b130432a76157b9055a61a3411ad.png']
2025-01-12 19:21:57,498 - INFO - models.responder - Generating response using model 'qwen'.
Downloading shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.59it/s]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00, 1.78s/it]
You shouldn't move a model that is dispatched using accelerate hooks.
2025-01-12 19:22:17,998 - INFO - models.model_loader - Qwen model loaded and cached.
2025-01-12 19:22:18,255 - ERROR - models.responder - Error generating response: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)
127.0.0.1 - - [12/Jan/2025 19:22:18] "POST /chat HTTP/1.1" 200 -
127.0.0.1 - - [12/Jan/2025 19:22:18] "POST /rename_session HTTP/1.1" 200 -

[wydnas]

I am seeing similar to this - I am using Ubuntu 22.04, miniforge3, python 3.10, RTX 3090 (24GB VRAM) - even when I only have 1 document indexed (not sure why it is reporting 3) I am hitting VRAM memory limits. The error is given below and is very similar to above.

2025-02-12 15:35:22,002 - INFO - main - Application started.

• Serving Flask app 'app'
• Debug mode: on
  WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
• Running on http://127.0.0.1:5050
  Press CTRL+C to quit
• Restarting with stat
  INFO 02-12 15:35:25 init.py:183] Automatically detected platform cuda.
  2025-02-12 15:35:26,599 - INFO - main - Application started.
• Debugger is active!
• Debugger PIN: 108-226-565
  Verbosity is set to 1 (active). Pass verbose=0 to make quieter.
  Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.25it/s]
  You are using in-memory collection. This means every image is stored in memory.
  You might want to rethink this if you have a large collection!
  Loaded 29 images from 1 JSON files.
  /home/sandy/miniforge3/envs/lGPTvis_0/lib/python3.10/site-packages/byaldi/colpali.py:162: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  self.indexed_embeddings.extend(torch.load(file))
  2025-02-12 15:35:55,189 - INFO - main - RAG model for session 850dcae3-9189-4c80-83a7-65a57032d7b7 loaded from index.
  2025-02-12 15:35:55,189 - INFO - main - Application initialized and indexes loaded.
  127.0.0.1 - - [12/Feb/2025 15:35:55] "GET / HTTP/1.1" 302 -
  127.0.0.1 - - [12/Feb/2025 15:35:55] "GET /chat HTTP/1.1" 200 -
  127.0.0.1 - - [12/Feb/2025 15:35:55] "GET /static/css/styles.css HTTP/1.1" 304 -
  127.0.0.1 - - [12/Feb/2025 15:35:55] "GET /favicon.ico HTTP/1.1" 404 -
  2025-02-12 15:36:09,351 - INFO - models.retriever - Retrieving documents for query: Do you have any information on impact breccias?
  2025-02-12 15:36:09,776 - INFO - models.retriever - Added image to list: images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_b779c00c35c4254605573ea46824eb3b.png
  2025-02-12 15:36:09,780 - INFO - models.retriever - Added image to list: images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_d2cfe8841e031d05a12cc0fb5c83759a.png
  2025-02-12 15:36:09,783 - INFO - models.retriever - Added image to list: images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_e3904b4e4703a39fc518c7d677f30664.png
  2025-02-12 15:36:09,783 - INFO - models.retriever - Total 3 documents retrieved. Image paths: ['images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_b779c00c35c4254605573ea46824eb3b.png', 'images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_d2cfe8841e031d05a12cc0fb5c83759a.png', 'images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_e3904b4e4703a39fc518c7d677f30664.png']
  2025-02-12 15:36:09,783 - INFO - main - Retrieved images: ['images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_b779c00c35c4254605573ea46824eb3b.png', 'images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_d2cfe8841e031d05a12cc0fb5c83759a.png', 'images/850dcae3-9189-4c80-83a7-65a57032d7b7/retrieved_e3904b4e4703a39fc518c7d677f30664.png']
  2025-02-12 15:36:09,783 - INFO - models.responder - Generating response using model 'qwen'.
  Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████| 5/5 [00:05<00:00, 1.14s/it]
  Some parameters are on the meta device because they were offloaded to the cpu.
  Using a slow image processor as use_fast is unset and a slow processor was saved with this model. use_fast=True will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with use_fast=False.
  You shouldn't move a model that is dispatched using accelerate hooks.
  2025-02-12 15:36:17,899 - ERROR - models.responder - Error generating response: You can't move a model that has some modules offloaded to cpu or disk.
  127.0.0.1 - - [12/Feb/2025 15:36:17] "POST /chat HTTP/1.1" 200 -

[wydnas]

OK the VRAM part might be a red herring as I changed the model from qwen to llama-vision and had the same error but this time VRAM was only sitting at 38.4%.... so that does not look like the root cause.