Curl multi check in - #1430
Curl multi check in#1430
Conversation
… the exception handling
…add code to check the time taken with the parallel data transfer.
… one request at a time. This can make the parallel data transfer work.
…o handle the parallel transfer.
…data transfer. Compiled only.
…e the log file is too big.
jgallagher59701
left a comment
jgallagher59701
left a comment
There was a problem hiding this comment.
Most of my requests are for comments since this is such a complex and important part of the server, I think it needs to be a bit easier to understand.
| memcpy(target_buffer, source_buffer, the_one_chunk->get_size()); | ||
| } | ||
|
|
||
| #if 0 |
There was a problem hiding this comment.
Lets remove this if it is not used. If we're saving it to use later with the compute threads, lets add a comment about that.
| } | ||
| #endif | ||
|
|
||
| struct CurlMultiTransfer { |
There was a problem hiding this comment.
I realized this is pretty complicated - when I was reading the code it took me a while to understand that 'CurlMultiTransfer' is ours and not libcurl's! Anyway, I think I see how this is used, but I think some explanation is needed. The 'super_chunk' is transferred by the 'dmrpp_easy_handle.' The dmrpp_easy_handle unique_ptr has a custom deleter (that's worth pointing out).
The 'super_chunk_internal' is there to actually read the data.
Also, I think 're_try' should be 'retry.'
|
|
|
2 participants
Description
Reference ticket: HYRAX-2265
Use curl-multi APIs to manage the parallel data transfer.
Tasks