Fix/copilot concurrency limit

[ApfelPresse]

Why are the changes needed?

Before this change multipart upload/download could start one goroutine per file.
With many small files, this created too much parallel work, drove CPU usage up, and in some cases caused jobs to fail.

This PR adds a configurable concurrency limit, so the number of parallel transfers is capped (GOMAXPROCS * concurrency-per-cpu, with a maximum limit).
That makes behavior more stable in Kubernetes/container environments while still allowing tuning for throughput.

What changes were proposed in this pull request?

1. Added bounded concurrency control for upload and download flows:
   • New CLI flag: --concurrency-per-cpu (default: data.DefaultConcurrencyPerCPU)
   • Effective worker cap: GOMAXPROCS * concurrency-per-cpu, bounded by MaxConcurrency
   • Applied to both download and upload scheduling paths via semaphore-based limiting
2. Improved HTTP download behavior:
   • DownloadFileFromHTTP now returns an error for non-2xx responses
   • Added logging for HTTP download start/status

How was this patch tested?

1. Added unit tests for:
   • sanitizeConcurrencyPerCPU
   • getConcurrency
   • acquireSemaphore (including context cancellation)
2. Updated existing uploader tests for the new constructor parameter.
3. Ran this fix branch in our environment for several days with real workloads (including many small files). CPU load stayed stable and we did not see regressions.

Labels

fixed
changed

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Docs link

Flytecopilot readme.md file

[codecov]

Codecov Report

❌ Patch coverage is 78.18182% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.33%. Comparing base (6f35206) to head (ecde474).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
flytecopilot/data/download.go	76.92%	5 Missing and 4 partials ⚠️
flytecopilot/data/upload.go	75.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7730      +/-   ##
==========================================
+ Coverage   56.50%   57.33%   +0.82%     
==========================================
  Files         772      931     +159     
  Lines       52285    58361    +6076     
==========================================
+ Hits        29546    33463    +3917     
- Misses      19974    21837    +1863     
- Partials     2765     3061     +296     

Flag	Coverage Δ
unittests-datacatalog	53.62% <ø> (ø)
unittests-flyteadmin	53.24% <ø> (ø)
unittests-flytecopilot	52.70% <78.18%> (+4.64%)	⬆️
unittests-flytectl	64.11% <ø> (?)
unittests-flyteidl	76.63% <ø> (ø)
unittests-flyteplugins	60.45% <ø> (ø)
unittests-flytepropeller	53.84% <ø> (ø)
unittests-flytestdlib	64.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pingsutw]

Is it ready for review?

[ApfelPresse]

Is it ready for review?

Thanks for checking in! I havent had much time to work on this recently. The code itself is pretty much finished but I still wanted to add a few more tests to improve the coverage since the codecov report was still failing.
I will try to get that sorted out and move it out of draft once its ready for review.