speeding up fetch

AGENTS.md

@@ -21,8 +21,10 @@ decodes the JWT token for user ID, queries the GraphQL API for workout data, and
 - **API Structure**: WeightXReps uses GraphQL at `/api/graphql` for all data operations. Authentication via JWT tokens in Authorization headers.
 - **Authentication**: Login mutation `login(u: $u, p: $p)` returns a JWT token containing user ID in the `id` field.
 - **Data Retrieval**:
-  - `JDay` query fetches workouts by user ID and date (YMD format: YYYY-MM-DD). Includes structured data like eblocks (exercise blocks), sets, exercises, and a pre-formatted log.
-  - `jrange` query fetches a range of workout days around a given date, with configurable count (max 32). Returns days with workouts in the range.
+  - `JDay` query fetches workouts by user ID and date (YMD format: YYYY-MM-DD). Includes structured data like eblocks (exercise blocks), sets, exercises, and a pre-formatted log (comments live here).
+  - `jrange` query fetches range data between `ymd - range*7` and `ymd`. **`range` is in weeks, max 32** (32 weeks ≈ 224 days), not a workout-day count. Returns days with workouts in the window. `jrange` includes eblocks/sets/exercises but **not** the `log` field, so it cannot replace `jday` for cache-accurate fetches.
+  - GraphQL aliases work: one request can fetch many `jday` selections (`d0: jday(...) { ... } d1: jday(...) { ... }`).
+  - `downloadLogs` returns all of the current user's logs as `JEditorData` in one request (used by the website export). Not used by fetch because converting JEditorData can diverge from `jday` cache format.
 - **Data Formatting**: Workout logs are formatted text with #exercise prefixes and compressed set notations (e.g., 135x5x3 or 445x1,3). Formatting logic found in client code.
 - **Color Scheme**: Website editor uses specific RGB colors for syntax highlighting:
   - Date: #9D4EDD (157,78,221)
@@ -31,7 +33,7 @@ decodes the JWT token for user ID, queries the GraphQL API for workout data, and
   - Weights: #FF7900 (255,121,0)
   - Reps: #00BBF9 (0,187,249)
   - Sets: #F15BB5 (241,91,181)
-- **Performance**: Reuse HTTP client across requests to maintain connection pooling and avoid TCP overhead. Implemented concurrent API fetching for multiple workouts and session-level caching of user preferences to reduce latency and redundant calls.
+- **Performance**: Reuse HTTP client across requests to maintain connection pooling and avoid TCP overhead. Session-level caching of user preferences. Fetch packs up to 10 `jday` queries per GraphQL request (aliases) and runs 8 requests concurrently (`JDAY_BATCH_SIZE=10`, `FETCH_CONCURRENCY=8`). Sequential `jday` fetching of 152 workouts took ~19s; batched+concurrent takes ~1.2s for the workout downloads (~3.4s including date listing/auth). HTTP client uses a larger idle pool, TCP_NODELAY, and gzip.
 - **Rust Implementation**: Used reqwest for HTTP with client reuse, serde for JSON, base64 for JWT decoding, ansi_term for colors, atty for TTY detection. Handled GraphQL responses, error checking, and inline color application during text generation.
 
 ## Referenced Links
@@ -72,7 +74,7 @@ You can look in `weightxreps-client/src/data/generated---db-types-and-hooks.tsx`
 - Ensures ordered output in list commands by buffering concurrent async requests to maintain sequence.
 - Local caching of structured workout data in XDG_CACHE_HOME/wxrust/{uid}/yyyy-mm-dd.txt for offline access and performance.
 - Cache format is text, same as what is formatted for output using `format_workout_for_cache`
-- Bulk fetch command to download workouts from server into cache, with options for diff, force, and file import
+- Bulk fetch command to download workouts from server into cache, with options for diff, force, file import, and `--stats` (prints `X workouts, Y seconds, x.y T/s, x.y MB/s` after the progress line). Fetch skips already-cached dates by checking file existence (no parse). Network fetches use batched concurrent `jday` queries. HTTP response body sizes are counted in `api::transfer_stats`.
 - Progress bars for long-running operations using indicatif
 - Side-by-side diff display using similar crate for comparing local and server workouts
 - Parsing and formatting support for RPE (@ syntax), BW exercises (BW, BW+, BW-), lb/kg units
@@ -85,7 +87,8 @@ You can look in `weightxreps-client/src/data/generated---db-types-and-hooks.tsx`
 
 ## Dependencies Added
 
-- `reqwest` (0.12) with JSON features for HTTP requests and client reuse.
+- `reqwest` (0.12) with JSON and gzip features for HTTP requests and client reuse.
+- `futures` (0.3) for concurrent batched GraphQL streams (`buffer_unordered`).
 - `serde` (1.0) with derive for JSON serialization/deserialization.
 - `base64` (0.21) for JWT payload decoding.
 - `tokio` (1) for async runtime.
@@ -196,6 +199,8 @@ Unlike the C version which shows separate tables per filter, the Rust implementa
 
 ## Future Improvements
 
+- Parallelize `get_dates` jrange pagination (currently sequential week-windows).
+- Optionally use `downloadLogs` for full-history `fetch` with no date filter.
 - Add support for year/month range queries.
 - Add export options (JSON, CSV).
 - Support for user profile and goals queries.

Cargo.toml

@@ -11,8 +11,9 @@ name = "wxrust"
 path = "src/lib.rs"
 
 [dependencies]
-reqwest = { version = "0.12", features = ["json"] }
+reqwest = { version = "0.12", features = ["json", "gzip"] }
 tokio = { version = "1", features = ["full"] }
+futures = "0.3"
 serde_json = "1.0"
 serde = { version = "1.0", features = ["derive"] }
 base64 = "0.21"

README.md

@@ -18,7 +18,7 @@ A Rust CLI tool to extract and display workouts from the WeightXReps.net website
 - Supports detailed views, summaries, and date filtering
 - Handles JWT tokens and GraphQL queries efficiently
 - Caches workout data locally in XDG cache directory for improved performance
-- Fetches and caches workouts in bulk with progress indication
+- Fetches and caches workouts in bulk with progress indication, using batched concurrent GraphQL requests
 - Compares local cache with server versions
 - Imports workouts from text export files
 
@@ -91,6 +91,9 @@ Create a `credentials.txt` file with your WeightXReps account email on the first
 - Show diff between local and server: `wxrust fetch --diff 2025-10`
 - Force re-download: `wxrust fetch --force 2025`
 - Import from text export file: `wxrust fetch --file export.txt`
+- Print transfer rate after fetch: `wxrust fetch --stats 2026`
+
+`fetch` downloads workouts concurrently (10 dates per GraphQL request, 8 requests in flight) and skips dates that are already cached unless `--force` is used. `--stats` prints `X workouts, Y seconds, x.y T/s, x.y MB/s` after the progress line.
 
 #### Table Command (PR Progression)
 

src/api.rs

@@ -2,6 +2,28 @@ use async_trait::async_trait;
 use serde::de::DeserializeOwned;
 use ansi_term::Colour;
 use tokio::sync::OnceCell;
+use std::sync::atomic::{AtomicU64, Ordering};
+
+static TRANSFER_BYTES: AtomicU64 = AtomicU64::new(0);
+static TRANSFER_REQUESTS: AtomicU64 = AtomicU64::new(0);
+
+pub fn reset_transfer_stats() {
+    TRANSFER_BYTES.store(0, Ordering::Relaxed);
+    TRANSFER_REQUESTS.store(0, Ordering::Relaxed);
+}
+
+/// Returns `(requests, bytes)` recorded since the last reset.
+pub fn transfer_stats() -> (u64, u64) {
+    (
+        TRANSFER_REQUESTS.load(Ordering::Relaxed),
+        TRANSFER_BYTES.load(Ordering::Relaxed),
+    )
+}
+
+fn record_transfer(bytes: usize) {
+    TRANSFER_BYTES.fetch_add(bytes as u64, Ordering::Relaxed);
+    TRANSFER_REQUESTS.fetch_add(1, Ordering::Relaxed);
+}
 
 use crate::models::{GraphQLRequest, GraphQLResponse, WorkoutRequest, WorkoutResponse, UserBasicInfoData, User};
 use crate::formatters::STDERR_COLOR_ENABLED;
@@ -67,7 +89,11 @@ pub struct ReqwestClient {
 impl ReqwestClient {
     pub fn new_with_verbose(verbose: bool) -> Self {
         ReqwestClient {
-            client: reqwest::Client::new(),
+            client: reqwest::Client::builder()
+                .pool_max_idle_per_host(32)
+                .tcp_nodelay(true)
+                .build()
+                .expect("failed to build HTTP client"),
             verbose,
             user_info: OnceCell::new(),
         }
@@ -86,6 +112,7 @@ impl ApiClient for ReqwestClient {
             .await?;
         let status = response.status();
         let text = response.text().await?;
+        record_transfer(text.len());
         log_verbose_response(&text, status, self.verbose);
         let body: GraphQLResponse<crate::models::LoginData> = serde_json::from_str(&text)?;
         Ok(body)
@@ -106,6 +133,7 @@ impl ApiClient for ReqwestClient {
             .await?;
         let status = response.status();
         let text = response.text().await?;
+        record_transfer(text.len());
         log_verbose_response(&text, status, self.verbose);
         let body: GraphQLResponse<T> = serde_json::from_str(&text)?;
         Ok(body)

src/fetch.rs

@@ -1,19 +1,39 @@
-use crate::api::ReqwestClient;
+use crate::api::ApiClient;
 use crate::formatters;
 use crate::models;
 use crate::parsers;
 use crate::utils;
 use crate::workouts;
 use regex::Regex;
 use std::fs;
+use std::time::Instant;
+
+pub fn format_transfer_stats(transfers: u64, bytes: u64, elapsed_secs: f64) -> String {
+    let (tps, mbps) = if elapsed_secs > 0.0 {
+        (
+            transfers as f64 / elapsed_secs,
+            (bytes as f64 / 1_000_000.0) / elapsed_secs,
+        )
+    } else {
+        (0.0, 0.0)
+    };
+    format!(
+        "{} workouts, {:.2} seconds, {:.1} T/s, {:.2} MB/s",
+        transfers,
+        elapsed_secs.max(0.0),
+        tps,
+        mbps
+    )
+}
 
-pub async fn fetch_command(
-    data_access: &crate::api::DataAccess<'_, ReqwestClient>,
+pub async fn fetch_command<C: ApiClient>(
+    data_access: &crate::api::DataAccess<'_, C>,
     dates: &[String],
     diff: bool,
     force: bool,
     file: Option<&str>,
     verbose: bool,
+    stats: bool,
 ) -> Result<(), String> {
     let uid = data_access.uid.ok_or("No user ID available")?;
 
@@ -28,35 +48,102 @@ pub async fn fetch_command(
         return Ok(());
     }
 
+    if diff {
+        return fetch_diff(data_access, uid, &dates_to_fetch, verbose, stats).await;
+    }
+
     let pb = utils::create_progress_bar(dates_to_fetch.len() as u64);
 
+    let mut need = Vec::new();
     for date in &dates_to_fetch {
-        pb.set_message(format!("Fetching {}", date));
-
-        if diff {
-            let server_jday = workouts::get_jday(data_access, date, verbose).await?;
-            let local_jday = workouts::lookup_cached_jday(uid, date, verbose);
-
-            if let Some(local) = local_jday {
-                show_diff(date, &local, &server_jday);
-            } else {
-                println!("{}: No local cache, server version:", date);
-                let user_wants_kg = workouts::resolve_user_wants_kg(data_access).await;
-                let workout = formatters::format_workout(date, &server_jday, user_wants_kg);
-                print!("{}", workout);
-            }
-        } else if !force && workouts::lookup_cached_jday(uid, date, verbose).is_some() {
+        if !force && workouts::cached_jday_exists(uid, date) {
             pb.println(format!("{} already cached, skipping", date));
+            pb.inc(1);
         } else {
-            let jday = workouts::get_jday(data_access, date, verbose).await?;
-            workouts::write_cached_jday(uid, date, &jday);
+            need.push(date.clone());
         }
+    }
 
-        pb.inc(1);
+    let mut elapsed_secs = 0.0;
+    let mut bytes = 0u64;
+    if !need.is_empty() {
+        crate::api::reset_transfer_stats();
+        let start = Instant::now();
+        // --force must hit the network even when cache files already exist.
+        let fetch_access = without_cache(data_access, force);
+        workouts::get_jdays_with_callback(
+            &fetch_access,
+            &need,
+            workouts::JDAY_BATCH_SIZE,
+            workouts::FETCH_CONCURRENCY,
+            verbose,
+            |date, jday| {
+                workouts::write_cached_jday(uid, date, jday);
+                pb.set_message(format!("Fetching {}", date));
+                pb.inc(1);
+            },
+        )
+        .await?;
+        elapsed_secs = start.elapsed().as_secs_f64();
+        let (_requests, received) = crate::api::transfer_stats();
+        bytes = received;
+        if verbose {
+            eprintln!(
+                "Fetched {} workouts in {:.2}s ({} per request, concurrency {})",
+                need.len(),
+                elapsed_secs,
+                workouts::JDAY_BATCH_SIZE,
+                workouts::FETCH_CONCURRENCY
+            );
+        }
     }
 
     pb.finish_with_message("Done");
 
+    if stats {
+        println!("{}", format_transfer_stats(need.len() as u64, bytes, elapsed_secs));
+    }
+
+    Ok(())
+}
+
+async fn fetch_diff<C: ApiClient>(
+    data_access: &crate::api::DataAccess<'_, C>,
+    uid: u32,
+    dates: &[String],
+    verbose: bool,
+    stats: bool,
+) -> Result<(), String> {
+    let pb = utils::create_progress_bar(dates.len() as u64);
+    pb.set_message("Fetching");
+    let user_wants_kg = workouts::resolve_user_wants_kg(data_access).await;
+
+    crate::api::reset_transfer_stats();
+    let start = Instant::now();
+    // Diff compares against the server, so skip local cache for the download.
+    let fetch_access = without_cache(data_access, true);
+    let fetched = workouts::get_jdays(&fetch_access, dates, verbose).await?;
+    let elapsed_secs = start.elapsed().as_secs_f64();
+    let (_requests, bytes) = crate::api::transfer_stats();
+    pb.inc(dates.len() as u64);
+
+    pb.finish_and_clear();
+
+    if stats {
+        println!("{}", format_transfer_stats(fetched.len() as u64, bytes, elapsed_secs));
+    }
+
+    for (date, server_jday) in fetched {
+        let local_jday = workouts::lookup_cached_jday(uid, &date, verbose);
+        if let Some(local) = local_jday {
+            show_diff(&date, &local, &server_jday);
+        } else {
+            println!("{}: No local cache, server version:", date);
+            let workout = formatters::format_workout(&date, &server_jday, user_wants_kg);
+            print!("{}", workout);
+        }
+    }
+
     Ok(())
 }
 
@@ -112,8 +199,22 @@ fn parse_file_export(content: &str) -> Result<Vec<(String, models::JDay)>, Strin
     Ok(workouts_list)
 }
 
-async fn get_dates_to_fetch(
-    data_access: &crate::api::DataAccess<'_, ReqwestClient>,
+fn without_cache<'a, C: ApiClient>(
+    data_access: &crate::api::DataAccess<'a, C>,
+    skip_cache: bool,
+) -> crate::api::DataAccess<'a, C> {
+    crate::api::DataAccess {
+        client: data_access.client,
+        token: data_access.token,
+        uid: data_access.uid,
+        use_network: data_access.use_network,
+        use_cache: data_access.use_cache && !skip_cache,
+        write_cache: data_access.write_cache,
+    }
+}
+
+async fn get_dates_to_fetch<C: ApiClient>(
+    data_access: &crate::api::DataAccess<'_, C>,
     dates: &[String],
 ) -> Result<Vec<String>, String> {
     if dates.is_empty() {

src/main.rs

@@ -134,6 +134,9 @@ struct FetchArgs {
     #[arg(long, value_name = "FILE")]
     file: Option<String>,
 
+    #[arg(long, help = "Print transfer rate (T/s, MB/s) after fetch")]
+    stats: bool,
+
     dates: Vec<String>,
 }
 
@@ -225,6 +228,7 @@ async fn handle_fetch(
         fetch_args.force,
         fetch_args.file.as_deref(),
         verbose,
+        fetch_args.stats,
     ).await {
         utils::exit_with_error(e);
     }

src/models.rs

@@ -67,6 +67,9 @@ pub struct WorkoutData {
     pub jday: Option<JDay>,
 }
 
+/// GraphQL alias map used by batched `jday` queries (`d0`, `d1`, ...).
+pub type BatchJDayData = std::collections::HashMap<String, Option<JDay>>;
+
 #[derive(Deserialize, Serialize, Debug, Clone)]
 #[allow(dead_code)]
 pub struct JDay {