fix(asap-tools): batch ClickHouse JSON-lines bulk load instead of a single oversized INSERT

[akanksha-akkihal]

Summary

Bulk-loading JSON-lines datasets into ClickHouse previously piped the entire file through a single INSERT ... FORMAT JSONEachRow, which is unreliable for the multi-GB datasets used in the ClickHouse benchmark. This PR loads in bounded batches, waits for ClickHouse to be reachable before loading, and fails with an actionable message when input data is malformed.

Changes

• Batched loader (experiment_utils/services/json/loader.py): streams the
  file and inserts in bounded batches (default 100k rows) via
  docker exec clickhouse-client, instead of one giant INSERT.
• Readiness check: poll ClickHouse HTTP /ping until it returns Ok.
  before loading; on timeout, raise with the tail of the container logs.
• Input validation: each line is checked for valid JSON and embedded null
  bytes, so malformed input fails with the offending line number and a preview.
• MV-safe init: when init_sql_file is provided, it owns DROP/CREATE
  for its own objects — the standalone DROP TABLE no longer runs first and
  break schemas with dependent materialized views.
• Both the ClickBench and custom JSON-lines paths share the batched loader.

[milindsrivastava1997]

Code review findings on this PR:

1. asap-tools/experiments/experiment_utils/services/clickhouse_service.py:317 — The new ClickHouse HTTP readiness wait runs after the DROP TABLE / init-SQL execution, not before it, so it can't protect the very calls it was added to guard. start() runs _exec_sql_file/DROP TABLE IF EXISTS first and only calls _ensure_clickhouse_http_ready(url) afterward. If ClickHouse's HTTP port isn't up yet, the SQL exec calls (which curl the same url) raise immediately, before the readiness wait ever runs. Currently masked because the sole caller already waits via ClickHouseService._wait_for_service_ready() first.

2. asap-tools/experiments/experiment_utils/services/json/loader.py:106 — max_rows no longer bounds how much of the file is read/validated: the early-exit check compares against total, which only advances inside flush() at batch boundaries (default batch_size=100_000), not per line read. With e.g. --max-rows 1000 and the default batch size on a file with >100k lines, the loader reads/validates up to ~100k lines before truncating and stopping — a bad line anywhere in that window aborts the whole load even though only 1000 rows were requested. Sibling h2o/loader.py avoids this by checking the raw per-line index, not a post-flush counter.

3. asap-tools/experiments/experiment_utils/services/clickhouse_service.py:313 (plausible) — The unconditional DROP TABLE IF EXISTS that used to run before every load is now only issued inside the BUILTIN_DDL_FILES branch, so dataset_name="custom" with no init_sql_file no longer drops/resets the table between runs — repeated runs would append instead of starting clean.

4. asap-tools/experiments/experiment_utils/services/clickhouse_service.py:413 — _ensure_clickhouse_http_ready reimplements the same poll-/ping-with-retries loop that DockerServiceBase._wait_for_service_ready / ClickHouseService.is_healthy() already provide, instead of reusing that mechanism. Not a bug, but risks the two copies drifting out of sync.

5. asap-tools/experiments/experiment_utils/services/clickhouse_service.py:439 — The timeout-path log-fetch hardcodes the container name "clickhouse-server" instead of using ClickHouseService.CONTAINER_NAME, unlike _load_json_batched nearby which does it correctly.

Will follow up with fixes for these.