fix: resolve catalog pagination limits and add GHCR CI/CD#2592
Open
kelvinzer0 wants to merge 19 commits into
Open
fix: resolve catalog pagination limits and add GHCR CI/CD#2592kelvinzer0 wants to merge 19 commits into
kelvinzer0 wants to merge 19 commits into
Conversation
- Increase default catalog limit from 10 to 50 products - Increase pagination loops from 4 to 20 for larger catalogs - Remove arbitrary 20-item cap on getCollections (now default 100) - Add cursor parameter to validation schemas for manual pagination - Add cursor support to getCollectionsDto - Add GitHub Actions workflow for GHCR publishing Fixes evolution-foundation#2589
Contributor
Reviewer's GuideAdjusts WhatsApp Business catalog pagination defaults and exposes cursor-based pagination, while adding a GitHub Actions workflow to build and publish Docker images to GHCR. File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The new
cursorparameter is exposed in the DTO and schema but only used infetchCatalog; if collections also support cursor-based pagination it would be good to wirecursorthroughfetchCollectionsfor consistency and functionality. - The increased hard-coded limits in
fetchCatalog(limit=50, up to 20 loops) andfetchCollections(limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `cursor` parameter is exposed in the DTO and schema but only used in `fetchCatalog`; if collections also support cursor-based pagination it would be good to wire `cursor` through `fetchCollections` for consistency and functionality.
- The increased hard-coded limits in `fetchCatalog` (limit=50, up to 20 loops) and `fetchCollections` (limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
- Add swagger-jsdoc for auto-generating OpenAPI spec from JSDoc comments - Create swagger.config.ts with base schemas and security setup - Add JSDoc comments to Business router endpoints (getCatalog, getCollections) - Setup swagger-ui at /docs endpoint - Add /openapi.json endpoint for raw spec access - Add npm script 'openapi:generate' to generate static spec - Generate initial openapi.json with Business endpoints Access: - Swagger UI: http://localhost:8080/docs - OpenAPI JSON: http://localhost:8080/openapi.json
- Fix package.json import in swagger.config.ts (use readFileSync instead of import) - Fix package.json import in generate-openapi.ts - Fix prettier formatting in main.ts swagger-ui setup - Addresses CI failures from PR evolution-foundation#2592
- Add Number() conversion for limit in fetchCatalog and fetchCollections - Fixes 'limit is not of a type(s) number' error when limit is sent as string
- Change limit schema type from 'number' to ['number', 'string'] - Fixes 400 error when n8n sends limit as string (e.g. "50" instead of 50) - Service layer already converts with Number() for safety
- fetchBusinessProfile failure no longer blocks catalog/collections fetch - Both endpoints now continue even if business profile check fails - Prevents returning isBusiness:false when profile API is temporarily down
Changes: - Fix protocolMessage parsing regression - Performance improvements - Meta Coexistence support - Less automation detection (reduced bans) - Fix session recovery issues
|
🔴 👎 A correção da paginação do catálogo é válida, mas o PR mistura escopo: aponta para |
Adds a new CatalogBrowserService that lazily launches a Puppeteer
browser session to fetch catalog & collections via web.whatsapp.com's
internal WPP API, bypassing Baileys' protocol-level truncation.
Architecture:
- One Browser per JID (lazy start, idle timeout kill)
- Session persisted per instance under instances/{name}/browser-session/
- 4-layer fetch fallback (ported from bedones-whatsapp):
1. queryCatalog with pagination cursor
2. CatalogStore.findQuery
3. WPP.catalog.getMyCatalog
4. WPP.catalog.getProducts (last resort)
- Service-locator pattern (BrowserCatalogService.getInstance()) so the
non-NestJS-managed BaileysStartupService can access it
API:
- /business/getCatalog/{instance} now accepts { provider: 'browser' }
- /business/getCollections/{instance} now accepts { provider: 'browser' }
- Default behavior (no provider field) unchanged — uses Baileys
Docker:
- Adds chromium + nss + freetype + harfbuzz + font-noto-emoji
- Sets PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
- Adds CATALOG_BROWSER_ENABLED (default false) + tunables
Config (env vars):
- CATALOG_BROWSER_ENABLED (default false)
- CATALOG_BROWSER_IDLE_TIMEOUT_MS (default 600000 = 10 min)
- CATALOG_BROWSER_MAX_SESSIONS (default 5)
- CATALOG_BROWSER_HEADLESS (default true)
Why: WhatsApp's anti-bot on protocol level (Baileys) is much stricter
than browser level. Bedones-whatsapp proved browser-based fetch works
for full catalogs. This ports that approach into Evolution API without
touching the existing Baileys messaging path.
Refs: investigated with bedones-whatsapp (whatsapp-web.js implementation)
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI npm ci was failing because package-lock.json was out of sync after adding puppeteer-core to package.json. Regenerate lock file. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…Logger Evolution API doesn't use NestJS — it uses Express with manual DI. Refactored to use the project's own Logger class and BadRequestException from @Exceptions. Removed catalog-browser.module.ts (NestJS module, not needed). Now compiles clean against the existing codebase. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI lint was failing on 17 prettier issues + 1 duplicate import in business.router.ts. Auto-fixed via eslint --fix. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Docker Hub builds were failing due to missing DOCKER_USERNAME/DOCKER_PASSWORD secrets. GHCR (ghcr.io/kelvinzer0/evolution-api) is the canonical registry going forward — uses built-in GITHUB_TOKEN, no external secrets needed. Removed: - publish_docker_image.yml (tag-based Docker Hub) - publish_docker_image_homolog.yml (develop branch Docker Hub) - publish_docker_image_latest.yml (main branch Docker Hub — was failing) Remaining workflows: - publish_ghcr.yml (main + tags + PRs) - check_code_quality.yml - security.yml Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Chromium was crashing on launch with 'Protocol error (Target.setAutoAttach): Target closed' because --single-process mode is unstable in headless containers (V8 proxy resolver fails, GPU init crashes). Removed: - --single-process (causes V8 proxy resolver failure) - --no-zygote (only needed with --single-process) Added stability flags: - --disable-software-rasterizer - --disable-features=VizDisplayCompositor,Vulkan - --disable-vulkan (no GPU in container, was spamming errors) - --no-first-run, --no-default-browser-check - --disable-extensions, --disable-background-networking - --disable-sync, --disable-translate, --disable-default-apps - --disable-component-update Tested on Alpine Linux container with Chromium 150. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Previous failed launches left SingletonLock files + orphan chromium
processes in userDataDir, blocking new launches with 'profile appears
to be in use by another Chromium process' error.
Added:
- cleanStaleLocks() method called before every browser launch:
- Removes SingletonLock, SingletonCookie, SingletonSocket files
- Kills orphan chromium processes via pkill
- browser.on('disconnected') handler to clean up internal state
- protocolTimeout: 60000ms for slow container startup
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Without wa-js injection, window.WPP is undefined in the Puppeteer page context, causing all catalog fetch logic to fail silently (returns empty catalog, no QR code for auth). Root cause: whatsapp-web.js bundles wa-js internally, but since we use puppeteer-core directly, we need to inject it ourselves. Changes: - Added @wppconnect/wa-js@^3.22.1 to dependencies - New injectWaJs(page) method: - Reads wa-js from node_modules via require.resolve - Injects via page.evaluate(waJsCode) - Waits for WPP.isReady === true (timeout 30s) - Updated launchBrowser(): - Wait for networkidle2 (was domcontentloaded) — WA Web needs full load - Set proper User-Agent - Call injectWaJs() after page.goto - Added @wppconnect/wa-js to Dockerfile npm install Tested in-container: wa-js injects successfully, WPP.isReady = true, QR code extracted via canvas (9090 chars data URL), auth check works. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes critical pagination limitations in the WhatsApp Business catalog API that prevent fetching catalogs with more than 50 products.
Changes
whatsapp.baileys.service.ts:4904)whatsapp.baileys.service.ts:4927)getCollections- now defaults to 100 (whatsapp.baileys.service.ts:4974)cursorparameter to validation schemas for manual pagination control (business.schema.ts)cursortogetCollectionsDto(business.dto.ts)Problem
The current implementation has these limitations:
getCatalog: Default limit=10 with max 4 loops = 50 products maxgetCollections: Hard-capped at 20 items per collectioncursorparameter stripped by validation schema, preventing manual paginationSolution
Testing
getCatalogandgetCollectionsnow return complete dataCloses #2589
Summary by Sourcery
Adjust WhatsApp Business catalog APIs to support larger, cursor-based pagination and add automated container publishing to GitHub Container Registry.
New Features:
Enhancements:
CI: