data-transformed

Clusterflick is a free, open-source site listing every film showing in London, from 450+ venues in one search.

This repository is the stage of the data pipeline that turns each venue's raw listings into one standard format and matches every film to The Movie Database (TMDB). See the Clusterflick organisation for how the pieces fit together.

This repository contains the automated workflow for transforming raw cinema data into a standardized format for the Clusterflick project.

Purpose

The transformation workflow processes raw cinema event data retrieved from various cinema websites and converts it into a unified, standardized cinema data format. This includes:

• Data normalization: Converting venue-specific data formats into a consistent schema
• Movie matching: Attempting to match each event to a movie in The Movie Database (TMDB)
• Categorization: When movie matching fails, events are categorized appropriately
• Quality assurance: Ensuring data integrity and consistency across all venues

How It Works

A Prepare Transform job runs first, once per run, after the latest raw data from clusterflick/data-retrieved has been downloaded. It builds the inputs every venue shares - the first-seen history from the last 10 days of clusterflick/data-combined releases, yesterday's release from this repository, and what every external source (Eventbrite, OutSavvy, ...) lists at each venue. It shares them as the transform-inputs artifact:

npm run transform-prepare

Each job then downloads that artifact and the latest raw data from the clusterflick/data-retrieved repository's most recent release, and executes the transform command for each venue:

npm run transform -- <venue-identifier>

This command:

• Reads the raw data for the specified venue
• Normalizes the data structure
• Attempts to match events to movies in TMDB using the Movie Database API
• Categorizes events that cannot be matched to movies
• Generates standardized output files

Schedule

The workflow is automatically triggered when the data retrieval workflow completes successfully. It can also be triggered manually via workflow dispatch if needed.

Workflow Structure

Transform commands are organized into parallel job groups to:

• Minimize total wall clock time
• Prevent cascading failures (one command failure won't stop others)
• Enable individual job restarts when issues occur

Maintenance

Adding New Venues

When a new venue is added to the system, this workflow must be updated to include the transformation step. To add a venue:

1. Identify the appropriate job group (or create a new one if needed)
2. Add a new step to run the transform command for the venue:

   - name: venue-identifier
     run: npm run transform -- venue-identifier

3. Ensure the job downloads the correct raw data files from the retrieval repository

Dependencies

The workflow requires several API keys configured as GitHub secrets:

• MOVIEDB_API_KEY - For matching events to movies in The Movie Database
• GEMINI_API_KEY - For AI-powered categorization when movie matching fails
• OPENAI_API_KEY - For identifying the films in multi-film events (double bills, marathons), which clusterflick/scripts sends to OpenAI by default. Setting the repository variable CAPABLE_LLM_PROVIDER to gemini sends them to Gemini instead
• PAT - Personal Access Token for triggering downstream workflows

Licence

The code in this repository is licensed under the MIT licence.

The data published in this repository's releases is not — it is licensed separately, under CC BY 4.0, with a carve-out for the third-party metadata it carries. If you use it, credit Clusterflick:

Screening data from Clusterflick (CC BY 4.0)

The themoviedb and themoviedbs fields hold metadata from The Movie Database and are excluded from that grant — Clusterflick has no right to sublicense them. See LICENSE-DATA for the exact terms, or clusterflick.com/data-licence for the readable version.