Wikipedia-USAS-processing

This repository contains various DataTrove pipelines, filters, formatters, and helper functions for processing the HuggingFaceFW finewiki dataset, in various languages listed in the languages section, to create a synthetic (silver labelled) training dataset for USAS semantic tags and Multi Word Expression (MWE) identification for these languages.

For more information on the filtering and processing, see the filtering and processing section below and for more information about the data we use see the data section below.

The commands we ran to process the data for the journal paper can be found in the commands used to create the original dataset section below.

Setup

You can either use the dev container with your favourite editor, e.g. VSCode. Or you can create your setup locally below we demonstrate both.

In both cases they share the same tools, of which these tools are:

uv for Python packaging and development
make (OPTIONAL) for automation of tasks, not strictly required but makes life easier.

Dev Container

A dev container uses a docker container to create the required development environment, the Dockerfile we use for this dev container can be found at ./.devcontainer/Dockerfile. To run it locally it requires docker to be installed, you can also run it in a cloud based code editor, for a list of supported editors/cloud editors see the following webpage.

To run for the first time on a local VSCode editor (a slightly more detailed and better guide on the VSCode website):

Ensure docker is running.
Ensure the VSCode Dev Containers extension is installed in your VSCode editor.
Open the command pallete CMD + SHIFT + P and then select Dev Containers: Rebuild and Reopen in Container

You should now have everything you need to develop, uv, make, for VSCode various extensions like Pylance, etc.

If you have any trouble see the VSCode website..

Local

To run locally first ensure you have the following tools installted locally:

uv for Python packaging and development. (version 0.9.6)
make (OPTIONAL) for automation of tasks, not strictly required but makes life easier.
- Ubuntu: apt-get install make
- Mac: Xcode command line tools includes make else you can use brew.
- Windows: Various solutions proposed in this blog post on how to install on Windows, inclduing Cygwin, and Windows Subsystem for Linux.

When developing on the project you will want to install the Python package locally in editable format with all the extra requirements, this can be done like so:

uv sync --all-extras

Linting

Linting and formatting with ruff it is a replacement for tools like Flake8, isort, Black etc, and we us ty for type checking.

To run the linting:

make lint

Tests

To run the tests (uses pytest and coverage) and generate a coverage report:

make test

HuggingFace Authentication

Before processing or uploading to the HuggingFace hub please authenticate using a token from huggingface.co/settings/tokens;

hf auth login

or by using a token that is set within ./.env, read using dotenv, e.g.

HF_TOKEN="HUGGINGFACE_TOKEN_KEY_VALUE"

Set the relevant permissions, the minimum for this is repository is "read" only permission, if you want to upload the created synthetic silver labelled dataset to HuggingFace please ensure that you have allowed write permission to the namespace/repository you are going to upload too on HuggingFace.

Data

The data will be coming from HuggingFaceFW finewiki dataset and will be filtered so that each Wikipedia article is either rated as a "Good Articles" (GA) or "Featured Articles" (FA) by an editor, we hope that this will remove articles that might be incomplete or require additional editing. This filtering is inspired by Conia et al. 2024 whereby they found training on data from only "featured" and "good" articles performed similarly to training on the far larger Wikipedia articles that contained non-good and non-featured articles thus showing that training on smaller amounts of data is as affective and more efficient. The "Featured" and "Good" article can be defined differently for each Wikipedia language site as stated in the English site definition within the following article. The list of GA and FA can be found at the HuggingFace dataset ucrelnlp/wikipedia-ga-fa-ids.

Languages

The languages that this repository covers and supports, this table is also available in machine readable format at ./wikipedia_processing/data/usas_wikipedia_processing.yaml (languages that have the value of True for the key training). These languages have been selected based on semantic tagging support for the given language and the number of GA and FA articles available for the given language.

Language	ISO 639-3
English	eng
Dutch	nld
Spanish	spa
Danish	dan
Italian	ita
Portuguese	por
Chinese	zho
Finnish	fin

Filtering and Processing

Each Wikipedia article from HuggingFaceFW finewiki goes through the following pipeline;

The Wikipedia article ID and title match an article within the GA or FA articles (taken from the ucrelnlp/wikipedia-ga-fa-ids dataset).
The Wikipedia article is not an article that is in the test data. (manual list of Wikipedia article URLs)
Remove Wikipedia family tree tables, mathematical equations, and tables from the article text.
Remove any Markdown formatting like headers using mistune Python package from the article text.
Remove articles that contain less than 50 tokens based off a language specific tokenizer.
Apply exact and then MinHash de-duplication.
Sentence split using language specific sentence splitters (spaCy sentence splitters are installed when the processing script is running).
USAS semantic and when the tagger supports it MWE identification using PyMUSAS Rule Based languages specific taggers per sentence (The spaCy tokenizers, lemmatizers, and POS taggers as well as the PyMUSAS tagger are installed when the processing script is running).
- Each sentence has all leading and trailing whitespace removed, any tokens identified as whitespace are kept as tokens but with no USAS tags, and all PUNCT tags are mapped onto Z9 tags. If a Z99 USAS tag (the unmatched tag) is produced by any of the rule based taggers this USAS tag is not kept, it is removed.

The processed data will contain the following fields:

text - the processed article text.
id - dawiki/1171348
page_id - Page ID 1171348
title - Article title.
url - https://da.wikipedia.org/wiki/El_Salvador_ved_sommer-OL_2024
version (int|string) - revision/version identifier of the page (comes from HuggingFaceFW finewiki) 1167219203
start_end_sentence_character_indexes - list of start and end character offsets for each sentence, e.g. [[0, 10], [11, 15]] the first sentence is between text[0:10].
tokens - list of a list of tokens whereby the inner list represents the tokens for a given sentence, e.g. tokens[0] would contain all of the tokens in the first sentence.
tags - list of a list of a list of USAS tags that were predicted by the PyMUSAS Rule Based languages specific tagger. The inner list represents the most likely USAS tags for the given token, e.g. tags[0][0] will contain a list of most likely USAS tags for the first token in the first sentence, in most cases it will only contain one USAS tag. When it contains more than one USAS tag this represents a token in which the meaning is a combination of the given predicted USAS tags. Some tokens will contain no USAS tags as the Rule Based tagger cannot make prediction for all tokens. Tags within each group are de-duplicated after any tag_mapper renaming is applied.
other_tags - list of a list of a list of a list of USAS tags, one level deeper than tags, containing every other valid USAS tag group for a token that was not its most likely tag group, e.g. other_tags[0][0] will contain a list of the other valid USAS tag groups for the first token in the first sentence, and other_tags[0][0][0] the tags within the first of those groups. Keeping each group as its own inner list (rather than merging them together) preserves which tags PyMUSAS considered part of the same combined meaning. Most tokens will contain no other USAS tag groups, in which case the outer list is empty. As with tags, tags within each group are de-duplicated after any tag_mapper renaming is applied. They are ordered by the most likely USAS tag group.
mwes - list of a list of MWE labels that were predicted by the PyMUSAS Rule Based languages specific tagger, these always relate to the most likely USAS tags. The MWE labels denote at the sentence level which tokens are MWEs, e.g. mwes[0][0] represent all of the MWE labels for the first token in the first sentence, if it contains 1 and mwes[0][1] also contains 1 then the first token and second token in the first sentence are a MWE. If more than one label occurs then MWEs are overlapping which should not be the case with PyMUSAS taggers. MWEs can be dis-continuous. The index of MWE labels always start at 1 and reset per sentence, e.g. the first sentence can contain a MWE label of 1 and so can the second sentence, but they will be different MWEs as MWEs are constrained to occur within a single sentence; they cannot span sentence boundaries.

The final data is written as zstd-compressed Parquet files rather than JSONL, as Parquet gives better compression on the repetitive, deeply nested tokens/tags/other_tags/mwes fields and native HuggingFace Hub Dataset Viewer support.

Train/validation split

Each language's documents are split into train and validation subsets, written to separate train/validation subfolders (there is no split field/column in the data itself, the split is entirely determined by which subfolder a file is in). The validation split is X% of the language's documents, or N documents, whichever is reached first (controlled by the --validation-percentage/-v and --max-validation-documents/-n options of build_usas_wikipedia_dataset.py below) rather than a flat percentage. A flat percentage would give under-resourced languages (some have as few as ~200 articles) a tiny handful of validation documents while giving well-resourced languages like English potentially thousands, which is more than needed for a useful validation set and just eats into training data. Capping at min(X% of total, N) keeps small languages at their natural percentage-based split (they will never reach the N cap) while bounding well-resourced languages' validation set to a sane absolute size. The split assignment is deterministic (hashed from each document's page ID), so re-running the pipeline reproduces the same split.

Filtering and Processing script

processing_scripts/build_usas_wikipedia_dataset.py is the pipeline entry point. It writes the final output either to a local directory or directly to a HuggingFace Hub dataset repository — pass exactly one of --output-dir or --hf-dataset-repo-id.

Writing locally, as Parquet files under ./local_da/data/da/{train,validation}/:

uv run processing_scripts/build_usas_wikipedia_dataset.py da ./log_data/ --output-dir ./local_da/

Uploading directly to a HuggingFace Hub dataset repository (see HuggingFace Authentication above):

uv run processing_scripts/build_usas_wikipedia_dataset.py da ./log_data/ --hf-dataset-repo-id ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia

Some options worth knowing about (run --help for the full list):

--max-final-output-file-size/-e - maximum size in MB of the final Parquet shards (default 200MB); distinct from --max-output-file-size/-s, which only governs intermediate staging files used during processing.
--validation-percentage/-v and --max-validation-documents/-n - control the train/validation split described above.
--private/--public - whether a Hub repository created by --hf-dataset-repo-id is private (default: public).

An example of processing the Danish Wikipedia data locally but saving the data to the HuggingFace ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia repository.

uv run processing_scripts/build_usas_wikipedia_dataset.py da ./log_data/ -o -w 5 -t 2 -m 0.85 --hf-dataset-repo-id ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia --public

Running on Slurm

By default the script runs each pipeline stage (reading, dedup, tagging, etc.) as local multiprocessing workers. Pass --executor slurm to instead submit each stage as a Slurm job array via DataTrove's SlurmPipelineExecutor, which is useful when a single machine doesn't have enough CPUs/memory to process a language in reasonable time. Stages still run in the same dependency order — one stage's Slurm job array only starts once the previous stage's finishes, enforced via Slurm --dependency chaining between the stages' sbatch calls, not by this script waiting in-process.

Because of that chaining, SlurmPipelineExecutor submits every stage's job and then returns immediately — it does not poll Slurm for completion. So with --executor slurm this script itself exits as soon as all stages are submitted, well before the actual Slurm jobs finish; a clean exit (code 0) only means submission succeeded, not that processing has completed. Track real progress with squeue/sacct or by watching logging_dir/.

The Slurm sbatch command help page can be found at; https://slurm.schedmd.com/sbatch.html

--slurm-partition and --slurm-time are required when --executor slurm is set; every other --slurm-* option is optional and only used with --executor slurm (passing any of them with the default --executor local is an error).

uv run processing_scripts/build_usas_wikipedia_dataset.py da ./log_data/ \
    --output-dir ./local_da/ \
    --executor slurm \
    --slurm-partition compute \
    --slurm-time 4:00:00 \
    --slurm-cpus-per-task 4 \
    --slurm-mem-per-cpu-gb 4 \
    -w 32

Some other Slurm options worth knowing about (run --help for the full list):

--slurm-venv-path/--slurm-condaenv - activate a virtualenv or conda environment in each Slurm job before running; mutually exclusive with each other.
--slurm-qos - Slurm QOS to submit jobs under (default normal).
--slurm-mail-user/--slurm-mail-type - get emailed on job events, e.g. --slurm-mail-type FAIL.
--slurm-sbatch-args - JSON object of any additional raw sbatch arguments, e.g. '{"account": "myaccount"}'.

Understanding tasks, workers, and resources on Slurm

The pipeline runs as a chain of dependent stages (reading → initial processing → exact dedup → MinHash dedup → post-processing/tagging → stats merge). With --executor slurm, each stage is submitted as its own Slurm job array, and a stage's job array is only submitted once the previous stage's array has fully finished — stages never run concurrently with each other.

Within a single stage's job array:

tasks (derived from -w/--number-of-workers × -t/--tasks-multiplier) is the array size — the number of independent work units (array indices) that stage is split into, e.g. sbatch --array=0-159 for 160 tasks.
workers (essentially -w/--number-of-workers) is a concurrency throttle on that array, e.g. --array=0-159%32 means at most 32 array indices run at the same instant. It is not a node count, and -w 32 does not reserve 32 nodes.
--slurm-cpus-per-task and --slurm-mem-per-cpu-gb are the resources requested per array index (per task), e.g. 4 CPUs and 4GB/CPU = 16GB for that one task. Slurm's scheduler then places each task on whatever node has free capacity — it may pack many concurrently-running tasks onto a single large node, or spread them across several smaller ones, entirely independently of workers. If you need specific node placement, pass the relevant raw flags via --slurm-sbatch-args.
--slurm-time is a per-task limit, not a shared budget: every task gets its own fresh clock starting when that task begins running, regardless of when the array was submitted or how long earlier tasks took. A task started 2 hours after the array began still gets the full time limit from its own start.
Because of the concurrency throttle, a stage's total wall-clock time can exceed the per-task time limit: e.g. 160 tasks at 32 concurrent, each taking close to the 4-hour limit, is roughly ceil(160 / 32) × 4h ≈ 20h for that stage to fully drain, even though no single task exceeds 4 hours.
With --executor local, -w/--number-of-workers is additionally capped by the CPU count of the machine running the script (os.process_cpu_count()), since local workers are real concurrent processes on that machine. This cap does not apply with --executor slurm — there, -w is only a concurrency throttle on the job array (see above) and is not limited by the submission host's core count.
-j/--tasks-per-job (Slurm-only, default 1) shrinks the array size itself: instead of one array index per task, each array index runs tasks_per_job tasks sequentially, one after another, inside the same allocation. A stage with 160 tasks and -j 5 submits sbatch --array=0-31 (32 array indices) instead of 0-159 — the same 160 tasks still all get run, just 5-at-a-time-in-series per array index rather than each getting its own index. This doesn't change -w/-t, concurrency, or resource requests per index at all; it only reduces how many array indices Slurm has to create in the first place, at the cost of each index taking up to tasks_per_job times as long. Useful when a Slurm admin caps the total number of array indices you can have submitted (running or pending) at once — see --max-number-of-parallel-tasks below, which sets this automatically across a multi-language run.

Uploading multiple languages to the same Hub repository

All languages can share a single Hub dataset repository so users can pick a language, by running the script once per language against the same --hf-dataset-repo-id. Each run writes into its own data/<wikipedia_language_code>/{train,validation}/ path within the repo, so nothing is overwritten between languages.

For the Hub Dataset Viewer to expose each language as a selectable config with its train/validation splits, add a configs: block to the dataset repository's own README (its "dataset card"), keyed by wikipedia_language_code for consistency with ./wikipedia_processing/data/usas_wikipedia_processing.yaml and the CLI, e.g.:

configs:
  - config_name: da
    data_files:
      - split: train
        path: "data/da/train/*.parquet"
      - split: validation
        path: "data/da/validation/*.parquet"
  - config_name: nl
    data_files:
      - split: train
        path: "data/nl/train/*.parquet"
      - split: validation
        path: "data/nl/validation/*.parquet"

Running every training language at once

processing_scripts/run_all_training_languages.py automates the "run once per language" pattern above: it reads every language with training: true (from the packaged wikipedia_processing/data/usas_wikipedia_processing.yaml by default, or a custom file via --languages-file), and for each one launches build_usas_wikipedia_dataset.py as an independent, concurrently-running subprocess, using the same Python interpreter this script is itself running under (sys.executable) so it works from a Slurm login node or any other environment, as long as this script was launched with an interpreter that already has the project's dependencies (e.g. via uv run, or an activated venv). This only affects the process that submits/polls each language's Slurm job array; the environment each pipeline stage actually runs in on the compute nodes is still controlled independently via --slurm-venv-path/--slurm-condaenv.

Rather than using the same fixed -w/-t for every language, it first checks each language's finewiki dataset shard count and scales -w/-t off that (via --shard-tasks-multiplier, clamped between --min-tasks-per-language and --max-tasks-per-language, and capped at --max-workers-per-language concurrent workers) — so small languages aren't handed more Slurm tasks than they have data to fill, and large languages get proportionally more parallelism. Any options this script doesn't declare itself (--executor, --slurm-partition, --slurm-time, --overwrite, etc.) are forwarded verbatim to every language's invocation.

# Preview the computed shard counts / -w / -t / commands without launching anything:
uv run processing_scripts/run_all_training_languages.py ./log_data --dry-run \
    --executor slurm --slurm-partition compute --slurm-time 6:00:00

# Real run, uploading every language to the same shared Hub repo:
uv run processing_scripts/run_all_training_languages.py ./log_data \
    --executor slurm --slurm-partition compute --slurm-time 6:00:00 \
    --slurm-cpus-per-task 4 --slurm-mem-per-cpu-gb 4

With --executor local, each language's process blocks until its own full stage chain finishes (same as a single-language run), so this command blocks until every language finishes — run it under tmux/screen/nohup for a real multi-hour run.

With --executor slurm this is not the case: DataTrove's SlurmPipelineExecutor submits each stage as a chained sbatch job array and returns as soon as submission succeeds, without waiting for the jobs to actually run (see Running on Slurm above). So this command exits almost immediately after every stage for every language has been submitted — the finished successfully/failed messages it prints only reflect whether submission succeeded, not whether the Slurm jobs themselves have completed. Use squeue/sacct, or tail logging_dir/<wikipedia_code>/, to track the actual runs. Per-language stdout/stderr from the driver process itself is captured under logging_dir/driver/<wikipedia_code>.log.

Staying under a Slurm submitted-jobs quota with `--max-number-of-parallel-tasks`

Because every stage of a language's pipeline is chained by Slurm --dependency rather than by this script waiting in-process, launching several languages within seconds of each other (see --stagger-seconds) means all of their stages get submitted to Slurm essentially at once — most stay pending on their dependency, but a Slurm admin's MaxSubmitJobsPerUser-style quota typically counts pending and running array indices alike. That total is not something -w/-t can control (they only throttle concurrency of already-submitted indices, see above); the only way to actually shrink how many array indices get created is -j/--tasks-per-job.

--max-number-of-parallel-tasks automates this: given a budget, it estimates each training language's own peak simultaneously-submitted array-index count (summed across its ~11 chained stages), proportionally shrinks each language's share of that budget when it doesn't fit, and works out a -j/--tasks-per-job value per language that brings its peak down to its share — appending --tasks-per-job to that language's forwarded build_usas_wikipedia_dataset.py command. It never touches -w/-t or the actual shard-scaled data-processing parallelism; the only effect is on how many Slurm array indices get created to do that same work. Every language has a fixed floor of 11 array indices (once -j is large enough that every stage collapses to a single index), so the budget must be at least 11 × (number of training languages).

# Cap the combined peak submitted array indices across all languages at 300:
uv run processing_scripts/run_all_training_languages.py ./log_data --dry-run \
    --executor slurm --slurm-partition compute --slurm-time 6:00:00 \
    --max-number-of-parallel-tasks 300

Only meaningful with --executor slurm; it has no effect with --executor local.

Deduplicating an existing dataset

processing_scripts/deduplicate_wikipedia_dataset.py is a post-hoc fix-up script for a dataset already built and uploaded by build_usas_wikipedia_dataset.py (e.g. ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia) — it does not re-run any of the filtering/tagging pipeline above, it only reads and rewrites each language's already-processed train/validation Parquet output. It fixes two issues that can occur, especially after re-running the pipeline for a language (e.g. to add more articles) or on a highly-parallel Slurm run:

Duplicate ids. For each language, every row of train and validation (combined) is grouped by id. Where an id occurs more than once, only the row with the highest version is kept and the rest are dropped. If the duplicate copies span both train and validation, the surviving row always ends up in validation (moved out of train if that's where the highest-version copy originally was); duplicates confined to a single split just keep the surviving row in that same split.
Validation split overflow. TrainValidationSplitAnnotator (see Train/validation split above) is meant to cap the whole validation split at --max-validation-documents, but divides that cap evenly across DataTrove ranks with a max(1, ...) floor — so whenever a language's rank/task count exceeds --max-validation-documents, every rank keeps at least 1 validation document regardless, and the real total ends up close to the rank count instead of the intended cap (English hit this after a 27-task Slurm run capped at 20: it ended up with 27 validation documents, not 20). This script re-caps the post-dedup validation split at --max-validation-documents/-n (default 20), moving any excess back into train deterministically (same page_id hash the annotator itself uses), without needing to re-run the whole pipeline.

Optionally, pass --max-validation-percentage/-p to also enforce the full min(X% of total, N) rule from Train/validation split above (mirroring build_usas_wikipedia_dataset.py's --validation-percentage), rather than only capping an oversized validation split: each language's validation split is rebalanced to exactly min(round(total_post_dedup_documents × p / 100), --max-validation-documents), moving rows from train into validation (or the reverse) as needed, ranked by the same deterministic page_id hash. Omitting -p (the default) keeps the shrink-only overflow fix above unchanged.

By default the script only reports what it would change — pass --output-dir to also write the deduplicated Parquet locally, and/or --push to commit the result back to the Hub repository (this replaces each processed language's existing train/validation Parquet shards in one commit per language, so no stale duplicate shards are left behind).

# Report duplicate/overflow counts for every language without writing anything:
uv run processing_scripts/deduplicate_wikipedia_dataset.py

# Write deduplicated Parquet for Danish only to a local directory, without pushing:
uv run processing_scripts/deduplicate_wikipedia_dataset.py -l da --output-dir ./local_dedup

# Deduplicate every language and push the result back to the Hub:
uv run processing_scripts/deduplicate_wikipedia_dataset.py --push

# Deduplicate every language, ensures that either 20 documents or 10% of documents
# are in the validation split, and push the result back to the Hub:
uv run processing_scripts/deduplicate_wikipedia_dataset.py -p 10 --push

Some options worth knowing about (run --help for the full list):

-l/--language - restrict processing to specific language(s) (repeatable); defaults to every config found in --hf-dataset-repo-id.
-n/--max-validation-documents - the validation split cap to enforce (default 20); should match the value the data was originally built with.
-p/--max-validation-percentage - optional (default: unset); when given, rebalances each language's validation split, growing or shrinking it as needed, to min(round(total_post_dedup_documents × p / 100), --max-validation-documents), mirroring build_usas_wikipedia_dataset.py's --validation-percentage. Omitted by default, which keeps the shrink-only --max-validation-documents overflow-cap behavior above.
-e/--max-output-file-size - target maximum size in GB per output Parquet shard, pre-compression (default 1.0; the actual file size will be a lot smaller due to compression); larger splits are written as multiple shard files instead of one.

Dataset statistics

processing_scripts/dataset_statistics.py reports per-language, per-split statistics for a dataset already built and uploaded by build_usas_wikipedia_dataset.py (e.g. ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia) — like deduplicate_wikipedia_dataset.py above, it only reads the already-processed train/validation Parquet output, it does not re-run any of the filtering/tagging pipeline. Each language is shown by its full display name (e.g. "Danish", via language_display_name), sorted alphabetically. For each language, plus a "Total" language aggregating every language together, it reports the following for whichever split(s) --split selects (see below):

Number of articles and number of sentences (Sentences (M), in millions, rounded to 3 decimal places).
Number of tokens (Tokens (M), in millions, rounded to 3 decimal places).
Number of labelled tokens (Labelled Tokens (M), tokens with at least one USAS tag, in millions rounded to 3 decimal places) and Labels per Token — the average number of USAS tag labels per token, counting both the tags and other_tags columns (both are positive labels when training).
Multi Tag Membership (%) — the percentage of USAS tag labels that belong to a "multi tag membership" group. A "multi tag membership" group is any tag group — a labelled token's tags entry, or an individual group within its other_tags entry — that itself contains more than one USAS tag, e.g. tags[0][0] is ["A3", "M6"]. Every tag within such a group counts towards both the numerator and the denominator (the total count of individual tag labels across tags and other_tags), so this always falls between 0% and 100%.
Number of unique USAS tags (from both tags and other_tags).
Number of Multi-Word Expressions (MWEs (M), in millions, rounded to 3 decimal places).
MWE Tokens (%) — the percentage of tokens that are part of at least one MWE. This can be higher than simply dividing the "Number of Multi-Word Expressions" by the "Number of tokens" as each MWE contains more than one token thus each of those tokens in the MWE count towards the MWE token count.

It reads HF_TOKEN from the environment the same way as HuggingFace Authentication above (via .env/python-dotenv), needed if --hf-dataset-repo-id is private.

# Print a table for every language in the default dataset (train + validation + combined total):
uv run processing_scripts/dataset_statistics.py

# Report statistics for a single language's train split only, omitting the MWE columns, and also export to CSV and LaTeX:
uv run processing_scripts/dataset_statistics.py -l da --split train -x number_of_mwes -x mwe_token_percentage \
    --output-csv ./stats.csv --output-latex ./stats.tex

Some options worth knowing about (run --help for the full list):

-l/--language - restrict to specific language(s) (repeatable); defaults to every config found in --hf-dataset-repo-id.
-s/--split - train, validation, all (default; reports train, validation, and their combined total as separate rows), or combined (loads both splits but reports only their combined total, without the separate train/validation rows).
-x/--exclude-column - omit specific column(s) (repeatable) from the table, CSV, and LaTeX output, e.g. -x number_of_mwes. Column names match the dict keys used internally (run --help to see the full list of valid values).
--output-csv - also write the table to a CSV file, with raw unformatted numeric values (unlike the console table, which adds , thousands separators).
--output-latex - also write the table as a LaTeX tabular environment (booktabs-style rules), with the same human-readable, escaped headers shown in the console table.

Initial Dataset Statistics

┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Language   ┃ Split      ┃ Articles ┃ Sentences (M) ┃ Tokens (M) ┃ Labelled Tokens (M) ┃ Labels per Token ┃ Multi Tag Membership (%) ┃ Unique Tags ┃ MWEs (M) ┃ MWE Tokens (%) ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ Chinese    │ train      │ 2,787    │ 0.670         │ 15.463     │ 10.010              │ 2.90             │ 21.12                    │ 215         │ 0.047    │ 0.62           │
│ Chinese    │ validation │ 20       │ 0.007         │ 0.149      │ 0.100               │ 3.05             │ 22.01                    │ 214         │ 0.000    │ 0.64           │
│ Danish     │ train      │ 168      │ 0.062         │ 1.278      │ 0.968               │ 1.28             │ 13.42                    │ 213         │ 0.033    │ 6.14           │
│ Danish     │ validation │ 19       │ 0.006         │ 0.138      │ 0.102               │ 1.24             │ 13.37                    │ 211         │ 0.003    │ 5.74           │
│ Dutch      │ train      │ 358      │ 0.149         │ 2.599      │ 1.743               │ 1.59             │ 10.87                    │ 211         │ 0.000    │ 0.00           │
│ Dutch      │ validation │ 20       │ 0.009         │ 0.160      │ 0.110               │ 1.65             │ 11.38                    │ 211         │ 0.000    │ 0.00           │
│ English    │ train      │ 49,198   │ 7.134         │ 182.734    │ 170.030             │ 1.68             │ 11.10                    │ 217         │ 13.785   │ 17.57          │
│ English    │ validation │ 20       │ 0.004         │ 0.106      │ 0.099               │ 1.64             │ 11.05                    │ 212         │ 0.008    │ 17.65          │
│ Finnish    │ train      │ 845      │ 0.216         │ 3.356      │ 2.454               │ 1.35             │ 16.21                    │ 209         │ 0.000    │ 0.00           │
│ Finnish    │ validation │ 20       │ 0.005         │ 0.072      │ 0.053               │ 1.35             │ 12.95                    │ 207         │ 0.000    │ 0.00           │
│ Italian    │ train      │ 1,141    │ 0.329         │ 9.485      │ 7.776               │ 1.67             │ 11.99                    │ 219         │ 0.094    │ 2.13           │
│ Italian    │ validation │ 20       │ 0.005         │ 0.154      │ 0.127               │ 1.74             │ 12.83                    │ 216         │ 0.002    │ 2.25           │
│ Portuguese │ train      │ 3,449    │ 0.760         │ 17.784     │ 13.795              │ 2.16             │ 15.24                    │ 218         │ 0.132    │ 1.61           │
│ Portuguese │ validation │ 20       │ 0.004         │ 0.102      │ 0.080               │ 2.15             │ 16.67                    │ 215         │ 0.001    │ 1.45           │
│ Spanish    │ train      │ 4,561    │ 0.917         │ 30.044     │ 24.078              │ 1.52             │ 1.03                     │ 219         │ 0.067    │ 0.47           │
│ Spanish    │ validation │ 20       │ 0.003         │ 0.106      │ 0.086               │ 1.55             │ 1.13                     │ 219         │ 0.000    │ 0.52           │
│ Total      │ train      │ 62,507   │ 10.237        │ 262.745    │ 230.853             │ 1.76             │ 11.51                    │ 220         │ 14.158   │ 12.52          │
│ Total      │ validation │ 159      │ 0.044         │ 0.987      │ 0.758               │ 1.84             │ 14.21                    │ 220         │ 0.014    │ 3.35           │
│ Total      │ total      │ 62,666   │ 10.281        │ 263.732    │ 231.611             │ 1.76             │ 11.52                    │ 220         │ 14.173   │ 12.49          │
└────────────┴────────────┴──────────┴───────────────┴────────────┴─────────────────────┴──────────────────┴──────────────────────────┴─────────────┴──────────┴────────────────┘

Token count distribution

processing_scripts/token_count_distribution.py plots and tabulates the distribution of token counts per sentence and per article for a dataset already built and uploaded by build_usas_wikipedia_dataset.py (e.g. ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia) — like dataset_statistics.py above, it only reads the already-processed train/validation Parquet output, it does not re-run any of the filtering/tagging pipeline. For each language (from --split, default train) it produces:

A histogram (PNG) of tokens-per-sentence, and a separate histogram of tokens-per-article, each with every language overlaid as its own colored, density-normalized step curve on a shared log-scaled x-axis, so differently-sized corpora stay comparable by shape rather than raw count.
A quantile table (25/50/75/90/95/99%, plus the maximum observed count) for each granularity, one row per language plus a final Macro Avg row — the unweighted mean of each language's own values (equal weight per language, regardless of corpus size).

It reads HF_TOKEN from the environment the same way as HuggingFace Authentication above, needed if --hf-dataset-repo-id is private.

# Histograms + Markdown tables (printed to console) for every language in the default dataset's train split:
uv run processing_scripts/token_count_distribution.py

# Two languages, train + validation combined, tables exported as LaTeX:
uv run processing_scripts/token_count_distribution.py -l da -l en --split all --format latex \
    --output-table-sentences ./data/tables/sentence_quantiles.tex \
    --output-table-articles ./data/tables/article_quantiles.tex

Some options worth knowing about (run --help for the full list):

-l/--language - restrict to specific language(s) (repeatable); defaults to every config found in --hf-dataset-repo-id.
-s/--split - train (default), validation, or all (combines both splits).
-f/--format - markdown (default) or latex for the quantile tables.
--output-histogram-sentences/--output-histogram-articles - PNG output paths for the two histograms (default under data/plots/).
--output-table-sentences/--output-table-articles - optional paths to write each quantile table to; defaults to printing to the console.

USAS tag distribution

processing_scripts/usas_tag_distribution.py tabulates the distribution of individual USAS tags for a dataset already built and uploaded by build_usas_wikipedia_dataset.py (e.g. ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia) — like dataset_statistics.py above, it only reads the already-processed train/validation Parquet output, it does not re-run any of the filtering/tagging pipeline. Tags are counted from both the tags and other_tags columns, since both are positive labels when training (other_tags holds every other valid tag group PyMUSAS considered besides the most likely one in tags). For each language (from --split, default train) it produces four tables, each with one column per language (sorted by language name via language_display_name) plus a final Macro Avg column — the unweighted mean of each language's own value (equal weight per language, regardless of corpus size):

The full major tag (first character of a USAS tag, e.g. A3 and A1 are both major tag A) distribution.
The top --top-bottom-count most common individual tags.
The bottom --top-bottom-count least common individual tags.
A five-number summary (Min, P25, P50, P75, Max) of how spread out individual tags' raw counts and percentages are within each language, with each cell showing both, e.g. 120 (12.0%).

It reads HF_TOKEN from the environment the same way as HuggingFace Authentication above, needed if --hf-dataset-repo-id is private.

# Markdown tables (printed to console) for every language in the default dataset's train split:
uv run processing_scripts/usas_tag_distribution.py

# Two languages, train + validation combined, top/bottom 5 tags, tables exported as LaTeX:
uv run processing_scripts/usas_tag_distribution.py -l da -l en --split all --top-bottom-count 5 --format latex \
    --output-table-major ./data/tables/major_tags.tex \
    --output-table-top ./data/tables/top_tags.tex \
    --output-table-bottom ./data/tables/bottom_tags.tex \
    --output-table-summary ./data/tables/tag_summary.tex

Some options worth knowing about (run --help for the full list):

-l/--language - restrict to specific language(s) (repeatable); defaults to every config found in --hf-dataset-repo-id.
-s/--split - train (default), validation, or all (combines both splits).
-n/--top-bottom-count - number of most-common (top) and least-common (bottom) individual tags to report (default 10).
-f/--format - markdown (default) or latex for the distribution tables.
--output-table-major/--output-table-top/--output-table-bottom/--output-table-summary - optional paths to write each distribution table to; defaults to printing to the console.

Pipeline runtime and peak node usage

processing_scripts/report_pipeline_runtime_and_nodes.py summarises, per language, how long a completed run of the pipeline took and how many compute nodes it could use in parallel. It reads the logging_dir that build_usas_wikipedia_dataset.py / run_all_training_languages.py wrote (./log_data/ in the examples above) — it does not touch the dataset or re-run any processing. For each log_data/<wikipedia_language_code>/ folder (the driver/ folder is skipped) it derives:

Total wall-clock — the span between the earliest and latest timestamp across every stage's */logs/task_*.log. This is end-to-end run time and includes the Slurm queue gaps between the dependent stages, not just compute time.
Max nodes — the widest #SBATCH --array=0-(N-1)%M directive across the language's stages. Every array task requests --nodes=1 (see Understanding tasks, workers, and resources on Slurm above), so this is the most nodes the language can occupy at once. The minimum is always 1 — stages run as a dependency chain and each task is an independent single-core job — so only the maximum is reported.

The table is printed as Markdown by default, or as a LaTeX booktabs tabular with --format latex.

# Markdown table for the local log_data folder, rows ordered by language name alphabetically:
uv run processing_scripts/report_pipeline_runtime_and_nodes.py ./log_data

# LaTeX table, ordered by descending node count, written to a file:
uv run processing_scripts/report_pipeline_runtime_and_nodes.py ./log_data \
    --format latex --sort-by nodes --output-file ./data/tables/pipeline_runtime.tex

Some options worth knowing about (run --help for the full list):

-f/--format - markdown (default) or latex.
-s/--sort-by - row ordering: runtime (descending), nodes (descending), or language (default, A–Z).
-o/--output-file - write the table to a file instead of stdout.

Runtime / node table for the original dataset run

| Language   | Max nodes | Total wall-clock |
| :--------- | --------: | :--------------- |
| English    |        15 | 2h 49m 54s       |
| Portuguese |         3 | 2h 33m 41s       |
| Chinese    |         5 | 57m 17s          |
| Spanish    |         4 | 55m 59s          |
| Danish     |         2 | 33m 26s          |
| Finnish    |         2 | 29m 35s          |
| Italian    |         4 | 28m 2s           |
| Dutch      |         2 | 23m 37s          |

Documents filtered at each pipeline stage

processing_scripts/report_pipeline_document_funnel.py summarises, per language, how many documents each filtering stage removed on a completed run. Like the runtime report above it only reads the log_data/ folder — specifically each stage's stats.json (total documents in, forwarded documents out) — and never touches the dataset. Six pipeline blocks discard documents, in this order:

Stage folder	Block	Removes
`reading`	Lambda	pages that are not Wikipedia "Good"/"Featured" articles
`reading`	Simple URL Filter	held-out test-set URLs
`initial_process`	Empty text filter	documents empty after the markdown → plain-text conversion
`initial_process`	Minimum Words Document Filter	documents below the min-word threshold
`exact_dedup_filter`	exact-deduplication	exact-duplicate documents
`minhash_dedup_filter`	MinHash stage 4	near-duplicate documents (MinHash threshold)

Two views are available:

--view dropped (default) — documents removed at each stage, the total removed, and the count (with percentage of the FineWiki input) that survived.
--view survived — the FineWiki input count and the number of documents still alive after each stage.

A stage that dropped nothing for every language is omitted (the empty-text filter is normally dormant); pass --all-stages to keep every column. The table is Markdown by default, or a LaTeX booktabs tabular with --format latex.

# Per-stage drop counts for the local log_data folder, largest corpus first:
uv run processing_scripts/report_pipeline_document_funnel.py ./log_data

# Surviving-document funnel as a LaTeX table:
uv run processing_scripts/report_pipeline_document_funnel.py ./log_data \
    --view survived --format latex --output-file ./data/tables/pipeline_funnel.tex

Some options worth knowing about (run --help for the full list):

-v/--view - dropped (default) or survived.
-f/--format - markdown (default) or latex.
-s/--sort-by - row ordering: input (descending FineWiki input), kept (descending), or language (default, A–Z).
--all-stages - keep filter-stage columns that dropped nothing for any language.
-o/--output-file - write the table to a file instead of stdout.

Per-stage drop counts for the original dataset run

| Language   | Good/Featured | Test URL | Min words | Exact dedup | MinHash dedup | Total removed |            Kept |
| :--------- | ------------: | -------: | --------: | ----------: | ------------: | ------------: | --------------: |
| Chinese    |     1,291,263 |        0 |     1,526 |         196 |           155 |     1,293,140 |  2,815 (0.217%) |
| Danish     |       291,764 |        0 |         0 |           7 |             3 |       291,774 |    187 (0.064%) |
| Dutch      |     2,072,477 |        0 |         0 |           5 |             5 |     2,072,487 |   378 (0.0182%) |
| English    |     6,562,224 |        4 |         6 |       1,608 |         1,571 |     6,565,413 | 49,242 (0.744%) |
| Finnish    |       571,963 |        0 |         0 |          30 |            42 |       572,035 |    865 (0.151%) |
| Italian    |     1,798,167 |        0 |         0 |         178 |           252 |     1,798,597 | 1,162 (0.0646%) |
| Portuguese |     1,131,646 |        0 |         0 |         167 |           100 |     1,131,913 |  3,470 (0.306%) |
| Spanish    |     1,943,867 |        0 |         0 |         265 |           243 |     1,944,375 |  4,590 (0.236%) |

Estimating deduplication loss

deduplicate_wikipedia_dataset.py above never records how many documents its id-based dedup actually removed from a given build. processing_scripts/report_deduplication_loss.py estimates that count after the fact, purely by diffing two already-generated LaTeX tables — it makes no HuggingFace Hub or log_data calls itself:

The "Kept" column of report_pipeline_document_funnel.py --view dropped (default: data/tables/pipeline_funnel.tex) — documents surviving the pipeline's own GA/FA, test-URL, min-words, exact-dedup, and MinHash-dedup filters, per language.
The "Articles" column of dataset_statistics.py --output-latex (default: data/tables/overall_dataset_statistics.tex) — the final published article count per language, summed across whichever splits are present (a language's separate train/validation rows are added together; an explicit combined row, e.g. from --split all/--split combined, is used as-is instead of being double-counted).

documents_after_filtering − final_articles is then the number of documents removed afterwards — exactly what the id-based dedup step does to a freshly-built dataset. Caveat: this is only a valid measurement when the funnel table's log_data run is the same, complete run that produced the final dataset. If the dataset was instead built from several separate pipeline runs merged together (see Commands used to create the original dataset below), the funnel table only reflects one of those runs, and the diff will not isolate dedup-by-id loss — it will also pick up every document contributed by the other runs.

A language present in only one table, or with an unavailable ("n/a") "Kept" count, is skipped and reported separately; a final "Total (matched languages)" row sums the figures across every included language.

# Report using the default table paths:
uv run processing_scripts/report_deduplication_loss.py

# Report using explicit paths, and also save to CSV:
uv run processing_scripts/report_deduplication_loss.py \
    data/tables/pipeline_funnel.tex data/tables/overall_dataset_statistics.tex \
    --output-csv data/tables/dedup_loss.csv

Some options worth knowing about (run --help for the full list):

Two positional arguments - the funnel table path and the statistics table path, in that order; both default to the paths shown above.
--output-csv - also write the report to a CSV file, with raw unformatted numeric values.
--output-latex - also write the report as a LaTeX tabular environment (booktabs-style rules).

Output: Estimated document loss from id-based deduplication

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Language                  ┃ Documents After Filtering ┃ Final Articles ┃ Dropped ┃ Dropped (%) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Chinese                   │ 2,815                     │ 2,807          │ 8       │ 0.28        │
│ Danish                    │ 187                       │ 187            │ 0       │ 0.00        │
│ Dutch                     │ 378                       │ 378            │ 0       │ 0.00        │
│ English                   │ 49,242                    │ 49,218         │ 24      │ 0.05        │
│ Finnish                   │ 865                       │ 865            │ 0       │ 0.00        │
│ Italian                   │ 1,162                     │ 1,161          │ 1       │ 0.09        │
│ Portuguese                │ 3,470                     │ 3,469          │ 1       │ 0.03        │
│ Spanish                   │ 4,590                     │ 4,581          │ 9       │ 0.20        │
│ Total (matched languages) │ 62,709                    │ 62,666         │ 43      │ 0.07        │
└───────────────────────────┴───────────────────────────┴────────────────┴─────────┴─────────────┘

Clearing stale shards from a Hub dataset repository

HuggingFaceDatasetWriter (used by build_usas_wikipedia_dataset.py when uploading directly to the Hub) only ever adds/overwrites the specific Parquet shard files it writes — it never deletes pre-existing files in the repo. If a previous run for a language wrote more shards than a later re-run produces (e.g. an earlier, larger run left data/da/train/003.parquet behind), those extra shards are silently left in the repo and included in the dataset by anyone loading it.

processing_scripts/clear_hub_dataset_shards.py clears a language's (or the whole repository's) existing shards out first, so a re-run starts from a clean slate. It defaults to a dry run that only lists what would be deleted — pass --delete to actually remove the files, which happens as a single commit (create_commit with a CommitOperationDelete per file). Since Hub dataset repositories are Git-backed, deleted files remain recoverable from the repo's commit history (e.g. via revision=<commit-sha> when loading, or HfApi.list_repo_commits) as long as that commit stays reachable — i.e. until the branch is force-pushed/rewritten or the commit is otherwise garbage-collected.

# Report what would be deleted for Danish, without deleting anything:
uv run processing_scripts/clear_hub_dataset_shards.py -l da

# Actually delete Danish's existing shards, with a confirmation prompt:
uv run processing_scripts/clear_hub_dataset_shards.py -l da --delete

# Delete every language's shards without a confirmation prompt:
uv run processing_scripts/clear_hub_dataset_shards.py --delete --yes

Some options worth knowing about (run --help for the full list):

-l/--language - restrict deletion to specific language(s) (repeatable), matching <path-in-repo>/<language>/; defaults to clearing every file under --path-in-repo.
--path-in-repo - repo-relative folder prefix shards are written under (default data, matching the layout described in Uploading multiple languages to the same Hub repository above).
-y/--yes - skip the confirmation prompt before deleting; only used with --delete.

Commands used to create the original `ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia` dataset

These commands were used to create the original ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia datasets that was used in the journal paper;

# This was ran on a SLURM cluster whereby the python executable had this code base installed via `pip install .`
python processing_scripts/run_all_training_languages.py ./log_data --executor slurm --slurm-partition cpu-48h --slurm-time 30:00:00 --hf-dataset-repo-id ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia --slurm-mem-per-cpu-gb 5 --slurm-venv-path /mnt/nfs/homes/mooreap1/wikipedia-USAS-processing/venv/bin/python --slurm-cpus-per-task 1 --max-workers-per-language 30 --max-tasks-per-language 30 --min-tasks-per-language 2 --languages-file ./usas_wikipedia_processing.yaml --slurm-sbatch-args "{\"nice\": 100}" --max-number-of-parallel-tasks 180 --shard-tasks-multiplier 1 --randomize-start-duration 65 --min-hash-threshold 0.85 --overwrite
# This command was ran after all of the languages had been processed, it was ran locally not on the SLURM cluster.
# this de-duplicated with respect to the Wikipedia Article/Page ID and re-balances the train and validation
# split.
uv run processing_scripts/deduplicate_wikipedia_dataset.py -p 10 --push

The first command used python rather than uv as we ran it on our SLURM cluster, in essence most of the time as we had a hard limit on the number of tasks that a user could submit to SLURM in one go (inclduing tasks that are scheduled but not running), we ended up running this command multiple times but processing different languages using --languages-file ./usas_wikipedia_processing.yaml file to state which languages ran via setting training to False for languages that we did not want to process data for. This command used 1 CPU with 5GB of RAM in total per task which is more than enough for this processing setup. Afterwards we ran the de-duplicating with respect to the Wikipedia Article/Page ID keeping the most recent version of the article script locally and ensured that each language had either 10% or at most 20 of the articles as validation data whichever was lower.

The various dataset statistics that compliment this dataset can be generated using the following commands;

# LaTeX table, ordered by language name alphabetically, written to a file:
uv run processing_scripts/report_pipeline_runtime_and_nodes.py ./log_data \
    --format latex --sort-by language --output-file ./data/tables/pipeline_runtime.tex

# Dropped-document funnel as a LaTeX table:
uv run processing_scripts/report_pipeline_document_funnel.py ./log_data \
    --view dropped --format latex --output-file ./data/tables/pipeline_funnel.tex --sort-by language

# Article and Sentence token statistics
uv run processing_scripts/token_count_distribution.py --split train --format latex --output-histogram-sentences data/plots/token_count_per_sentence_histogram.png --output-histogram-articles data/plots/token_count_per_article_histogram.png --output-table-sentences ./data/tables/token_count_per_sentence.tex --output-table-articles ./data/tables/token_count_per_article.tex

    
# Overall dataset statistics
uv run processing_scripts/dataset_statistics.py --hf-dataset-repo-id "ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia" --split all --hf-dataset-revision "main" --output-latex ./data/tables/overall_dataset_statistics.tex

# Overall dataset statistics total only values
uv run processing_scripts/dataset_statistics.py --hf-dataset-repo-id "ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia" --split combined --hf-dataset-revision "main" --output-latex ./data/tables/combined_overall_dataset_statistics.tex

# The number of documents removed from de-duplication using Wikipedia Article/Page ID table
uv run processing_scripts/report_deduplication_loss.py --output-latex ./data/tables/de_duplication_using_wikipedia_article_id.tex

# Tag distribution statistics
uv run processing_scripts/usas_tag_distribution.py --hf-dataset-repo-id "ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia" --hf-dataset-revision "main" --split all --top-bottom-count 5 --format latex --output-table-major ./data/tables/major_tag_distribution.tex --output-table-top ./data/tables/top_tags_distributi
on.tex --output-table-bottom ./data/tables/bottom_tags_distribution.tex --output-table-summary ./data/tables/tag_frequency_summary.tex

The README for the dataset that is used in the HuggingFace Hub can be found at ./data/template_readmes/Multilingual_USAS_Labelled_Silver_Wikipedia.md

Benchmarking spaCy model speed

processing_scripts/benchmark_da_spacy_model_speed.py compares the processing speed of the two Danish spaCy models (da_core_news_lg and da_core_news_trf) across a sweep of sentence counts, at a fixed sentence length of 21 tokens — both figures derived from the observed corpus statistics (an average of 362 sentences and 7556 tokens per document, i.e. ~21 tokens/sentence). It caps the tokens processed for any single benchmarked point at 100,000 by default.

Results can be shown as a graph, a console table, or both, and the table can additionally be exported as a LaTeX tabular:

# Save a comparison graph (default: ./da_spacy_model_speed.png)
uv run processing_scripts/benchmark_da_spacy_model_speed.py

# Print a table to the console instead
uv run processing_scripts/benchmark_da_spacy_model_speed.py --format table

# Also export the table as LaTeX, independent of --format
uv run processing_scripts/benchmark_da_spacy_model_speed.py --latex-output ./results.tex

Run --help for the full list of options, including --token-length, --max-tokens, and --num-points.

NOTE: the arguments used to produce the table and plot in the paper;

uv run processing_scripts/benchmark_da_spacy_model_speed.py --token-length 21 --max-tokens 100000 --num-points 8 --format plot --output data/plots/da_spacy_model_speed.png --latex-output data/tables/da_spacy_model_speed.tex

License

The code is licensed under Apache License Version 2.0.

Claude settings

For those that use Anthropic's Claude we have shared some suggested settings, see ./.claude folder that are enforced within this project but can be easily adjusted or removed if you prefer to use your own settings or the default settings of Claude. The project level settings for Claude, can be found at ./.claude/settings.json are auto generated by running the following script;

cd .claude/hooks && uv run generate_settings.py > ../settings.json

This script creates a settings file with;

Numerous Deny permissions that have come from the list of files, stated in ./.claude/hooks/sensitive_patterns.py, that you do not want Claude to write/edit/read.
A pre-hook, ./.claude/hooks/block_sensitive_files.py, that catches any write/edit/read to the list of files that the Deny permissions might miss, e.g. a call to Python using Bash.

To note this pre-hook and Deny permissions would not stop Claude from write/edit/read if Claude requests the file through an unusual regex pattern like e*v to get the .env file, but this is a best effort try to reduce Claude's access to these more sensitive files. Generally speaking if you are using API keys reduce the scope as much as possible and limit the time and resource access while developing.

Name		Name	Last commit message	Last commit date
.claude		.claude
.devcontainer		.devcontainer
data/template_readmes		data/template_readmes
processing_scripts		processing_scripts
tests		tests
wikipedia_processing		wikipedia_processing
.dockerignore		.dockerignore
.gitignore		.gitignore
.python-version		.python-version
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
coding_style_format_example.py		coding_style_format_example.py
makefile		makefile
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Navigation Menu

Uh oh!

Repository files navigation

Wikipedia-USAS-processing

Setup

Dev Container

Local

Linting

Tests

HuggingFace Authentication

Data

Languages

Filtering and Processing

Train/validation split

Filtering and Processing script

Running on Slurm

Understanding tasks, workers, and resources on Slurm

Uploading multiple languages to the same Hub repository

Running every training language at once

Staying under a Slurm submitted-jobs quota with `--max-number-of-parallel-tasks`

Deduplicating an existing dataset

Dataset statistics

Token count distribution

USAS tag distribution

Pipeline runtime and peak node usage

Documents filtered at each pipeline stage

Estimating deduplication loss

Clearing stale shards from a Hub dataset repository

Commands used to create the original `ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia` dataset

Benchmarking spaCy model speed

License

Claude settings

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Wikipedia-USAS-processing

Setup

Dev Container

Local

Linting

Tests

HuggingFace Authentication

Data

Languages

Filtering and Processing

Train/validation split

Filtering and Processing script

Running on Slurm

Understanding tasks, workers, and resources on Slurm

Uploading multiple languages to the same Hub repository

Running every training language at once

Staying under a Slurm submitted-jobs quota with --max-number-of-parallel-tasks

Deduplicating an existing dataset

Dataset statistics

Token count distribution

USAS tag distribution

Pipeline runtime and peak node usage

Documents filtered at each pipeline stage

Estimating deduplication loss

Clearing stale shards from a Hub dataset repository

Commands used to create the original ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia dataset

Benchmarking spaCy model speed

License

Claude settings

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Staying under a Slurm submitted-jobs quota with `--max-number-of-parallel-tasks`

Commands used to create the original `ucrelnlp/Multilingual-USAS-Labelled-Silver-Wikipedia` dataset