DistilKaggle

A Distilled Dataset of Kaggle Jupyter Notebooks

MSR 2024
21st IEEE/ACM International Conference on Mining Software Repositories
Data and Tool Showcase Track

Download the dataset on Zenodo →

Overview

DistilKaggle is a curated dataset of publicly available Python Jupyter notebooks from Kaggle, spanning September 2015–October 2023. It brings together extracted code cells, Markdown cells, and notebook metrics to support research on notebook code quality, documentation, and structure.

The dataset was distilled from a collection of Kaggle notebooks identified using MetaKaggle metadata. The accompanying paper identifies 34 metrics, enabling researchers to select and analyze notebooks beyond the filters available on Kaggle.

DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks
Mojtaba Mostafavi Ghahfarokhi, Arash Asgari, Mohammad Abolnejadian, and Abbas Heydarnoori
MSR 2024 · Data and Tool Showcase Track · Lisbon, Portugal

Contents

The DistilKaggle dataset consists of three main CSV files:

1. code.csv: Contains over 12 million rows of code cells extracted from Kaggle notebooks. Each row is identified by the notebook's kernel ID and cell index, preserving its place within the source notebook.
2. markdown.csv: Contains over 5 million rows of Markdown cells extracted from Kaggle notebooks. As in code.csv, each row includes the kernel ID and cell index.
3. notebook_metrics.csv: Provides notebook features and metrics described in the accompanying paper for over 517,000 Python notebooks.

The code and Markdown cell dataset covers 542,051 notebooks, as reported in the paper. The metrics file covers a subset of these notebooks.

The files are available individually or together in DistilKaggle.tar.gz. You do not need both the archive and the individual files.

Usage

Researchers can use DistilKaggle to analyze notebook content, study relationships between code and documentation, or select notebooks based on their metrics without downloading the original notebook collection.

• Use notebook_metrics.csv for notebook-level analyses and filtering.
• Use code.csv and markdown.csv to examine cell content and ordering within notebooks.
• See the example analysis for predicting an author's Kaggle Performance Tier from notebook metrics.

Quick start

Download notebook_metrics.csv into your working directory and install pandas:

pip install pandas

Load the metrics and inspect the available columns:

import pandas as pd

metrics = pd.read_csv("notebook_metrics.csv")
print(metrics.shape)
print(metrics.head())
print(metrics.columns.tolist())

To explore a cell CSV without loading the whole file into memory, read a sample:

code_sample = pd.read_csv("code.csv", nrows=1000)
print(code_sample.head())

Repository structure

The downloadable dataset is hosted on Zenodo. This repository contains supporting metadata, processing scripts, and an example analysis.

Path	Purpose
0/–5/	Directories organized by Kaggle Performance Tier, containing notebook metadata, download logs, and tier-specific processing scripts
utility/1_aggregate_dataset.py	Aggregates data across Performance Tiers
utility/2_application.ipynb	Example analysis predicting an author's Performance Tier from notebook metrics

Citation

If you use DistilKaggle in your research, please cite the accompanying paper:

@inproceedings{mostafavi-msr2024-DistilKaggle,
  author    = {Mostafavi Ghahfarokhi, Mojtaba and Asgari, Arash and
               Abolnejadian, Mohammad and Heydarnoori, Abbas},
  title     = {{DistilKaggle}: A Distilled Dataset of {Kaggle} {Jupyter} Notebooks},
  booktitle = {Proceedings of the 21st International Conference on Mining Software Repositories},
  series    = {MSR '24},
  year      = {2024},
  pages     = {647--651},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  location  = {Lisbon, Portugal},
  doi       = {10.1145/3643991.3644882},
  url       = {https://doi.org/10.1145/3643991.3644882}
}

The dataset is archived separately on Zenodo: 10.5281/zenodo.10317389.

License and contact

• Dataset: CC BY 4.0, as specified in the Zenodo record.
• Repository code: MIT License.

For questions about the dataset, contact Mohammad Abolnejadian.