Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 7 additions & 10 deletions docs/concepts/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,21 +37,18 @@ For contributors, it offers insight into the design principles and architecture

### What’s in this section

The **Concepts** section introduces the core ideas that define Texera’s design and operation:
- **[How Texera runs your workflow](how-texera-runs-your-workflow/):** pipelining, blocking operators, regions, workers and execution mode.

- **Workflows:** How users visually build and manage data pipelines.
- **Operators:** The modular units that perform data transformations.
- **Execution Engine:** The core component that executes workflows efficiently.
- **Data Model:** How Texera represents, stores, and streams data.
- **Architecture:** The high-level structure connecting frontend, backend, and execution layers.
More concept pages are still being written. Until then, these pages cover the same ground:

Each page below explores one of these areas in more depth, explaining how Texera’s internal components work together to support flexible, scalable, and interactive data analytics.
- **Workflows and operators:** [Guide for how to use Texera](/docs/tutorials/guide-for-how-to-use-texera/) and the [operator reference](/docs/reference/operators/).
- **Execution engine internals:** [Engine](/docs/reference/engine/).
- **Data storage:** [Storage](/docs/reference/storage/).
- **Architecture:** [Project structure](/docs/reference/project-structure/) and the [Overview](/docs/overview/).

---

### When to read this section

If you’re new to Texera, start with the **[Overview](/docs/overview/)** page to understand what the platform does.
If you’re new to Texera, start with the **[Overview](/docs/overview/)** page to understand what the platform does.
Then come here to learn *how it works under the hood*.

If you’re contributing to Texera or integrating it with other systems, the detailed concept pages — such as **Engine**, **Operator Framework**, and **Architecture** — will help you understand Texera’s internal design and extension points.
69 changes: 69 additions & 0 deletions docs/concepts/how-texera-runs-your-workflow/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

---
title: "How Texera Runs Your Workflow"
weight: 10
---

This page explains why some operators start early, some wait, and some finish first. You don't need it to use Texera, but it helps when a run looks "stuck".

## Data flows like an assembly line

Texera doesn't wait for one operator to finish before starting the next. Rows flow down the arrows in small batches, so a Filter can work on the first rows while the CSV reader is still reading the rest. This is called **pipelining**.

## Some operators must see everything first

Some operators can't output anything until they have seen **all** their input. You can't sort a list you haven't finished reading. These are **blocking** operators:

- **Sort**, **Aggregate**, **Distinct**
- Set operators: **Intersect**, **Difference**, **Symmetric Difference**
- **Join**: it reads one input fully first (the *build* side) and then streams the other
- Machine learning **training** operators

While a blocking operator reads, the operators after it show 0 rows. That's normal. Python UDFs written with `UDFTableOperator` behave the same way: they output only after an input is complete.

## Regions

Texera splits a workflow into **regions** at blocking points. Operators in one region run together; a region starts after the regions it depends on finish. Between regions, results are saved (**materialized**) so the next region can read them.

To see regions, click the **Layers** button in the editor toolbar and check **Regions**. On a narrow window the button can be hidden; widen the window if you don't see it.

<img src="layers-menu.png" alt="Layers menu with Grid, Regions, Workers, Status and Performance" width="160">

After you click **Run**, each region is drawn as a shaded area. Here Sort splits the workflow into two regions:

![Two regions: CSV reader and Sort in one, Filter in another](regions.png)

| Shade | Region is… |
|---|---|
| Gray | Waiting |
| Blue or yellow | Running (blue: reading an input that others wait for) |
| Green | Finished |

## Workers

An operator can run as several copies (**workers**) in parallel, each handling part of the data. For Python and other UDFs, set **Worker count** in the property panel. To show the count on each box, check **Workers** in the **Layers** menu.

## Execution mode

The **Settings** tab in the editor's left panel has **Execution Mode**:

- **Pipelined** (default): rows flow between operators as described above.
- **Materialized**: every operator finishes and saves its output before the next one starts. Usually slower, because operators don't overlap.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
43 changes: 11 additions & 32 deletions docs/examples/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,51 +27,30 @@ weight: 80
This section showcases example workflows, use cases, and demonstrations to help you understand Texera in action.
{{% /pageinfo %}}

Texera makes it easy to design and execute data analytics workflows visually.
Here, you’ll find a collection of **example workflows** that highlight Texera’s capabilities — from data ingestion and transformation to visualization and machine learning.
Texera makes it easy to design and execute data analytics workflows visually.

---

## 🧩 Example Workflows

Each example demonstrates how Texera operators can be combined to solve different types of data problems.
Step-by-step examples you can follow today:

- **Text Analytics Workflow**
Analyze text data using tokenization, keyword extraction, and word cloud visualization.
→ See the [Text Analytics Example](/docs/examples/text-analytics/)
- **[Filter and aggregate a CSV file](/docs/tutorials/create-dataset-upload-data/)**: upload data, keep the rows you need, and compute averages.
- **[Add a column with Python](/docs/tutorials/guide-to-use-python-udf/)**: write a small Python UDF, and see what common errors mean.
- **[Bring a Jupyter notebook into Texera](/docs/tutorials/migrate-jupyter-notebook/)**: turn notebook cells into a workflow.

- **Join and Filter Workflow**
Combine multiple datasets using joins and filters to create complex data pipelines.
→ See the [Join Operator Example](/docs/examples/join-and-filter/)

- **Machine Learning Workflow**
Build end-to-end ML workflows with data preprocessing, model training, and evaluation operators.
→ See the [Machine Learning Example](/docs/examples/machine-learning/)

- **Visualization Workflow**
Explore Texera’s interactive visual operators like scatter plots, histograms, and word clouds.
→ See the [Visualization Example](/docs/examples/visualization/)
Pages for text analytics, joins, machine learning and visualization examples are planned. Until then, see the [operator reference](/docs/reference/operators/).

---

## 💡 How to Run the Examples

To try these examples on your local Texera instance:
## 💡 Ready-made examples

1. Launch Texera following the [Getting Started](/docs/getting-started/) guide.
2. Open the **Workflow Editor** in your browser at `http://localhost:4200`.
3. Import an example workflow file (`.json`) into the editor.
4. Run the workflow to see Texera’s operators and data visualizations in action.
- **Your own Texera ([Docker install](/docs/getting-started/installing-using-docker/))**: start it with `docker compose --profile examples up` to pre-create two example workflows and their datasets.
- **Any Texera**: open **Hub → Workflows** to browse workflows others have made public.
- **Import a workflow file**: on **Your Work → Workflows**, click the cloud upload icon next to **Create Workflow** and choose a workflow `.json` or `.zip` file.

---

## 🧠 Want to Contribute an Example?

If you’ve built your own workflow and want to share it:
- Fork the [Texera repository](https://github.com/apache/texera).
- Add your workflow under `examples/` with a short README.
- Submit a pull request following our [Contribution Guidelines](/docs/contribution-guidelines/).

---

These examples are a great starting point for learning Texera’s **visual programming model** and understanding how different operators interact to form powerful data pipelines.
Write a tutorial page under `docs/tutorials/` with screenshots and a sample data file, then submit a pull request following our [Contribution Guidelines](/docs/contribution-guidelines/).
7 changes: 2 additions & 5 deletions docs/getting-started/ui-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,9 @@ Once logged in, you should see the following page:

### Navigation Bar

On the left sidebar, you can switch between different resource modules:
The left sidebar switches between Hub, Your Work (workflows, datasets, computing units and more) and, for admins, Admin. Each item is described in [The dashboard](/docs/tutorials/guide-for-how-to-use-texera/#the-dashboard).

- **Workflows** — manage workflow projects.
- **Datasets** — upload and manage data files.
- **Quota** — check usage statistics and resource consumption.
- **Admin** — manage system users (visible only to admins).
For a walkthrough of the workflow editor, see the [Guide for how to use Texera](/docs/tutorials/guide-for-how-to-use-texera/).

{{< alert color="success" title="Tip" >}}
Hover over the navigation icons to see quick tooltips for each section.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tags: [user-defined-functions, python]

[Home](../../../) > [User Defined Functions](../../) > [Python](../)

> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors.

### Input Properties

| Property | Requirement | Type | Default | Description |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tags: [user-defined-functions, python]

[Home](../../../) > [User Defined Functions](../../) > [Python](../)

> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors.

### Input Properties

| Property | Requirement | Type | Default | Description |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tags: [user-defined-functions, python]

[Home](../../../) > [User Defined Functions](../../) > [Python](../)

> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors.

### Input Properties

| Property | Requirement | Type | Default | Description |
Expand Down
67 changes: 67 additions & 0 deletions docs/tutorials/computing-units.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

---
title: "Computing Units"
weight: 15
---

A **computing unit** is the machine that runs your workflows. You pick one in the workflow editor's top bar before clicking **Run** ([how](../guide-for-how-to-use-texera/#run-a-workflow)). You can see all of yours under **Your Work → Compute**.

## Create one

Click **+ Computing Unit** in the top bar, or **Create Computing Unit** on the Compute page. What you see depends on how your Texera is set up:

- **Create Computing Unit** (hosted Texera): Texera starts a new machine for you. Choose:

| Field | What to pick |
|---|---|
| **Select RAM Size** | Memory. Start with the smallest; pick more if a run fails on large data. |
| **Select #CPU Core(s)** | More cores help only when operators run with several workers. |
| **Select #GPU(s)** | Only for deep learning in Python. Shown only if GPUs are available. |
| **Image** | The software installed on the unit. Keep **Default** unless told otherwise. Fixed for the unit's lifetime. |
| **Advanced Settings** | Shared memory (raise it for PyTorch) and JVM heap. Leave the defaults unless an admin tells you otherwise. |

- **Connect to a Local Computing Unit** (Texera on your own computer): enter a name and keep the suggested address.

A new unit shows a gold dot while it starts. You can select it once the dot turns green.

## Status dots

| Dot | Meaning |
|---|---|
| Green | Running: ready to use |
| Gold | Starting up, or shutting down |
| Red | Unavailable. Pick or create another unit |

## Tips

- **Reuse one unit** for all your workflows. You don't need one per workflow.
- **A new unit usually won't fix an error in your workflow**, such as a mistake in Python code or an operator setting. Read the error first ([where to find errors](../guide-for-how-to-use-texera/#where-to-find-errors)). A different unit *can* help when the problem is the unit itself: not enough memory, CPU or GPU, or a missing Python package or software image.
- **Terminating a unit stops it.** Your workflows and datasets are kept; the results of runs on it are lost. To get them back, run the workflow again on another unit.
- **Details**: click the eye icon next to a unit to see its CPU, memory and GPU limits. Local units have no limits and show `NaN`.
- **Python packages**: the **+** icon next to a unit manages its Python environments.

## Common messages

| Message | What to do |
|---|---|
| `You may only have N computing unit(s) running at the same time` | Reuse a running unit, terminate a Kubernetes unit, or disconnect a local unit you no longer need (trash icon). |
| `Insufficient CPU / memory / GPU available in the server` | Ask for less RAM, fewer cores or GPUs, or try again later. |
| **Run** is disabled | You only have read access to the selected unit. Pick a unit you own. |
54 changes: 0 additions & 54 deletions docs/tutorials/create-dataset-upload-data.md

This file was deleted.

Loading
Loading