From d284c20e3968863b39d9c2702301f47688ab44b6 Mon Sep 17 00:00:00 2001 From: lovromazgon <8320753+lovromazgon@users.noreply.github.com> Date: Mon, 19 Oct 2026 08:26:04 +0000 Subject: [PATCH] [automated] Update connectors list --- .../5-connectors/7-list/1-elasticsearch.mdx | 274 + .../{8-activemq.mdx => 10-activemq.mdx} | 0 .../7-list/{16-box.mdx => 18-box.mdx} | 0 .../7-list/{1-file.mdx => 2-file.mdx} | 18 +- .../7-list/{18-chaos.mdx => 20-chaos.mdx} | 0 .../7-list/{23-dropbox.mdx => 25-dropbox.mdx} | 0 .../{24-dynamodb.mdx => 26-dynamodb.mdx} | 0 .../{2-generator.mdx => 3-generator.mdx} | 36 +- ...0-google-drive.mdx => 31-google-drive.mdx} | 0 .../7-list/{34-http.mdx => 35-http.mdx} | 0 .../{36-influxdb.mdx => 37-influxdb.mdx} | 0 .../7-list/{37-kinesis.mdx => 38-kinesis.mdx} | 0 .../7-list/{3-kafka.mdx => 4-kafka.mdx} | 226 +- .../7-list/{39-mongo.mdx => 40-mongo.mdx} | 0 docs/1-using/5-connectors/7-list/40-mysql.mdx | 353 - .../7-list/{4-log.mdx => 5-log.mdx} | 34 +- docs/1-using/5-connectors/7-list/6-mysql.mdx | 611 + .../5-connectors/7-list/6-postgres.mdx | 320 - .../7-list/{5-pgvector.mdx => 7-pgvector.mdx} | 34 +- docs/1-using/5-connectors/7-list/7-s3.mdx | 326 - .../5-connectors/7-list/8-postgres.mdx | 742 + docs/1-using/5-connectors/7-list/9-s3.mdx | 691 + src/connectorgen/denied-connectors.json | 4 +- static/connectors.json | 27042 +++++++++------- .../.metadata.yaml | 2 + .../.metadata.yaml | 2 + .../.metadata.yaml | 2 + .../.metadata.yaml | 2 + .../connector.yaml | 259 + .../.metadata.yaml | 2 + .../connector.yaml | 259 + .../.metadata.yaml | 2 + .../connector.yaml | 156 + .../.metadata.yaml | 2 + .../connector.yaml | 156 + .../.metadata.yaml | 2 + .../connector.yaml | 229 + .../.metadata.yaml | 2 + .../connector.yaml | 362 + .../.metadata.yaml | 2 + .../connector.yaml | 362 + .../.metadata.yaml | 2 + .../connector.yaml | 574 + .../.metadata.yaml | 2 + .../connector.yaml | 96 + .../.metadata.yaml | 2 + .../connector.yaml | 186 + .../.metadata.yaml | 2 + .../connector.yaml | 193 + .../.metadata.yaml | 2 + .../connector.yaml | 193 + .../.metadata.yaml | 2 + .../connector.yaml | 291 + .../.metadata.yaml | 2 + .../connector.yaml | 320 + .../.metadata.yaml | 2 + .../connector.yaml | 320 + .../.metadata.yaml | 2 + .../connector.yaml | 320 + .../.metadata.yaml | 2 + .../connector.yaml | 542 + .../.metadata.yaml | 2 + .../connector.yaml | 568 + .../.metadata.yaml | 2 + .../connector.yaml | 163 + .../.metadata.yaml | 2 + .../connector.yaml | 583 + .../.metadata.yaml | 2 + .../connector.yaml | 689 + .../.metadata.yaml | 2 + .../connector.yaml | 744 + .../.metadata.yaml | 2 + .../connector.yaml | 739 + .../.metadata.yaml | 2 + .../connector.yaml | 657 + .../.metadata.yaml | 2 + .../connector.yaml | 748 + .../.metadata.yaml | 2 + .../connector.yaml | 417 + 79 files changed, 27746 insertions(+), 13149 deletions(-) create mode 100644 docs/1-using/5-connectors/7-list/1-elasticsearch.mdx rename docs/1-using/5-connectors/7-list/{8-activemq.mdx => 10-activemq.mdx} (100%) rename docs/1-using/5-connectors/7-list/{16-box.mdx => 18-box.mdx} (100%) rename docs/1-using/5-connectors/7-list/{1-file.mdx => 2-file.mdx} (65%) rename docs/1-using/5-connectors/7-list/{18-chaos.mdx => 20-chaos.mdx} (100%) rename docs/1-using/5-connectors/7-list/{23-dropbox.mdx => 25-dropbox.mdx} (100%) rename docs/1-using/5-connectors/7-list/{24-dynamodb.mdx => 26-dynamodb.mdx} (100%) rename docs/1-using/5-connectors/7-list/{2-generator.mdx => 3-generator.mdx} (79%) rename docs/1-using/5-connectors/7-list/{30-google-drive.mdx => 31-google-drive.mdx} (100%) rename docs/1-using/5-connectors/7-list/{34-http.mdx => 35-http.mdx} (100%) rename docs/1-using/5-connectors/7-list/{36-influxdb.mdx => 37-influxdb.mdx} (100%) rename docs/1-using/5-connectors/7-list/{37-kinesis.mdx => 38-kinesis.mdx} (100%) rename docs/1-using/5-connectors/7-list/{3-kafka.mdx => 4-kafka.mdx} (51%) rename docs/1-using/5-connectors/7-list/{39-mongo.mdx => 40-mongo.mdx} (100%) delete mode 100644 docs/1-using/5-connectors/7-list/40-mysql.mdx rename docs/1-using/5-connectors/7-list/{4-log.mdx => 5-log.mdx} (74%) create mode 100644 docs/1-using/5-connectors/7-list/6-mysql.mdx delete mode 100644 docs/1-using/5-connectors/7-list/6-postgres.mdx rename docs/1-using/5-connectors/7-list/{5-pgvector.mdx => 7-pgvector.mdx} (78%) delete mode 100644 docs/1-using/5-connectors/7-list/7-s3.mdx create mode 100644 docs/1-using/5-connectors/7-list/8-postgres.mdx create mode 100644 docs/1-using/5-connectors/7-list/9-s3.mdx create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.3.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.4.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.4.1/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.5.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.5.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.6.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-elasticsearch@v0.6.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-file@v0.10.10/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-file@v0.10.10/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-file@v0.10.9/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-file@v0.10.9/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-generator@v0.10.7/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-generator@v0.10.7/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.12.6/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.12.6/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.13.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.13.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.14.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-kafka@v0.14.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-log@v0.7.6/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-log@v0.7.6/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.1/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.1/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.2/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.2/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.3/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.3/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.4/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.1.4/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.2.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.2.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.3.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.3.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.4.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.4.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.11.0/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.11.0/connector.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.9.5/.metadata.yaml create mode 100644 static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.9.5/connector.yaml diff --git a/docs/1-using/5-connectors/7-list/1-elasticsearch.mdx b/docs/1-using/5-connectors/7-list/1-elasticsearch.mdx new file mode 100644 index 0000000..0a5f74c --- /dev/null +++ b/docs/1-using/5-connectors/7-list/1-elasticsearch.mdx @@ -0,0 +1,274 @@ +--- +IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. + +title: "elasticsearch" +description: "An Elasticsearch destination plugin for Conduit." +--- + +import ReactDiffViewer from 'react-diff-viewer-continued'; +import Chip from '@mui/material/Chip'; +import Box from "@mui/system/Box"; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import GitHubIcon from '@mui/icons-material/GitHub'; +import IconButton from '@mui/material/IconButton'; +import Stack from '@mui/material/Stack'; +import Link from '@mui/material/Link'; +import StarIcon from '@mui/icons-material/Star'; +import Tooltip from '@mui/material/Tooltip'; + +# elasticsearch + + + {/* Author info */} + + Author: Miquido, Meroxa, Inc. + + Conduit team logo + + + + {/* Divider */} + + + {/* GitHub link */} + + + + + {/* Stars */} + } + label="0" + size="small" + sx={{ height: 24 }} + /> + + +## Latest release + +- [conduit-connector-elasticsearch_0.6.0_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Darwin_arm64.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Darwin_x86_64.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Linux_arm64.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Linux_i386.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Linux_x86_64.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Windows_arm64.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Windows_i386.tar.gz) +- [conduit-connector-elasticsearch_0.6.0_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/conduit-connector-elasticsearch_0.6.0_Windows_x86_64.tar.gz) +- [elasticsearch_v0.6.0_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_darwin_amd64.tar.gz) +- [elasticsearch_v0.6.0_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_darwin_amd64.tar.gz.sigstore.json) +- [elasticsearch_v0.6.0_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_darwin_arm64.tar.gz) +- [elasticsearch_v0.6.0_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_darwin_arm64.tar.gz.sigstore.json) +- [elasticsearch_v0.6.0_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_linux_amd64.tar.gz) +- [elasticsearch_v0.6.0_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_linux_amd64.tar.gz.sigstore.json) +- [elasticsearch_v0.6.0_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_linux_arm64.tar.gz) +- [elasticsearch_v0.6.0_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-elasticsearch/releases/download/v0.6.0/elasticsearch_v0.6.0_linux_arm64.tar.gz.sigstore.json) + +## Description + +The Conduit plugin supporting Elasticsearch destination. + +## Source Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-source + type: source + plugin: "elasticsearch" + name: example-source + settings: + # The Elasticsearch host and port (e.g.: http://127.0.0.1:9200). + # Type: string + host: "" + # The version of the Elasticsearch service. One of: 5, 6, 7, 8. + # Type: string + version: "" + # Base64-encoded token for authorization; if set, overrides + # username/password and service token. + # Type: string + APIKey: "" + # The number of items stored in bulk in the index. The minimum value + # is `1`, maximum value is `10000`. + # Type: int + batchSize: "1000" + # SHA256 hex fingerprint given by Elasticsearch on first launch. + # Type: string + certificateFingerprint: "" + # Endpoint for the Elastic Service (https://elastic.co/cloud). + # Type: string + cloudID: "" + # The sortby field for each index to be used by elasticsearch search + # api. + # Type: string + indexes.*.sortBy: "_seq_no" + # The sortOrder(asc or desc) for each index to be used by + # elasticsearch search api. + # Type: string + indexes.*.sortOrder: "asc" + # The password for HTTP Basic Authentication. + # Type: string + password: "" + # This period is used by workers to poll for new data at regular + # intervals. + # Type: duration + pollingPeriod: "5s" + # The maximum number of retries of failed operations. + # Type: int + retries: "0" + # Service token for authorization; if set, overrides + # username/password. + # Type: string + serviceToken: "" + # The username for HTTP Basic Authentication. + # Type: string + username: "" + # Maximum delay before an incomplete batch is read from the source. + # Type: duration + sdk.batch.delay: "0" + # Maximum size of batch before it gets read from the source. + # Type: int + sdk.batch.size: "0" + # Specifies whether to use a schema context name. If set to false, no + # schema context name will be used, and schemas will be saved with the + # subject name specified in the connector (not safe because of name + # conflicts). + # Type: bool + sdk.schema.context.enabled: "true" + # Schema context name to be used. Used as a prefix for all schema + # subject names. If empty, defaults to the connector ID. + # Type: string + sdk.schema.context.name: "" + # Whether to extract and encode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "false" + # The subject of the key schema. If the record metadata contains the + # field "opencdc.collection" it is prepended to the subject name and + # separated with a dot. + # Type: string + sdk.schema.extract.key.subject: "key" + # Whether to extract and encode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "true" + # The subject of the payload schema. If the record metadata contains + # the field "opencdc.collection" it is prepended to the subject name + # and separated with a dot. + # Type: string + sdk.schema.extract.payload.subject: "payload" + # The type of the payload schema. + # Type: string + sdk.schema.extract.type: "avro" +``` + +## Destination Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-destination + type: destination + plugin: "elasticsearch" + name: example-destination + settings: + # The Elasticsearch host and port (e.g.: http://127.0.0.1:9200). + # Type: string + host: "" + # The version of the Elasticsearch service. One of: 5, 6, 7, 8. + # Type: string + version: "" + # Base64-encoded token for authorization; if set, overrides + # username/password and service token. + # Type: string + APIKey: "" + # The number of items stored in bulk in the index. The minimum value + # is `1`, maximum value is `10 000`. + # Type: int + bulkSize: "1000" + # SHA256 hex fingerprint given by Elasticsearch on first launch. + # Type: string + certificateFingerprint: "" + # Endpoint for the Elastic Service (https://elastic.co/cloud). + # Type: string + cloudID: "" + # The name of the index to write the data to. + # Type: string + index: "{{ index .Metadata "opencdc.collection" }}" + # The password for HTTP Basic Authentication. + # Type: string + password: "" + # The maximum number of retries of failed operations. The minimum + # value is `0` which disabled retry logic. The maximum value is `255. + # Type: int + retries: "0" + # Service token for authorization; if set, overrides + # username/password. + # Type: string + serviceToken: "" + # The name of the index's type to write the data to. + # Type: string + type: "" + # The username for HTTP Basic Authentication. + # Type: string + username: "" + # Maximum delay before an incomplete batch is written to the + # destination. + # Type: duration + sdk.batch.delay: "0" + # Maximum size of batch before it gets written to the destination. + # Type: int + sdk.batch.size: "0" + # Allow bursts of at most X records (0 or less means that bursts are + # not limited). Only takes effect if a rate limit per second is set. + # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the + # effective batch size will be equal to `sdk.rate.burst`. + # Type: int + sdk.rate.burst: "0" + # Maximum number of records written per second (0 means no rate + # limit). + # Type: float + sdk.rate.perSecond: "0" + # The format of the output record. See the Conduit documentation for a + # full list of supported formats + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + # Type: string + sdk.record.format: "opencdc/json" + # Options to configure the chosen output record format. Options are + # normally key=value pairs separated with comma (e.g. + # opt1=val2,opt2=val2), except for the `template` record format, where + # options are a Go template. + # Type: string + sdk.record.format.options: "" + # Whether to extract and decode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "true" + # Whether to extract and decode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "true" +``` diff --git a/docs/1-using/5-connectors/7-list/8-activemq.mdx b/docs/1-using/5-connectors/7-list/10-activemq.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/8-activemq.mdx rename to docs/1-using/5-connectors/7-list/10-activemq.mdx diff --git a/docs/1-using/5-connectors/7-list/16-box.mdx b/docs/1-using/5-connectors/7-list/18-box.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/16-box.mdx rename to docs/1-using/5-connectors/7-list/18-box.mdx diff --git a/docs/1-using/5-connectors/7-list/1-file.mdx b/docs/1-using/5-connectors/7-list/2-file.mdx similarity index 65% rename from docs/1-using/5-connectors/7-list/1-file.mdx rename to docs/1-using/5-connectors/7-list/2-file.mdx index 87b5293..9924d3b 100644 --- a/docs/1-using/5-connectors/7-list/1-file.mdx +++ b/docs/1-using/5-connectors/7-list/2-file.mdx @@ -68,22 +68,6 @@ import Tooltip from '@mui/material/Tooltip'; ## Latest release -- [conduit-connector-file_0.10.8_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Darwin_arm64.tar.gz) -- [conduit-connector-file_0.10.8_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Darwin_x86_64.tar.gz) -- [conduit-connector-file_0.10.8_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Linux_arm64.tar.gz) -- [conduit-connector-file_0.10.8_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Linux_i386.tar.gz) -- [conduit-connector-file_0.10.8_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Linux_x86_64.tar.gz) -- [conduit-connector-file_0.10.8_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Windows_arm64.tar.gz) -- [conduit-connector-file_0.10.8_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Windows_i386.tar.gz) -- [conduit-connector-file_0.10.8_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/conduit-connector-file_0.10.8_Windows_x86_64.tar.gz) -- [file_v0.10.8_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_darwin_amd64.tar.gz) -- [file_v0.10.8_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_darwin_amd64.tar.gz.sigstore.json) -- [file_v0.10.8_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_darwin_arm64.tar.gz) -- [file_v0.10.8_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_darwin_arm64.tar.gz.sigstore.json) -- [file_v0.10.8_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_linux_amd64.tar.gz) -- [file_v0.10.8_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_linux_amd64.tar.gz.sigstore.json) -- [file_v0.10.8_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_linux_arm64.tar.gz) -- [file_v0.10.8_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-file/releases/download/v0.10.8/file_v0.10.8_linux_arm64.tar.gz.sigstore.json) ## Description @@ -192,7 +176,7 @@ pipelines: sdk.rate.perSecond: "0" # The format of the output record. See the Conduit documentation for a # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). # Type: string sdk.record.format: "opencdc/json" # Options to configure the chosen output record format. Options are diff --git a/docs/1-using/5-connectors/7-list/18-chaos.mdx b/docs/1-using/5-connectors/7-list/20-chaos.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/18-chaos.mdx rename to docs/1-using/5-connectors/7-list/20-chaos.mdx diff --git a/docs/1-using/5-connectors/7-list/23-dropbox.mdx b/docs/1-using/5-connectors/7-list/25-dropbox.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/23-dropbox.mdx rename to docs/1-using/5-connectors/7-list/25-dropbox.mdx diff --git a/docs/1-using/5-connectors/7-list/24-dynamodb.mdx b/docs/1-using/5-connectors/7-list/26-dynamodb.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/24-dynamodb.mdx rename to docs/1-using/5-connectors/7-list/26-dynamodb.mdx diff --git a/docs/1-using/5-connectors/7-list/2-generator.mdx b/docs/1-using/5-connectors/7-list/3-generator.mdx similarity index 79% rename from docs/1-using/5-connectors/7-list/2-generator.mdx rename to docs/1-using/5-connectors/7-list/3-generator.mdx index 079c0d3..09e6130 100644 --- a/docs/1-using/5-connectors/7-list/2-generator.mdx +++ b/docs/1-using/5-connectors/7-list/3-generator.mdx @@ -68,22 +68,22 @@ import Tooltip from '@mui/material/Tooltip'; ## Latest release -- [conduit-connector-generator_0.10.6_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Darwin_arm64.tar.gz) -- [conduit-connector-generator_0.10.6_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Darwin_x86_64.tar.gz) -- [conduit-connector-generator_0.10.6_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Linux_arm64.tar.gz) -- [conduit-connector-generator_0.10.6_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Linux_i386.tar.gz) -- [conduit-connector-generator_0.10.6_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Linux_x86_64.tar.gz) -- [conduit-connector-generator_0.10.6_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Windows_arm64.tar.gz) -- [conduit-connector-generator_0.10.6_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Windows_i386.tar.gz) -- [conduit-connector-generator_0.10.6_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/conduit-connector-generator_0.10.6_Windows_x86_64.tar.gz) -- [generator_v0.10.6_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_darwin_amd64.tar.gz) -- [generator_v0.10.6_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_darwin_amd64.tar.gz.sigstore.json) -- [generator_v0.10.6_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_darwin_arm64.tar.gz) -- [generator_v0.10.6_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_darwin_arm64.tar.gz.sigstore.json) -- [generator_v0.10.6_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_linux_amd64.tar.gz) -- [generator_v0.10.6_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_linux_amd64.tar.gz.sigstore.json) -- [generator_v0.10.6_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_linux_arm64.tar.gz) -- [generator_v0.10.6_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.6/generator_v0.10.6_linux_arm64.tar.gz.sigstore.json) +- [conduit-connector-generator_0.10.7_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Darwin_arm64.tar.gz) +- [conduit-connector-generator_0.10.7_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Darwin_x86_64.tar.gz) +- [conduit-connector-generator_0.10.7_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Linux_arm64.tar.gz) +- [conduit-connector-generator_0.10.7_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Linux_i386.tar.gz) +- [conduit-connector-generator_0.10.7_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Linux_x86_64.tar.gz) +- [conduit-connector-generator_0.10.7_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Windows_arm64.tar.gz) +- [conduit-connector-generator_0.10.7_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Windows_i386.tar.gz) +- [conduit-connector-generator_0.10.7_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/conduit-connector-generator_0.10.7_Windows_x86_64.tar.gz) +- [generator_v0.10.7_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_darwin_amd64.tar.gz) +- [generator_v0.10.7_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_darwin_amd64.tar.gz.sigstore.json) +- [generator_v0.10.7_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_darwin_arm64.tar.gz) +- [generator_v0.10.7_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_darwin_arm64.tar.gz.sigstore.json) +- [generator_v0.10.7_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_linux_amd64.tar.gz) +- [generator_v0.10.7_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_linux_amd64.tar.gz.sigstore.json) +- [generator_v0.10.7_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_linux_arm64.tar.gz) +- [generator_v0.10.7_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-generator/releases/download/v0.10.7/generator_v0.10.7_linux_arm64.tar.gz.sigstore.json) ## Description @@ -215,8 +215,8 @@ pipelines: # generated (0 means no rate limit). # Type: float rate: "" - # The time it takes to 'read' a record. Deprecated: use `rate` - # instead. + # The time it takes to 'read' a record. + # Deprecated: use `rate` instead. # Type: duration readTime: "" # Number of records to be generated (0 means infinite). diff --git a/docs/1-using/5-connectors/7-list/30-google-drive.mdx b/docs/1-using/5-connectors/7-list/31-google-drive.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/30-google-drive.mdx rename to docs/1-using/5-connectors/7-list/31-google-drive.mdx diff --git a/docs/1-using/5-connectors/7-list/34-http.mdx b/docs/1-using/5-connectors/7-list/35-http.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/34-http.mdx rename to docs/1-using/5-connectors/7-list/35-http.mdx diff --git a/docs/1-using/5-connectors/7-list/36-influxdb.mdx b/docs/1-using/5-connectors/7-list/37-influxdb.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/36-influxdb.mdx rename to docs/1-using/5-connectors/7-list/37-influxdb.mdx diff --git a/docs/1-using/5-connectors/7-list/37-kinesis.mdx b/docs/1-using/5-connectors/7-list/38-kinesis.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/37-kinesis.mdx rename to docs/1-using/5-connectors/7-list/38-kinesis.mdx diff --git a/docs/1-using/5-connectors/7-list/3-kafka.mdx b/docs/1-using/5-connectors/7-list/4-kafka.mdx similarity index 51% rename from docs/1-using/5-connectors/7-list/3-kafka.mdx rename to docs/1-using/5-connectors/7-list/4-kafka.mdx index e47584c..48c72bc 100644 --- a/docs/1-using/5-connectors/7-list/3-kafka.mdx +++ b/docs/1-using/5-connectors/7-list/4-kafka.mdx @@ -68,22 +68,8 @@ import Tooltip from '@mui/material/Tooltip'; ## Latest release -- [conduit-connector-kafka_0.12.5_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Darwin_arm64.tar.gz) -- [conduit-connector-kafka_0.12.5_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Darwin_x86_64.tar.gz) -- [conduit-connector-kafka_0.12.5_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Linux_arm64.tar.gz) -- [conduit-connector-kafka_0.12.5_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Linux_i386.tar.gz) -- [conduit-connector-kafka_0.12.5_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Linux_x86_64.tar.gz) -- [conduit-connector-kafka_0.12.5_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Windows_arm64.tar.gz) -- [conduit-connector-kafka_0.12.5_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Windows_i386.tar.gz) -- [conduit-connector-kafka_0.12.5_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/conduit-connector-kafka_0.12.5_Windows_x86_64.tar.gz) -- [kafka_v0.12.5_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_darwin_amd64.tar.gz) -- [kafka_v0.12.5_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_darwin_amd64.tar.gz.sigstore.json) -- [kafka_v0.12.5_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_darwin_arm64.tar.gz) -- [kafka_v0.12.5_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_darwin_arm64.tar.gz.sigstore.json) -- [kafka_v0.12.5_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_linux_amd64.tar.gz) -- [kafka_v0.12.5_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_linux_amd64.tar.gz.sigstore.json) -- [kafka_v0.12.5_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_linux_arm64.tar.gz) -- [kafka_v0.12.5_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.12.5/kafka_v0.12.5_linux_arm64.tar.gz.sigstore.json) +- [kafka_v0.14.0_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.14.0/kafka_v0.14.0_darwin_arm64.tar.gz) +- [kafka_v0.14.0_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-kafka/releases/download/v0.14.0/kafka_v0.14.0_darwin_arm64.tar.gz.sigstore.json) ## Description @@ -95,8 +81,26 @@ consumer's offset in Kafka. Internally, though, we're not saving the offset as the position: instead, we're saving the consumer group ID, since that's all which is needed for Kafka to find the offsets for our consumer. -A source is getting associated with a consumer group ID the first time the -`Read()` method is called. +A source is associated with a consumer group ID when it is first opened. Set +`groupID` to choose it; otherwise a random one is generated and kept in the +position. + +### Headers + +Each Kafka header is emitted as a metadata entry `kafka.header.` holding +the header value. Metadata is a string map, so a key that repeats in one +record keeps its last value. + +A header whose key or value is not valid UTF-8 is emitted unchanged, but a +record carrying it cannot be marshalled as an OpenCDC protobuf record: it +fails wherever records cross a protobuf boundary (a standalone connector or +processor, the API), and JSON encoders replace the invalid bytes with U+FFFD. +The source logs `KAFKA_LEGACY_HEADER_NOT_UTF8` at WARN once per header key, +with the key as base64 in `header_key_b64` (the value is never logged), and +reports the count as `kafka_source_legacy_header_not_utf8_total` in the logs +and at teardown. A future version will omit such entries by default, after +the deprecation period +([#319](https://github.com/ConduitIO/conduit-connector-kafka/issues/319)). ## Destination @@ -122,6 +126,160 @@ Batching can also be configured using connector SDK provided options: - `sdk.batch.delay`: maximum delay before an incomplete batch is written to the destination (defaults to 0, no limit) +## Delivery semantics + +These are the guarantees that the connector's CI checks with the SDK +acceptance suite v2 and a kill -9 test, and the conditions under which they +hold. + +### Source guarantees + +- **At-least-once.** The source commits a record's offset to its consumer group + only after Conduit acknowledged the record. It never commits an offset it + has not been acked for. +- **Commits are batched**, so a restart after a crash re-delivers records: + offsets are committed after every `commitOffsetsSize` acks (default 1000), + every `commitOffsetsDelay` (default 5s), and when the pipeline stops + gracefully. After a crash (kill -9, power loss) the source re-delivers at + most the records acked since the last commit, fewer than `commitOffsetsSize` + when commits succeed. Setting `commitOffsetsSize` to 1 commits on every ack + and lowers throughput. +- **Ordering:** records of one partition are delivered in offset order, so + records with the same key (which Kafka keeps in one partition) arrive in the + order they were written. Re-delivery after a restart repeats a suffix of each + partition, in order. There is no order between partitions. +- **Resume uses the consumer group**, not the offset in the Conduit position. + If the group's committed offsets are gone (the group was deleted, or the + pipeline was stopped longer than the broker's `offsets.retention.minutes`, + 7 days by default), the source starts where `readFromBeginning` says: from + the beginning of the topic (re-delivery), or from the end, which **skips the + records written while the pipeline was stopped**. Use + `readFromBeginning: true` where skipping is not acceptable. +- **Restart after a crash can pause.** The crashed instance stays a member of + the consumer group until the broker's session timeout expires (45s by + default), and the partitions it held are not consumed until then. + +### Destination guarantees + +- **At-least-once.** A record is acknowledged to Conduit only after the broker + confirmed it. With `acks: all` (the default) that means the in-sync replicas + have it. `acks: one` acknowledges after the leader alone has it, and a leader + failure can lose acknowledged records; `acks: none` acknowledges without any + confirmation. Only `acks: all` keeps the at-least-once guarantee. +- **Not idempotent.** A record that is replayed after a crash is written again. + The idempotent producer (on with `acks: all`) removes duplicates from its own + retries of a produce request, not from a record sent twice. +- **Ordering:** records with the same key go to the same partition (key hash) + in the order Conduit wrote them, as long as the partition count of the topic + does not change. One exception: if the broker rejects a record after it was + enqueued (for example `MESSAGE_TOO_LARGE`), the later records of the same + batch may already be written. Conduit is told the batch failed from that + record on, and when the records are replayed the rejected record is stored + after them, together with duplicates of them. +- **Writes can wait for an unreachable broker without a bound** when + `deliveryTimeout` is 0 (the default). Cancelling the pipeline ends the wait + and nacks the records; nothing unconfirmed is acknowledged. +- **Large records:** a record must fit in `batchBytes` (default 1000012 bytes) + and in the topic's `max.message.bytes`, after the output format is applied. + The default `opencdc/json` format base64-encodes raw payloads, which adds + about a third, so with the defaults the largest raw payload is about + 730 KiB (roughly 750000 bytes, less the record's key and metadata). Larger + records need a higher `batchBytes` and a higher `max.message.bytes` on the + topic (and on the broker, if its `message.max.bytes` is lower). The + acceptance suite's 1 MiB records run with both set to 4 MiB. + +### Acceptance suite claims + +The acceptance suite v2 run (revision 2.0.1) claims per-key ordering for the +source, a redelivery bound of 1000 records after a restart, and a +non-idempotent destination. + +- **Redelivery bound.** The suite's default bound is 10. The source's bound, + read from the code, is `commitOffsetsSize`: offsets are committed + synchronously once that many acks have accumulated, so at most + `commitOffsetsSize`-1 acked records are uncommitted when the process dies. + The suite's restart case and the kill -9 test did not exceed it, which + shows it is not violated in those runs; it is not a measured maximum. The + bound of 1000 holds only for `commitOffsetsSize` up to 1000 (the default). + A larger value raises the bound with it, and such a configuration is + outside what the certification covers. +- **Destination per-key ordering is not claimed**, and the suite's ordering + case for the destination is skipped, because of the exception above: a + record the broker rejects after it was enqueued is stored after later + records of its batch when the batch is replayed. +- **Killing a source in-process.** The suite kills a source inside the test + process, which leaves its consumer group membership alive. The test closes + the killed source's client without committing, which releases its + partitions. A real process death sends no LeaveGroup: the broker drops the + member only when its session times out (see "Restart after a crash can + pause" above). The kill -9 test covers a real process death. + +## Mirroring (Kafka→Kafka) + +To copy a topic faithfully, preserving key, value, headers, timestamp and +partition, set `emitMirrorMetadata: true` on the source and `mirror: true` on +the destination: + +```yaml +version: 2.2 +pipelines: + - id: mirror + status: running + connectors: + - id: source + plugin: "kafka" + type: source + settings: + servers: "localhost:9092" + topics: "orders" + readFromBeginning: "true" + emitMirrorMetadata: "true" + - id: destination + plugin: "kafka" + type: destination + settings: + servers: "localhost:9092" + topic: "orders.mirror" + mirror: "true" +``` + +`mirror` writes each record to the destination partition named by the source +partition, so a consumer of the copy sees the same per-partition order as the +source. Rules and caveats: + +- The destination topic must already exist (mirror mode does not auto-create + it) and have at least as many partitions as the source topic; otherwise the + first record fails with `KAFKA_PARTITION_COUNT_MISMATCH`, and a record whose + partition is missing fails with `KAFKA_PARTITION_OUT_OF_RANGE`. A + destination with more partitions is allowed: the extra partitions stay + empty, and key-hash locality is not preserved. +- `mirror` requires `acks: all` and rejects `deliveryTimeout` and a user-set + `sdk.record.format`; it writes the record value bytes verbatim, bypassing + the output format. +- Per-partition order is preserved until the first produce error after a + record is enqueued (for example the broker rejects it); after that the + partition may hold later records before an earlier one, and the connector + logs a WARN with `kafka_destination_post_enqueue_errors_total`. The remedy + is to re-seed the affected partition. +- A mirror pipeline must keep the default error policy (a DLQ with + `window-size: 0` or a nack threshold above 0 turns a configuration error + into silent record loss). +- Timestamp topic-config checks are not yet enforced: a destination topic + with `message.timestamp.type=LogAppendTime`, a short `retention.ms`, or + `message.timestamp.before/after.max.ms` bounds may rewrite, expire or + reject preserved timestamps. The connector preserves the timestamp but + does not yet fail on these topic configurations. + +## Redpanda + +The kafka connector is the Redpanda profile: the retired labs `redpanda` +connector is replaced by this connector pointed at a Redpanda cluster +(`servers` set to the Redpanda Kafka API address). CI runs the integration, +acceptance-v2 and kill -9 tests against both Apache Kafka and Redpanda. Mirror +mode works on both brokers; Redpanda does not return or enforce the +`message.timestamp.before/after.max.ms` bounds, so only the timestamp type and +retention rules can differ from Apache Kafka. + ## Source Parameters ```yaml @@ -163,6 +321,17 @@ pipelines: # be committed at a time. # Type: int commitOffsetsSize: "1000" + # EmitMirrorMetadata, when true, additionally emits lossless mirror + # metadata on every record: kafka.partition, kafka.partitions, + # kafka.offset, kafka.timestamp, kafka.timestamp.type, kafka.headers, + # kafka.key.null and kafka.value.null. It is off by default because + # metadata is carried to every destination, and destinations that + # serialize it write different bytes for pipelines that never asked + # for mirroring. Enable it together with the destination's mirror + # option to preserve header, partition and timestamp across a + # Kafka-to-Kafka copy. + # Type: bool + emitMirrorMetadata: "" # GroupID defines the consumer group id. # Type: string groupID: "" @@ -255,6 +424,12 @@ pipelines: # writes, all = wait for the full ISR to acknowledge the writes. # Type: string acks: "all" + # AllowLossyHeaders, valid only with mirror, writes headers rebuilt + # from kafka.header. entries when kafka.headers is absent (repeated + # keys collapsed, order undefined). Without it, mirror mode fails a + # record whose kafka.headers is missing. + # Type: bool + allowLossyHeaders: "" # BatchBytes limits the maximum size of a request in bytes before # being sent to a partition. This mirrors Kafka's max.message.bytes. # Type: int @@ -284,6 +459,19 @@ pipelines: # presented by the server and any host name in that certificate. # Type: bool insecureSkipVerify: "" + # Mirror enables Kafka-to-Kafka mirror mode: it writes the record's + # key, value, headers, timestamp and partition to Kafka verbatim from + # the mirror metadata the source emits when its emitMirrorMetadata + # option is on. It requires acks=all, rejects deliveryTimeout > 0 and + # a user-set sdk.record.format, and disables topic auto-creation. See + # the kafka mirror passthrough design in ConduitIO/conduit. + # Type: bool + mirror: "" + # ProduceTimeout is a connector-side bound on a Write in mirror mode, + # because deliveryTimeout must stay unset there. Minimum 30s, maximum + # 1h. + # Type: duration + produceTimeout: "2m" # Mechanism configures the connector to use SASL authentication. If # empty, no authentication will be performed. # Type: string @@ -323,7 +511,7 @@ pipelines: sdk.rate.perSecond: "0" # The format of the output record. See the Conduit documentation for a # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). # Type: string sdk.record.format: "opencdc/json" # Options to configure the chosen output record format. Options are diff --git a/docs/1-using/5-connectors/7-list/39-mongo.mdx b/docs/1-using/5-connectors/7-list/40-mongo.mdx similarity index 100% rename from docs/1-using/5-connectors/7-list/39-mongo.mdx rename to docs/1-using/5-connectors/7-list/40-mongo.mdx diff --git a/docs/1-using/5-connectors/7-list/40-mysql.mdx b/docs/1-using/5-connectors/7-list/40-mysql.mdx deleted file mode 100644 index cbf9eab..0000000 --- a/docs/1-using/5-connectors/7-list/40-mysql.mdx +++ /dev/null @@ -1,353 +0,0 @@ ---- -IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. - -title: "mysql" -description: "A Conduit Connector for MySQL" ---- - -import ReactDiffViewer from 'react-diff-viewer-continued'; -import Chip from '@mui/material/Chip'; -import Box from "@mui/system/Box"; -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import GitHubIcon from '@mui/icons-material/GitHub'; -import IconButton from '@mui/material/IconButton'; -import Stack from '@mui/material/Stack'; -import Link from '@mui/material/Link'; -import StarIcon from '@mui/icons-material/Star'; -import Tooltip from '@mui/material/Tooltip'; - -# mysql - - - {/* Author info */} - - Author: Meroxa, Inc. - - Conduit team logo - - - - {/* Divider */} - - - {/* GitHub link */} - - - - - {/* Stars */} - } - label="2" - size="small" - sx={{ height: 24 }} - /> - - -## Latest release - -- [conduit-connector-mysql_0.3.0_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Darwin_arm64.tar.gz) -- [conduit-connector-mysql_0.3.0_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Darwin_x86_64.tar.gz) -- [conduit-connector-mysql_0.3.0_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Linux_arm64.tar.gz) -- [conduit-connector-mysql_0.3.0_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Linux_x86_64.tar.gz) -- [conduit-connector-mysql_0.3.0_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Windows_arm64.tar.gz) -- [conduit-connector-mysql_0.3.0_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/conduitio-labs/conduit-connector-mysql/releases/download/v0.3.0/conduit-connector-mysql_0.3.0_Windows_x86_64.tar.gz) - -## Description - -## Source - -A source connector pulls data from an external resource and pushes it to -downstream resources via Conduit. - -### Snapshot mode - -Snapshot mode is the first stage of the source sync process. It reads all rows -from the configured tables as record snapshots. - -In snapshot mode, the record payload consists of -[opencdc.StructuredData](https://conduit.io/docs/using/opencdc-record#structured-data), -with each key being a column and each value being that column's value. - -### Change Data Capture mode - -When the connector switches to CDC mode, it starts streaming changes from the -obtained position at the start of the snapshot. It uses the row-based binlog format -to capture detailed changes at the individual row level. - -### Unsafe snapshot - -By default, the connector will error out if it finds a table that has no primary key and no specified sorting column specified, as we can't guarantee that the snapshot will be consistent. Table changes during the snapshot will be however captured by CDC mode. - -As of writing, the unsafe snapshot is implemented using batches with `LIMIT` and `OFFSET`, so expect it to be slow for large tables. You can optimize the snapshot by specifying a sorting column (see source configuration parameters). -The position of the table is currently not recorded, so the unsafe snapshot will restart from zero every time. - -### Schema - -The source connector uses [avro](https://avro.apache.org/docs/1.11.1/specification/) to decode mysql rows. Here's the MySQL datatype to avro datatype equivalence that the connector uses: - -| MySQL Type | Avro Type | -| ---------- | --------- | -| TINYINT | int | -| SMALLINT | int | -| MEDIUMINT | int | -| INT | int | -| BIGINT | long | -| DECIMAL | double | -| NUMERIC | double | -| FLOAT | double | -| DOUBLE | double | -| BIT | bytes | -| CHAR | string | -| VARCHAR | string | -| TINYTEXT | string | -| TEXT | string | -| MEDIUMTEXT | string | -| LONGTEXT | string | -| BINARY | bytes | -| VARBINARY | bytes | -| TINYBLOB | bytes | -| BLOB | bytes | -| MEDIUMBLOB | bytes | -| LONGBLOB | bytes | -| DATE | string | -| TIME | string | -| DATETIME | string | -| TIMESTAMP | string | -| YEAR | long | -| ENUM | string | -| SET | string | -| JSON | string | - -### Record structure - -Records produced by the MySQL source connector contain the following [opencdc record structure](https://conduit.io/docs/using/opencdc-record): - -* **Operation**: Indicates the type of change (`create`, `update`, `delete`, `snapshot`). -* **Payload**: - * `Before`: (Only present on CDC `update` operations) `opencdc.StructuredData` representing the row state before the change. - * `After`: `opencdc.StructuredData` representing the row state after the change (or the current state for `snapshot` and `create`). -* **Key**: Identifies the specific row. See Snapshot/CDC sections below for details. -* **Position**: Represents the point in the data stream. It's a JSON object containing *either* a `snapshot_position` field *or* a `cdc_position` field, structured as described below. -* **Metadata**: Contains standard OpenCDC fields plus: - * `mysql.server.id`: (CDC only) The originating MySQL server ID from the replication event header. - * [Key](https://conduit.io/docs/using/opencdc-record/#opencdckeyschema) and [payload](https://conduit.io/docs/using/opencdc-record/#opencdcpayloadschema) schema registry data. - -#### Snapshot Records - -* **Operation**: `snapshot`. -* **Payload `After`**: `opencdc.StructuredData` with all column values. -* **Key**: - * With Primary Key(s): `opencdc.StructuredData` using primary key columns and values. - * Without Primary Key(s) (using `LIMIT`/`OFFSET`): `opencdc.RawData` string `"_"`. -* **Position (`snapshot_position` field)**: A JSON object containing: - * `snapshots`: (object) Maps table names to their specific snapshot position object. - * *(Single Primary Key Table)*: Contains `last_read` (any) and `snapshot_end` (any). - * *(Multiple Primary Key Table)*: Contains an array of objects, each with `key_name` (string), `last_read` (any), and `snapshot_end` (any). - * *(No Primary Key Table)*: Table entry might be missing or empty (offset is not persisted). - -#### CDC Records - -* **Operation**: Either `create`, `update` or `delete`. -* **Key**: - * With Primary Key(s): `opencdc.StructuredData` using primary key columns and values (from the `After` state). - * Without Primary Key(s): `opencdc.RawData` string `"_"`. -* **Payload**: - * `Before`: (For `update` only) `opencdc.StructuredData` with pre-update column values. - * `After`: `opencdc.StructuredData` with post-change column values (for `create`, `update`) or pre-deletion values (for `delete`). -* **Position (`cdc_position` field)**: Encodes the binlog location in a JSON object with the following fields. - * `name`: (string) Binlog file name. - * `pos`: (uint32) Position within the binlog file. - * `prev`: (object, optional) Position of the preceding event, containing `name` (string) and `pos` (uint32). Omitted if not applicable. - * `idx`: (int, optional) Row index within a potentially multi-row MySQL replication event. Omitted if zero or not applicable. - -## Destination - -The MySQL destination takes a slice of `opencdc.Record` and writes them in batches to MySQL. It will use the `opencdc.collection` field in the metadata to determine the table to write to. - -### Upsert Behavior - -If the target table contains a column with a unique constraint (this includes PRIMARY KEY and UNIQUE indexes), records will be upserted. Otherwise, they will be appended. Support for updating tables without unique constraints is tracked [here](https://github.com/conduitio-labs/conduit-connector-mysql/issues/66). - -If the target table already contains a record with the same key, the Destination will upsert with its current received values. Because Keys must be unique, this can overwrite and thus potentially lose data, so keys should be assigned correctly from the Source. - -If a unique key is not present in the target table, the record will be simply appended. - -## Requirements and compatibility - -The connector is tested against MySQL v8.0. Compatibility with older versions isn't guaranteed. - -### MySQL Server Requirements: - -- Binary Log (binlog) must be enabled. -- Binlog format must be set to ROW. -- Binlog row image must be set to FULL. -- Tables must have **sortable** primary keys. - -### MySQL User Privileges: - -For Snapshot and CDC modes, the following privileges are required: - -- SELECT -- LOCK TABLES -- RELOAD -- REPLICATION CLIENT -- REPLICATION SLAVE - - -## Source Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-source - type: source - plugin: "mysql" - name: example-source - settings: - # The connection string for the MySQL database. - # Type: string - dsn: "" - # Represents the tables to read from. - By default, no tables are - # included, but can be modified by adding a comma-separated string of - # regex patterns. - They are applied in the order that they are - # provided, so the final regex supersedes all previous ones. - To - # include all tables, use "*". You can then filter that list by adding - # a comma-separated string of regex patterns. - To set an "include" - # regex, add "+" or nothing in front of the regex. - To set an - # "exclude" regex, add "-" in front of the regex. - e.g. "-.*meta$, - # wp_postmeta" will exclude all tables ending with "meta" but include - # the table "wp_postmeta". - # Type: string - tables: "" - # Disables verbose cdc driver logs. - # Type: bool - cdc.disableLogs: "" - # Controls whether the snapshot is done. - # Type: bool - snapshot.enabled: "true" - # Limits how many rows should be retrieved on each database fetch on - # snapshot mode. - # Type: int - snapshot.fetchSize: "10000" - # Allows a snapshot of a table with neither a primary key nor a - # defined sorting column. The opencdc.Position won't record the last - # record read from a table. - # Type: bool - snapshot.unsafe: "" - # Allows to force using a custom column to sort the snapshot. - # Type: string - tableConfig.*.sortingColumn: "" - # Maximum delay before an incomplete batch is read from the source. - # Type: duration - sdk.batch.delay: "1s" - # Maximum size of batch before it gets read from the source. - # Type: int - sdk.batch.size: "10000" - # Specifies whether to use a schema context name. If set to false, no - # schema context name will be used, and schemas will be saved with the - # subject name specified in the connector (not safe because of name - # conflicts). - # Type: bool - sdk.schema.context.enabled: "true" - # Schema context name to be used. Used as a prefix for all schema - # subject names. If empty, defaults to the connector ID. - # Type: string - sdk.schema.context.name: "" - # Whether to extract and encode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "true" - # The subject of the key schema. If the record metadata contains the - # field "opencdc.collection" it is prepended to the subject name and - # separated with a dot. - # Type: string - sdk.schema.extract.key.subject: "key" - # Whether to extract and encode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "true" - # The subject of the payload schema. If the record metadata contains - # the field "opencdc.collection" it is prepended to the subject name - # and separated with a dot. - # Type: string - sdk.schema.extract.payload.subject: "payload" - # The type of the payload schema. - # Type: string - sdk.schema.extract.type: "avro" -``` - -## Destination Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-destination - type: destination - plugin: "mysql" - name: example-destination - settings: - # The connection string for the MySQL database. - # Type: string - dsn: "" - # Maximum delay before an incomplete batch is written to the - # destination. - # Type: duration - sdk.batch.delay: "0" - # Maximum size of batch before it gets written to the destination. - # Type: int - sdk.batch.size: "0" - # Allow bursts of at most X records (0 or less means that bursts are - # not limited). Only takes effect if a rate limit per second is set. - # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the - # effective batch size will be equal to `sdk.rate.burst`. - # Type: int - sdk.rate.burst: "0" - # Maximum number of records written per second (0 means no rate - # limit). - # Type: float - sdk.rate.perSecond: "0" - # The format of the output record. See the Conduit documentation for a - # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). - # Type: string - sdk.record.format: "opencdc/json" - # Options to configure the chosen output record format. Options are - # normally key=value pairs separated with comma (e.g. - # opt1=val2,opt2=val2), except for the `template` record format, where - # options are a Go template. - # Type: string - sdk.record.format.options: "" - # Whether to extract and decode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "true" - # Whether to extract and decode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "true" -``` diff --git a/docs/1-using/5-connectors/7-list/4-log.mdx b/docs/1-using/5-connectors/7-list/5-log.mdx similarity index 74% rename from docs/1-using/5-connectors/7-list/4-log.mdx rename to docs/1-using/5-connectors/7-list/5-log.mdx index 7f78c3c..60f2f10 100644 --- a/docs/1-using/5-connectors/7-list/4-log.mdx +++ b/docs/1-using/5-connectors/7-list/5-log.mdx @@ -68,22 +68,22 @@ import Tooltip from '@mui/material/Tooltip'; ## Latest release -- [conduit-connector-log_0.7.5_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Darwin_arm64.tar.gz) -- [conduit-connector-log_0.7.5_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Darwin_x86_64.tar.gz) -- [conduit-connector-log_0.7.5_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Linux_arm64.tar.gz) -- [conduit-connector-log_0.7.5_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Linux_i386.tar.gz) -- [conduit-connector-log_0.7.5_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Linux_x86_64.tar.gz) -- [conduit-connector-log_0.7.5_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Windows_arm64.tar.gz) -- [conduit-connector-log_0.7.5_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Windows_i386.tar.gz) -- [conduit-connector-log_0.7.5_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/conduit-connector-log_0.7.5_Windows_x86_64.tar.gz) -- [log_v0.7.5_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_darwin_amd64.tar.gz) -- [log_v0.7.5_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_darwin_amd64.tar.gz.sigstore.json) -- [log_v0.7.5_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_darwin_arm64.tar.gz) -- [log_v0.7.5_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_darwin_arm64.tar.gz.sigstore.json) -- [log_v0.7.5_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_linux_amd64.tar.gz) -- [log_v0.7.5_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_linux_amd64.tar.gz.sigstore.json) -- [log_v0.7.5_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_linux_arm64.tar.gz) -- [log_v0.7.5_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.5/log_v0.7.5_linux_arm64.tar.gz.sigstore.json) +- [conduit-connector-log_0.7.6_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Darwin_arm64.tar.gz) +- [conduit-connector-log_0.7.6_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Darwin_x86_64.tar.gz) +- [conduit-connector-log_0.7.6_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Linux_arm64.tar.gz) +- [conduit-connector-log_0.7.6_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Linux_i386.tar.gz) +- [conduit-connector-log_0.7.6_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Linux_x86_64.tar.gz) +- [conduit-connector-log_0.7.6_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Windows_arm64.tar.gz) +- [conduit-connector-log_0.7.6_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Windows_i386.tar.gz) +- [conduit-connector-log_0.7.6_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/conduit-connector-log_0.7.6_Windows_x86_64.tar.gz) +- [log_v0.7.6_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_darwin_amd64.tar.gz) +- [log_v0.7.6_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_darwin_amd64.tar.gz.sigstore.json) +- [log_v0.7.6_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_darwin_arm64.tar.gz) +- [log_v0.7.6_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_darwin_arm64.tar.gz.sigstore.json) +- [log_v0.7.6_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_linux_amd64.tar.gz) +- [log_v0.7.6_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_linux_amd64.tar.gz.sigstore.json) +- [log_v0.7.6_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_linux_arm64.tar.gz) +- [log_v0.7.6_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-log/releases/download/v0.7.6/log_v0.7.6_linux_arm64.tar.gz.sigstore.json) ## Description @@ -152,7 +152,7 @@ pipelines: sdk.rate.perSecond: "0" # The format of the output record. See the Conduit documentation for a # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). # Type: string sdk.record.format: "opencdc/json" # Options to configure the chosen output record format. Options are diff --git a/docs/1-using/5-connectors/7-list/6-mysql.mdx b/docs/1-using/5-connectors/7-list/6-mysql.mdx new file mode 100644 index 0000000..475b65a --- /dev/null +++ b/docs/1-using/5-connectors/7-list/6-mysql.mdx @@ -0,0 +1,611 @@ +--- +IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. + +title: "mysql" +description: "A Conduit Connector for MySQL" +--- + +import ReactDiffViewer from 'react-diff-viewer-continued'; +import Chip from '@mui/material/Chip'; +import Box from "@mui/system/Box"; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import GitHubIcon from '@mui/icons-material/GitHub'; +import IconButton from '@mui/material/IconButton'; +import Stack from '@mui/material/Stack'; +import Link from '@mui/material/Link'; +import StarIcon from '@mui/icons-material/Star'; +import Tooltip from '@mui/material/Tooltip'; + +# mysql + + + {/* Author info */} + + Author: Meroxa, Inc. + + Conduit team logo + + + + {/* Divider */} + + + {/* GitHub link */} + + + + + {/* Stars */} + } + label="2" + size="small" + sx={{ height: 24 }} + /> + + +## Latest release + +- [conduit-connector-mysql_0.5.0_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Darwin_arm64.tar.gz) +- [conduit-connector-mysql_0.5.0_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Darwin_x86_64.tar.gz) +- [conduit-connector-mysql_0.5.0_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Linux_arm64.tar.gz) +- [conduit-connector-mysql_0.5.0_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Linux_x86_64.tar.gz) +- [conduit-connector-mysql_0.5.0_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Windows_arm64.tar.gz) +- [conduit-connector-mysql_0.5.0_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/conduit-connector-mysql_0.5.0_Windows_x86_64.tar.gz) +- [mysql_v0.5.0_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_darwin_amd64.tar.gz) +- [mysql_v0.5.0_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_darwin_amd64.tar.gz.sigstore.json) +- [mysql_v0.5.0_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_darwin_arm64.tar.gz) +- [mysql_v0.5.0_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_darwin_arm64.tar.gz.sigstore.json) +- [mysql_v0.5.0_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_linux_amd64.tar.gz) +- [mysql_v0.5.0_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_linux_amd64.tar.gz.sigstore.json) +- [mysql_v0.5.0_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_linux_arm64.tar.gz) +- [mysql_v0.5.0_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-mysql/releases/download/v0.5.0/mysql_v0.5.0_linux_arm64.tar.gz.sigstore.json) + +## Description + +## Source + +A source connector pulls data from an external resource and pushes it to +downstream resources via Conduit. + +### Snapshot mode + +Snapshot mode is the first stage of the source sync process. It reads all rows +from the configured tables as record snapshots. + +In snapshot mode, the record payload consists of +[opencdc.StructuredData](https://conduit.io/docs/using/opencdc-record#structured-data), +with each key being a column and each value being that column's value. + +### Change Data Capture mode + +When the connector switches to CDC mode, it starts streaming changes from the +obtained position at the start of the snapshot. It uses the row-based binlog format +to capture detailed changes at the individual row level. + +### Tables without a primary key + +The snapshot reads each table in key order and records the last key read in the +position, so it can resume exactly where it stopped and read a consistent view. +That needs a primary key, or a column you name with +`tableConfig..sortingColumn`. If a table has neither, the connector +fails to start with error `mysql.snapshot.table_without_key`, naming the table +and the config path `tableConfig.
.sortingColumn`. To fix it, do one of: + +- add a primary key to the table; +- set `tableConfig.
.sortingColumn` to a column that is `NOT NULL` and the + only column of a primary key or unique index. Anything else is rejected at + start with `mysql.snapshot.sorting_column_invalid`, because duplicate or NULL + values can make the snapshot skip rows; +- exclude the table. With a pattern list, a later `-` rule removes tables from an + earlier match, for example `tables: "*,-^audit_log$"` snapshots everything + except `audit_log`. With an explicit list, remove the table from it; +- set `snapshot.enabled: false` to stream changes only. Rows that already exist + are then not copied. CDC works for tables without a primary key; their record + keys are derived from the binlog position. + +The key and sorting-column checks run only when a snapshot is about to run. A +pipeline restarting in CDC mode is not affected. + +#### Migrating from `snapshot.unsafe` + +The `snapshot.unsafe` setting has been removed. It snapshotted keyless tables +with `LIMIT`/`OFFSET`, recorded no position (so every restart began again from +zero), and could skip or repeat rows when the table changed during the +snapshot. In practice it also applied without the setting: a keyless table was +snapshotted this way even when `snapshot.unsafe` was not set. + +- `snapshot.unsafe: true` now fails configuration validation with error + `mysql.config.unsafe_snapshot_removed`, **including for pipelines that are + already in CDC mode**: validation does not look at the position. Remove the + setting before upgrading. +- `snapshot.unsafe: false` is accepted and ignored. +- After upgrading, a pipeline that still has a keyless table and will run a + snapshot fails to start with `mysql.snapshot.table_without_key` until you use + one of the options above. + +### Schema + +The source connector uses [avro](https://avro.apache.org/docs/1.11.1/specification/) to decode mysql rows. Here's the MySQL datatype to avro datatype equivalence that the connector uses: + +| MySQL Type | Avro Type | +| ---------- | --------- | +| TINYINT | int | +| SMALLINT | int | +| MEDIUMINT | int | +| INT | int | +| BIGINT | long | +| DECIMAL | double | +| NUMERIC | double | +| FLOAT | double | +| DOUBLE | double | +| BIT | bytes | +| CHAR | string | +| VARCHAR | string | +| TINYTEXT | string | +| TEXT | string | +| MEDIUMTEXT | string | +| LONGTEXT | string | +| BINARY | bytes | +| VARBINARY | bytes | +| TINYBLOB | bytes | +| BLOB | bytes | +| MEDIUMBLOB | bytes | +| LONGBLOB | bytes | +| DATE | string | +| TIME | string | +| DATETIME | string | +| TIMESTAMP | string | +| YEAR | long | +| ENUM | string | +| SET | string | +| JSON | string | + +### Record structure + +Records produced by the MySQL source connector contain the following [opencdc record structure](https://conduit.io/docs/using/opencdc-record): + +* **Operation**: Indicates the type of change (`create`, `update`, `delete`, `snapshot`). +* **Payload**: + * `Before`: (Only present on CDC `update` operations) `opencdc.StructuredData` representing the row state before the change. + * `After`: `opencdc.StructuredData` representing the row state after the change (or the current state for `snapshot` and `create`). +* **Key**: Identifies the specific row. See Snapshot/CDC sections below for details. +* **Position**: Represents the point in the data stream. It's a JSON object containing *either* a `snapshot_position` field *or* a `cdc_position` field, structured as described below. +* **Metadata**: Contains standard OpenCDC fields plus: + * `mysql.server.id`: (CDC only) The originating MySQL server ID from the replication event header. + * [Key](https://conduit.io/docs/using/opencdc-record/#opencdckeyschema) and [payload](https://conduit.io/docs/using/opencdc-record/#opencdcpayloadschema) schema registry data. + +#### Snapshot Records + +* **Operation**: `snapshot`. +* **Payload `After`**: `opencdc.StructuredData` with all column values. +* **Key**: `opencdc.StructuredData` using the primary key columns (or the configured `tableConfig.
.sortingColumn`) and their values. +* **Position (`snapshot_position` field)**: A JSON object containing: + * `snapshots`: (object) Maps table names to their specific snapshot position object. + * *(Single Primary Key Table)*: Contains `last_read` (any) and `snapshot_end` (any). + * *(Multiple Primary Key Table)*: Contains an array of objects, each with `key_name` (string), `last_read` (any), and `snapshot_end` (any). + +#### CDC Records + +* **Operation**: Either `create`, `update` or `delete`. +* **Key**: + * With Primary Key(s): `opencdc.StructuredData` using primary key columns and values (from the `After` state). + * Without Primary Key(s): `opencdc.RawData` string `"_"`. +* **Payload**: + * `Before`: (For `update` only) `opencdc.StructuredData` with pre-update column values. + * `After`: `opencdc.StructuredData` with post-change column values (for `create`, `update`) or pre-deletion values (for `delete`). +* **Position (`cdc_position` field)**: Encodes the binlog location in a JSON object with the following fields. + * `name`: (string) Binlog file name. + * `pos`: (uint32) Position within the binlog file. + * `prev`: (object, optional) Position of the preceding event, containing `name` (string) and `pos` (uint32). Omitted if not applicable. + * `idx`: (int, optional) Row index within a potentially multi-row MySQL replication event. Omitted if zero or not applicable. + +## Destination + +The MySQL destination takes a slice of `opencdc.Record` and writes them in batches to MySQL. It will use the `opencdc.collection` field in the metadata to determine the table to write to. + +### Upsert Behavior + +If the target table contains a column with a unique constraint (this includes PRIMARY KEY and UNIQUE indexes), records will be upserted. Otherwise, they will be appended. Support for updating tables without unique constraints is tracked [here](https://github.com/conduitio-labs/conduit-connector-mysql/issues/66). + +If the target table already contains a record with the same key, the Destination will upsert with its current received values. Because Keys must be unique, this can overwrite and thus potentially lose data, so keys should be assigned correctly from the Source. + +If a unique key is not present in the target table, the record will be simply appended. + +With several unique indexes (including functional and prefix indexes), `ON DUPLICATE KEY UPDATE` overwrites the row matched by any of them, including that row's primary key. Functional unique indexes are read but not evaluated: the destination cannot compute their expressions, so it neither type-checks their inputs nor uses them to decide whether a record can be matched. + +### Columns and schema changes + +Each record is written with exactly the fields in its `.Payload.After`, whatever the other records in the batch contain: + +- A field the table has no column for fails the write with `mysql.destination.unknown_column`. The destination does not create or alter tables, and it never drops a field. Add the column first, or remove the field with a processor. +- A field the record does not have is not written. An inserted row gets the column's default; an existing row keeps the column's current value. A field set to `null` writes `NULL`. + +Table and column names are quoted, so reserved words work. A collection of the form `db.table` writes to `table` in database `db`. + +**Security.** Table and column names come from the records. Quoting keeps them from changing the SQL, but a record's collection can name any database the DSN user can write to, through `db.table`. Grant the destination's user access only to the databases it should write. Quoting is only sound in character sets where no character can end in a backtick byte, so a connection using `sjis`, `cp932`, `gbk`, `gb18030` or `big5` is refused with `mysql.destination.unsafe_charset` (checked in the DSN, when the destination opens, and on the connection of every write). Use `utf8mb4`, the driver's default. + +### Deletes + +Delete records are matched on the target table's primary key, so the table must have one: a delete for a table without a primary key fails with `mysql.destination.delete_without_primary_key` instead of removing rows. The record key must hold a non-null value for every primary-key column (`mysql.destination.key_column_missing` otherwise), either as structured data or as a single JSON object. Each value must be one scalar: a list or an object fails with `mysql.destination.key_value_not_scalar` (it would match several rows or none). Key values are converted to the type of their key column, which is read for every write, so an `ALTER TABLE` on a running pipeline is picked up without a restart. The same rules apply to the values an upsert writes into the columns of any unique index (the primary key and `UNIQUE` keys), because those values decide which row `ON DUPLICATE KEY UPDATE` overwrites: `{"id":1.5}` for a `BIGINT` key is refused instead of being rounded to 2 and overwriting row 2. A record must be fully non-null in at least one unique index, otherwise it fails with `mysql.destination.key_column_missing`: a null never conflicts, so every redelivery would insert another row. A null in one unique column is fine while another unique index identifies the row (a nullable `UNIQUE` email next to a primary key). Exceptions, which append a row on every redelivery: a null or absent `AUTO_INCREMENT` primary key, which generates a new id, and a table without a unique index. Field names are resolved to columns by the MySQL server itself (through `information_schema`), so a field names exactly the column a statement would use: `ID` names `id`, and so do characters MySQL folds (for example the Kelvin sign names `k`), while `Å¿` names no column. A record with two fields that name one column (`id` and `ID`) fails with `mysql.destination.duplicate_column`. A delete for a table that does not exist fails with `mysql.destination.table_not_found`; table names are case-sensitive unless the server sets `lower_case_table_names`. + +- **Integer columns** (`TINYINT` to `BIGINT`, signed or `UNSIGNED`) take a whole number within the column's range (`5`, `5.0`, `5e0` or the string `"5"`). A fraction, a non-numeric string or a value outside the range fails with `mysql.destination.key_value_type_mismatch` instead of matching nothing or the wrong row. A boolean matches as MySQL compares it: `true` is 1, which is what a `TINYINT(1)` key needs. +- **String columns** (`CHAR`, `VARCHAR`, the `TEXT` types, `BINARY`, `VARBINARY`) take a string. An integer is sent as its decimal text and a raw JSON number as written (so `1e3` matches the row `'1e3'`), because MySQL compares a number with a string column as a double and `1000` would match `'1000'`, `'01000'`, `'1000.0'` and `' 1000'`. A float is accepted only if it is whole and below 1e15, where MySQL's text for it is the plain integer; other floats and booleans fail. +- **`ENUM` and `SET` columns** take the member name as a string. A number fails, because MySQL would compare it with the member's index. +- **`BIT(M)` columns** take a whole number from 0 to 2^M-1. +- **Other columns** (`DECIMAL`, floating point, temporal) get the value as it is; a raw JSON number keeps its literal, which MySQL compares exactly with a `DECIMAL` column. + +Integer values in a raw JSON key are matched exactly. Structured keys that reach a standalone connector over the plugin protocol have already passed through a double, so integers above 2^53 sent that way are rounded before the destination sees them ([ConduitIO/conduit#3100](https://github.com/ConduitIO/conduit/issues/3100)). A record whose operation is not create, update, delete or snapshot fails with `mysql.destination.unsupported_operation`. + +### Atomicity and acknowledgements + +Each write is a single transaction. If any record in the write fails, nothing from that write is stored and none of its records are acknowledged; the records are redelivered after the pipeline recovers. Upserts on a primary or unique key make the redelivery idempotent. Redelivering a record whose `AUTO_INCREMENT` key is null or absent, or a record for a table with no unique key, appends another row. + +The destination sets `wait_timeout=60` on its own connections (unless the DSN sets `wait_timeout`), and the pool drops connections idle for half of it. If the connector dies in the middle of a write and the server does not see the connection close (a NAT, a proxy, a half-open TCP connection), the uncommitted transaction and its row locks end within that time instead of the server default of 8 hours, so the redelivered records do not wait on them. + +## Delivery semantics and limits + +The source connector is in beta. This section states what it guarantees and what it does not. + +### Delivery guarantees + +- **At-least-once.** A record is acknowledged upstream only after the destination has + acknowledged it, and the position carried on a record is what Conduit persists. After a + crash or restart you can receive records again, never fewer. Make destinations idempotent + (upsert by key). +- **Snapshot to CDC handoff.** The connector captures the binlog position at the start of + the snapshot and begins CDC from it, so changes made while the snapshot runs are delivered + after it. Rows can therefore appear twice: once as a `snapshot` record and again as a + CDC change. +- **Restart.** A restart during the snapshot resumes the snapshot from the last + acknowledged key per table. After a CDC restart, CDC resumes after the last acknowledged + row, also in the middle of a multi-row binlog event, in a later binlog event of one large + statement, and inside a compressed transaction (`binlog_transaction_compression=ON`). After + upgrading from an older version, restarts re-read the binlog file of the saved position from + its start to find the saved event, until the first record is acknowledged under the new + version. Rows are delivered again (duplicates, never a gap) in three cases, each logged as a + warning: the saved position was written right after a restart in the middle of an event (that + event is delivered again), an older version's position points inside a compressed transaction + (its binlog file is delivered again from its start), or the captured tables changed while the + saved position was inside a compressed transaction (that transaction is delivered again). +- **Ordering.** CDC records are delivered in binlog order. During the snapshot, rows are + ordered by key within a table; tables are read concurrently, so there is no ordering across + tables. + +### Limits + +- **No GTID support.** Positions are a binlog file name and offset on the server the + connector read from. They are not portable between servers: a replica, or a replica + promoted after a failover, has different binlog files and offsets. **Failing over the + source database breaks resume.** The connector may fail to start, or may not resume at the + matching point. After a failover, treat the pipeline as needing a new snapshot: clear its + position and expect to re-read the data. GTID-based positions are not supported in this + release. +- **Tables added after the pipeline entered CDC are not snapshotted.** A snapshot runs only + on a fresh start. The position records which tables were snapshotted, and a restart checks + the configuration against it: + - If you add a table to an explicit `tables` list (plain table names), or switch + `snapshot.enabled` from `false` to `true`, the connector refuses to start with + `mysql.snapshot.tables_added_after_cdc`, naming the tables. Start a new pipeline for them, + or reset this pipeline's position to snapshot everything again (expect duplicates). + - If `tables` uses a pattern (`*`, a regex, an exclude, or an anchored name such as + `^orders$`), the connector logs one warning with the same code and then delivers the new + tables **from CDC only**: their existing rows are not snapshotted and are never delivered. + - A plain name is a regex that is not anchored, so `orders` also matches `orders_archive`. + It is still treated as an explicit list, which is the safe side. Anchor it as `^orders$` + to make it a pattern. + - A table removed from `tables` and added back later counts as added: changes made while it + was excluded were never captured. For an explicit list this means a table that is absent + at one restart and present at the next errors. + - A position written by an older version cannot be checked. The connector adopts the + configured tables as **unverified**, never as snapshotted, and logs + `mysql.snapshot.tables_unverified` on every start until the position is reset. Tables added + afterwards are detected. + - With more than 200 tables the position stores a hash of the table set instead of the + names. A change to the set then cannot be named or given a direction (a table that was only + removed also triggers it). The error says that "the configured table set changed", and + the way out is to restore the previous `tables` list or reset the position; removing tables + from the list does not clear it. +- **A key column that was retyped restarts its table's snapshot.** The position records the + type of each key column. If the saved key does not fit the column's type now (an `INT` key + column that became `VARCHAR`, or `VARCHAR` that became `DECIMAL`, whose values order + differently), the table's snapshot restarts from its first row with a warning. Rows already + delivered are delivered again; nothing is skipped. +- **Upgrading in the middle of a snapshot re-reads text, decimal and binary key tables.** + A snapshot position written by an older version does not record the key type, so a table + whose key is text, binary, `DECIMAL`, `TIME`, `ENUM`, `SET` or `BIT` cannot be checked for a + retype and restarts from its first row after the upgrade; its partial progress is re-read. + Integer, date and time keys resume where they stopped. Pipelines already in CDC are not + affected. +- **Binlog retention is your responsibility.** If the persisted position has been purged + from the server, the connector cannot resume. Keep `binlog_expire_logs_seconds` longer than + the longest outage you want to survive. +- **No heartbeat.** The connector does not send or consume heartbeat events. The position + is carried on records, so while the configured tables are quiet the persisted position does + not advance, even if the server keeps writing and rotating binlogs for other tables. A long + quiet period followed by binlog purging can leave the stored position pointing at a purged + file. +- **DDL during a running pipeline is not handled explicitly and is not tested.** Running + `ALTER TABLE` (or other schema changes) on a captured table while the snapshot or CDC is + running may produce records that do not match the schema registered earlier, or an error. + Stop the pipeline, change the schema, and verify the result before resuming. +- **Tables must be orderable for the snapshot.** See "Tables without a primary key". +- **Type mapping has known gaps.** `DECIMAL` and `NUMERIC` map to Avro `double` and can lose + precision beyond what a 64-bit float holds. SQL `NULL` is delivered as the column type's + zero value (`0`, `""`), so it cannot be told apart from a real zero. Date and time types are + delivered as strings. A full type-mapping design is pending; do not rely on exact fidelity + for these types yet. +- **`BIGINT UNSIGNED` above 2^63-1 stops the source.** The column maps to Avro `long`, which + cannot hold values above 9223372036854775807. When the snapshot or CDC reads such a value, + the source stops with `mysql.types.unsigned_overflow`, naming the table, column and value. + No record is emitted for that row and the position does not move past it. Exclude the + table, or bring the values into range, and restart. Versions before this one emitted a + wrong number in the snapshot (18446744073709551615 came out as 3474867090068222261) and + failed with an encoder error in CDC. + +## Requirements and compatibility + +The connector is tested against MySQL v8.0. Compatibility with older versions isn't guaranteed. + +### MySQL Server Requirements: + +- Binary Log (binlog) must be enabled. +- Binlog format must be set to ROW. +- Binlog row image must be set to FULL. +- Recommended: `binlog_row_metadata=FULL` (MySQL 8.0.1+, MariaDB 10.5+). Without it the connector cannot + detect every schema change that happened while it was stopped; see `mysql.cdc.row_columns_mismatch` below. +- Tables must have **sortable** primary keys. + +### Partial row images + +`binlog_row_image` is also a session variable. A writing session that sets it to `MINIMAL` or `NOBLOB` logs +only part of the row, even when the server value is `FULL`: + +- `MINIMAL`, UPDATE: the before image holds only the primary key, and the after image holds only the changed + columns (the primary key is in it only if it changed). +- `MINIMAL`, DELETE: the before image holds only the primary key. +- `MINIMAL`, INSERT: the image holds only the columns the statement names. Every column it omits is skipped, + including columns with a `DEFAULT` and nullable columns, so an ORM that leaves defaulted columns out halts on + its first insert. An INSERT that names every column does not halt. +- `NOBLOB`: `BLOB` and `TEXT` columns that an UPDATE or DELETE did not change are skipped, and so is a `BLOB` + that an INSERT omits. An INSERT that omits a non-`BLOB` column with a default is logged in full. + +A skipped column is unknown. It is not NULL and it is not the column default: the binlog carries no value for +it, and the schema default may have changed since the row was written, so the connector does not fill it in. + +The CDC source halts on the first such event instead of delivering zero values over real data. The error has +the code `mysql.cdc.row_image_skipped_columns` and names the table, the action, the binlog file and position, and +the columns that were left out. Snapshot rows are not affected. If the event sits inside a compressed +transaction (`binlog_transaction_compression=ON`) the exact position is not available and the error says so. + +The source also stops with the code `mysql.cdc.partial_json_unsupported` when a writer logs partial JSON updates +(`binlog_row_value_options=PARTIAL_JSON`), which the connector cannot read. Set the option to an empty value for +the writers. + +Two more CDC halts are resolved by resetting the position, so the recovery below (deletes) applies to them too: + +- `mysql.cdc.row_columns_mismatch`: a binlog event has a different number of values than its table has + columns, even after the connector re-read the table's columns (it does that once on a mismatch, because its + cached column list can be out of date after a DDL its parser skips, such as `ADD COLUMN ... INVISIBLE`). + The connector decodes events with the table's current columns, so an event written before an `ADD COLUMN` + or `DROP COLUMN` and read after it (the connector was stopped or lagging across the DDL) cannot be mapped. + It also happens when the connector's user cannot see every column: column privileges, or a generated + invisible primary key hidden by `show_gipk_in_create_table_and_information_schema=OFF`, which halts by + design. Make hidden columns visible first. Then restart the pipeline; if it halts again on the same event, + reset the position to snapshot again. +- `mysql.cdc.row_columns_mismatch` also covers an event whose number of values fits the table but that was + written under other columns: after a `DROP COLUMN` plus an `ADD COLUMN` (or a column changing type) ran + before the connector read the event, every value after the dropped column would land in the wrong column. + The connector compares the types the event was written with against the table's columns (by family, so + `INT` to `BIGINT` or a longer `VARCHAR` is fine) and halts on a difference, also after a restart, so + restarting does not help: reset the position. With `binlog_row_metadata=FULL` the event also carries its + column names, and the connector compares those too; a renamed column then halts as well, because a rename + cannot be told from a drop plus an add. **Without `FULL`, a column dropped and another of the same type + family added at the same position, while the connector is stopped or lagging, is not detected, and the + values are mapped onto the new column.** `binlog_row_metadata=FULL` (MySQL 8.0.1+, MariaDB 10.5+; the default + is `MINIMAL`) closes that for events written after it is set, at the cost of a larger binlog. Stop the + pipeline before changing a captured table's columns, or set it. +- `mysql.cdc.resume_event_not_found`: on a restart, the binlog event the saved position names was not where + the position says. This should not happen; report it with the logged position, and reset the position to + continue. + +#### Recovery + +**Resuming means a re-snapshot, and a snapshot does not carry deletes.** Rows deleted at the source while the +pipeline was halted would stay in the destination. Do one of the following, for ALL captured tables, not only +the table named in the error: + +- truncate the destination tables, then clear the connector's position to re-snapshot; or +- clear the connector's position to re-snapshot, and after the snapshot has completed reconcile the destination + against the source, deletes included. + +Do not reconcile first and re-snapshot afterwards: rows deleted at the source between the reconcile and the +start of the snapshot would stay in the destination. + +1. Stop the cause. Set `binlog_row_image = 'FULL'` for the writers, and for the server use + `SET PERSIST binlog_row_image = 'FULL'` (`SET GLOBAL` does not survive a restart). Then reconnect every + writer: existing sessions, pooled connections included, keep the value they started with. +2. The event is already in the binlog, so every restart halts on it. Find its transaction with + `SHOW BINLOG EVENTS IN ''` (the error gives the start position of the event); with GTIDs enabled, + the `Gtid` event just before it names the GTID. +3. Truncate the destination for all captured tables, or plan to reconcile it after the snapshot (see above). +4. Clear the connector's position to re-snapshot. +5. If you chose to reconcile, do it once the snapshot has completed, including deletes, for all captured tables. + +A lighter way to resume after the event is tracked in +[#212](https://github.com/ConduitIO/conduit-connector-mysql/issues/212). + +#### Upgrading + +If the binlog after the stored position already holds an event from a `MINIMAL` or `NOBLOB` session, including +an event that an older version had partly emitted, the upgraded connector halts on it after the restart. Older +versions delivered such events with zero values. + +### MySQL User Privileges: + +For Snapshot and CDC modes, the following privileges are required: + +- SELECT +- LOCK TABLES +- RELOAD +- REPLICATION CLIENT +- REPLICATION SLAVE + +## Source Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-source + type: source + plugin: "mysql" + name: example-source + settings: + # The connection string for the MySQL database. + # Type: string + dsn: "" + # Represents the tables to read from. - By default, no tables are + # included, but can be modified by adding a comma-separated string of + # regex patterns. - They are applied in the order that they are + # provided, so the final regex supersedes all previous ones. - To + # include all tables, use "*". You can then filter that list by adding + # a comma-separated string of regex patterns. - To set an "include" + # regex, add "+" or nothing in front of the regex. - To set an + # "exclude" regex, add "-" in front of the regex. - e.g. "-.*meta$, + # wp_postmeta" will exclude all tables ending with "meta" but include + # the table "wp_postmeta". + # Type: string + tables: "" + # Disables verbose cdc driver logs. + # Type: bool + cdc.disableLogs: "" + # Controls whether the snapshot is done. + # Type: bool + snapshot.enabled: "true" + # Limits how many rows should be retrieved on each database fetch on + # snapshot mode. + # Type: int + snapshot.fetchSize: "10000" + # Removed. The unsafe snapshot no longer exists: true fails validation + # with a migration message, false is accepted and ignored. Tables + # without a primary key need tableConfig.
.sortingColumn. + # Type: bool + snapshot.unsafe: "" + # Allows to force using a custom column to sort the snapshot. The + # column must be NOT NULL and the only column of a primary key or + # unique index. + # Type: string + tableConfig.*.sortingColumn: "" + # Maximum delay before an incomplete batch is read from the source. + # Type: duration + sdk.batch.delay: "1s" + # Maximum size of batch before it gets read from the source. + # Type: int + sdk.batch.size: "10000" + # Specifies whether to use a schema context name. If set to false, no + # schema context name will be used, and schemas will be saved with the + # subject name specified in the connector (not safe because of name + # conflicts). + # Type: bool + sdk.schema.context.enabled: "true" + # Schema context name to be used. Used as a prefix for all schema + # subject names. If empty, defaults to the connector ID. + # Type: string + sdk.schema.context.name: "" + # Whether to extract and encode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "true" + # The subject of the key schema. If the record metadata contains the + # field "opencdc.collection" it is prepended to the subject name and + # separated with a dot. + # Type: string + sdk.schema.extract.key.subject: "key" + # Whether to extract and encode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "true" + # The subject of the payload schema. If the record metadata contains + # the field "opencdc.collection" it is prepended to the subject name + # and separated with a dot. + # Type: string + sdk.schema.extract.payload.subject: "payload" + # The type of the payload schema. + # Type: string + sdk.schema.extract.type: "avro" +``` + +## Destination Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-destination + type: destination + plugin: "mysql" + name: example-destination + settings: + # The connection string for the MySQL database. + # Type: string + dsn: "" + # Maximum delay before an incomplete batch is written to the + # destination. + # Type: duration + sdk.batch.delay: "0" + # Maximum size of batch before it gets written to the destination. + # Type: int + sdk.batch.size: "0" + # Allow bursts of at most X records (0 or less means that bursts are + # not limited). Only takes effect if a rate limit per second is set. + # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the + # effective batch size will be equal to `sdk.rate.burst`. + # Type: int + sdk.rate.burst: "0" + # Maximum number of records written per second (0 means no rate + # limit). + # Type: float + sdk.rate.perSecond: "0" + # The format of the output record. See the Conduit documentation for a + # full list of supported formats + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + # Type: string + sdk.record.format: "opencdc/json" + # Options to configure the chosen output record format. Options are + # normally key=value pairs separated with comma (e.g. + # opt1=val2,opt2=val2), except for the `template` record format, where + # options are a Go template. + # Type: string + sdk.record.format.options: "" + # Whether to extract and decode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "true" + # Whether to extract and decode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "true" +``` diff --git a/docs/1-using/5-connectors/7-list/6-postgres.mdx b/docs/1-using/5-connectors/7-list/6-postgres.mdx deleted file mode 100644 index 6b301aa..0000000 --- a/docs/1-using/5-connectors/7-list/6-postgres.mdx +++ /dev/null @@ -1,320 +0,0 @@ ---- -IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. - -title: "postgres" -description: "Conduit connector for PostgreSQL" ---- - -import ReactDiffViewer from 'react-diff-viewer-continued'; -import Chip from '@mui/material/Chip'; -import Box from "@mui/system/Box"; -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import GitHubIcon from '@mui/icons-material/GitHub'; -import IconButton from '@mui/material/IconButton'; -import Stack from '@mui/material/Stack'; -import Link from '@mui/material/Link'; -import StarIcon from '@mui/icons-material/Star'; -import Tooltip from '@mui/material/Tooltip'; - -# postgres - - - {/* Author info */} - - Author: Meroxa, Inc. - - Conduit team logo - - - - {/* Divider */} - - - {/* GitHub link */} - - - - - {/* Stars */} - } - label="20" - size="small" - sx={{ height: 24 }} - /> - - -## Latest release - -- [conduit-connector-postgres_0.14.3_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Darwin_arm64.tar.gz) -- [conduit-connector-postgres_0.14.3_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Darwin_x86_64.tar.gz) -- [conduit-connector-postgres_0.14.3_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Linux_arm64.tar.gz) -- [conduit-connector-postgres_0.14.3_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Linux_i386.tar.gz) -- [conduit-connector-postgres_0.14.3_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Linux_x86_64.tar.gz) -- [conduit-connector-postgres_0.14.3_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Windows_arm64.tar.gz) -- [conduit-connector-postgres_0.14.3_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Windows_i386.tar.gz) -- [conduit-connector-postgres_0.14.3_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/conduit-connector-postgres_0.14.3_Windows_x86_64.tar.gz) -- [postgres_v0.14.3_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_darwin_amd64.tar.gz) -- [postgres_v0.14.3_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_darwin_amd64.tar.gz.sigstore.json) -- [postgres_v0.14.3_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_darwin_arm64.tar.gz) -- [postgres_v0.14.3_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_darwin_arm64.tar.gz.sigstore.json) -- [postgres_v0.14.3_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_linux_amd64.tar.gz) -- [postgres_v0.14.3_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_linux_amd64.tar.gz.sigstore.json) -- [postgres_v0.14.3_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_linux_arm64.tar.gz) -- [postgres_v0.14.3_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-postgres/releases/download/v0.14.3/postgres_v0.14.3_linux_arm64.tar.gz.sigstore.json) - -## Description - -## Source - -The Postgres Source Connector connects to a database with the provided `url` and -starts creating records for each change detected in the provided tables. - -Upon starting, the source takes a snapshot of the provided tables in the database, -then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. - -### Snapshot - -When the connector first starts, snapshot mode is enabled. The connector acquires -a read-only lock on the tables, and then reads all rows of the tables into Conduit. -Once all rows in that initial snapshot are read the connector releases its lock and -switches into CDC mode. - -This behavior is enabled by default, but can be turned off by adding -`"snapshotMode": "never"` to the Source configuration. - -### Change Data Capture - -This connector implements Change Data Capture (CDC) features for PostgreSQL by -creating a logical replication slot and a publication that listens to changes in the -configured tables. Every detected change is converted into a record. If there are no -records available, the connector blocks until a record is available or the connector -receives a stop signal. - -#### Logical Replication Configuration - -When the connector switches to CDC mode, it attempts to run the initial setup commands -to create its logical replication slot and publication. It will connect to an existing -slot if one with the configured name exists. - -The Postgres user specified in the connection URL must have sufficient privileges to -run all of these setup commands, or it will fail. - -Example pipeline configuration that's using logical replication: - -```yaml -version: 2.2 -pipelines: - - id: pg-to-log - status: running - connectors: - - id: pg - type: source - plugin: builtin:postgres - settings: - url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" - tables: "users" - cdcMode: "logrepl" - logrepl.publicationName: "examplepub" - logrepl.slotName: "exampleslot" - - id: log - type: destination - plugin: builtin:log - settings: - level: info -``` - -:warning: When the connector or pipeline is deleted, the connector will automatically -attempt to delete the replication slot and publication. This is the default behaviour -and can be disabled by setting `logrepl.autoCleanup` to `false`. - -### Key Handling - -The connector will automatically look up the primary key column for the specified tables -and use them as the key value. If that can't be determined, the connector will return -an error. - -## Destination - -The Postgres Destination takes a Conduit record and stores it using a SQL statement. -The Destination is designed to handle different payloads and keys. Because of this, -each record is individually parsed and upserted. - -### Handling record operations - -Based on the `Operation` field in the record, the destination will either insert, -update or delete the record in the target table. Snapshot records are always inserted. - -If the target table already contains a record with the same key as a record being -inserted, the record will be updated (upserted). This can overwrite and thus potentially -lose data, so keys should be assigned correctly from the Source. - -If the target table does not contain a record with the same key as a record being -deleted, the record will be ignored. - -If there is no key, the record will be simply appended. - - -## Source Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-source - type: source - plugin: "postgres" - name: example-source - settings: - # Tables is a List of table names to read from, separated by a comma, - # e.g.:"table1,table2". Use "*" if you'd like to listen to all tables. - # Type: string - tables: "" - # URL is the connection string for the Postgres database. - # Type: string - url: "" - # CDCMode determines how the connector should listen to changes. - # Type: string - cdcMode: "auto" - # LogreplAutoCleanup determines if the replication slot and - # publication should be removed when the connector is deleted. - # Type: bool - logrepl.autoCleanup: "true" - # LogreplPublicationName determines the publication name in case the - # connector uses logical replication to listen to changes (see - # CDCMode). - # Type: string - logrepl.publicationName: "conduitpub" - # LogreplSlotName determines the replication slot name in case the - # connector uses logical replication to listen to changes (see - # CDCMode). Can only contain lower-case letters, numbers, and the - # underscore character. - # Type: string - logrepl.slotName: "conduitslot" - # WithAvroSchema determines whether the connector should attach an - # avro schema on each record. - # Type: bool - logrepl.withAvroSchema: "true" - # Snapshot fetcher size determines the number of rows to retrieve at a - # time. - # Type: int - snapshot.fetchSize: "50000" - # SnapshotMode is whether the plugin will take a snapshot of the - # entire table before starting cdc mode. - # Type: string - snapshotMode: "initial" - # Maximum delay before an incomplete batch is read from the source. - # Type: duration - sdk.batch.delay: "0" - # Maximum size of batch before it gets read from the source. - # Type: int - sdk.batch.size: "0" - # Specifies whether to use a schema context name. If set to false, no - # schema context name will be used, and schemas will be saved with the - # subject name specified in the connector (not safe because of name - # conflicts). - # Type: bool - sdk.schema.context.enabled: "true" - # Schema context name to be used. Used as a prefix for all schema - # subject names. If empty, defaults to the connector ID. - # Type: string - sdk.schema.context.name: "" - # Whether to extract and encode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "false" - # The subject of the key schema. If the record metadata contains the - # field "opencdc.collection" it is prepended to the subject name and - # separated with a dot. - # Type: string - sdk.schema.extract.key.subject: "key" - # Whether to extract and encode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "false" - # The subject of the payload schema. If the record metadata contains - # the field "opencdc.collection" it is prepended to the subject name - # and separated with a dot. - # Type: string - sdk.schema.extract.payload.subject: "payload" - # The type of the payload schema. - # Type: string - sdk.schema.extract.type: "avro" -``` - -## Destination Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-destination - type: destination - plugin: "postgres" - name: example-destination - settings: - # URL is the connection string for the Postgres database. - # Type: string - url: "" - # Table is used as the target table into which records are inserted. - # Type: string - table: "{{ index .Metadata "opencdc.collection" }}" - # Maximum delay before an incomplete batch is written to the - # destination. - # Type: duration - sdk.batch.delay: "0" - # Maximum size of batch before it gets written to the destination. - # Type: int - sdk.batch.size: "0" - # Allow bursts of at most X records (0 or less means that bursts are - # not limited). Only takes effect if a rate limit per second is set. - # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the - # effective batch size will be equal to `sdk.rate.burst`. - # Type: int - sdk.rate.burst: "0" - # Maximum number of records written per second (0 means no rate - # limit). - # Type: float - sdk.rate.perSecond: "0" - # The format of the output record. See the Conduit documentation for a - # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). - # Type: string - sdk.record.format: "opencdc/json" - # Options to configure the chosen output record format. Options are - # normally key=value pairs separated with comma (e.g. - # opt1=val2,opt2=val2), except for the `template` record format, where - # options are a Go template. - # Type: string - sdk.record.format.options: "" - # Whether to extract and decode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "true" - # Whether to extract and decode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "true" -``` diff --git a/docs/1-using/5-connectors/7-list/5-pgvector.mdx b/docs/1-using/5-connectors/7-list/7-pgvector.mdx similarity index 78% rename from docs/1-using/5-connectors/7-list/5-pgvector.mdx rename to docs/1-using/5-connectors/7-list/7-pgvector.mdx index 2905900..b017c72 100644 --- a/docs/1-using/5-connectors/7-list/5-pgvector.mdx +++ b/docs/1-using/5-connectors/7-list/7-pgvector.mdx @@ -68,22 +68,22 @@ import Tooltip from '@mui/material/Tooltip'; ## Latest release -- [conduit-connector-pgvector_0.1.0_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Darwin_arm64.tar.gz) -- [conduit-connector-pgvector_0.1.0_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Darwin_x86_64.tar.gz) -- [conduit-connector-pgvector_0.1.0_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Linux_arm64.tar.gz) -- [conduit-connector-pgvector_0.1.0_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Linux_i386.tar.gz) -- [conduit-connector-pgvector_0.1.0_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Linux_x86_64.tar.gz) -- [conduit-connector-pgvector_0.1.0_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Windows_arm64.tar.gz) -- [conduit-connector-pgvector_0.1.0_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Windows_i386.tar.gz) -- [conduit-connector-pgvector_0.1.0_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/conduit-connector-pgvector_0.1.0_Windows_x86_64.tar.gz) -- [pgvector_v0.1.0_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_darwin_amd64.tar.gz) -- [pgvector_v0.1.0_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_darwin_amd64.tar.gz.sigstore.json) -- [pgvector_v0.1.0_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_darwin_arm64.tar.gz) -- [pgvector_v0.1.0_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_darwin_arm64.tar.gz.sigstore.json) -- [pgvector_v0.1.0_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_linux_amd64.tar.gz) -- [pgvector_v0.1.0_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_linux_amd64.tar.gz.sigstore.json) -- [pgvector_v0.1.0_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_linux_arm64.tar.gz) -- [pgvector_v0.1.0_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.0/pgvector_v0.1.0_linux_arm64.tar.gz.sigstore.json) +- [conduit-connector-pgvector_0.1.1_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Darwin_arm64.tar.gz) +- [conduit-connector-pgvector_0.1.1_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Darwin_x86_64.tar.gz) +- [conduit-connector-pgvector_0.1.1_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Linux_arm64.tar.gz) +- [conduit-connector-pgvector_0.1.1_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Linux_i386.tar.gz) +- [conduit-connector-pgvector_0.1.1_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Linux_x86_64.tar.gz) +- [conduit-connector-pgvector_0.1.1_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Windows_arm64.tar.gz) +- [conduit-connector-pgvector_0.1.1_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Windows_i386.tar.gz) +- [conduit-connector-pgvector_0.1.1_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/conduit-connector-pgvector_0.1.1_Windows_x86_64.tar.gz) +- [pgvector_v0.1.1_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_darwin_amd64.tar.gz) +- [pgvector_v0.1.1_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_darwin_amd64.tar.gz.sigstore.json) +- [pgvector_v0.1.1_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_darwin_arm64.tar.gz) +- [pgvector_v0.1.1_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_darwin_arm64.tar.gz.sigstore.json) +- [pgvector_v0.1.1_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_linux_amd64.tar.gz) +- [pgvector_v0.1.1_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_linux_amd64.tar.gz.sigstore.json) +- [pgvector_v0.1.1_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_linux_arm64.tar.gz) +- [pgvector_v0.1.1_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-pgvector/releases/download/v0.1.1/pgvector_v0.1.1_linux_arm64.tar.gz.sigstore.json) ## Description @@ -192,7 +192,7 @@ pipelines: sdk.rate.perSecond: "0" # The format of the output record. See the Conduit documentation for a # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). # Type: string sdk.record.format: "opencdc/json" # Options to configure the chosen output record format. Options are diff --git a/docs/1-using/5-connectors/7-list/7-s3.mdx b/docs/1-using/5-connectors/7-list/7-s3.mdx deleted file mode 100644 index e0a12e7..0000000 --- a/docs/1-using/5-connectors/7-list/7-s3.mdx +++ /dev/null @@ -1,326 +0,0 @@ ---- -IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. - -title: "s3" -description: "An S3 source and destination plugin for Conduit, written in Go." ---- - -import ReactDiffViewer from 'react-diff-viewer-continued'; -import Chip from '@mui/material/Chip'; -import Box from "@mui/system/Box"; -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import GitHubIcon from '@mui/icons-material/GitHub'; -import IconButton from '@mui/material/IconButton'; -import Stack from '@mui/material/Stack'; -import Link from '@mui/material/Link'; -import StarIcon from '@mui/icons-material/Star'; -import Tooltip from '@mui/material/Tooltip'; - -# s3 - - - {/* Author info */} - - Author: Meroxa, Inc. - - Conduit team logo - - - - {/* Divider */} - - - {/* GitHub link */} - - - - - {/* Stars */} - } - label="9" - size="small" - sx={{ height: 24 }} - /> - - -## Latest release - -- [conduit-connector-s3_0.9.4_Darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Darwin_arm64.tar.gz) -- [conduit-connector-s3_0.9.4_Darwin_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Darwin_x86_64.tar.gz) -- [conduit-connector-s3_0.9.4_Linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Linux_arm64.tar.gz) -- [conduit-connector-s3_0.9.4_Linux_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Linux_i386.tar.gz) -- [conduit-connector-s3_0.9.4_Linux_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Linux_x86_64.tar.gz) -- [conduit-connector-s3_0.9.4_Windows_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Windows_arm64.tar.gz) -- [conduit-connector-s3_0.9.4_Windows_i386.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Windows_i386.tar.gz) -- [conduit-connector-s3_0.9.4_Windows_x86_64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/conduit-connector-s3_0.9.4_Windows_x86_64.tar.gz) -- [s3_v0.9.4_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_darwin_amd64.tar.gz) -- [s3_v0.9.4_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_darwin_amd64.tar.gz.sigstore.json) -- [s3_v0.9.4_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_darwin_arm64.tar.gz) -- [s3_v0.9.4_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_darwin_arm64.tar.gz.sigstore.json) -- [s3_v0.9.4_linux_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_linux_amd64.tar.gz) -- [s3_v0.9.4_linux_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_linux_amd64.tar.gz.sigstore.json) -- [s3_v0.9.4_linux_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_linux_arm64.tar.gz) -- [s3_v0.9.4_linux_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.9.4/s3_v0.9.4_linux_arm64.tar.gz.sigstore.json) - -## Description - -The S3 connector is one of [Conduit](https://github.com/ConduitIO/conduit) -builtin plugins. It provides both, a source and a destination S3 connectors. - -## Source - -The S3 Source Connector connects to a S3 bucket with the provided -configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and -`aws.region`. If the bucket doesn't exist, or the permissions fail, then an -error will occur. After that, the `Open` method is called to start the -connection from the provided position. - -### Change Data Capture (CDC) - -This connector implements CDC features for S3 by scanning the bucket for changes -every `pollingPeriod` and detecting any change that happened after a certain -timestamp. These changes (update, delete, create) are then inserted into a -buffer that is checked on each Read request. - -* To capture "delete" and "update", the S3 bucket versioning must be enabled. -* To capture "create" actions, the bucket versioning doesn't matter. - -#### Position Handling - -The connector goes through two modes. - -* Snapshot mode: which loops through the S3 bucket and returns the objects that - are already in there. The _Position_ during this mode is the object key - attached to an underscore, an "s" for snapshot, and the _maxLastModifiedDate_ - found so far. As an example: "thisIsAKey_s12345", which makes the connector - know at what mode it is and what object it last read. The - _maxLastModifiedDate_ will be used when changing to CDC mode, the iterator - will capture changes that happened after that. - -* CDC mode: this mode iterates through the S3 bucket every `pollingPeriod` and - captures new actions made on the bucket. the _Position_ during this mode is - the object key attached to an underscore, a "c" for CDC, and the object's - _lastModifiedDate_ in seconds. As an example: "thisIsAKey_c1634049397". This - position is used to return only the actions with a _lastModifiedDate_ higher - than the last record returned, which will ensure that no duplications are in - place. - -### Record Keys - -The S3 object key uniquely identifies the objects in an Amazon S3 bucket, which -is why a record key is the key read from the S3 bucket. - -## Destination - -The S3 Destination Connector connects to an S3 bucket with the provided -configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and -`aws.region`. If the permissions fail, the connector will not be ready for -writing to S3. - -### Writer - -The S3 destination writer has a buffer with the size of `bufferSize`, for each -time `Write` is called, a new record is added to the buffer. When the buffer is -full, all the records from it will be written to the S3 bucket, and an ack -function will be called for each record after being written. - -## Source Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-source - type: source - plugin: "s3" - name: example-source - settings: - # AWS access key id. - # Type: string - aws.accessKeyId: "" - # the AWS S3 bucket name. - # Type: string - aws.bucket: "" - # the AWS S3 bucket region - # Type: string - aws.region: "" - # AWS secret access key. - # Type: string - aws.secretAccessKey: "" - # The endpoint to connect to. Set this when using an S3-compatible - # object store such as MinIO (for example http://localhost:9000). - # Leave empty to use the default AWS endpoints. The AWS_ENDPOINT_URL - # environment variable is also honored when this is empty. Note that - # an http:// endpoint sends every request, including the SigV4 - # Authorization header that carries the access key ID, in the clear: - # use https:// for anything that is not on localhost. - # Type: string - aws.endpoint: "" - # Use path-style addressing (http://endpoint/bucket/key) instead of - # virtual-hosted addressing (http://bucket.endpoint/key). Set this to - # true when using an S3-compatible store that does not support - # virtual-hosted addressing. The default MinIO setup (started without - # MINIO_DOMAIN) is one: it ignores the bucket in the host name and - # takes the first path segment as the bucket instead. What comes back - # depends on the key: with a prefix configured the remaining segments - # are still a key, so PutObject fails with a 404 NoSuchBucket; without - # a prefix there is no key left and MinIO answers with a 400 - # MalformedXML. Leave false for AWS S3. - # Type: bool - aws.pathStyle: "false" - # polling period for the CDC mode, formatted as a time.Duration - # string. - # Type: duration - pollingPeriod: "1s" - # the S3 key prefix. - # Type: string - prefix: "" - # Maximum delay before an incomplete batch is read from the source. - # Type: duration - sdk.batch.delay: "0" - # Maximum size of batch before it gets read from the source. - # Type: int - sdk.batch.size: "0" - # Specifies whether to use a schema context name. If set to false, no - # schema context name will be used, and schemas will be saved with the - # subject name specified in the connector (not safe because of name - # conflicts). - # Type: bool - sdk.schema.context.enabled: "true" - # Schema context name to be used. Used as a prefix for all schema - # subject names. If empty, defaults to the connector ID. - # Type: string - sdk.schema.context.name: "" - # Whether to extract and encode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "false" - # The subject of the key schema. If the record metadata contains the - # field "opencdc.collection" it is prepended to the subject name and - # separated with a dot. - # Type: string - sdk.schema.extract.key.subject: "key" - # Whether to extract and encode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "false" - # The subject of the payload schema. If the record metadata contains - # the field "opencdc.collection" it is prepended to the subject name - # and separated with a dot. - # Type: string - sdk.schema.extract.payload.subject: "payload" - # The type of the payload schema. - # Type: string - sdk.schema.extract.type: "avro" -``` - -## Destination Parameters - -```yaml -version: 2.2 -pipelines: - - id: example - status: running - connectors: - - id: example-destination - type: destination - plugin: "s3" - name: example-destination - settings: - # AWS access key id. - # Type: string - aws.accessKeyId: "" - # the AWS S3 bucket name. - # Type: string - aws.bucket: "" - # the AWS S3 bucket region - # Type: string - aws.region: "" - # AWS secret access key. - # Type: string - aws.secretAccessKey: "" - # the destination format, either "json" or "parquet". - # Type: string - format: "" - # The endpoint to connect to. Set this when using an S3-compatible - # object store such as MinIO (for example http://localhost:9000). - # Leave empty to use the default AWS endpoints. The AWS_ENDPOINT_URL - # environment variable is also honored when this is empty. Note that - # an http:// endpoint sends every request, including the SigV4 - # Authorization header that carries the access key ID, in the clear: - # use https:// for anything that is not on localhost. - # Type: string - aws.endpoint: "" - # Use path-style addressing (http://endpoint/bucket/key) instead of - # virtual-hosted addressing (http://bucket.endpoint/key). Set this to - # true when using an S3-compatible store that does not support - # virtual-hosted addressing. The default MinIO setup (started without - # MINIO_DOMAIN) is one: it ignores the bucket in the host name and - # takes the first path segment as the bucket instead. What comes back - # depends on the key: with a prefix configured the remaining segments - # are still a key, so PutObject fails with a 404 NoSuchBucket; without - # a prefix there is no key left and MinIO answers with a 400 - # MalformedXML. Leave false for AWS S3. - # Type: bool - aws.pathStyle: "false" - # the S3 key prefix. - # Type: string - prefix: "" - # Maximum delay before an incomplete batch is written to the - # destination. - # Type: duration - sdk.batch.delay: "0" - # Maximum size of batch before it gets written to the destination. - # Type: int - sdk.batch.size: "0" - # Allow bursts of at most X records (0 or less means that bursts are - # not limited). Only takes effect if a rate limit per second is set. - # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the - # effective batch size will be equal to `sdk.rate.burst`. - # Type: int - sdk.rate.burst: "0" - # Maximum number of records written per second (0 means no rate - # limit). - # Type: float - sdk.rate.perSecond: "0" - # The format of the output record. See the Conduit documentation for a - # full list of supported formats - # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). - # Type: string - sdk.record.format: "opencdc/json" - # Options to configure the chosen output record format. Options are - # normally key=value pairs separated with comma (e.g. - # opt1=val2,opt2=val2), except for the `template` record format, where - # options are a Go template. - # Type: string - sdk.record.format.options: "" - # Whether to extract and decode the record key with a schema. - # Type: bool - sdk.schema.extract.key.enabled: "true" - # Whether to extract and decode the record payload with a schema. - # Type: bool - sdk.schema.extract.payload.enabled: "true" -``` diff --git a/docs/1-using/5-connectors/7-list/8-postgres.mdx b/docs/1-using/5-connectors/7-list/8-postgres.mdx new file mode 100644 index 0000000..57c35a7 --- /dev/null +++ b/docs/1-using/5-connectors/7-list/8-postgres.mdx @@ -0,0 +1,742 @@ +--- +IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. + +title: "postgres" +description: "Conduit connector for PostgreSQL" +--- + +import ReactDiffViewer from 'react-diff-viewer-continued'; +import Chip from '@mui/material/Chip'; +import Box from "@mui/system/Box"; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import GitHubIcon from '@mui/icons-material/GitHub'; +import IconButton from '@mui/material/IconButton'; +import Stack from '@mui/material/Stack'; +import Link from '@mui/material/Link'; +import StarIcon from '@mui/icons-material/Star'; +import Tooltip from '@mui/material/Tooltip'; + +# postgres + + + {/* Author info */} + + Author: Meroxa, Inc. + + Conduit team logo + + + + {/* Divider */} + + + {/* GitHub link */} + + + + + {/* Stars */} + } + label="20" + size="small" + sx={{ height: 24 }} + /> + + +## Latest release + + +## Description + +## Connection and SCRAM authentication + +Source and destination connect with the `url` setting. Over TLS with SCRAM +authentication, the connector does not use channel binding unless you ask for it: +`channel_binding=prefer`, the default of the pgx driver since 5.9, is mapped to +`disable`, which is how the driver behaved in earlier releases. Channel binding +(SCRAM-SHA-256-PLUS) fails against a server with an Ed25519 certificate and behind +a proxy that terminates TLS and passes the SCRAM exchange through. Add +`channel_binding=require` to the URL, or set `PGCHANNELBINDING=require` in the +environment of the Conduit process, to require it. The driver reads +`PGCHANNELBINDING` when the URL has no `channel_binding`. An explicit `prefer`, +in the URL or in the environment, is also mapped to `disable`; `require` and +`disable` are kept. + +``` +postgres://user:pass@host:5432/db?sslmode=require&channel_binding=require +``` + +### Upgrading from v0.14.x: connection string parsing + +This version uses pgx 5.11, which parses connection strings like libpq does. +Most URLs are unaffected. Check yours if it relies on one of these (the full +list is in the pgx 5.11.0 changelog): + +- `+` in a query value is literal, no longer a space; encode a space as `%20`. +- If a query parameter is repeated, the last one wins (the first used to). +- An IPv6 host in a URL needs brackets: `postgres://[::1]/db`. +- In a `key=value` connection string a backslash escapes the next character, + so a Windows path must be written `sslcert=C:\\path\\to\\cert`. +- Default locations of the client certificate, key and `.pgpass` file are + resolved from `$HOME` instead of the OS account's home directory. + +## Source + +The Postgres Source Connector connects to a database with the provided `url` and +starts creating records for each change detected in the provided tables. + +Upon starting, the source takes a snapshot of the provided tables in the database, +then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. + +### Snapshot + +When the connector first starts, snapshot mode is enabled. The connector acquires +a read-only lock on the tables, and then reads all rows of the tables into Conduit. +Once all rows in that initial snapshot are read the connector releases its lock and +switches into CDC mode. + +This behavior is enabled by default, but can be turned off by adding +`"snapshotMode": "never"` to the Source configuration. + +### Change Data Capture + +This connector implements Change Data Capture (CDC) features for PostgreSQL by +creating a logical replication slot and a publication that listens to changes in the +configured tables. Every detected change is converted into a record. If there are no +records available, the connector blocks until a record is available or the connector +receives a stop signal. + +#### Logical Replication Configuration + +When the connector switches to CDC mode, it attempts to run the initial setup commands +to create its logical replication slot and publication. It will connect to an existing +slot if one with the configured name exists. + +The Postgres user specified in the connection URL must have sufficient privileges to +run all of these setup commands, or it will fail. + +Example pipeline configuration that's using logical replication: + +```yaml +version: 2.2 +pipelines: + - id: pg-to-log + status: running + connectors: + - id: pg + type: source + plugin: builtin:postgres + settings: + url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" + tables: "users" + cdcMode: "logrepl" + logrepl.publicationName: "examplepub" + logrepl.slotName: "exampleslot" + - id: log + type: destination + plugin: builtin:log + settings: + level: info +``` + +:warning: When the connector or pipeline is deleted, the connector will automatically +attempt to delete the replication slot and publication. This is the default behaviour +and can be disabled by setting `logrepl.autoCleanup` to `false`. + +### Key Handling + +The connector will automatically look up the primary key column for the specified tables +and use them as the key value. If that can't be determined, the connector will return +an error. + +The table's `REPLICA IDENTITY` must carry the primary key in deletes. `DEFAULT` and +`FULL` always do. `USING INDEX` does only if the index includes the primary-key +column; otherwise Postgres sends deletes without the key, and they cannot be applied +downstream. The source refuses to open on such a table with the error code +`postgres.replica_identity.excludes_key`. Fix it with +`ALTER TABLE
REPLICA IDENTITY FULL`, or use a unique index that includes the +primary key. + +### Before Image of Updates and Deletes + +`Payload.Before` is set only when Postgres sends the whole old row, which it does +under `REPLICA IDENTITY FULL`. Under `DEFAULT` and `USING INDEX`, Postgres sends only +the identity columns of the old row, so deletes and key-changing updates carry no +`Before`. The deleted row's key is in the record `Key`. A key-changing update carries +the new key in `Key`; the old key is not carried. If you need the old row or the old +key, set `REPLICA IDENTITY FULL` on the table. + +:warning: The Postgres destination does not delete the old row when a primary key +changes. It upserts on the new key, so a primary-key-changing `UPDATE` leaves the row +under its old key in the destination next to the new one. This happens with or without +Avro and under `REPLICA IDENTITY FULL` as well. It is a known destination bug +([#386](https://github.com/ConduitIO/conduit-connector-postgres/issues/386)). Until it +is fixed, avoid changing primary keys on tables replicated to Postgres, or reconcile +the old rows by hand. + +### Delete With a Missing Key + +**Symptom.** The pipeline fails and keeps failing after restarts with +`postgres.replica_identity.excludes_key: delete on table "
" (WAL LSN , in the +transaction that commits at LSN ) arrived without a value for key column ...`. +The replication slot retains WAL while it is stuck. + +**Diagnosis.** The table's `REPLICA IDENTITY` was changed while the pipeline was +running, to `USING INDEX` on an index whose key columns do not include the primary key +(columns in `INCLUDE` do not count), or the identity index was dropped. Postgres then +sends the delete without the key. Check with +`SELECT relreplident FROM pg_class WHERE oid = '
'::regclass` (`i` is +`USING INDEX`). + +**Remediation.** + +1. Fix the identity so that new deletes carry the key: + `ALTER TABLE
REPLICA IDENTITY FULL`, or `REPLICA IDENTITY DEFAULT`, or + `REPLICA IDENTITY USING INDEX ` on a unique index that has the primary key + among its key columns. This does not repair the change already in the WAL. A restart + alone fails on it again. +2. Stop the pipeline. Postgres refuses to read or advance a slot while a connection + uses it. +3. Find the row. The error carries no row values, because they would end up in logs. + Read the stuck transaction from the slot instead, as a superuser or a role with + `REPLICATION`. Peeking does not consume anything. `` is the commit LSN `` + from the error with its low hex part incremented by one (`0/40C5318` becomes + `0/40C5319`); the error prints it: + + ```sql + SELECT xid, lsn, encode(data, 'escape') AS message + FROM pg_logical_slot_peek_binary_changes( + '', '', NULL, + 'proto_version', '1', 'publication_names', ''); + ``` + + Use the values of `logrepl.slotName` and `logrepl.publicationName` (defaults + `conduitslot` and `conduitpub`). + The output can start with earlier transactions that the pipeline already delivered + but Postgres has not yet been told about (the connector reports its position + periodically). They can contain the same value as the stuck delete, so do not read + them. Read only the transaction (`xid`) that contains the `D` row whose `lsn` is the + WAL LSN `` in the error. + Each row is one pgoutput message. `B` and `C` begin and commit the transaction. `R` + names a table and its columns. `I`, `U` and `D` are the changes. A `K` tuple follows + the table's object id: a 2-byte column count, then one entry per column, `n` for NULL + or `t`, a 4-byte length and the text value. For this error the only non-null columns + are those of the identity index, so the readable text is the value to look for, for + example `222` in a `D` row that shows `K ... n t ... 222 n`. Postgres only accepts a + unique index as a replica identity, so that value identifies the row in the + destination. Only tables in the publication appear, which are the tables the pipeline + captures. +4. Delete that row from the destination yourself. The change has no key, so the pipeline + cannot do it. Then work out what else step 5 would skip. It skips every change the + peek lists before the stuck one, and the other changes of the stuck transaction. + Postgres has not been told they are done, and the pipeline may have read them + without the destination writing them before it stopped. Get the source connector's + last acknowledged position from Conduit: + `GET /v1/connectors/`, field `sourceState.position`. It is + base64 of JSON, for example + `{"version":2,"type":2,"last_lsn":"0/40845E8","tx_commit_lsn":"0/40846E8","tx_seq":1}` + (decode it with `base64 -d`). Conduit sets it only when the destination acknowledges a + record, never when the source reads one, so `last_lsn` is the WAL LSN of the last + change the destination has confirmed. If Conduit was restarted since the pipeline + stopped, it reports the last position it saved, which can be slightly older; that errs + on the safe side. Find the peek row of kind `I`, `U` or `D` whose `lsn` equals + `last_lsn`. Ignore `B` and `C` rows: a `C` row can show the same number as the first + change after it. If several rows share that `lsn` (a `COPY` or another statement + that changes many rows can do that), use `tx_seq`. It numbers the transaction's + `I`, `U`, `D` and `T` (truncate) rows in the order listed, starting at 1 after its + `B` row. `R` (relation), `Y` (type) and `O` (origin) rows are not counted. The row + with that number is the one to match, and it is an `I`, `U` or `D` row. The matched row + itself and everything listed before it are in the destination. For each change + listed after it, up to the stuck one and including the rest of the stuck transaction, + check the destination for its rows and apply any that are missing by hand (`I` and + `U` carry the full new row; `D` carries the identity columns). If no row matches, + treat every listed change as unconfirmed. A position without `last_lsn` (a pipeline + still in its snapshot) matches nothing. +5. Skip the transaction. Use the same `` as in step 3. Writing it as a literal is + deliberate: `''::pg_lsn + 1` only works from Postgres 14. + - Postgres 11 or later: + `SELECT pg_replication_slot_advance('', '');` + - Postgres 10 has no such function. Consume the changes up to `` instead. This + skips exactly the same changes and needs no new snapshot: + `SELECT count(*) FROM pg_logical_slot_get_binary_changes('', '', NULL, 'proto_version', '1', 'publication_names', '');` + + Advancing to `` is what skips the transaction. Advancing to the change LSN + (``) or to the commit LSN `` itself does not: Postgres replays the transaction + and the pipeline fails on the same delete. +6. Start the pipeline. + +**Skipping the transaction skips that change, and it skips the whole transaction, +including its other changes to captured tables.** Skipping data is your decision and the +connector cannot check it for you. + +### Unchanged TOASTed Columns + +Postgres stores large field values (`text`, `bytea`, `jsonb`, and other +variable-length types past a size threshold) out-of-line using +[TOAST](https://www.postgresql.org/docs/current/storage-toast.html). When a row is +updated but a TOASTed column is **not** changed, Postgres omits that column's value +from the logical-replication message to avoid re-transmitting large unchanged data. + +For such columns, this connector **omits the field from the record payload** rather +than emitting it as `NULL`. Emitting `NULL` would be silent data corruption — a +downstream sink cannot distinguish "the value was set to NULL" from "the value did +not change". Omission preserves that distinction: absence means "unchanged". + +What this means for your destination: + +- **Partial-update / upsert sinks** (including this connector's own Postgres + destination, which builds its `ON CONFLICT DO UPDATE SET` clause only from the + fields present in the payload) handle omission correctly — an absent field is left + untouched in the target row. +- **Full-row / full-document replace sinks** must treat an absent field as + "unchanged" and preserve the existing target value. If a sink cannot do this, set + `REPLICA IDENTITY FULL` on the source table. Under `REPLICA IDENTITY FULL` the old + row image carries every column, and this connector backfills unchanged TOASTed + values from it so the payload is always complete. + +### Value Representation + +Values do not depend on the time zone of the machine running the connector or on the +version of the Postgres driver. The same row gives the same record everywhere, for +snapshot and CDC. + +| Postgres type | Without an Avro schema | With an Avro schema (`logrepl.withAvroSchema`, the default) | +| --- | --- | --- | +| `timestamp` | The wall clock stored in Postgres, as a time with no offset information (`2024-01-02T03:04:05Z` for `2024-01-02 03:04:05`). | `local-timestamp-micros` of the same wall clock. | +| `timestamptz` | The instant, in UTC (`Z`). | `timestamp-micros`. | +| `date` | Midnight UTC of the date. | `date`. | +| `tsvector` | Postgres text form, `'a':1 'cat':3A,4B`. | `string`, same text. | +| `infinity`, `-infinity` | The strings `infinity` and `-infinity`. | The source halts with error code `postgres.source.temporal_not_representable`. | +| Years before 1 BC or after 9999 AD | Postgres text, for example `0044-03-15 12:00:00 BC`. | As above, up to what 64-bit microseconds hold; beyond that the same error. | +| Arrays, ranges, multiranges of these | Element and bound values as above. | Not supported by the Avro schema. | + +A range or multirange is a JSON object `{"Lower":...,"Upper":...,"LowerType":105, +"UpperType":101,"Valid":true}`, and a multirange is an array of them. `LowerType` and +`UpperType` are the ASCII codes of the bound kind: `105` (`i`) inclusive, `101` (`e`) +exclusive, `85` (`U`) unbounded (the bound is `null`), `69` (`E`) empty range. Bounds +follow the rules above. Arrays of `tsmultirange`, `tstzmultirange` and `datemultirange` +are emitted as the Postgres text of the array. + +If the source halts with `postgres.source.temporal_not_representable`, set +`logrepl.withAvroSchema` to `false`. Removing the row does not help a change that is +already in the WAL, because it is read again after every restart. + +### Numeric Columns + +A `numeric` column with a precision and scale, `numeric(p,s)`, is an Avro +`decimal(p,s)` and its values are `big.Rat`. + +A `numeric` column **without** a precision and scale can hold any value Postgres +can store, which no Avro `decimal` can. With `logrepl.withAvroSchema: true` (the +default) it is an Avro `string` holding the exact text of the value, with its +scale (`1.50` stays `1.50`), and `NaN`, `Infinity` and +`-Infinity` as those three words: **a consumer of such a column must handle +them**, a plain decimal parser will reject them. Earlier versions used `decimal(65535, 65531)`, which could not encode a +value of 10000 or more and stopped the pipeline. The first record after an +upgrade registers a new schema version for the table; consumers must read the +field as a string. This connector's destination writes the string to a `numeric` +column exactly. + +**Known issue:** a zero with a display scale reads as `0` in the snapshot and as +`0.000` (its stored text) in change data capture. The values are numerically +equal; only the text differs. Tracked in +[#391](https://github.com/ConduitIO/conduit-connector-postgres/issues/391). + +So the Go type of an unconstrained `numeric` depends on the setting: a `string` +with `logrepl.withAvroSchema: true`, a `big.Rat` with `false`. Changing a column +between `text`/`varchar` and an unconstrained numeric is accepted by the +`evolve` schema-drift policy only when `withAvroSchema` is `true` (the Avro +shape does not change); with `false` the value's type changes, so it halts. + +Without an Avro schema, a `numeric` is a `big.Rat`; `NaN`, `Infinity` and +`-Infinity` are the strings `"NaN"`, `"Infinity"` and `"-Infinity"`, and the same +applies to the elements of `numeric[]` (finite elements are unchanged). A `NULL` +stays `NULL`. + +`NaN` in a `numeric(p,s)` column (the infinities cannot be stored there) cannot +be an Avro decimal, and the connector never replaces it with `NULL` or a number: +the source stops with the error code `postgres.source.numeric_not_representable`, +which names the table, the column, the value and the row's key. To continue: + +- **Change data capture:** restart once with `logrepl.withAvroSchema: false` + from the same position. The value is delivered as a string. **Then set it + back to `true` as soon as that record has been delivered.** Without an Avro + schema a finite `numeric` is a `big.Rat`, which a standalone connector + cannot send to Conduit (#378): the next finite numeric would stop the + pipeline. Built-in pipelines are not affected. + On a standalone connector the SDK also fails the whole read batch on the + first record it cannot send, so if the NaN/Infinity record or its batch + contains any finite numeric, it is never delivered with Avro off and this + workaround does not help (#378 is the fix). While Avro is off, + JSON-serializing destinations receive a finite numeric as the string `"3/2"`. +- **Snapshot:** `UPDATE` the row to a finite value (or `NULL`) and restart. +- Removing the row, or adding a `CHECK` constraint, does not help for a change + that is already in the WAL: it is read again after every restart. Do not drop + the replication slot to get past it, that loses the position. A constraint + only keeps new bad values out. + +### Limitations + +- **`TRUNCATE` is not propagated.** OpenCDC has no truncate operation, so a + `TRUNCATE` on a published table produces no record: rows it removed remain in + the destination. If the publication publishes `truncate` (the one the + connector creates does by default), the connector logs a `WARN` ("TRUNCATE was NOT + propagated") naming the table and the LSN, at most once a minute per table. + A restricted or user-supplied publication (for example + `publish = 'insert, update, delete'`) never sends the message, so there is + no warning and no other signal. The `suppressed` field counts the TRUNCATEs + of that table the rate limit swallowed since the previous warning; when + another table's warning evicts an expired entry that still holds a count, a + one-line summary with that count is logged. Treat the log lines as a signal, + not a precise counter: a count still pending when the process exits is + lost. Inserts, updates and deletes around the `TRUNCATE` are delivered as + usual. A configurable policy for TRUNCATE is planned. Until then, treat the + warning as a signal to reconcile the destination by hand. + +## Destination + +The Postgres Destination takes a Conduit record and stores it using a SQL statement. +The Destination is designed to handle different payloads and keys. Because of this, +each record is individually parsed and upserted. + +### Handling record operations + +Based on the `Operation` field in the record, the destination will either insert, +update or delete the record in the target table. Snapshot records are always inserted. + +If the target table already contains a record with the same key as a record being +inserted, the record will be updated (upserted). This can overwrite and thus potentially +lose data, so keys should be assigned correctly from the Source. + +If the target table does not contain a record with the same key as a record being +deleted, the record will be ignored. + +If there is no key, the record will be simply appended. + +### Keys, columns and acknowledgements + +- **Upsert key.** With a structured record key (or a raw key that is a JSON object), + a create, update or snapshot record is `INSERT ... ON CONFLICT () DO + UPDATE SET` every other field of the record. The key columns must carry a primary + key or unique constraint, otherwise Postgres rejects the statement. When a record has + no field besides its key columns (a table whose columns are all key columns), the + statement is `ON CONFLICT DO NOTHING`, so a redelivery is a no-op. +- **Columns.** Each record is written with exactly its own fields. A field the table has + no column for fails the write; the destination does not create or alter tables. A + field the record lacks is not written: an insert gets the column default, an existing + row keeps its value. A create, update or snapshot record without fields fails with + `postgres.destination.payload_empty`. +- **Key values** must be one non-null scalar per key column, for upserts and deletes. A + null value fails with `postgres.destination.key_column_missing` (a delete would match + every row with a NULL there, an upsert would append a new row on every redelivery), and + a list or object fails with `postgres.destination.key_value_not_scalar` (a delete would + match several rows or none). +- **Deletes** need a key and delete the rows matching every key column. +- **Raw JSON** keys and payloads are decoded without going through `float64`: an integer + literal is an `int64` (or `uint64`), and any other number keeps its literal, so + `numeric`, `text` and `json`/`jsonb` columns store it exactly. Integer columns take a + whole value in any form (`1e3`, `7.0`) and reject a fraction; `real` and `float8` + columns store the nearest float. Structured data that arrives over the plugin protocol + has already passed through a double (ConduitIO/conduit#3100), so integers above 2^53 + sent that way are rounded before the destination sees them. +- **Names** of the table and columns are quoted as Postgres identifiers, so reserved + words, upper case and double quotes work. A name with a NUL byte or longer than 63 bytes + fails with `postgres.destination.invalid_identifier`, because Postgres would address a + different table or column. The table name is one identifier: a dot is part of the name, + not a schema separator. +- **Acknowledgements.** A write is one implicit transaction. If any statement or the + commit fails, nothing from that write is stored and no record of it is acknowledged; + the records are redelivered (at-least-once). Keyed upserts make redelivery + idempotent; keyless inserts append again. + +### Schema-drift markers + +When the Postgres source halts on schema drift it emits a drift marker: a create +record with no key and no payload, with the metadata `postgres.schema.drift=true`. +The destination skips and acknowledges a marker without writing a row, but only +when the record's position proves the Postgres source produced it as a marker for +the table named in `postgres.schema.drift.table`. The metadata alone is not proof, +because any processor or upstream system can set it. + +- A record that carries the metadata but not a marker's position, and has a key + or a payload, is written as an ordinary record. One WARN per batch reports it. +- A record shaped like a marker (no key, no payload) without a marker's position + stops the pipeline with `postgres.destination.drift_marker_unverified`. Nothing + in that batch is written or acknowledged. +- A proven marker that a processor added a key or fields to is still skipped as + a marker, with a WARN that names the added fields. + +A marker loses its position when it passes through another system, such as Kafka +or a file, because records read from there carry that system's position. In a +Postgres to Kafka to Postgres setup, drop drift markers before the hop with a +filter processor on the `postgres.schema.drift` metadata. + +The proof relies on the engine not letting processors change a record's +position. Conduit's classic engine refuses that; the arch-v2 engine lets a +processor rewrite positions, so there a processor could forge a marker. The +forged record would be skipped, which a processor with that power could do to +any record anyway. +### Column type changes on a running pipeline + +The destination reuses prepared statements (pgx's `cache_statement` query +mode, the fast mode) and checks a per-table schema fingerprint with every +batch. After `ALTER TABLE ... ALTER COLUMN ... TYPE` the fingerprint changes, +the batch is rolled back, the statement cache is cleared, and the batch is +retried with the new column types. A value the new type cannot hold exactly +fails the write instead of being converted; a widening change is written +type-exactly without an error. + +This keeps cached statements' single round trip per batch. On one dedicated +machine (benchi `destination-exec-mode`: keyed upserts into a two-column +table) describing every statement instead cost 35% fewer rows per second at +`sdk.batch.size: 1`, the default, and 16% fewer at 1000. `exec` and +`simple_protocol`, which never cache parameter types (PgBouncer in +transaction mode needs one of them), are kept when the URL sets them, as is +`describe_exec`. + +### Numeric columns + +Decimal values (`big.Rat`) are sent to Postgres as exact decimal strings, and +Postgres applies the column's precision and scale itself when it stores the +value. The destination never reads or caches a column's scale, so an +`ALTER TABLE ... ALTER COLUMN ... TYPE numeric(p,s)` takes effect on the next +write without restarting the pipeline, and the table is resolved through the +connection's `search_path` like the rest of the statement. + +Postgres rounds half away from zero when a value has more fractional digits +than the column's scale, and rejects a value that exceeds the column's +precision (`numeric field overflow`). A rejected batch is not acknowledged. + +A decimal that has no finite expansion, such as 1/3, is truncated toward zero +after 1001 fractional digits before it is sent. So is a decimal that terminates +but needs more than 16383 fractional digits, which Postgres rejects on every +numeric column. Postgres allows a column scale of at most 1000, so the +truncation never changes the result of the column's rounding. An unconstrained +`numeric` column stores all 1001 digits. + +The destination no longer checks that the target column is numeric. The general +rule: any column whose text input accepts the decimal string stores it. +Postgres converts the string, and the examples below were checked on +Postgres 17.5: + +- `integer`, `smallint`, `bigint`: stored if the value is whole. A fractional + value is rejected, also on a destination that was already running when the + column was changed to an integer type (see "Column type changes on a running + pipeline"). +- `real`, `float8`: stored as the nearest binary float. +- `money`: rounded to the locale's fraction digits (1/3 is stored as `$0.33`). +- `oid`: stored if the value is a whole number (7 is stored as `7`). +- `text`: stored as the decimal string. `varchar(n)` and `char(n)`: rejected + with 22001 if the string is longer than `n`. +- `boolean`: 1 is stored as `true` and 0 as `false`; other values are rejected. +- `date`: 20261012 is stored as `2026-10-12`. +- `timestamp`: 20261012 is stored as `2026-10-12 00:00:00`. +- `timestamptz`: 20261012 is stored as `2026-10-12 00:00:00+00` (session time + zone UTC). +- `interval`: 12345 is stored as `03:25:45`. +- `json`, `jsonb`: 1/4 is stored as the JSON number `0.25`. +- `bytea`: 5/2 is stored as `\x322e35`, the ASCII bytes of `2.5`. +- `bit(n)`: stored if the decimal string is exactly `n` binary digits (101 is + stored as `101` in a `bit(3)`). +- `time`, `inet`: rejected. + +Every accepted conversion to a non-numeric type is a silent coercion if the +schema maps a numeric field to the wrong column, so check the mapping. + +## Source Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-source + type: source + plugin: "postgres" + name: example-source + settings: + # Tables is a List of table names to read from, separated by a comma, + # e.g.:"table1,table2". Use "*" if you'd like to listen to all tables. + # Type: string + tables: "" + # URL is the connection string for the Postgres database. With TLS, + # SCRAM authentication does not use channel binding by default + # (`channel_binding=disable`); add `channel_binding=require` to the + # URL to require SCRAM-SHA-256-PLUS. + # Type: string + url: "" + # CDCMode determines how the connector should listen to changes. + # Type: string + cdcMode: "auto" + # LogreplAutoCleanup determines if the replication slot and + # publication should be removed when the connector is deleted. + # Type: bool + logrepl.autoCleanup: "true" + # LogreplImportDebeziumOffset starts the pipeline from a Debezium + # Postgres connector's committed offset instead of a snapshot. It is + # written by `conduit migrate kafka-connect --cutover`, not by hand. + # It applies only while the pipeline has no position: the first + # acknowledged record writes an ordinary position and the setting is + # ignored from then on (remove it once the import is complete). + # Requires snapshotMode "never" and cdcMode "logrepl", and an existing + # slot and publication, which import mode never creates. Requires + # Conduit v0.19.0 or later, which acks a record only after its + # position is persisted. + # Type: string + logrepl.import.debeziumOffset: "" + # LogreplPublicationName determines the publication name in case the + # connector uses logical replication to listen to changes (see + # CDCMode). + # Type: string + logrepl.publicationName: "conduitpub" + # LogreplSchemaDriftPolicy determines how the connector handles schema + # drift (a DDL that changes a table's shape during CDC; DBZ-3 B1): - + # "halt" (default): emit a drift marker record, checkpoint before it, + # and stop with a terminal error until the pipeline is restarted to + # approve the change. - "evolve": accept additive changes silently; + # incompatible (narrowing) changes still halt. "dlq" is reserved for a + # future version and rejected at validation. + # Type: string + logrepl.schemaDrift.policy: "halt" + # LogreplSlotName determines the replication slot name in case the + # connector uses logical replication to listen to changes (see + # CDCMode). Can only contain lower-case letters, numbers, and the + # underscore character. + # Type: string + logrepl.slotName: "conduitslot" + # WithAvroSchema determines whether the connector should attach an + # avro schema on each record. + # Type: bool + logrepl.withAvroSchema: "true" + # Snapshot fetcher size determines the number of rows to retrieve at a + # time. + # Type: int + snapshot.fetchSize: "50000" + # SnapshotMode is whether the plugin will take a snapshot of the + # entire table before starting cdc mode. + # Type: string + snapshotMode: "initial" + # Maximum delay before an incomplete batch is read from the source. + # Type: duration + sdk.batch.delay: "0" + # Maximum size of batch before it gets read from the source. + # Type: int + sdk.batch.size: "0" + # Specifies whether to use a schema context name. If set to false, no + # schema context name will be used, and schemas will be saved with the + # subject name specified in the connector (not safe because of name + # conflicts). + # Type: bool + sdk.schema.context.enabled: "true" + # Schema context name to be used. Used as a prefix for all schema + # subject names. If empty, defaults to the connector ID. + # Type: string + sdk.schema.context.name: "" + # Whether to extract and encode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "false" + # The subject of the key schema. If the record metadata contains the + # field "opencdc.collection" it is prepended to the subject name and + # separated with a dot. + # Type: string + sdk.schema.extract.key.subject: "key" + # Whether to extract and encode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "false" + # The subject of the payload schema. If the record metadata contains + # the field "opencdc.collection" it is prepended to the subject name + # and separated with a dot. + # Type: string + sdk.schema.extract.payload.subject: "payload" + # The type of the payload schema. + # Type: string + sdk.schema.extract.type: "avro" +``` + +## Destination Parameters + +```yaml +version: 2.2 +pipelines: + - id: example + status: running + connectors: + - id: example-destination + type: destination + plugin: "postgres" + name: example-destination + settings: + # URL is the connection string for the Postgres database. With TLS, + # SCRAM authentication does not use channel binding by default + # (`channel_binding=disable`); add `channel_binding=require` to the + # URL to require SCRAM-SHA-256-PLUS. The destination reuses prepared + # statements (cache_statement) and checks a per-table schema + # fingerprint with every batch, so a column type change on a running + # pipeline invalidates the cache and the next write sees the new type + # — fast and correct. Set `default_query_exec_mode` to + # `describe_exec`, `exec` or `simple_protocol` to override; + # `cache_statement` and `cache_describe` are also guarded and safe. + # Type: string + url: "" + # Table is used as the target table into which records are inserted. + # Type: string + table: "{{ index .Metadata "opencdc.collection" }}" + # Maximum delay before an incomplete batch is written to the + # destination. + # Type: duration + sdk.batch.delay: "0" + # Maximum size of batch before it gets written to the destination. + # Type: int + sdk.batch.size: "0" + # Allow bursts of at most X records (0 or less means that bursts are + # not limited). Only takes effect if a rate limit per second is set. + # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the + # effective batch size will be equal to `sdk.rate.burst`. + # Type: int + sdk.rate.burst: "0" + # Maximum number of records written per second (0 means no rate + # limit). + # Type: float + sdk.rate.perSecond: "0" + # The format of the output record. See the Conduit documentation for a + # full list of supported formats + # (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + # Type: string + sdk.record.format: "opencdc/json" + # Options to configure the chosen output record format. Options are + # normally key=value pairs separated with comma (e.g. + # opt1=val2,opt2=val2), except for the `template` record format, where + # options are a Go template. + # Type: string + sdk.record.format.options: "" + # Whether to extract and decode the record key with a schema. + # Type: bool + sdk.schema.extract.key.enabled: "true" + # Whether to extract and decode the record payload with a schema. + # Type: bool + sdk.schema.extract.payload.enabled: "true" +``` diff --git a/docs/1-using/5-connectors/7-list/9-s3.mdx b/docs/1-using/5-connectors/7-list/9-s3.mdx new file mode 100644 index 0000000..748b290 --- /dev/null +++ b/docs/1-using/5-connectors/7-list/9-s3.mdx @@ -0,0 +1,691 @@ +--- +IMPORTANT: This file was generated using src/connectorgen/main.go. DO NOT EDIT. + +title: "s3" +description: "An S3 source and destination plugin for Conduit, written in Go." +--- + +import ReactDiffViewer from 'react-diff-viewer-continued'; +import Chip from '@mui/material/Chip'; +import Box from "@mui/system/Box"; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import GitHubIcon from '@mui/icons-material/GitHub'; +import IconButton from '@mui/material/IconButton'; +import Stack from '@mui/material/Stack'; +import Link from '@mui/material/Link'; +import StarIcon from '@mui/icons-material/Star'; +import Tooltip from '@mui/material/Tooltip'; + +# s3 + + + {/* Author info */} + + Author: Meroxa, Inc. + + Conduit team logo + + + + {/* Divider */} + + + {/* GitHub link */} + + + + + {/* Stars */} + } + label="9" + size="small" + sx={{ height: 24 }} + /> + + +## Latest release + +- [s3_v0.11.0_darwin_amd64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.11.0/s3_v0.11.0_darwin_amd64.tar.gz) +- [s3_v0.11.0_darwin_amd64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.11.0/s3_v0.11.0_darwin_amd64.tar.gz.sigstore.json) +- [s3_v0.11.0_darwin_arm64.tar.gz](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.11.0/s3_v0.11.0_darwin_arm64.tar.gz) +- [s3_v0.11.0_darwin_arm64.tar.gz.sigstore.json](https://conduit.gateway.scarf.sh/connector/download/ConduitIO/conduit-connector-s3/releases/download/v0.11.0/s3_v0.11.0_darwin_arm64.tar.gz.sigstore.json) + +## Description + +The S3 connector is one of [Conduit](https://github.com/ConduitIO/conduit) +builtin plugins. It provides both, a source and a destination S3 connectors. + +## Source + +The S3 Source Connector connects to a S3 bucket with the provided +configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and +`aws.region`. If the bucket doesn't exist, or the permissions fail, then an +error will occur. After that, the `Open` method is called to start the +connection from the provided position. + +### Change Data Capture (CDC) + +This connector implements CDC features for S3 by scanning the bucket for changes +every `pollingPeriod` and detecting any change that happened after a certain +timestamp. These changes (update, delete, create) are then inserted into a +buffer that is checked on each Read request. + +* To capture "delete" and "update", the S3 bucket versioning must be enabled. +* To capture "create" actions, the bucket versioning doesn't matter. + +#### Position Handling + +The connector goes through two modes. + +* Snapshot mode: which loops through the S3 bucket and returns the objects that + are already in there. The _Position_ during this mode is the object key + attached to an underscore, an "s" for snapshot, and the _maxLastModifiedDate_ + found so far. As an example: "thisIsAKey_s12345", which makes the connector + know at what mode it is and what object it last read. The + _maxLastModifiedDate_ will be used when changing to CDC mode, the iterator + will capture changes that happened after that. + +* CDC mode: this mode iterates through the S3 bucket every `pollingPeriod` and + captures new actions made on the bucket. the _Position_ during this mode is + the object key attached to an underscore, a "c" for CDC, and the object's + _lastModifiedDate_ in seconds. As an example: "thisIsAKey_c1634049397". This + position is used to resume from the last record returned. + + **CDC mode is listing based and best effort, bounded by `lookbackPeriod`.** + It detects changes by listing the bucket, which cannot be lossless: an + object that becomes visible in a listing late can be missed. Lossless S3 + change capture needs bucket event notifications, which this connector does + not use. Every poll selects the objects modified since `lookbackPeriod` + (default 5m) before the newest object seen so far, and skips versions it + already delivered, so objects that become visible late, within that + window, are picked up: S3 stamps an object with the time its upload began, + a multipart upload with the time it was initiated, and lists in key order. + The object store's own timestamps decide, so the clock of the machine + running Conduit does not matter, with one exception: the newest timestamp + is capped at this machine's clock, so one bogus timestamp from the future + (a store clock that stepped forward, or a misbehaving endpoint) cannot hide + the objects written after it. The cap lowers the window's start, which + causes redelivery. If this machine's clock is behind the store's by more + than `lookbackPeriod` the source logs a warning with the code + `s3.source.host_clock_behind_store`: the window is stretched by the skew. + + Delivery is at-least-once. After a restart the objects modified within + `lookbackPeriod` of the newest object are delivered again. The snapshot + hands over to CDC from the newest object that existed when the snapshot + started (found by one extra listing of the prefix, one request per 1000 + keys, at the start of the snapshot), minus `lookbackPeriod`, so CDC + redelivers the objects written while the snapshot ran, however long it + took, plus those within `lookbackPeriod` before it started. On a bucket + that was bulk loaded recently this can be every object of the load. + + The versions remembered for deduplication are limited to about 256 MiB + (peak memory of a poll is about 1.5 times that, candidates included). If + the objects inside the window need more, the window is **shortened** + instead of stopping the source: the oldest remembered entries are + forgotten by whole seconds and the window starts just above them. The + source logs a warning with the code `s3.source.dedupe_window_shrunk` + naming the effective lookback and the number of forgotten entries. While + the window is shortened, an object that becomes visible late by more than + the effective lookback can be missed, so the tolerance for late-visible + objects is lower than `lookbackPeriod`; lower `lookbackPeriod` to avoid + the shrinking. The source stops, with the error + `s3.source.dedupe_window_overflow` (after a warning), only if the versions + sharing a single second of `LastModified` do not fit, or when entries + stamped at or after this machine's present second (a store clock that is + ahead, or bogus timestamps from the future) take the budget: a shortened + window never starts later than the present second, because that would hide + the objects written from now on, so the source stops instead (the message + names the clock skew and says to fix the clock). A store clock that is ahead + by less than `lookbackPeriod` is normal and does not stop the source as long + as the entries stamped in the present and the future fit the budget. A + future-stamped object can therefore raise the start of the window at most + up to the present second, never past it. The shortened window is not + remembered across restarts: after a restart the window is rebuilt from the + position and shortened again, so a restart delivers the window's objects + again (duplicates, never a skip). When more objects share a second than + one poll can hand over, they are delivered over several polls, oldest + first, and the newest tracked entries may be dropped from tracking to make + room; those are delivered again later (duplicates). A catch-up after + downtime is delivered in bounded batches, oldest first. A resume position + with a timestamp in the future, which only an older release could have + written, is clamped to this machine's clock with a warning. + + Known limits (listing cannot see these, or sees them too late): + * an upload that takes longer than `lookbackPeriod` to complete is missed + when newer objects were seen in the meantime (multipart uploads carry + their initiation time); + * replica buckets whose replication lag exceeds `lookbackPeriod`; + * an identical re-put of an object within the same second is not seen as a + change; + * in an unversioned bucket an overwrite is reported as a create (there is + no older version to compare with) and a delete is not reported; + * in a versioned bucket intermediate versions that appear and are + superseded within one poll are not emitted, only the latest; removing a + delete marker (undelete) and permanently deleting the latest version are + not emitted; + * a GetObject that fails for another reason than the object having been + deleted, for example an object archived to Glacier or Deep Archive, + stops the source with that error. + + Objects deleted between being listed and being read are skipped, in the + snapshot and in CDC. Values of `lookbackPeriod` above 1h log a warning and + above 24h are rejected. + +### Record Keys + +The S3 object key uniquely identifies the objects in an Amazon S3 bucket, which +is why a record key is the key read from the S3 bucket. + +## Destination + +The S3 Destination Connector connects to an S3 bucket with the provided +configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and +`aws.region`. If the permissions fail, the connector will not be ready for +writing to S3. + +### Writes and delivery + +Every batch the SDK hands to the destination (`sdk.batch.size` records, or +fewer after `sdk.batch.delay`; one record per batch by default) is written +with one `PutObject` per object, and the records are acknowledged only after +`PutObject` succeeded. Delivery is at-least-once: a batch that is retried +after a failure or a restart is written again as a new object, so a reader +can see a record twice, never zero times. Object names are +`--.`: the write time in nanoseconds (so +objects still sort by write time), a random 16-hex-character id per +destination instance, and that instance's write counter. They are unique by +construction, also across pipelines or Conduit instances writing to the same +bucket and prefix in the same clock tick, and not deterministic (a retried +batch gets a new name). With `conditionalWrites` (default true) every +`PutObject` carries `If-None-Match: *`, so a store never replaces an existing +object. The records of a refused write are not acknowledged: + +- `s3.destination.key_exists` (412): the key exists. Either another writer + used it, or the AWS SDK retried this writer's own `PutObject` after the + response to a successful first attempt was lost; in that case the records + are already stored, and redelivering them writes a duplicate under a new + key (at-least-once, nothing is lost). +- `s3.destination.conditional_conflict` (409): a conflicting operation on + the key was in progress. Transient; a retry writes under a new key. +- `s3.destination.conditional_writes_unsupported`: the store rejects the + header (501, or a 400/405 that names it). Set `conditionalWrites: false`; + keys stay unique by construction. + +A store that silently ignores `If-None-Match` writes unconditionally, with +no warning; the protection then rests on the unique names alone. Tested on +MinIO; AWS S3 documents support for conditional writes. + +### Formats + +- `json`: one JSON object per line with `Operation`, `Position`, `Payload`, + `Key` and `Metadata`; payload and key are JSON strings. A record whose key, + payload, position or metadata is not valid UTF-8 is refused with the error + `s3.destination.invalid_utf8` instead of being stored with replaced bytes. +- `parquet`: the same fields as string columns. +- `bytearray`: the payload bytes of every record, unchanged, each followed + by `format.bytearray.separator` (default a newline). Structured payloads + are written as their JSON encoding. A record without a payload (a delete, + a Kafka tombstone) or with an empty one is refused with the error + `s3.destination.null_value`; drop such records with a `filter` processor + if they are expected. A Kafka tombstone arrives as an empty value, so an + empty Kafka value cannot be told apart from it and is refused too. The + separator is not escaped inside payloads: a payload that contains it + cannot be split back unambiguously (as with Kafka Connect's + `ByteArrayFormat`). + +`compression` is `none` or `gzip` for `json` and `bytearray` (gzip appends +`.gz` to the extension and sets the content type to `application/gzip`), and +`none`, `gzip` (default) or `snappy` for `parquet`, where it is the column +codec. + +When a record is refused, the records before it in the batch are written as +one object and acked, the refused record and the ones after it are nacked +(to the DLQ, if one is configured). + +### Coming from the Kafka Connect S3 sink + +- `ByteArrayFormat` is `format: bytearray`; `format.bytearray.separator` and + `format.bytearray.extension` have the same meaning (Kafka Connect's + extension default is `.bin` as well). +- `JsonFormat` for topics with JSON values is `format: bytearray` with + `format.bytearray.extension: .json`: the stored bytes are the record + values. Kafka Connect re-serializes values through its converter, so + whitespace and key order can differ, and `schemas.enable=true` envelopes + are not produced. +- `s3.compression.type: gzip` is `compression: gzip`; `parquet.codec` is + `compression` with `format: parquet` (Kafka Connect defaults to snappy, + this connector to gzip). +- `behavior.on.null.values` other than `fail` has no counterpart. + +### Server-side encryption + +By default the destination sends no server-side encryption header, so the +bucket's default encryption applies. On AWS every bucket encrypts new +objects with SSE-S3 unless it is configured otherwise, and a bucket whose +default is SSE-KMS gets SSE-KMS objects. An S3-compatible store without a +default encryption configuration stores objects unencrypted. When +`aws.serverSideEncryption` is empty, `Open` reads the bucket's default +encryption (`GetBucketEncryption`, permission +`s3:GetEncryptionConfiguration`) and logs a warning with the code +`s3.destination.bucket_unencrypted` if there is none. It never refuses to +start; without the permission the check is skipped (debug log). + +Set `aws.serverSideEncryption` to `AES256`, `aws:kms` or `aws:kms:dsse` to +request a mode explicitly, and `aws.sseKmsKeyId` to pick the KMS key for the +two KMS modes. An explicit mode overrides the bucket default. + +Upgrading from v0.9.5 or earlier: those versions sent `AES256` on every +write, whatever the bucket default was, so an SSE-KMS bucket default was +silently replaced by SSE-S3 (#1047). To keep the old behavior, set +`aws.serverSideEncryption: AES256`. + +### Partitioning (Kafka Connect S3 sink layout) + +`partitioner` selects the directory layout under `prefix`: + +| `partitioner` | Object key | +| --- | --- | +| `none` (default) | `/.` | +| `default` | `//partition=/.` | +| `field` | `//=/.../.` | +| `time`, `hourly`, `daily` | `//
.sortingColumn`. If a table has neither, the connector + fails to start with error `mysql.snapshot.table_without_key`, naming the table + and the config path `tableConfig.
.sortingColumn`. To fix it, do one of: + + - add a primary key to the table; + - set `tableConfig.
.sortingColumn` to a column that is `NOT NULL` and the + only column of a primary key or unique index. Anything else is rejected at + start with `mysql.snapshot.sorting_column_invalid`, because duplicate or NULL + values can make the snapshot skip rows; + - exclude the table. With a pattern list, a later `-` rule removes tables from an + earlier match, for example `tables: "*,-^audit_log$"` snapshots everything + except `audit_log`. With an explicit list, remove the table from it; + - set `snapshot.enabled: false` to stream changes only. Rows that already exist + are then not copied. CDC works for tables without a primary key; their record + keys are derived from the binlog position. + + The key and sorting-column checks run only when a snapshot is about to run. A + pipeline restarting in CDC mode is not affected. + + #### Migrating from `snapshot.unsafe` + + The `snapshot.unsafe` setting has been removed. It snapshotted keyless tables + with `LIMIT`/`OFFSET`, recorded no position (so every restart began again from + zero), and could skip or repeat rows when the table changed during the + snapshot. In practice it also applied without the setting: a keyless table was + snapshotted this way even when `snapshot.unsafe` was not set. + + - `snapshot.unsafe: true` now fails configuration validation with error + `mysql.config.unsafe_snapshot_removed`, **including for pipelines that are + already in CDC mode**: validation does not look at the position. Remove the + setting before upgrading. + - `snapshot.unsafe: false` is accepted and ignored. + - After upgrading, a pipeline that still has a keyless table and will run a + snapshot fails to start with `mysql.snapshot.table_without_key` until you use + one of the options above. + + ### Schema + + The source connector uses [avro](https://avro.apache.org/docs/1.11.1/specification/) to decode mysql rows. Here's the MySQL datatype to avro datatype equivalence that the connector uses: + + | MySQL Type | Avro Type | + | ---------- | --------- | + | TINYINT | int | + | SMALLINT | int | + | MEDIUMINT | int | + | INT | int | + | BIGINT | long | + | DECIMAL | double | + | NUMERIC | double | + | FLOAT | double | + | DOUBLE | double | + | BIT | bytes | + | CHAR | string | + | VARCHAR | string | + | TINYTEXT | string | + | TEXT | string | + | MEDIUMTEXT | string | + | LONGTEXT | string | + | BINARY | bytes | + | VARBINARY | bytes | + | TINYBLOB | bytes | + | BLOB | bytes | + | MEDIUMBLOB | bytes | + | LONGBLOB | bytes | + | DATE | string | + | TIME | string | + | DATETIME | string | + | TIMESTAMP | string | + | YEAR | long | + | ENUM | string | + | SET | string | + | JSON | string | + + ### Record structure + + Records produced by the MySQL source connector contain the following [opencdc record structure](https://conduit.io/docs/using/opencdc-record): + + * **Operation**: Indicates the type of change (`create`, `update`, `delete`, `snapshot`). + * **Payload**: + * `Before`: (Only present on CDC `update` operations) `opencdc.StructuredData` representing the row state before the change. + * `After`: `opencdc.StructuredData` representing the row state after the change (or the current state for `snapshot` and `create`). + * **Key**: Identifies the specific row. See Snapshot/CDC sections below for details. + * **Position**: Represents the point in the data stream. It's a JSON object containing *either* a `snapshot_position` field *or* a `cdc_position` field, structured as described below. + * **Metadata**: Contains standard OpenCDC fields plus: + * `mysql.server.id`: (CDC only) The originating MySQL server ID from the replication event header. + * [Key](https://conduit.io/docs/using/opencdc-record/#opencdckeyschema) and [payload](https://conduit.io/docs/using/opencdc-record/#opencdcpayloadschema) schema registry data. + + #### Snapshot Records + + * **Operation**: `snapshot`. + * **Payload `After`**: `opencdc.StructuredData` with all column values. + * **Key**: `opencdc.StructuredData` using the primary key columns (or the configured `tableConfig.
.sortingColumn`) and their values. + * **Position (`snapshot_position` field)**: A JSON object containing: + * `snapshots`: (object) Maps table names to their specific snapshot position object. + * *(Single Primary Key Table)*: Contains `last_read` (any) and `snapshot_end` (any). + * *(Multiple Primary Key Table)*: Contains an array of objects, each with `key_name` (string), `last_read` (any), and `snapshot_end` (any). + + #### CDC Records + + * **Operation**: Either `create`, `update` or `delete`. + * **Key**: + * With Primary Key(s): `opencdc.StructuredData` using primary key columns and values (from the `After` state). + * Without Primary Key(s): `opencdc.RawData` string `"_"`. + * **Payload**: + * `Before`: (For `update` only) `opencdc.StructuredData` with pre-update column values. + * `After`: `opencdc.StructuredData` with post-change column values (for `create`, `update`) or pre-deletion values (for `delete`). + * **Position (`cdc_position` field)**: Encodes the binlog location in a JSON object with the following fields. + * `name`: (string) Binlog file name. + * `pos`: (uint32) Position within the binlog file. + * `prev`: (object, optional) Position of the preceding event, containing `name` (string) and `pos` (uint32). Omitted if not applicable. + * `idx`: (int, optional) Row index within a potentially multi-row MySQL replication event. Omitted if zero or not applicable. + + ## Destination + + The MySQL destination takes a slice of `opencdc.Record` and writes them in batches to MySQL. It will use the `opencdc.collection` field in the metadata to determine the table to write to. + + ### Upsert Behavior + + If the target table contains a column with a unique constraint (this includes PRIMARY KEY and UNIQUE indexes), records will be upserted. Otherwise, they will be appended. Support for updating tables without unique constraints is tracked [here](https://github.com/conduitio-labs/conduit-connector-mysql/issues/66). + + If the target table already contains a record with the same key, the Destination will upsert with its current received values. Because Keys must be unique, this can overwrite and thus potentially lose data, so keys should be assigned correctly from the Source. + + If a unique key is not present in the target table, the record will be simply appended. + + With several unique indexes (including functional and prefix indexes), `ON DUPLICATE KEY UPDATE` overwrites the row matched by any of them, including that row's primary key. Functional unique indexes are read but not evaluated: the destination cannot compute their expressions, so it neither type-checks their inputs nor uses them to decide whether a record can be matched. + + ### Columns and schema changes + + Each record is written with exactly the fields in its `.Payload.After`, whatever the other records in the batch contain: + + - A field the table has no column for fails the write with `mysql.destination.unknown_column`. The destination does not create or alter tables, and it never drops a field. Add the column first, or remove the field with a processor. + - A field the record does not have is not written. An inserted row gets the column's default; an existing row keeps the column's current value. A field set to `null` writes `NULL`. + + Table and column names are quoted, so reserved words work. A collection of the form `db.table` writes to `table` in database `db`. + + **Security.** Table and column names come from the records. Quoting keeps them from changing the SQL, but a record's collection can name any database the DSN user can write to, through `db.table`. Grant the destination's user access only to the databases it should write. Quoting is only sound in character sets where no character can end in a backtick byte, so a connection using `sjis`, `cp932`, `gbk`, `gb18030` or `big5` is refused with `mysql.destination.unsafe_charset` (checked in the DSN, when the destination opens, and on the connection of every write). Use `utf8mb4`, the driver's default. + + ### Deletes + + Delete records are matched on the target table's primary key, so the table must have one: a delete for a table without a primary key fails with `mysql.destination.delete_without_primary_key` instead of removing rows. The record key must hold a non-null value for every primary-key column (`mysql.destination.key_column_missing` otherwise), either as structured data or as a single JSON object. Each value must be one scalar: a list or an object fails with `mysql.destination.key_value_not_scalar` (it would match several rows or none). Key values are converted to the type of their key column, which is read for every write, so an `ALTER TABLE` on a running pipeline is picked up without a restart. The same rules apply to the values an upsert writes into the columns of any unique index (the primary key and `UNIQUE` keys), because those values decide which row `ON DUPLICATE KEY UPDATE` overwrites: `{"id":1.5}` for a `BIGINT` key is refused instead of being rounded to 2 and overwriting row 2. A record must be fully non-null in at least one unique index, otherwise it fails with `mysql.destination.key_column_missing`: a null never conflicts, so every redelivery would insert another row. A null in one unique column is fine while another unique index identifies the row (a nullable `UNIQUE` email next to a primary key). Exceptions, which append a row on every redelivery: a null or absent `AUTO_INCREMENT` primary key, which generates a new id, and a table without a unique index. Field names are resolved to columns by the MySQL server itself (through `information_schema`), so a field names exactly the column a statement would use: `ID` names `id`, and so do characters MySQL folds (for example the Kelvin sign names `k`), while `Å¿` names no column. A record with two fields that name one column (`id` and `ID`) fails with `mysql.destination.duplicate_column`. A delete for a table that does not exist fails with `mysql.destination.table_not_found`; table names are case-sensitive unless the server sets `lower_case_table_names`. + + - **Integer columns** (`TINYINT` to `BIGINT`, signed or `UNSIGNED`) take a whole number within the column's range (`5`, `5.0`, `5e0` or the string `"5"`). A fraction, a non-numeric string or a value outside the range fails with `mysql.destination.key_value_type_mismatch` instead of matching nothing or the wrong row. A boolean matches as MySQL compares it: `true` is 1, which is what a `TINYINT(1)` key needs. + - **String columns** (`CHAR`, `VARCHAR`, the `TEXT` types, `BINARY`, `VARBINARY`) take a string. An integer is sent as its decimal text and a raw JSON number as written (so `1e3` matches the row `'1e3'`), because MySQL compares a number with a string column as a double and `1000` would match `'1000'`, `'01000'`, `'1000.0'` and `' 1000'`. A float is accepted only if it is whole and below 1e15, where MySQL's text for it is the plain integer; other floats and booleans fail. + - **`ENUM` and `SET` columns** take the member name as a string. A number fails, because MySQL would compare it with the member's index. + - **`BIT(M)` columns** take a whole number from 0 to 2^M-1. + - **Other columns** (`DECIMAL`, floating point, temporal) get the value as it is; a raw JSON number keeps its literal, which MySQL compares exactly with a `DECIMAL` column. + + Integer values in a raw JSON key are matched exactly. Structured keys that reach a standalone connector over the plugin protocol have already passed through a double, so integers above 2^53 sent that way are rounded before the destination sees them ([ConduitIO/conduit#3100](https://github.com/ConduitIO/conduit/issues/3100)). A record whose operation is not create, update, delete or snapshot fails with `mysql.destination.unsupported_operation`. + + ### Atomicity and acknowledgements + + Each write is a single transaction. If any record in the write fails, nothing from that write is stored and none of its records are acknowledged; the records are redelivered after the pipeline recovers. Upserts on a primary or unique key make the redelivery idempotent. Redelivering a record whose `AUTO_INCREMENT` key is null or absent, or a record for a table with no unique key, appends another row. + + The destination sets `wait_timeout=60` on its own connections (unless the DSN sets `wait_timeout`), and the pool drops connections idle for half of it. If the connector dies in the middle of a write and the server does not see the connection close (a NAT, a proxy, a half-open TCP connection), the uncommitted transaction and its row locks end within that time instead of the server default of 8 hours, so the redelivered records do not wait on them. + + ## Delivery semantics and limits + + The source connector is in beta. This section states what it guarantees and what it does not. + + ### Delivery guarantees + + - **At-least-once.** A record is acknowledged upstream only after the destination has + acknowledged it, and the position carried on a record is what Conduit persists. After a + crash or restart you can receive records again, never fewer. Make destinations idempotent + (upsert by key). + - **Snapshot to CDC handoff.** The connector captures the binlog position at the start of + the snapshot and begins CDC from it, so changes made while the snapshot runs are delivered + after it. Rows can therefore appear twice: once as a `snapshot` record and again as a + CDC change. + - **Restart.** A restart during the snapshot resumes the snapshot from the last + acknowledged key per table. After a CDC restart, CDC resumes after the last acknowledged + row, also in the middle of a multi-row binlog event, in a later binlog event of one large + statement, and inside a compressed transaction (`binlog_transaction_compression=ON`). After + upgrading from an older version, restarts re-read the binlog file of the saved position from + its start to find the saved event, until the first record is acknowledged under the new + version. Rows are delivered again (duplicates, never a gap) in three cases, each logged as a + warning: the saved position was written right after a restart in the middle of an event (that + event is delivered again), an older version's position points inside a compressed transaction + (its binlog file is delivered again from its start), or the captured tables changed while the + saved position was inside a compressed transaction (that transaction is delivered again). + - **Ordering.** CDC records are delivered in binlog order. During the snapshot, rows are + ordered by key within a table; tables are read concurrently, so there is no ordering across + tables. + + ### Limits + + - **No GTID support.** Positions are a binlog file name and offset on the server the + connector read from. They are not portable between servers: a replica, or a replica + promoted after a failover, has different binlog files and offsets. **Failing over the + source database breaks resume.** The connector may fail to start, or may not resume at the + matching point. After a failover, treat the pipeline as needing a new snapshot: clear its + position and expect to re-read the data. GTID-based positions are not supported in this + release. + - **Tables added after the pipeline entered CDC are not snapshotted.** A snapshot runs only + on a fresh start. The position records which tables were snapshotted, and a restart checks + the configuration against it: + - If you add a table to an explicit `tables` list (plain table names), or switch + `snapshot.enabled` from `false` to `true`, the connector refuses to start with + `mysql.snapshot.tables_added_after_cdc`, naming the tables. Start a new pipeline for them, + or reset this pipeline's position to snapshot everything again (expect duplicates). + - If `tables` uses a pattern (`*`, a regex, an exclude, or an anchored name such as + `^orders$`), the connector logs one warning with the same code and then delivers the new + tables **from CDC only**: their existing rows are not snapshotted and are never delivered. + - A plain name is a regex that is not anchored, so `orders` also matches `orders_archive`. + It is still treated as an explicit list, which is the safe side. Anchor it as `^orders$` + to make it a pattern. + - A table removed from `tables` and added back later counts as added: changes made while it + was excluded were never captured. For an explicit list this means a table that is absent + at one restart and present at the next errors. + - A position written by an older version cannot be checked. The connector adopts the + configured tables as **unverified**, never as snapshotted, and logs + `mysql.snapshot.tables_unverified` on every start until the position is reset. Tables added + afterwards are detected. + - With more than 200 tables the position stores a hash of the table set instead of the + names. A change to the set then cannot be named or given a direction (a table that was only + removed also triggers it). The error says that "the configured table set changed", and + the way out is to restore the previous `tables` list or reset the position; removing tables + from the list does not clear it. + - **A key column that was retyped restarts its table's snapshot.** The position records the + type of each key column. If the saved key does not fit the column's type now (an `INT` key + column that became `VARCHAR`, or `VARCHAR` that became `DECIMAL`, whose values order + differently), the table's snapshot restarts from its first row with a warning. Rows already + delivered are delivered again; nothing is skipped. + - **Upgrading in the middle of a snapshot re-reads text, decimal and binary key tables.** + A snapshot position written by an older version does not record the key type, so a table + whose key is text, binary, `DECIMAL`, `TIME`, `ENUM`, `SET` or `BIT` cannot be checked for a + retype and restarts from its first row after the upgrade; its partial progress is re-read. + Integer, date and time keys resume where they stopped. Pipelines already in CDC are not + affected. + - **Binlog retention is your responsibility.** If the persisted position has been purged + from the server, the connector cannot resume. Keep `binlog_expire_logs_seconds` longer than + the longest outage you want to survive. + - **No heartbeat.** The connector does not send or consume heartbeat events. The position + is carried on records, so while the configured tables are quiet the persisted position does + not advance, even if the server keeps writing and rotating binlogs for other tables. A long + quiet period followed by binlog purging can leave the stored position pointing at a purged + file. + - **DDL during a running pipeline is not handled explicitly and is not tested.** Running + `ALTER TABLE` (or other schema changes) on a captured table while the snapshot or CDC is + running may produce records that do not match the schema registered earlier, or an error. + Stop the pipeline, change the schema, and verify the result before resuming. + - **Tables must be orderable for the snapshot.** See "Tables without a primary key". + - **Type mapping has known gaps.** `DECIMAL` and `NUMERIC` map to Avro `double` and can lose + precision beyond what a 64-bit float holds. A `BIGINT UNSIGNED` value above 2^63-1 cannot + be encoded as Avro `long`; the connector logs a warning and passes the value through + unconverted. Date and time types are delivered as strings. A full type-mapping design is + pending; do not rely on exact fidelity for these types yet. + + ## Requirements and compatibility + + The connector is tested against MySQL v8.0. Compatibility with older versions isn't guaranteed. + + ### MySQL Server Requirements: + + - Binary Log (binlog) must be enabled. + - Binlog format must be set to ROW. + - Binlog row image must be set to FULL. + - Tables must have **sortable** primary keys. + + ### Partial row images + + `binlog_row_image` is also a session variable. A writing session that sets it to `MINIMAL` or `NOBLOB` logs + only part of the row, even when the server value is `FULL`: + + - `MINIMAL`, UPDATE: the before image holds only the primary key, and the after image holds only the changed + columns (the primary key is in it only if it changed). + - `MINIMAL`, DELETE: the before image holds only the primary key. + - `MINIMAL`, INSERT: the image holds only the columns the statement names. Every column it omits is skipped, + including columns with a `DEFAULT` and nullable columns, so an ORM that leaves defaulted columns out halts on + its first insert. An INSERT that names every column does not halt. + - `NOBLOB`: `BLOB` and `TEXT` columns that an UPDATE or DELETE did not change are skipped, and so is a `BLOB` + that an INSERT omits. An INSERT that omits a non-`BLOB` column with a default is logged in full. + + A skipped column is unknown. It is not NULL and it is not the column default: the binlog carries no value for + it, and the schema default may have changed since the row was written, so the connector does not fill it in. + + The CDC source halts on the first such event instead of delivering zero values over real data. The error has + the code `mysql.cdc.row_image_skipped_columns` and names the table, the action, the binlog file and position, and + the columns that were left out. Snapshot rows are not affected. If the event sits inside a compressed + transaction (`binlog_transaction_compression=ON`) the exact position is not available and the error says so. + + The source also stops with the code `mysql.cdc.partial_json_unsupported` when a writer logs partial JSON updates + (`binlog_row_value_options=PARTIAL_JSON`), which the connector cannot read. Set the option to an empty value for + the writers. + + Two more CDC halts are resolved by resetting the position, so the recovery below (deletes) applies to them too: + + - `mysql.cdc.row_columns_mismatch`: a binlog event has fewer values than its table now has columns. The + connector decodes events with the table's current columns, so an event written before an `ADD COLUMN` and + read after it (the connector was stopped or lagging across the DDL) cannot be mapped. Restarting does not + help: the event fails the same way every time. Reset the position to snapshot again. + - `mysql.cdc.resume_event_not_found`: on a restart, the binlog event the saved position names was not where + the position says. This should not happen; report it with the logged position, and reset the position to + continue. + + #### Recovery + + **Resuming means a re-snapshot, and a snapshot does not carry deletes.** Rows deleted at the source while the + pipeline was halted would stay in the destination. Do one of the following, for ALL captured tables, not only + the table named in the error: + + - truncate the destination tables, then clear the connector's position to re-snapshot; or + - clear the connector's position to re-snapshot, and after the snapshot has completed reconcile the destination + against the source, deletes included. + + Do not reconcile first and re-snapshot afterwards: rows deleted at the source between the reconcile and the + start of the snapshot would stay in the destination. + + 1. Stop the cause. Set `binlog_row_image = 'FULL'` for the writers, and for the server use + `SET PERSIST binlog_row_image = 'FULL'` (`SET GLOBAL` does not survive a restart). Then reconnect every + writer: existing sessions, pooled connections included, keep the value they started with. + 2. The event is already in the binlog, so every restart halts on it. Find its transaction with + `SHOW BINLOG EVENTS IN ''` (the error gives the start position of the event); with GTIDs enabled, + the `Gtid` event just before it names the GTID. + 3. Truncate the destination for all captured tables, or plan to reconcile it after the snapshot (see above). + 4. Clear the connector's position to re-snapshot. + 5. If you chose to reconcile, do it once the snapshot has completed, including deletes, for all captured tables. + + A lighter way to resume after the event is tracked in + [#212](https://github.com/ConduitIO/conduit-connector-mysql/issues/212). + + #### Upgrading + + If the binlog after the stored position already holds an event from a `MINIMAL` or `NOBLOB` session, including + an event that an older version had partly emitted, the upgraded connector halts on it after the restart. Older + versions delivered such events with zero values. + + ### MySQL User Privileges: + + For Snapshot and CDC modes, the following privileges are required: + + - SELECT + - LOCK TABLES + - RELOAD + - REPLICATION CLIENT + - REPLICATION SLAVE + version: v0.4.0 + author: Meroxa, Inc. + source: + parameters: + - name: dsn + description: The connection string for the MySQL database. + type: string + default: "" + validations: + - type: required + value: "" + - name: tables + description: |- + Represents the tables to read from. + - By default, no tables are included, but can be modified by adding a comma-separated string of regex patterns. + - They are applied in the order that they are provided, so the final regex supersedes all previous ones. + - To include all tables, use "*". You can then filter that list by adding a comma-separated string of regex patterns. + - To set an "include" regex, add "+" or nothing in front of the regex. + - To set an "exclude" regex, add "-" in front of the regex. + - e.g. "-.*meta$, wp_postmeta" will exclude all tables ending with "meta" but include the table "wp_postmeta". + type: string + default: "" + validations: + - type: required + value: "" + - name: cdc.disableLogs + description: Disables verbose cdc driver logs. + type: bool + default: "" + validations: [] + - name: snapshot.enabled + description: Controls whether the snapshot is done. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Limits how many rows should be retrieved on each database fetch on snapshot mode. + type: int + default: "10000" + validations: [] + - name: snapshot.unsafe + description: 'Removed. The unsafe snapshot no longer exists: true fails validation with a migration message, false is accepted and ignored. Tables without a primary key need tableConfig.
.sortingColumn.' + type: bool + default: "" + validations: [] + - name: tableConfig.*.sortingColumn + description: |- + Allows to force using a custom column to sort the snapshot. The column must be + NOT NULL and the only column of a primary key or unique index. + type: string + default: "" + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: 1s + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "10000" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: dsn + description: The connection string for the MySQL database. + type: string + default: "" + validations: + - type: required + value: "" + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/.metadata.yaml new file mode 100644 index 0000000..891987b --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:10.23154016Z +commitSHA: b8bff4420e9de7eeeebd069de18db7bef936c793 diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/connector.yaml new file mode 100644 index 0000000..8337213 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-mysql@v0.5.0/connector.yaml @@ -0,0 +1,568 @@ +version: "1.0" +specification: + name: mysql + summary: A Conduit Connector for MySQL + description: |- + ## Source + + A source connector pulls data from an external resource and pushes it to + downstream resources via Conduit. + + ### Snapshot mode + + Snapshot mode is the first stage of the source sync process. It reads all rows + from the configured tables as record snapshots. + + In snapshot mode, the record payload consists of + [opencdc.StructuredData](https://conduit.io/docs/using/opencdc-record#structured-data), + with each key being a column and each value being that column's value. + + ### Change Data Capture mode + + When the connector switches to CDC mode, it starts streaming changes from the + obtained position at the start of the snapshot. It uses the row-based binlog format + to capture detailed changes at the individual row level. + + ### Tables without a primary key + + The snapshot reads each table in key order and records the last key read in the + position, so it can resume exactly where it stopped and read a consistent view. + That needs a primary key, or a column you name with + `tableConfig.
.sortingColumn`. If a table has neither, the connector + fails to start with error `mysql.snapshot.table_without_key`, naming the table + and the config path `tableConfig.
.sortingColumn`. To fix it, do one of: + + - add a primary key to the table; + - set `tableConfig.
.sortingColumn` to a column that is `NOT NULL` and the + only column of a primary key or unique index. Anything else is rejected at + start with `mysql.snapshot.sorting_column_invalid`, because duplicate or NULL + values can make the snapshot skip rows; + - exclude the table. With a pattern list, a later `-` rule removes tables from an + earlier match, for example `tables: "*,-^audit_log$"` snapshots everything + except `audit_log`. With an explicit list, remove the table from it; + - set `snapshot.enabled: false` to stream changes only. Rows that already exist + are then not copied. CDC works for tables without a primary key; their record + keys are derived from the binlog position. + + The key and sorting-column checks run only when a snapshot is about to run. A + pipeline restarting in CDC mode is not affected. + + #### Migrating from `snapshot.unsafe` + + The `snapshot.unsafe` setting has been removed. It snapshotted keyless tables + with `LIMIT`/`OFFSET`, recorded no position (so every restart began again from + zero), and could skip or repeat rows when the table changed during the + snapshot. In practice it also applied without the setting: a keyless table was + snapshotted this way even when `snapshot.unsafe` was not set. + + - `snapshot.unsafe: true` now fails configuration validation with error + `mysql.config.unsafe_snapshot_removed`, **including for pipelines that are + already in CDC mode**: validation does not look at the position. Remove the + setting before upgrading. + - `snapshot.unsafe: false` is accepted and ignored. + - After upgrading, a pipeline that still has a keyless table and will run a + snapshot fails to start with `mysql.snapshot.table_without_key` until you use + one of the options above. + + ### Schema + + The source connector uses [avro](https://avro.apache.org/docs/1.11.1/specification/) to decode mysql rows. Here's the MySQL datatype to avro datatype equivalence that the connector uses: + + | MySQL Type | Avro Type | + | ---------- | --------- | + | TINYINT | int | + | SMALLINT | int | + | MEDIUMINT | int | + | INT | int | + | BIGINT | long | + | DECIMAL | double | + | NUMERIC | double | + | FLOAT | double | + | DOUBLE | double | + | BIT | bytes | + | CHAR | string | + | VARCHAR | string | + | TINYTEXT | string | + | TEXT | string | + | MEDIUMTEXT | string | + | LONGTEXT | string | + | BINARY | bytes | + | VARBINARY | bytes | + | TINYBLOB | bytes | + | BLOB | bytes | + | MEDIUMBLOB | bytes | + | LONGBLOB | bytes | + | DATE | string | + | TIME | string | + | DATETIME | string | + | TIMESTAMP | string | + | YEAR | long | + | ENUM | string | + | SET | string | + | JSON | string | + + ### Record structure + + Records produced by the MySQL source connector contain the following [opencdc record structure](https://conduit.io/docs/using/opencdc-record): + + * **Operation**: Indicates the type of change (`create`, `update`, `delete`, `snapshot`). + * **Payload**: + * `Before`: (Only present on CDC `update` operations) `opencdc.StructuredData` representing the row state before the change. + * `After`: `opencdc.StructuredData` representing the row state after the change (or the current state for `snapshot` and `create`). + * **Key**: Identifies the specific row. See Snapshot/CDC sections below for details. + * **Position**: Represents the point in the data stream. It's a JSON object containing *either* a `snapshot_position` field *or* a `cdc_position` field, structured as described below. + * **Metadata**: Contains standard OpenCDC fields plus: + * `mysql.server.id`: (CDC only) The originating MySQL server ID from the replication event header. + * [Key](https://conduit.io/docs/using/opencdc-record/#opencdckeyschema) and [payload](https://conduit.io/docs/using/opencdc-record/#opencdcpayloadschema) schema registry data. + + #### Snapshot Records + + * **Operation**: `snapshot`. + * **Payload `After`**: `opencdc.StructuredData` with all column values. + * **Key**: `opencdc.StructuredData` using the primary key columns (or the configured `tableConfig.
.sortingColumn`) and their values. + * **Position (`snapshot_position` field)**: A JSON object containing: + * `snapshots`: (object) Maps table names to their specific snapshot position object. + * *(Single Primary Key Table)*: Contains `last_read` (any) and `snapshot_end` (any). + * *(Multiple Primary Key Table)*: Contains an array of objects, each with `key_name` (string), `last_read` (any), and `snapshot_end` (any). + + #### CDC Records + + * **Operation**: Either `create`, `update` or `delete`. + * **Key**: + * With Primary Key(s): `opencdc.StructuredData` using primary key columns and values (from the `After` state). + * Without Primary Key(s): `opencdc.RawData` string `"_"`. + * **Payload**: + * `Before`: (For `update` only) `opencdc.StructuredData` with pre-update column values. + * `After`: `opencdc.StructuredData` with post-change column values (for `create`, `update`) or pre-deletion values (for `delete`). + * **Position (`cdc_position` field)**: Encodes the binlog location in a JSON object with the following fields. + * `name`: (string) Binlog file name. + * `pos`: (uint32) Position within the binlog file. + * `prev`: (object, optional) Position of the preceding event, containing `name` (string) and `pos` (uint32). Omitted if not applicable. + * `idx`: (int, optional) Row index within a potentially multi-row MySQL replication event. Omitted if zero or not applicable. + + ## Destination + + The MySQL destination takes a slice of `opencdc.Record` and writes them in batches to MySQL. It will use the `opencdc.collection` field in the metadata to determine the table to write to. + + ### Upsert Behavior + + If the target table contains a column with a unique constraint (this includes PRIMARY KEY and UNIQUE indexes), records will be upserted. Otherwise, they will be appended. Support for updating tables without unique constraints is tracked [here](https://github.com/conduitio-labs/conduit-connector-mysql/issues/66). + + If the target table already contains a record with the same key, the Destination will upsert with its current received values. Because Keys must be unique, this can overwrite and thus potentially lose data, so keys should be assigned correctly from the Source. + + If a unique key is not present in the target table, the record will be simply appended. + + With several unique indexes (including functional and prefix indexes), `ON DUPLICATE KEY UPDATE` overwrites the row matched by any of them, including that row's primary key. Functional unique indexes are read but not evaluated: the destination cannot compute their expressions, so it neither type-checks their inputs nor uses them to decide whether a record can be matched. + + ### Columns and schema changes + + Each record is written with exactly the fields in its `.Payload.After`, whatever the other records in the batch contain: + + - A field the table has no column for fails the write with `mysql.destination.unknown_column`. The destination does not create or alter tables, and it never drops a field. Add the column first, or remove the field with a processor. + - A field the record does not have is not written. An inserted row gets the column's default; an existing row keeps the column's current value. A field set to `null` writes `NULL`. + + Table and column names are quoted, so reserved words work. A collection of the form `db.table` writes to `table` in database `db`. + + **Security.** Table and column names come from the records. Quoting keeps them from changing the SQL, but a record's collection can name any database the DSN user can write to, through `db.table`. Grant the destination's user access only to the databases it should write. Quoting is only sound in character sets where no character can end in a backtick byte, so a connection using `sjis`, `cp932`, `gbk`, `gb18030` or `big5` is refused with `mysql.destination.unsafe_charset` (checked in the DSN, when the destination opens, and on the connection of every write). Use `utf8mb4`, the driver's default. + + ### Deletes + + Delete records are matched on the target table's primary key, so the table must have one: a delete for a table without a primary key fails with `mysql.destination.delete_without_primary_key` instead of removing rows. The record key must hold a non-null value for every primary-key column (`mysql.destination.key_column_missing` otherwise), either as structured data or as a single JSON object. Each value must be one scalar: a list or an object fails with `mysql.destination.key_value_not_scalar` (it would match several rows or none). Key values are converted to the type of their key column, which is read for every write, so an `ALTER TABLE` on a running pipeline is picked up without a restart. The same rules apply to the values an upsert writes into the columns of any unique index (the primary key and `UNIQUE` keys), because those values decide which row `ON DUPLICATE KEY UPDATE` overwrites: `{"id":1.5}` for a `BIGINT` key is refused instead of being rounded to 2 and overwriting row 2. A record must be fully non-null in at least one unique index, otherwise it fails with `mysql.destination.key_column_missing`: a null never conflicts, so every redelivery would insert another row. A null in one unique column is fine while another unique index identifies the row (a nullable `UNIQUE` email next to a primary key). Exceptions, which append a row on every redelivery: a null or absent `AUTO_INCREMENT` primary key, which generates a new id, and a table without a unique index. Field names are resolved to columns by the MySQL server itself (through `information_schema`), so a field names exactly the column a statement would use: `ID` names `id`, and so do characters MySQL folds (for example the Kelvin sign names `k`), while `Å¿` names no column. A record with two fields that name one column (`id` and `ID`) fails with `mysql.destination.duplicate_column`. A delete for a table that does not exist fails with `mysql.destination.table_not_found`; table names are case-sensitive unless the server sets `lower_case_table_names`. + + - **Integer columns** (`TINYINT` to `BIGINT`, signed or `UNSIGNED`) take a whole number within the column's range (`5`, `5.0`, `5e0` or the string `"5"`). A fraction, a non-numeric string or a value outside the range fails with `mysql.destination.key_value_type_mismatch` instead of matching nothing or the wrong row. A boolean matches as MySQL compares it: `true` is 1, which is what a `TINYINT(1)` key needs. + - **String columns** (`CHAR`, `VARCHAR`, the `TEXT` types, `BINARY`, `VARBINARY`) take a string. An integer is sent as its decimal text and a raw JSON number as written (so `1e3` matches the row `'1e3'`), because MySQL compares a number with a string column as a double and `1000` would match `'1000'`, `'01000'`, `'1000.0'` and `' 1000'`. A float is accepted only if it is whole and below 1e15, where MySQL's text for it is the plain integer; other floats and booleans fail. + - **`ENUM` and `SET` columns** take the member name as a string. A number fails, because MySQL would compare it with the member's index. + - **`BIT(M)` columns** take a whole number from 0 to 2^M-1. + - **Other columns** (`DECIMAL`, floating point, temporal) get the value as it is; a raw JSON number keeps its literal, which MySQL compares exactly with a `DECIMAL` column. + + Integer values in a raw JSON key are matched exactly. Structured keys that reach a standalone connector over the plugin protocol have already passed through a double, so integers above 2^53 sent that way are rounded before the destination sees them ([ConduitIO/conduit#3100](https://github.com/ConduitIO/conduit/issues/3100)). A record whose operation is not create, update, delete or snapshot fails with `mysql.destination.unsupported_operation`. + + ### Atomicity and acknowledgements + + Each write is a single transaction. If any record in the write fails, nothing from that write is stored and none of its records are acknowledged; the records are redelivered after the pipeline recovers. Upserts on a primary or unique key make the redelivery idempotent. Redelivering a record whose `AUTO_INCREMENT` key is null or absent, or a record for a table with no unique key, appends another row. + + The destination sets `wait_timeout=60` on its own connections (unless the DSN sets `wait_timeout`), and the pool drops connections idle for half of it. If the connector dies in the middle of a write and the server does not see the connection close (a NAT, a proxy, a half-open TCP connection), the uncommitted transaction and its row locks end within that time instead of the server default of 8 hours, so the redelivered records do not wait on them. + + ## Delivery semantics and limits + + The source connector is in beta. This section states what it guarantees and what it does not. + + ### Delivery guarantees + + - **At-least-once.** A record is acknowledged upstream only after the destination has + acknowledged it, and the position carried on a record is what Conduit persists. After a + crash or restart you can receive records again, never fewer. Make destinations idempotent + (upsert by key). + - **Snapshot to CDC handoff.** The connector captures the binlog position at the start of + the snapshot and begins CDC from it, so changes made while the snapshot runs are delivered + after it. Rows can therefore appear twice: once as a `snapshot` record and again as a + CDC change. + - **Restart.** A restart during the snapshot resumes the snapshot from the last + acknowledged key per table. After a CDC restart, CDC resumes after the last acknowledged + row, also in the middle of a multi-row binlog event, in a later binlog event of one large + statement, and inside a compressed transaction (`binlog_transaction_compression=ON`). After + upgrading from an older version, restarts re-read the binlog file of the saved position from + its start to find the saved event, until the first record is acknowledged under the new + version. Rows are delivered again (duplicates, never a gap) in three cases, each logged as a + warning: the saved position was written right after a restart in the middle of an event (that + event is delivered again), an older version's position points inside a compressed transaction + (its binlog file is delivered again from its start), or the captured tables changed while the + saved position was inside a compressed transaction (that transaction is delivered again). + - **Ordering.** CDC records are delivered in binlog order. During the snapshot, rows are + ordered by key within a table; tables are read concurrently, so there is no ordering across + tables. + + ### Limits + + - **No GTID support.** Positions are a binlog file name and offset on the server the + connector read from. They are not portable between servers: a replica, or a replica + promoted after a failover, has different binlog files and offsets. **Failing over the + source database breaks resume.** The connector may fail to start, or may not resume at the + matching point. After a failover, treat the pipeline as needing a new snapshot: clear its + position and expect to re-read the data. GTID-based positions are not supported in this + release. + - **Tables added after the pipeline entered CDC are not snapshotted.** A snapshot runs only + on a fresh start. The position records which tables were snapshotted, and a restart checks + the configuration against it: + - If you add a table to an explicit `tables` list (plain table names), or switch + `snapshot.enabled` from `false` to `true`, the connector refuses to start with + `mysql.snapshot.tables_added_after_cdc`, naming the tables. Start a new pipeline for them, + or reset this pipeline's position to snapshot everything again (expect duplicates). + - If `tables` uses a pattern (`*`, a regex, an exclude, or an anchored name such as + `^orders$`), the connector logs one warning with the same code and then delivers the new + tables **from CDC only**: their existing rows are not snapshotted and are never delivered. + - A plain name is a regex that is not anchored, so `orders` also matches `orders_archive`. + It is still treated as an explicit list, which is the safe side. Anchor it as `^orders$` + to make it a pattern. + - A table removed from `tables` and added back later counts as added: changes made while it + was excluded were never captured. For an explicit list this means a table that is absent + at one restart and present at the next errors. + - A position written by an older version cannot be checked. The connector adopts the + configured tables as **unverified**, never as snapshotted, and logs + `mysql.snapshot.tables_unverified` on every start until the position is reset. Tables added + afterwards are detected. + - With more than 200 tables the position stores a hash of the table set instead of the + names. A change to the set then cannot be named or given a direction (a table that was only + removed also triggers it). The error says that "the configured table set changed", and + the way out is to restore the previous `tables` list or reset the position; removing tables + from the list does not clear it. + - **A key column that was retyped restarts its table's snapshot.** The position records the + type of each key column. If the saved key does not fit the column's type now (an `INT` key + column that became `VARCHAR`, or `VARCHAR` that became `DECIMAL`, whose values order + differently), the table's snapshot restarts from its first row with a warning. Rows already + delivered are delivered again; nothing is skipped. + - **Upgrading in the middle of a snapshot re-reads text, decimal and binary key tables.** + A snapshot position written by an older version does not record the key type, so a table + whose key is text, binary, `DECIMAL`, `TIME`, `ENUM`, `SET` or `BIT` cannot be checked for a + retype and restarts from its first row after the upgrade; its partial progress is re-read. + Integer, date and time keys resume where they stopped. Pipelines already in CDC are not + affected. + - **Binlog retention is your responsibility.** If the persisted position has been purged + from the server, the connector cannot resume. Keep `binlog_expire_logs_seconds` longer than + the longest outage you want to survive. + - **No heartbeat.** The connector does not send or consume heartbeat events. The position + is carried on records, so while the configured tables are quiet the persisted position does + not advance, even if the server keeps writing and rotating binlogs for other tables. A long + quiet period followed by binlog purging can leave the stored position pointing at a purged + file. + - **DDL during a running pipeline is not handled explicitly and is not tested.** Running + `ALTER TABLE` (or other schema changes) on a captured table while the snapshot or CDC is + running may produce records that do not match the schema registered earlier, or an error. + Stop the pipeline, change the schema, and verify the result before resuming. + - **Tables must be orderable for the snapshot.** See "Tables without a primary key". + - **Type mapping has known gaps.** `DECIMAL` and `NUMERIC` map to Avro `double` and can lose + precision beyond what a 64-bit float holds. SQL `NULL` is delivered as the column type's + zero value (`0`, `""`), so it cannot be told apart from a real zero. Date and time types are + delivered as strings. A full type-mapping design is pending; do not rely on exact fidelity + for these types yet. + - **`BIGINT UNSIGNED` above 2^63-1 stops the source.** The column maps to Avro `long`, which + cannot hold values above 9223372036854775807. When the snapshot or CDC reads such a value, + the source stops with `mysql.types.unsigned_overflow`, naming the table, column and value. + No record is emitted for that row and the position does not move past it. Exclude the + table, or bring the values into range, and restart. Versions before this one emitted a + wrong number in the snapshot (18446744073709551615 came out as 3474867090068222261) and + failed with an encoder error in CDC. + + ## Requirements and compatibility + + The connector is tested against MySQL v8.0. Compatibility with older versions isn't guaranteed. + + ### MySQL Server Requirements: + + - Binary Log (binlog) must be enabled. + - Binlog format must be set to ROW. + - Binlog row image must be set to FULL. + - Recommended: `binlog_row_metadata=FULL` (MySQL 8.0.1+, MariaDB 10.5+). Without it the connector cannot + detect every schema change that happened while it was stopped; see `mysql.cdc.row_columns_mismatch` below. + - Tables must have **sortable** primary keys. + + ### Partial row images + + `binlog_row_image` is also a session variable. A writing session that sets it to `MINIMAL` or `NOBLOB` logs + only part of the row, even when the server value is `FULL`: + + - `MINIMAL`, UPDATE: the before image holds only the primary key, and the after image holds only the changed + columns (the primary key is in it only if it changed). + - `MINIMAL`, DELETE: the before image holds only the primary key. + - `MINIMAL`, INSERT: the image holds only the columns the statement names. Every column it omits is skipped, + including columns with a `DEFAULT` and nullable columns, so an ORM that leaves defaulted columns out halts on + its first insert. An INSERT that names every column does not halt. + - `NOBLOB`: `BLOB` and `TEXT` columns that an UPDATE or DELETE did not change are skipped, and so is a `BLOB` + that an INSERT omits. An INSERT that omits a non-`BLOB` column with a default is logged in full. + + A skipped column is unknown. It is not NULL and it is not the column default: the binlog carries no value for + it, and the schema default may have changed since the row was written, so the connector does not fill it in. + + The CDC source halts on the first such event instead of delivering zero values over real data. The error has + the code `mysql.cdc.row_image_skipped_columns` and names the table, the action, the binlog file and position, and + the columns that were left out. Snapshot rows are not affected. If the event sits inside a compressed + transaction (`binlog_transaction_compression=ON`) the exact position is not available and the error says so. + + The source also stops with the code `mysql.cdc.partial_json_unsupported` when a writer logs partial JSON updates + (`binlog_row_value_options=PARTIAL_JSON`), which the connector cannot read. Set the option to an empty value for + the writers. + + Two more CDC halts are resolved by resetting the position, so the recovery below (deletes) applies to them too: + + - `mysql.cdc.row_columns_mismatch`: a binlog event has a different number of values than its table has + columns, even after the connector re-read the table's columns (it does that once on a mismatch, because its + cached column list can be out of date after a DDL its parser skips, such as `ADD COLUMN ... INVISIBLE`). + The connector decodes events with the table's current columns, so an event written before an `ADD COLUMN` + or `DROP COLUMN` and read after it (the connector was stopped or lagging across the DDL) cannot be mapped. + It also happens when the connector's user cannot see every column: column privileges, or a generated + invisible primary key hidden by `show_gipk_in_create_table_and_information_schema=OFF`, which halts by + design. Make hidden columns visible first. Then restart the pipeline; if it halts again on the same event, + reset the position to snapshot again. + - `mysql.cdc.row_columns_mismatch` also covers an event whose number of values fits the table but that was + written under other columns: after a `DROP COLUMN` plus an `ADD COLUMN` (or a column changing type) ran + before the connector read the event, every value after the dropped column would land in the wrong column. + The connector compares the types the event was written with against the table's columns (by family, so + `INT` to `BIGINT` or a longer `VARCHAR` is fine) and halts on a difference, also after a restart, so + restarting does not help: reset the position. With `binlog_row_metadata=FULL` the event also carries its + column names, and the connector compares those too; a renamed column then halts as well, because a rename + cannot be told from a drop plus an add. **Without `FULL`, a column dropped and another of the same type + family added at the same position, while the connector is stopped or lagging, is not detected, and the + values are mapped onto the new column.** `binlog_row_metadata=FULL` (MySQL 8.0.1+, MariaDB 10.5+; the default + is `MINIMAL`) closes that for events written after it is set, at the cost of a larger binlog. Stop the + pipeline before changing a captured table's columns, or set it. + - `mysql.cdc.resume_event_not_found`: on a restart, the binlog event the saved position names was not where + the position says. This should not happen; report it with the logged position, and reset the position to + continue. + + #### Recovery + + **Resuming means a re-snapshot, and a snapshot does not carry deletes.** Rows deleted at the source while the + pipeline was halted would stay in the destination. Do one of the following, for ALL captured tables, not only + the table named in the error: + + - truncate the destination tables, then clear the connector's position to re-snapshot; or + - clear the connector's position to re-snapshot, and after the snapshot has completed reconcile the destination + against the source, deletes included. + + Do not reconcile first and re-snapshot afterwards: rows deleted at the source between the reconcile and the + start of the snapshot would stay in the destination. + + 1. Stop the cause. Set `binlog_row_image = 'FULL'` for the writers, and for the server use + `SET PERSIST binlog_row_image = 'FULL'` (`SET GLOBAL` does not survive a restart). Then reconnect every + writer: existing sessions, pooled connections included, keep the value they started with. + 2. The event is already in the binlog, so every restart halts on it. Find its transaction with + `SHOW BINLOG EVENTS IN ''` (the error gives the start position of the event); with GTIDs enabled, + the `Gtid` event just before it names the GTID. + 3. Truncate the destination for all captured tables, or plan to reconcile it after the snapshot (see above). + 4. Clear the connector's position to re-snapshot. + 5. If you chose to reconcile, do it once the snapshot has completed, including deletes, for all captured tables. + + A lighter way to resume after the event is tracked in + [#212](https://github.com/ConduitIO/conduit-connector-mysql/issues/212). + + #### Upgrading + + If the binlog after the stored position already holds an event from a `MINIMAL` or `NOBLOB` session, including + an event that an older version had partly emitted, the upgraded connector halts on it after the restart. Older + versions delivered such events with zero values. + + ### MySQL User Privileges: + + For Snapshot and CDC modes, the following privileges are required: + + - SELECT + - LOCK TABLES + - RELOAD + - REPLICATION CLIENT + - REPLICATION SLAVE + version: v0.4.0 + author: Meroxa, Inc. + source: + parameters: + - name: dsn + description: The connection string for the MySQL database. + type: string + default: "" + validations: + - type: required + value: "" + - name: tables + description: |- + Represents the tables to read from. + - By default, no tables are included, but can be modified by adding a comma-separated string of regex patterns. + - They are applied in the order that they are provided, so the final regex supersedes all previous ones. + - To include all tables, use "*". You can then filter that list by adding a comma-separated string of regex patterns. + - To set an "include" regex, add "+" or nothing in front of the regex. + - To set an "exclude" regex, add "-" in front of the regex. + - e.g. "-.*meta$, wp_postmeta" will exclude all tables ending with "meta" but include the table "wp_postmeta". + type: string + default: "" + validations: + - type: required + value: "" + - name: cdc.disableLogs + description: Disables verbose cdc driver logs. + type: bool + default: "" + validations: [] + - name: snapshot.enabled + description: Controls whether the snapshot is done. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Limits how many rows should be retrieved on each database fetch on snapshot mode. + type: int + default: "10000" + validations: [] + - name: snapshot.unsafe + description: 'Removed. The unsafe snapshot no longer exists: true fails validation with a migration message, false is accepted and ignored. Tables without a primary key need tableConfig.
.sortingColumn.' + type: bool + default: "" + validations: [] + - name: tableConfig.*.sortingColumn + description: |- + Allows to force using a custom column to sort the snapshot. The column must be + NOT NULL and the only column of a primary key or unique index. + type: string + default: "" + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: 1s + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "10000" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: dsn + description: The connection string for the MySQL database. + type: string + default: "" + validations: + - type: required + value: "" + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/.metadata.yaml new file mode 100644 index 0000000..7312850 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:17.030393895Z +commitSHA: 8c621aca0207284da17d5656e8fa59d434ac063c diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/connector.yaml new file mode 100644 index 0000000..7dcaee2 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-pgvector@v0.1.1/connector.yaml @@ -0,0 +1,163 @@ +version: "1.0" +specification: + name: pgvector + summary: Conduit destination connector for pgvector (Postgres vector store). + description: |- + The pgvector destination connector is the flagship vector sink for Conduit's + AI data pipeline (CDC -> chunk -> embed -> pgvector), keeping a RAG index + fresh from Postgres. + + It is a write-only connector. Each incoming embedding record is upserted into + a pgvector table keyed by the record's stable key (the chunk_id), using + `INSERT ... ON CONFLICT () DO UPDATE`, so at-least-once redelivery + converges to a single row rather than duplicating it. + + At pipeline start the connector validates the configured embedding + `dimension` against the target table's `vector` column and fails fast with a + stable, actionable error (`ai.vector_dimension_mismatch`) if they disagree, + instead of failing silently on the first write. + version: v0.1.1 + author: Meroxa, Inc. + destination: + parameters: + - name: dimension + description: |- + Dimension is the embedding dimension the upstream model produces. It is + validated against the target table's vector column at pipeline start + (fail fast). There is no safe default, so it is required. + type: int + default: "" + validations: + - type: required + value: "" + - name: url + description: URL is the connection string for the target Postgres/pgvector database. + type: string + default: "" + validations: + - type: required + value: "" + - name: idMetadataKey + description: |- + IDMetadataKey is the record metadata key carrying the deterministic + chunk id ({source_key}:{index}), set by the chunking processor + (default "ai.chunk.id") and preserved by the embedding processor. When + present on a record it takes precedence over Record.Key as the upsert + conflict target; when absent, Record.Key is used (slice-1 behavior), + so records written outside the chunking pipeline still work. + type: string + default: ai.chunk.id + validations: [] + - name: keyColumn + description: |- + KeyColumn is the primary-key column used as the ON CONFLICT target for + idempotent upserts. It holds the record's stable chunk_id. + type: string + default: id + validations: [] + - name: metadataColumn + description: |- + MetadataColumn is the JSONB column that receives the record's metadata. + Leave empty to disable metadata writes. + type: string + default: metadata + validations: [] + - name: sourceKeyColumn + description: |- + SourceKeyColumn is the TEXT column that stores the source record's + stable key, populated on every upsert. Deletes match on this column + (not KeyColumn) so a source-record delete removes every chunk row ever + derived from it, even when the chunk count has changed since the last + embed (design doc §5's orphan-avoidance rule). Leave empty to disable + source_key population, falling back to a delete-by-KeyColumn match — + the slice-1 behavior, which can orphan rows and is not recommended for + the RAG pipeline. + type: string + default: source_key + validations: [] + - name: sourceKeyMetadataKey + description: |- + SourceKeyMetadataKey is the record metadata key carrying the source + record's key. The chunking processor sets this (default + "ai.chunk.source_key") and the embedding processor preserves it. + Required on every record when SourceKeyColumn is enabled. + type: string + default: ai.chunk.source_key + validations: [] + - name: table + description: |- + Table is the target table into which embedding rows are upserted. + Supports a Go template evaluated per record (default: the record's + OpenCDC collection), matching the Postgres connector's convention. + type: string + default: '{{ index .Metadata "opencdc.collection" }}' + validations: [] + - name: vectorColumn + description: |- + VectorColumn is the name of the pgvector `vector` column on the target + table. + type: string + default: embedding + validations: [] + - name: vectorField + description: |- + VectorField is the record payload field that carries the embedding + vector (a numeric array). + type: string + default: vector + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/.metadata.yaml new file mode 100644 index 0000000..1c579cc --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:20.073866251Z +commitSHA: 62307b36bcb3ac9e77712b35d6e823dd398eacd2 diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/connector.yaml new file mode 100644 index 0000000..64f115e --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.14.4/connector.yaml @@ -0,0 +1,583 @@ +version: "1.0" +specification: + name: postgres + summary: Conduit connector for PostgreSQL + description: | + ## Source + + The Postgres Source Connector connects to a database with the provided `url` and + starts creating records for each change detected in the provided tables. + + Upon starting, the source takes a snapshot of the provided tables in the database, + then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. + + ### Snapshot + + When the connector first starts, snapshot mode is enabled. The connector acquires + a read-only lock on the tables, and then reads all rows of the tables into Conduit. + Once all rows in that initial snapshot are read the connector releases its lock and + switches into CDC mode. + + This behavior is enabled by default, but can be turned off by adding + `"snapshotMode": "never"` to the Source configuration. + + ### Change Data Capture + + This connector implements Change Data Capture (CDC) features for PostgreSQL by + creating a logical replication slot and a publication that listens to changes in the + configured tables. Every detected change is converted into a record. If there are no + records available, the connector blocks until a record is available or the connector + receives a stop signal. + + #### Logical Replication Configuration + + When the connector switches to CDC mode, it attempts to run the initial setup commands + to create its logical replication slot and publication. It will connect to an existing + slot if one with the configured name exists. + + The Postgres user specified in the connection URL must have sufficient privileges to + run all of these setup commands, or it will fail. + + Example pipeline configuration that's using logical replication: + + ```yaml + version: 2.2 + pipelines: + - id: pg-to-log + status: running + connectors: + - id: pg + type: source + plugin: builtin:postgres + settings: + url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" + tables: "users" + cdcMode: "logrepl" + logrepl.publicationName: "examplepub" + logrepl.slotName: "exampleslot" + - id: log + type: destination + plugin: builtin:log + settings: + level: info + ``` + + :warning: When the connector or pipeline is deleted, the connector will automatically + attempt to delete the replication slot and publication. This is the default behaviour + and can be disabled by setting `logrepl.autoCleanup` to `false`. + + ### Key Handling + + The connector will automatically look up the primary key column for the specified tables + and use them as the key value. If that can't be determined, the connector will return + an error. + + The table's `REPLICA IDENTITY` must carry the primary key in deletes. `DEFAULT` and + `FULL` always do. `USING INDEX` does only if the index includes the primary-key + column; otherwise Postgres sends deletes without the key, and they cannot be applied + downstream. The source refuses to open on such a table with the error code + `postgres.replica_identity.excludes_key`. Fix it with + `ALTER TABLE
REPLICA IDENTITY FULL`, or use a unique index that includes the + primary key. + + ### Numeric Columns + + A `numeric` column with a precision and scale, `numeric(p,s)`, is an Avro + `decimal(p,s)` and its values are `big.Rat`. + + A `numeric` column **without** a precision and scale can hold any value Postgres + can store, which no Avro `decimal` can. With `logrepl.withAvroSchema: true` (the + default) it is an Avro `string` holding the exact text of the value, with its + scale (`1.50` stays `1.50`), and `NaN`, `Infinity` and + `-Infinity` as those three words: **a consumer of such a column must handle + them**, a plain decimal parser will reject them. Earlier versions used `decimal(65535, 65531)`, which could not encode a + value of 10000 or more and stopped the pipeline. The first record after an + upgrade registers a new schema version for the table; consumers must read the + field as a string. This connector's destination writes the string to a `numeric` + column exactly. + + **Known issue:** a zero with a display scale reads as `0` in the snapshot and as + `0.000` (its stored text) in change data capture. The values are numerically + equal; only the text differs. Tracked in + [#391](https://github.com/ConduitIO/conduit-connector-postgres/issues/391). + + So the Go type of an unconstrained `numeric` depends on the setting: a `string` + with `logrepl.withAvroSchema: true`, a `big.Rat` with `false`. + + Without an Avro schema, a `numeric` is a `big.Rat`; `NaN`, `Infinity` and + `-Infinity` are the strings `"NaN"`, `"Infinity"` and `"-Infinity"`, and the same + applies to the elements of `numeric[]` (finite elements are unchanged). A `NULL` + stays `NULL`. + + `NaN` in a `numeric(p,s)` column (the infinities cannot be stored there) cannot + be an Avro decimal, and the connector never replaces it with `NULL` or a number: + the source stops with the error code `postgres.source.numeric_not_representable`, + which names the table, the column, the value and the row's key. To continue: + + - **Change data capture:** restart once with `logrepl.withAvroSchema: false` + from the same position. The value is delivered as a string. **Then set it + back to `true` as soon as that record has been delivered.** Without an Avro + schema a finite `numeric` is a `big.Rat`, which a standalone connector + cannot send to Conduit (#378): the next finite numeric would stop the + pipeline. Built-in pipelines are not affected. + On a standalone connector the SDK also fails the whole read batch on the + first record it cannot send, so if the NaN/Infinity record or its batch + contains any finite numeric, it is never delivered with Avro off and this + workaround does not help (#378 is the fix). While Avro is off, + JSON-serializing destinations receive a finite numeric as the string `"3/2"`. + - **Snapshot:** `UPDATE` the row to a finite value (or `NULL`) and restart. + - Removing the row, or adding a `CHECK` constraint, does not help for a change + that is already in the WAL: it is read again after every restart. Do not drop + the replication slot to get past it, that loses the position. A constraint + only keeps new bad values out. + + ### Infinite Dates and Timestamps + + `date`, `timestamp` and `timestamptz` columns can hold `infinity` and + `-infinity`. The Avro date and timestamp types cannot represent them. Earlier + versions wrote them as `1970-01-02` and `1969-12-31` (dates) without any error. + + - With `logrepl.withAvroSchema: true` (the default), the source stops with the + error code `postgres.source.temporal_not_representable`, which names the + table, the column and the value. Removing the row does not help once the + change is in the WAL: set `logrepl.withAvroSchema` to `false`. + - With `logrepl.withAvroSchema: false`, the values are emitted as the strings + `"infinity"` and `"-infinity"`. + + The same applies per element of `date[]`, `timestamp[]`, `timestamptz[]` and + `numeric[]` (for `numeric[]`, `NaN`, `Infinity` and `-Infinity` elements are + emitted as strings; finite elements are unchanged). The connector has no Avro + mapping for array columns, so with the default `logrepl.withAvroSchema: true` + a table with an array column already stops with a schema error before any + record is emitted. + + ### Before Image of Updates and Deletes + + `Payload.Before` is set only when Postgres sends the whole old row, which it does + under `REPLICA IDENTITY FULL`. Under `DEFAULT` and `USING INDEX`, Postgres sends only + the identity columns of the old row, so deletes and key-changing updates carry no + `Before`. The deleted row's key is in the record `Key`. A key-changing update carries + the new key in `Key`; the old key is not carried. If you need the old row or the old + key, set `REPLICA IDENTITY FULL` on the table. + + :warning: The Postgres destination does not delete the old row when a primary key + changes. It upserts on the new key, so a primary-key-changing `UPDATE` leaves the row + under its old key in the destination next to the new one. This happens with or without + Avro and under `REPLICA IDENTITY FULL` as well. It is a known destination bug + ([#386](https://github.com/ConduitIO/conduit-connector-postgres/issues/386)). Until it + is fixed, avoid changing primary keys on tables replicated to Postgres, or reconcile + the old rows by hand. + + ### Delete With a Missing Key + + **Symptom.** The pipeline fails and keeps failing after restarts with + `postgres.replica_identity.excludes_key: delete on table "
" (WAL LSN , in the + transaction that commits at LSN ) arrived without a value for key column ...`. + The replication slot retains WAL while it is stuck. + + **Diagnosis.** The table's `REPLICA IDENTITY` was changed while the pipeline was + running, to `USING INDEX` on an index whose key columns do not include the primary key + (columns in `INCLUDE` do not count), or the identity index was dropped. Postgres then + sends the delete without the key. Check with + `SELECT relreplident FROM pg_class WHERE oid = '
'::regclass` (`i` is + `USING INDEX`). + + **Remediation.** + + 1. Fix the identity so that new deletes carry the key: + `ALTER TABLE
REPLICA IDENTITY FULL`, or `REPLICA IDENTITY DEFAULT`, or + `REPLICA IDENTITY USING INDEX ` on a unique index that has the primary key + among its key columns. This does not repair the change already in the WAL. A restart + alone fails on it again. + 2. Stop the pipeline. Postgres refuses to read or advance a slot while a connection + uses it. + 3. Find the row. The error carries no row values, because they would end up in logs. + Read the stuck transaction from the slot instead, as a superuser or a role with + `REPLICATION`. Peeking does not consume anything. `` is the commit LSN `` + from the error with its low hex part incremented by one (`0/40C5318` becomes + `0/40C5319`); the error prints it: + + ```sql + SELECT xid, lsn, encode(data, 'escape') AS message + FROM pg_logical_slot_peek_binary_changes( + '', '', NULL, + 'proto_version', '1', 'publication_names', ''); + ``` + + Use the values of `logrepl.slotName` and `logrepl.publicationName` (defaults + `conduitslot` and `conduitpub`). + The output can start with earlier transactions that the pipeline already delivered + but Postgres has not yet been told about (the connector reports its position + periodically). They can contain the same value as the stuck delete, so do not read + them. Read only the transaction (`xid`) that contains the `D` row whose `lsn` is the + WAL LSN `` in the error. + Each row is one pgoutput message. `B` and `C` begin and commit the transaction. `R` + names a table and its columns. `I`, `U` and `D` are the changes. A `K` tuple follows + the table's object id: a 2-byte column count, then one entry per column, `n` for NULL + or `t`, a 4-byte length and the text value. For this error the only non-null columns + are those of the identity index, so the readable text is the value to look for, for + example `222` in a `D` row that shows `K ... n t ... 222 n`. Postgres only accepts a + unique index as a replica identity, so that value identifies the row in the + destination. Only tables in the publication appear, which are the tables the pipeline + captures. + 4. Delete that row from the destination yourself. The change has no key, so the pipeline + cannot do it. Then work out what else step 5 would skip. It skips every change the + peek lists before the stuck one, and the other changes of the stuck transaction. + Postgres has not been told they are done, and the pipeline may have read them + without the destination writing them before it stopped. Get the source connector's + last acknowledged position from Conduit: + `GET /v1/connectors/`, field `sourceState.position`. It is + base64 of JSON, for example + `{"version":2,"type":2,"last_lsn":"0/40845E8","tx_commit_lsn":"0/40846E8","tx_seq":1}` + (decode it with `base64 -d`). Conduit sets it only when the destination acknowledges a + record, never when the source reads one, so `last_lsn` is the WAL LSN of the last + change the destination has confirmed. If Conduit was restarted since the pipeline + stopped, it reports the last position it saved, which can be slightly older; that errs + on the safe side. Find the peek row of kind `I`, `U` or `D` whose `lsn` equals + `last_lsn`. Ignore `B` and `C` rows: a `C` row can show the same number as the first + change after it. If several rows share that `lsn` (a `COPY` or another statement + that changes many rows can do that), use `tx_seq`. It numbers the transaction's + `I`, `U`, `D` and `T` (truncate) rows in the order listed, starting at 1 after its + `B` row. `R` (relation), `Y` (type) and `O` (origin) rows are not counted. The row + with that number is the one to match, and it is an `I`, `U` or `D` row. The matched row + itself and everything listed before it are in the destination. For each change + listed after it, up to the stuck one and including the rest of the stuck transaction, + check the destination for its rows and apply any that are missing by hand (`I` and + `U` carry the full new row; `D` carries the identity columns). If no row matches, + treat every listed change as unconfirmed. A position without `last_lsn` (a pipeline + still in its snapshot) matches nothing. + 5. Skip the transaction. Use the same `` as in step 3. Writing it as a literal is + deliberate: `''::pg_lsn + 1` only works from Postgres 14. + - Postgres 11 or later: + `SELECT pg_replication_slot_advance('', '');` + - Postgres 10 has no such function. Consume the changes up to `` instead. This + skips exactly the same changes and needs no new snapshot: + `SELECT count(*) FROM pg_logical_slot_get_binary_changes('', '', NULL, 'proto_version', '1', 'publication_names', '');` + + Advancing to `` is what skips the transaction. Advancing to the change LSN + (``) or to the commit LSN `` itself does not: Postgres replays the transaction + and the pipeline fails on the same delete. + 6. Start the pipeline. + + **Skipping the transaction skips that change, and it skips the whole transaction, + including its other changes to captured tables.** Skipping data is your decision and the + connector cannot check it for you. + + ### Unchanged TOASTed Columns + + Postgres stores large field values (`text`, `bytea`, `jsonb`, and other + variable-length types past a size threshold) out-of-line using + [TOAST](https://www.postgresql.org/docs/current/storage-toast.html). When a row is + updated but a TOASTed column is **not** changed, Postgres omits that column's value + from the logical-replication message to avoid re-transmitting large unchanged data. + + For such columns, this connector **omits the field from the record payload** rather + than emitting it as `NULL`. Emitting `NULL` would be silent data corruption — a + downstream sink cannot distinguish "the value was set to NULL" from "the value did + not change". Omission preserves that distinction: absence means "unchanged". + + What this means for your destination: + + - **Partial-update / upsert sinks** (including this connector's own Postgres + destination, which builds its `ON CONFLICT DO UPDATE SET` clause only from the + fields present in the payload) handle omission correctly — an absent field is left + untouched in the target row. + - **Full-row / full-document replace sinks** must treat an absent field as + "unchanged" and preserve the existing target value. If a sink cannot do this, set + `REPLICA IDENTITY FULL` on the source table. Under `REPLICA IDENTITY FULL` the old + row image carries every column, and this connector backfills unchanged TOASTed + values from it so the payload is always complete. + + With the default `logrepl.withAvroSchema: true` and the default + `REPLICA IDENTITY`, an update that leaves a TOASTed column unchanged still stops the + source: the field is omitted, but the Avro payload schema requires it, so the record + cannot be encoded + ([#383](https://github.com/ConduitIO/conduit-connector-postgres/issues/383)). Set + `REPLICA IDENTITY FULL` on such tables. + + ### Limitations + + - **`TRUNCATE` is not propagated.** OpenCDC has no truncate operation, so a + `TRUNCATE` on a published table produces no record: rows it removed remain in + the destination. If the publication publishes `truncate` (the one the + connector creates does by default), the connector logs a `WARN` ("TRUNCATE was NOT + propagated") naming the table and the LSN, at most once a minute per table. + A restricted or user-supplied publication (for example + `publish = 'insert, update, delete'`) never sends the message, so there is + no warning and no other signal. The `suppressed` field counts the TRUNCATEs + of that table the rate limit swallowed since the previous warning; when + another table's warning evicts an expired entry that still holds a count, a + one-line summary with that count is logged. Treat the log lines as a signal, + not a precise counter: a count still pending when the process exits is + lost. Inserts, updates and deletes around the `TRUNCATE` are delivered as + usual. A configurable policy for TRUNCATE is planned. Until then, treat the + warning as a signal to reconcile the destination by hand. + + ## Destination + + The Postgres Destination takes a Conduit record and stores it using a SQL statement. + The Destination is designed to handle different payloads and keys. Because of this, + each record is individually parsed and upserted. + + ### Handling record operations + + Based on the `Operation` field in the record, the destination will either insert, + update or delete the record in the target table. Snapshot records are always inserted. + + If the target table already contains a record with the same key as a record being + inserted, the record will be updated (upserted). This can overwrite and thus potentially + lose data, so keys should be assigned correctly from the Source. + + If the target table does not contain a record with the same key as a record being + deleted, the record will be ignored. + + If there is no key, the record will be simply appended. + + ### Numeric columns + + Decimal values (`big.Rat`) are sent to Postgres as exact decimal strings, and + Postgres applies the column's precision and scale itself when it stores the + value. The destination never reads or caches a column's scale, so an + `ALTER TABLE ... ALTER COLUMN ... TYPE numeric(p,s)` takes effect on the next + write without restarting the pipeline, and the table is resolved through the + connection's `search_path` like the rest of the statement. + + Postgres rounds half away from zero when a value has more fractional digits + than the column's scale, and rejects a value that exceeds the column's + precision (`numeric field overflow`). A rejected batch is not acknowledged. + + A decimal that has no finite expansion, such as 1/3, is truncated toward zero + after 1001 fractional digits before it is sent. So is a decimal that terminates + but needs more than 16383 fractional digits, which Postgres rejects on every + numeric column. Postgres allows a column scale of at most 1000, so the + truncation never changes the result of the column's rounding. An unconstrained + `numeric` column stores all 1001 digits. + + The destination no longer checks that the target column is numeric. The general + rule: any column whose text input accepts the decimal string stores it. + Postgres converts the string, and the examples below were checked on + Postgres 17.5: + + - `integer`, `smallint`, `bigint`: stored if the value is whole. A fractional + value is rejected, except on a destination that was already running when the + column was changed to an integer type: the cached prepared statement still + types the parameter as `numeric`, so 5/2 is silently rounded to 3 + (see #367). + - `real`, `float8`: stored as the nearest binary float. + - `money`: rounded to the locale's fraction digits (1/3 is stored as `$0.33`). + - `oid`: stored if the value is a whole number (7 is stored as `7`). + - `text`: stored as the decimal string. `varchar(n)` and `char(n)`: rejected + with 22001 if the string is longer than `n`. + - `boolean`: 1 is stored as `true` and 0 as `false`; other values are rejected. + - `date`: 20261012 is stored as `2026-10-12`. + - `timestamp`: 20261012 is stored as `2026-10-12 00:00:00`. + - `timestamptz`: 20261012 is stored as `2026-10-12 00:00:00+00` (session time + zone UTC). + - `interval`: 12345 is stored as `03:25:45`. + - `json`, `jsonb`: 1/4 is stored as the JSON number `0.25`. + - `bytea`: 5/2 is stored as `\x322e35`, the ASCII bytes of `2.5`. + - `bit(n)`: stored if the decimal string is exactly `n` binary digits (101 is + stored as `101` in a `bit(3)`). + - `time`, `inet`: rejected. + + Every accepted conversion to a non-numeric type is a silent coercion if the + schema maps a numeric field to the wrong column, so check the mapping. + version: v0.14.4 + author: Meroxa, Inc. + source: + parameters: + - name: tables + description: |- + Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2". + Use "*" if you'd like to listen to all tables. + type: string + default: "" + validations: + - type: required + value: "" + - name: url + description: URL is the connection string for the Postgres database. + type: string + default: "" + validations: + - type: required + value: "" + - name: cdcMode + description: CDCMode determines how the connector should listen to changes. + type: string + default: auto + validations: + - type: inclusion + value: auto,logrepl + - name: logrepl.autoCleanup + description: |- + LogreplAutoCleanup determines if the replication slot and publication should be + removed when the connector is deleted. + type: bool + default: "true" + validations: [] + - name: logrepl.publicationName + description: |- + LogreplPublicationName determines the publication name in case the + connector uses logical replication to listen to changes (see CDCMode). + type: string + default: conduitpub + validations: [] + - name: logrepl.slotName + description: |- + LogreplSlotName determines the replication slot name in case the + connector uses logical replication to listen to changes (see CDCMode). + Can only contain lower-case letters, numbers, and the underscore character. + type: string + default: conduitslot + validations: + - type: regex + value: ^[a-z0-9_]+$ + - name: logrepl.withAvroSchema + description: |- + WithAvroSchema determines whether the connector should attach an avro schema on each + record. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Snapshot fetcher size determines the number of rows to retrieve at a time. + type: int + default: "50000" + validations: [] + - name: snapshotMode + description: SnapshotMode is whether the plugin will take a snapshot of the entire table before starting cdc mode. + type: string + default: initial + validations: + - type: inclusion + value: initial,never + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: url + description: URL is the connection string for the Postgres database. + type: string + default: "" + validations: + - type: required + value: "" + - name: table + description: Table is used as the target table into which records are inserted. + type: string + default: '{{ index .Metadata "opencdc.collection" }}' + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduit.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/.metadata.yaml new file mode 100644 index 0000000..3b1f73f --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:19.414929015Z +commitSHA: 1a985d7bcf539378580ca6caa69c1aa5d2b640ad diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/connector.yaml new file mode 100644 index 0000000..886b2f5 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.15.0/connector.yaml @@ -0,0 +1,689 @@ +version: "1.0" +specification: + name: postgres + summary: Conduit connector for PostgreSQL + description: |- + ## Connection and SCRAM authentication + + Source and destination connect with the `url` setting. Over TLS with SCRAM + authentication, the connector does not use channel binding unless you ask for it: + `channel_binding=prefer`, the default of the pgx driver since 5.9, is mapped to + `disable`, which is how the driver behaved in earlier releases. Channel binding + (SCRAM-SHA-256-PLUS) fails against a server with an Ed25519 certificate and behind + a proxy that terminates TLS and passes the SCRAM exchange through. Add + `channel_binding=require` to the URL, or set `PGCHANNELBINDING=require` in the + environment of the Conduit process, to require it. The driver reads + `PGCHANNELBINDING` when the URL has no `channel_binding`. An explicit `prefer`, + in the URL or in the environment, is also mapped to `disable`; `require` and + `disable` are kept. + + ``` + postgres://user:pass@host:5432/db?sslmode=require&channel_binding=require + ``` + + ### Upgrading from v0.14.x: connection string parsing + + This version uses pgx 5.11, which parses connection strings like libpq does. + Most URLs are unaffected. Check yours if it relies on one of these (the full + list is in the pgx 5.11.0 changelog): + + - `+` in a query value is literal, no longer a space; encode a space as `%20`. + - If a query parameter is repeated, the last one wins (the first used to). + - An IPv6 host in a URL needs brackets: `postgres://[::1]/db`. + - In a `key=value` connection string a backslash escapes the next character, + so a Windows path must be written `sslcert=C:\\path\\to\\cert`. + - Default locations of the client certificate, key and `.pgpass` file are + resolved from `$HOME` instead of the OS account's home directory. + + ## Source + + The Postgres Source Connector connects to a database with the provided `url` and + starts creating records for each change detected in the provided tables. + + Upon starting, the source takes a snapshot of the provided tables in the database, + then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. + + ### Snapshot + + When the connector first starts, snapshot mode is enabled. The connector acquires + a read-only lock on the tables, and then reads all rows of the tables into Conduit. + Once all rows in that initial snapshot are read the connector releases its lock and + switches into CDC mode. + + This behavior is enabled by default, but can be turned off by adding + `"snapshotMode": "never"` to the Source configuration. + + ### Change Data Capture + + This connector implements Change Data Capture (CDC) features for PostgreSQL by + creating a logical replication slot and a publication that listens to changes in the + configured tables. Every detected change is converted into a record. If there are no + records available, the connector blocks until a record is available or the connector + receives a stop signal. + + #### Logical Replication Configuration + + When the connector switches to CDC mode, it attempts to run the initial setup commands + to create its logical replication slot and publication. It will connect to an existing + slot if one with the configured name exists. + + The Postgres user specified in the connection URL must have sufficient privileges to + run all of these setup commands, or it will fail. + + Example pipeline configuration that's using logical replication: + + ```yaml + version: 2.2 + pipelines: + - id: pg-to-log + status: running + connectors: + - id: pg + type: source + plugin: builtin:postgres + settings: + url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" + tables: "users" + cdcMode: "logrepl" + logrepl.publicationName: "examplepub" + logrepl.slotName: "exampleslot" + - id: log + type: destination + plugin: builtin:log + settings: + level: info + ``` + + :warning: When the connector or pipeline is deleted, the connector will automatically + attempt to delete the replication slot and publication. This is the default behaviour + and can be disabled by setting `logrepl.autoCleanup` to `false`. + + ### Key Handling + + The connector will automatically look up the primary key column for the specified tables + and use them as the key value. If that can't be determined, the connector will return + an error. + + The table's `REPLICA IDENTITY` must carry the primary key in deletes. `DEFAULT` and + `FULL` always do. `USING INDEX` does only if the index includes the primary-key + column; otherwise Postgres sends deletes without the key, and they cannot be applied + downstream. The source refuses to open on such a table with the error code + `postgres.replica_identity.excludes_key`. Fix it with + `ALTER TABLE
REPLICA IDENTITY FULL`, or use a unique index that includes the + primary key. + + ### Before Image of Updates and Deletes + + `Payload.Before` is set only when Postgres sends the whole old row, which it does + under `REPLICA IDENTITY FULL`. Under `DEFAULT` and `USING INDEX`, Postgres sends only + the identity columns of the old row, so deletes and key-changing updates carry no + `Before`. The deleted row's key is in the record `Key`. A key-changing update carries + the new key in `Key`; the old key is not carried. If you need the old row or the old + key, set `REPLICA IDENTITY FULL` on the table. + + :warning: The Postgres destination does not delete the old row when a primary key + changes. It upserts on the new key, so a primary-key-changing `UPDATE` leaves the row + under its old key in the destination next to the new one. This happens with or without + Avro and under `REPLICA IDENTITY FULL` as well. It is a known destination bug + ([#386](https://github.com/ConduitIO/conduit-connector-postgres/issues/386)). Until it + is fixed, avoid changing primary keys on tables replicated to Postgres, or reconcile + the old rows by hand. + + ### Delete With a Missing Key + + **Symptom.** The pipeline fails and keeps failing after restarts with + `postgres.replica_identity.excludes_key: delete on table "
" (WAL LSN , in the + transaction that commits at LSN ) arrived without a value for key column ...`. + The replication slot retains WAL while it is stuck. + + **Diagnosis.** The table's `REPLICA IDENTITY` was changed while the pipeline was + running, to `USING INDEX` on an index whose key columns do not include the primary key + (columns in `INCLUDE` do not count), or the identity index was dropped. Postgres then + sends the delete without the key. Check with + `SELECT relreplident FROM pg_class WHERE oid = '
'::regclass` (`i` is + `USING INDEX`). + + **Remediation.** + + 1. Fix the identity so that new deletes carry the key: + `ALTER TABLE
REPLICA IDENTITY FULL`, or `REPLICA IDENTITY DEFAULT`, or + `REPLICA IDENTITY USING INDEX ` on a unique index that has the primary key + among its key columns. This does not repair the change already in the WAL. A restart + alone fails on it again. + 2. Stop the pipeline. Postgres refuses to read or advance a slot while a connection + uses it. + 3. Find the row. The error carries no row values, because they would end up in logs. + Read the stuck transaction from the slot instead, as a superuser or a role with + `REPLICATION`. Peeking does not consume anything. `` is the commit LSN `` + from the error with its low hex part incremented by one (`0/40C5318` becomes + `0/40C5319`); the error prints it: + + ```sql + SELECT xid, lsn, encode(data, 'escape') AS message + FROM pg_logical_slot_peek_binary_changes( + '', '', NULL, + 'proto_version', '1', 'publication_names', ''); + ``` + + Use the values of `logrepl.slotName` and `logrepl.publicationName` (defaults + `conduitslot` and `conduitpub`). + The output can start with earlier transactions that the pipeline already delivered + but Postgres has not yet been told about (the connector reports its position + periodically). They can contain the same value as the stuck delete, so do not read + them. Read only the transaction (`xid`) that contains the `D` row whose `lsn` is the + WAL LSN `` in the error. + Each row is one pgoutput message. `B` and `C` begin and commit the transaction. `R` + names a table and its columns. `I`, `U` and `D` are the changes. A `K` tuple follows + the table's object id: a 2-byte column count, then one entry per column, `n` for NULL + or `t`, a 4-byte length and the text value. For this error the only non-null columns + are those of the identity index, so the readable text is the value to look for, for + example `222` in a `D` row that shows `K ... n t ... 222 n`. Postgres only accepts a + unique index as a replica identity, so that value identifies the row in the + destination. Only tables in the publication appear, which are the tables the pipeline + captures. + 4. Delete that row from the destination yourself. The change has no key, so the pipeline + cannot do it. Then work out what else step 5 would skip. It skips every change the + peek lists before the stuck one, and the other changes of the stuck transaction. + Postgres has not been told they are done, and the pipeline may have read them + without the destination writing them before it stopped. Get the source connector's + last acknowledged position from Conduit: + `GET /v1/connectors/`, field `sourceState.position`. It is + base64 of JSON, for example + `{"version":2,"type":2,"last_lsn":"0/40845E8","tx_commit_lsn":"0/40846E8","tx_seq":1}` + (decode it with `base64 -d`). Conduit sets it only when the destination acknowledges a + record, never when the source reads one, so `last_lsn` is the WAL LSN of the last + change the destination has confirmed. If Conduit was restarted since the pipeline + stopped, it reports the last position it saved, which can be slightly older; that errs + on the safe side. Find the peek row of kind `I`, `U` or `D` whose `lsn` equals + `last_lsn`. Ignore `B` and `C` rows: a `C` row can show the same number as the first + change after it. If several rows share that `lsn` (a `COPY` or another statement + that changes many rows can do that), use `tx_seq`. It numbers the transaction's + `I`, `U`, `D` and `T` (truncate) rows in the order listed, starting at 1 after its + `B` row. `R` (relation), `Y` (type) and `O` (origin) rows are not counted. The row + with that number is the one to match, and it is an `I`, `U` or `D` row. The matched row + itself and everything listed before it are in the destination. For each change + listed after it, up to the stuck one and including the rest of the stuck transaction, + check the destination for its rows and apply any that are missing by hand (`I` and + `U` carry the full new row; `D` carries the identity columns). If no row matches, + treat every listed change as unconfirmed. A position without `last_lsn` (a pipeline + still in its snapshot) matches nothing. + 5. Skip the transaction. Use the same `` as in step 3. Writing it as a literal is + deliberate: `''::pg_lsn + 1` only works from Postgres 14. + - Postgres 11 or later: + `SELECT pg_replication_slot_advance('', '');` + - Postgres 10 has no such function. Consume the changes up to `` instead. This + skips exactly the same changes and needs no new snapshot: + `SELECT count(*) FROM pg_logical_slot_get_binary_changes('', '', NULL, 'proto_version', '1', 'publication_names', '');` + + Advancing to `` is what skips the transaction. Advancing to the change LSN + (``) or to the commit LSN `` itself does not: Postgres replays the transaction + and the pipeline fails on the same delete. + 6. Start the pipeline. + + **Skipping the transaction skips that change, and it skips the whole transaction, + including its other changes to captured tables.** Skipping data is your decision and the + connector cannot check it for you. + + ### Unchanged TOASTed Columns + + Postgres stores large field values (`text`, `bytea`, `jsonb`, and other + variable-length types past a size threshold) out-of-line using + [TOAST](https://www.postgresql.org/docs/current/storage-toast.html). When a row is + updated but a TOASTed column is **not** changed, Postgres omits that column's value + from the logical-replication message to avoid re-transmitting large unchanged data. + + For such columns, this connector **omits the field from the record payload** rather + than emitting it as `NULL`. Emitting `NULL` would be silent data corruption — a + downstream sink cannot distinguish "the value was set to NULL" from "the value did + not change". Omission preserves that distinction: absence means "unchanged". + + What this means for your destination: + + - **Partial-update / upsert sinks** (including this connector's own Postgres + destination, which builds its `ON CONFLICT DO UPDATE SET` clause only from the + fields present in the payload) handle omission correctly — an absent field is left + untouched in the target row. + - **Full-row / full-document replace sinks** must treat an absent field as + "unchanged" and preserve the existing target value. If a sink cannot do this, set + `REPLICA IDENTITY FULL` on the source table. Under `REPLICA IDENTITY FULL` the old + row image carries every column, and this connector backfills unchanged TOASTed + values from it so the payload is always complete. + + ### Value Representation + + Values do not depend on the time zone of the machine running the connector or on the + version of the Postgres driver. The same row gives the same record everywhere, for + snapshot and CDC. + + | Postgres type | Without an Avro schema | With an Avro schema (`logrepl.withAvroSchema`, the default) | + | --- | --- | --- | + | `timestamp` | The wall clock stored in Postgres, as a time with no offset information (`2024-01-02T03:04:05Z` for `2024-01-02 03:04:05`). | `local-timestamp-micros` of the same wall clock. | + | `timestamptz` | The instant, in UTC (`Z`). | `timestamp-micros`. | + | `date` | Midnight UTC of the date. | `date`. | + | `tsvector` | Postgres text form, `'a':1 'cat':3A,4B`. | `string`, same text. | + | `infinity`, `-infinity` | The strings `infinity` and `-infinity`. | The source halts with error code `postgres.source.temporal_not_representable`. | + | Years before 1 BC or after 9999 AD | Postgres text, for example `0044-03-15 12:00:00 BC`. | As above, up to what 64-bit microseconds hold; beyond that the same error. | + | Arrays, ranges, multiranges of these | Element and bound values as above. | Not supported by the Avro schema. | + + A range or multirange is a JSON object `{"Lower":...,"Upper":...,"LowerType":105, + "UpperType":101,"Valid":true}`, and a multirange is an array of them. `LowerType` and + `UpperType` are the ASCII codes of the bound kind: `105` (`i`) inclusive, `101` (`e`) + exclusive, `85` (`U`) unbounded (the bound is `null`), `69` (`E`) empty range. Bounds + follow the rules above. Arrays of `tsmultirange`, `tstzmultirange` and `datemultirange` + are emitted as the Postgres text of the array. + + If the source halts with `postgres.source.temporal_not_representable`, set + `logrepl.withAvroSchema` to `false`. Removing the row does not help a change that is + already in the WAL, because it is read again after every restart. + + ### Numeric Columns + + A `numeric` column with a precision and scale, `numeric(p,s)`, is an Avro + `decimal(p,s)` and its values are `big.Rat`. + + A `numeric` column **without** a precision and scale can hold any value Postgres + can store, which no Avro `decimal` can. With `logrepl.withAvroSchema: true` (the + default) it is an Avro `string` holding the exact text of the value, with its + scale (`1.50` stays `1.50`), and `NaN`, `Infinity` and + `-Infinity` as those three words: **a consumer of such a column must handle + them**, a plain decimal parser will reject them. Earlier versions used `decimal(65535, 65531)`, which could not encode a + value of 10000 or more and stopped the pipeline. The first record after an + upgrade registers a new schema version for the table; consumers must read the + field as a string. This connector's destination writes the string to a `numeric` + column exactly. + + **Known issue:** a zero with a display scale reads as `0` in the snapshot and as + `0.000` (its stored text) in change data capture. The values are numerically + equal; only the text differs. Tracked in + [#391](https://github.com/ConduitIO/conduit-connector-postgres/issues/391). + + So the Go type of an unconstrained `numeric` depends on the setting: a `string` + with `logrepl.withAvroSchema: true`, a `big.Rat` with `false`. Changing a column + between `text`/`varchar` and an unconstrained numeric is accepted by the + `evolve` schema-drift policy only when `withAvroSchema` is `true` (the Avro + shape does not change); with `false` the value's type changes, so it halts. + + Without an Avro schema, a `numeric` is a `big.Rat`; `NaN`, `Infinity` and + `-Infinity` are the strings `"NaN"`, `"Infinity"` and `"-Infinity"`, and the same + applies to the elements of `numeric[]` (finite elements are unchanged). A `NULL` + stays `NULL`. + + `NaN` in a `numeric(p,s)` column (the infinities cannot be stored there) cannot + be an Avro decimal, and the connector never replaces it with `NULL` or a number: + the source stops with the error code `postgres.source.numeric_not_representable`, + which names the table, the column, the value and the row's key. To continue: + + - **Change data capture:** restart once with `logrepl.withAvroSchema: false` + from the same position. The value is delivered as a string. **Then set it + back to `true` as soon as that record has been delivered.** Without an Avro + schema a finite `numeric` is a `big.Rat`, which a standalone connector + cannot send to Conduit (#378): the next finite numeric would stop the + pipeline. Built-in pipelines are not affected. + On a standalone connector the SDK also fails the whole read batch on the + first record it cannot send, so if the NaN/Infinity record or its batch + contains any finite numeric, it is never delivered with Avro off and this + workaround does not help (#378 is the fix). While Avro is off, + JSON-serializing destinations receive a finite numeric as the string `"3/2"`. + - **Snapshot:** `UPDATE` the row to a finite value (or `NULL`) and restart. + - Removing the row, or adding a `CHECK` constraint, does not help for a change + that is already in the WAL: it is read again after every restart. Do not drop + the replication slot to get past it, that loses the position. A constraint + only keeps new bad values out. + + ### Limitations + + - **`TRUNCATE` is not propagated.** OpenCDC has no truncate operation, so a + `TRUNCATE` on a published table produces no record: rows it removed remain in + the destination. If the publication publishes `truncate` (the one the + connector creates does by default), the connector logs a `WARN` ("TRUNCATE was NOT + propagated") naming the table and the LSN, at most once a minute per table. + A restricted or user-supplied publication (for example + `publish = 'insert, update, delete'`) never sends the message, so there is + no warning and no other signal. The `suppressed` field counts the TRUNCATEs + of that table the rate limit swallowed since the previous warning; when + another table's warning evicts an expired entry that still holds a count, a + one-line summary with that count is logged. Treat the log lines as a signal, + not a precise counter: a count still pending when the process exits is + lost. Inserts, updates and deletes around the `TRUNCATE` are delivered as + usual. A configurable policy for TRUNCATE is planned. Until then, treat the + warning as a signal to reconcile the destination by hand. + + ## Destination + + The Postgres Destination takes a Conduit record and stores it using a SQL statement. + The Destination is designed to handle different payloads and keys. Because of this, + each record is individually parsed and upserted. + + ### Handling record operations + + Based on the `Operation` field in the record, the destination will either insert, + update or delete the record in the target table. Snapshot records are always inserted. + + If the target table already contains a record with the same key as a record being + inserted, the record will be updated (upserted). This can overwrite and thus potentially + lose data, so keys should be assigned correctly from the Source. + + If the target table does not contain a record with the same key as a record being + deleted, the record will be ignored. + + If there is no key, the record will be simply appended. + + ### Keys, columns and acknowledgements + + - **Upsert key.** With a structured record key (or a raw key that is a JSON object), + a create, update or snapshot record is `INSERT ... ON CONFLICT () DO + UPDATE SET` every other field of the record. The key columns must carry a primary + key or unique constraint, otherwise Postgres rejects the statement. When a record has + no field besides its key columns (a table whose columns are all key columns), the + statement is `ON CONFLICT DO NOTHING`, so a redelivery is a no-op. + - **Columns.** Each record is written with exactly its own fields. A field the table has + no column for fails the write; the destination does not create or alter tables. A + field the record lacks is not written: an insert gets the column default, an existing + row keeps its value. A create, update or snapshot record without fields fails with + `postgres.destination.payload_empty`. + - **Key values** must be one non-null scalar per key column, for upserts and deletes. A + null value fails with `postgres.destination.key_column_missing` (a delete would match + every row with a NULL there, an upsert would append a new row on every redelivery), and + a list or object fails with `postgres.destination.key_value_not_scalar` (a delete would + match several rows or none). + - **Deletes** need a key and delete the rows matching every key column. + - **Raw JSON** keys and payloads are decoded without going through `float64`: an integer + literal is an `int64` (or `uint64`), and any other number keeps its literal, so + `numeric`, `text` and `json`/`jsonb` columns store it exactly. Integer columns take a + whole value in any form (`1e3`, `7.0`) and reject a fraction; `real` and `float8` + columns store the nearest float. Structured data that arrives over the plugin protocol + has already passed through a double (ConduitIO/conduit#3100), so integers above 2^53 + sent that way are rounded before the destination sees them. + - **Names** of the table and columns are quoted as Postgres identifiers, so reserved + words, upper case and double quotes work. A name with a NUL byte or longer than 63 bytes + fails with `postgres.destination.invalid_identifier`, because Postgres would address a + different table or column. The table name is one identifier: a dot is part of the name, + not a schema separator. + - **Acknowledgements.** A write is one implicit transaction. If any statement or the + commit fails, nothing from that write is stored and no record of it is acknowledged; + the records are redelivered (at-least-once). Keyed upserts make redelivery + idempotent; keyless inserts append again. + + ### Numeric columns + + Decimal values (`big.Rat`) are sent to Postgres as exact decimal strings, and + Postgres applies the column's precision and scale itself when it stores the + value. The destination never reads or caches a column's scale, so an + `ALTER TABLE ... ALTER COLUMN ... TYPE numeric(p,s)` takes effect on the next + write without restarting the pipeline, and the table is resolved through the + connection's `search_path` like the rest of the statement. + + Postgres rounds half away from zero when a value has more fractional digits + than the column's scale, and rejects a value that exceeds the column's + precision (`numeric field overflow`). A rejected batch is not acknowledged. + + A decimal that has no finite expansion, such as 1/3, is truncated toward zero + after 1001 fractional digits before it is sent. So is a decimal that terminates + but needs more than 16383 fractional digits, which Postgres rejects on every + numeric column. Postgres allows a column scale of at most 1000, so the + truncation never changes the result of the column's rounding. An unconstrained + `numeric` column stores all 1001 digits. + + The destination no longer checks that the target column is numeric. The general + rule: any column whose text input accepts the decimal string stores it. + Postgres converts the string, and the examples below were checked on + Postgres 17.5: + + - `integer`, `smallint`, `bigint`: stored if the value is whole. A fractional + value is rejected, except on a destination that was already running when the + column was changed to an integer type: the cached prepared statement still + types the parameter as `numeric`, so 5/2 is silently rounded to 3 + (see #367). + - `real`, `float8`: stored as the nearest binary float. + - `money`: rounded to the locale's fraction digits (1/3 is stored as `$0.33`). + - `oid`: stored if the value is a whole number (7 is stored as `7`). + - `text`: stored as the decimal string. `varchar(n)` and `char(n)`: rejected + with 22001 if the string is longer than `n`. + - `boolean`: 1 is stored as `true` and 0 as `false`; other values are rejected. + - `date`: 20261012 is stored as `2026-10-12`. + - `timestamp`: 20261012 is stored as `2026-10-12 00:00:00`. + - `timestamptz`: 20261012 is stored as `2026-10-12 00:00:00+00` (session time + zone UTC). + - `interval`: 12345 is stored as `03:25:45`. + - `json`, `jsonb`: 1/4 is stored as the JSON number `0.25`. + - `bytea`: 5/2 is stored as `\x322e35`, the ASCII bytes of `2.5`. + - `bit(n)`: stored if the decimal string is exactly `n` binary digits (101 is + stored as `101` in a `bit(3)`). + - `time`, `inet`: rejected. + + Every accepted conversion to a non-numeric type is a silent coercion if the + schema maps a numeric field to the wrong column, so check the mapping. + version: v0.15.0 + author: Meroxa, Inc. + source: + parameters: + - name: tables + description: |- + Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2". + Use "*" if you'd like to listen to all tables. + type: string + default: "" + validations: + - type: required + value: "" + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + type: string + default: "" + validations: + - type: required + value: "" + - name: cdcMode + description: CDCMode determines how the connector should listen to changes. + type: string + default: auto + validations: + - type: inclusion + value: auto,logrepl + - name: logrepl.autoCleanup + description: |- + LogreplAutoCleanup determines if the replication slot and publication should be + removed when the connector is deleted. + type: bool + default: "true" + validations: [] + - name: logrepl.import.debeziumOffset + description: |- + LogreplImportDebeziumOffset starts the pipeline from a Debezium Postgres + connector's committed offset instead of a snapshot. It is written by + `conduit migrate kafka-connect --cutover`, not by hand. It applies only + while the pipeline has no position: the first acknowledged record writes + an ordinary position and the setting is ignored from then on (remove it + once the import is complete). Requires snapshotMode "never" and cdcMode + "logrepl", and an existing slot and publication, which import mode + never creates. Requires Conduit v0.19.0 or later, which acks a record + only after its position is persisted. + type: string + default: "" + validations: [] + - name: logrepl.publicationName + description: |- + LogreplPublicationName determines the publication name in case the + connector uses logical replication to listen to changes (see CDCMode). + type: string + default: conduitpub + validations: [] + - name: logrepl.schemaDrift.policy + description: |- + LogreplSchemaDriftPolicy determines how the connector handles schema + drift (a DDL that changes a table's shape during CDC; DBZ-3 B1): + - "halt" (default): emit a drift marker record, checkpoint before it, + and stop with a terminal error until the pipeline is restarted to + approve the change. + - "evolve": accept additive changes silently; incompatible (narrowing) + changes still halt. + "dlq" is reserved for a future version and rejected at validation. + type: string + default: halt + validations: [] + - name: logrepl.slotName + description: |- + LogreplSlotName determines the replication slot name in case the + connector uses logical replication to listen to changes (see CDCMode). + Can only contain lower-case letters, numbers, and the underscore character. + type: string + default: conduitslot + validations: + - type: regex + value: ^[a-z0-9_]+$ + - name: logrepl.withAvroSchema + description: |- + WithAvroSchema determines whether the connector should attach an avro schema on each + record. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Snapshot fetcher size determines the number of rows to retrieve at a time. + type: int + default: "50000" + validations: [] + - name: snapshotMode + description: SnapshotMode is whether the plugin will take a snapshot of the entire table before starting cdc mode. + type: string + default: initial + validations: + - type: inclusion + value: initial,never + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + type: string + default: "" + validations: + - type: required + value: "" + - name: table + description: Table is used as the target table into which records are inserted. + type: string + default: '{{ index .Metadata "opencdc.collection" }}' + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/.metadata.yaml new file mode 100644 index 0000000..d3590ba --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:18.675610636Z +commitSHA: 97f5f7afed97339ed7388696c6418a9fba5a8a74 diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/connector.yaml new file mode 100644 index 0000000..f50f71f --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.16.0/connector.yaml @@ -0,0 +1,744 @@ +version: "1.0" +specification: + name: postgres + summary: Conduit connector for PostgreSQL + description: |- + ## Connection and SCRAM authentication + + Source and destination connect with the `url` setting. Over TLS with SCRAM + authentication, the connector does not use channel binding unless you ask for it: + `channel_binding=prefer`, the default of the pgx driver since 5.9, is mapped to + `disable`, which is how the driver behaved in earlier releases. Channel binding + (SCRAM-SHA-256-PLUS) fails against a server with an Ed25519 certificate and behind + a proxy that terminates TLS and passes the SCRAM exchange through. Add + `channel_binding=require` to the URL, or set `PGCHANNELBINDING=require` in the + environment of the Conduit process, to require it. The driver reads + `PGCHANNELBINDING` when the URL has no `channel_binding`. An explicit `prefer`, + in the URL or in the environment, is also mapped to `disable`; `require` and + `disable` are kept. + + ``` + postgres://user:pass@host:5432/db?sslmode=require&channel_binding=require + ``` + + ### Upgrading from v0.14.x: connection string parsing + + This version uses pgx 5.11, which parses connection strings like libpq does. + Most URLs are unaffected. Check yours if it relies on one of these (the full + list is in the pgx 5.11.0 changelog): + + - `+` in a query value is literal, no longer a space; encode a space as `%20`. + - If a query parameter is repeated, the last one wins (the first used to). + - An IPv6 host in a URL needs brackets: `postgres://[::1]/db`. + - In a `key=value` connection string a backslash escapes the next character, + so a Windows path must be written `sslcert=C:\\path\\to\\cert`. + - Default locations of the client certificate, key and `.pgpass` file are + resolved from `$HOME` instead of the OS account's home directory. + + ## Source + + The Postgres Source Connector connects to a database with the provided `url` and + starts creating records for each change detected in the provided tables. + + Upon starting, the source takes a snapshot of the provided tables in the database, + then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. + + ### Snapshot + + When the connector first starts, snapshot mode is enabled. The connector acquires + a read-only lock on the tables, and then reads all rows of the tables into Conduit. + Once all rows in that initial snapshot are read the connector releases its lock and + switches into CDC mode. + + This behavior is enabled by default, but can be turned off by adding + `"snapshotMode": "never"` to the Source configuration. + + ### Change Data Capture + + This connector implements Change Data Capture (CDC) features for PostgreSQL by + creating a logical replication slot and a publication that listens to changes in the + configured tables. Every detected change is converted into a record. If there are no + records available, the connector blocks until a record is available or the connector + receives a stop signal. + + #### Logical Replication Configuration + + When the connector switches to CDC mode, it attempts to run the initial setup commands + to create its logical replication slot and publication. It will connect to an existing + slot if one with the configured name exists. + + The Postgres user specified in the connection URL must have sufficient privileges to + run all of these setup commands, or it will fail. + + Example pipeline configuration that's using logical replication: + + ```yaml + version: 2.2 + pipelines: + - id: pg-to-log + status: running + connectors: + - id: pg + type: source + plugin: builtin:postgres + settings: + url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" + tables: "users" + cdcMode: "logrepl" + logrepl.publicationName: "examplepub" + logrepl.slotName: "exampleslot" + - id: log + type: destination + plugin: builtin:log + settings: + level: info + ``` + + :warning: When the connector or pipeline is deleted, the connector will automatically + attempt to delete the replication slot and publication. This is the default behaviour + and can be disabled by setting `logrepl.autoCleanup` to `false`. + + ### Key Handling + + The connector will automatically look up the primary key column for the specified tables + and use them as the key value. If that can't be determined, the connector will return + an error. + + The table's `REPLICA IDENTITY` must carry the primary key in deletes. `DEFAULT` and + `FULL` always do. `USING INDEX` does only if the index includes the primary-key + column; otherwise Postgres sends deletes without the key, and they cannot be applied + downstream. The source refuses to open on such a table with the error code + `postgres.replica_identity.excludes_key`. Fix it with + `ALTER TABLE
REPLICA IDENTITY FULL`, or use a unique index that includes the + primary key. + + ### Before Image of Updates and Deletes + + `Payload.Before` is set only when Postgres sends the whole old row, which it does + under `REPLICA IDENTITY FULL`. Under `DEFAULT` and `USING INDEX`, Postgres sends only + the identity columns of the old row, so deletes and key-changing updates carry no + `Before`. The deleted row's key is in the record `Key`. A key-changing update carries + the new key in `Key`; the old key is not carried. If you need the old row or the old + key, set `REPLICA IDENTITY FULL` on the table. + + :warning: The Postgres destination does not delete the old row when a primary key + changes. It upserts on the new key, so a primary-key-changing `UPDATE` leaves the row + under its old key in the destination next to the new one. This happens with or without + Avro and under `REPLICA IDENTITY FULL` as well. It is a known destination bug + ([#386](https://github.com/ConduitIO/conduit-connector-postgres/issues/386)). Until it + is fixed, avoid changing primary keys on tables replicated to Postgres, or reconcile + the old rows by hand. + + ### Delete With a Missing Key + + **Symptom.** The pipeline fails and keeps failing after restarts with + `postgres.replica_identity.excludes_key: delete on table "
" (WAL LSN , in the + transaction that commits at LSN ) arrived without a value for key column ...`. + The replication slot retains WAL while it is stuck. + + **Diagnosis.** The table's `REPLICA IDENTITY` was changed while the pipeline was + running, to `USING INDEX` on an index whose key columns do not include the primary key + (columns in `INCLUDE` do not count), or the identity index was dropped. Postgres then + sends the delete without the key. Check with + `SELECT relreplident FROM pg_class WHERE oid = '
'::regclass` (`i` is + `USING INDEX`). + + **Remediation.** + + 1. Fix the identity so that new deletes carry the key: + `ALTER TABLE
REPLICA IDENTITY FULL`, or `REPLICA IDENTITY DEFAULT`, or + `REPLICA IDENTITY USING INDEX ` on a unique index that has the primary key + among its key columns. This does not repair the change already in the WAL. A restart + alone fails on it again. + 2. Stop the pipeline. Postgres refuses to read or advance a slot while a connection + uses it. + 3. Find the row. The error carries no row values, because they would end up in logs. + Read the stuck transaction from the slot instead, as a superuser or a role with + `REPLICATION`. Peeking does not consume anything. `` is the commit LSN `` + from the error with its low hex part incremented by one (`0/40C5318` becomes + `0/40C5319`); the error prints it: + + ```sql + SELECT xid, lsn, encode(data, 'escape') AS message + FROM pg_logical_slot_peek_binary_changes( + '', '', NULL, + 'proto_version', '1', 'publication_names', ''); + ``` + + Use the values of `logrepl.slotName` and `logrepl.publicationName` (defaults + `conduitslot` and `conduitpub`). + The output can start with earlier transactions that the pipeline already delivered + but Postgres has not yet been told about (the connector reports its position + periodically). They can contain the same value as the stuck delete, so do not read + them. Read only the transaction (`xid`) that contains the `D` row whose `lsn` is the + WAL LSN `` in the error. + Each row is one pgoutput message. `B` and `C` begin and commit the transaction. `R` + names a table and its columns. `I`, `U` and `D` are the changes. A `K` tuple follows + the table's object id: a 2-byte column count, then one entry per column, `n` for NULL + or `t`, a 4-byte length and the text value. For this error the only non-null columns + are those of the identity index, so the readable text is the value to look for, for + example `222` in a `D` row that shows `K ... n t ... 222 n`. Postgres only accepts a + unique index as a replica identity, so that value identifies the row in the + destination. Only tables in the publication appear, which are the tables the pipeline + captures. + 4. Delete that row from the destination yourself. The change has no key, so the pipeline + cannot do it. Then work out what else step 5 would skip. It skips every change the + peek lists before the stuck one, and the other changes of the stuck transaction. + Postgres has not been told they are done, and the pipeline may have read them + without the destination writing them before it stopped. Get the source connector's + last acknowledged position from Conduit: + `GET /v1/connectors/`, field `sourceState.position`. It is + base64 of JSON, for example + `{"version":2,"type":2,"last_lsn":"0/40845E8","tx_commit_lsn":"0/40846E8","tx_seq":1}` + (decode it with `base64 -d`). Conduit sets it only when the destination acknowledges a + record, never when the source reads one, so `last_lsn` is the WAL LSN of the last + change the destination has confirmed. If Conduit was restarted since the pipeline + stopped, it reports the last position it saved, which can be slightly older; that errs + on the safe side. Find the peek row of kind `I`, `U` or `D` whose `lsn` equals + `last_lsn`. Ignore `B` and `C` rows: a `C` row can show the same number as the first + change after it. If several rows share that `lsn` (a `COPY` or another statement + that changes many rows can do that), use `tx_seq`. It numbers the transaction's + `I`, `U`, `D` and `T` (truncate) rows in the order listed, starting at 1 after its + `B` row. `R` (relation), `Y` (type) and `O` (origin) rows are not counted. The row + with that number is the one to match, and it is an `I`, `U` or `D` row. The matched row + itself and everything listed before it are in the destination. For each change + listed after it, up to the stuck one and including the rest of the stuck transaction, + check the destination for its rows and apply any that are missing by hand (`I` and + `U` carry the full new row; `D` carries the identity columns). If no row matches, + treat every listed change as unconfirmed. A position without `last_lsn` (a pipeline + still in its snapshot) matches nothing. + 5. Skip the transaction. Use the same `` as in step 3. Writing it as a literal is + deliberate: `''::pg_lsn + 1` only works from Postgres 14. + - Postgres 11 or later: + `SELECT pg_replication_slot_advance('', '');` + - Postgres 10 has no such function. Consume the changes up to `` instead. This + skips exactly the same changes and needs no new snapshot: + `SELECT count(*) FROM pg_logical_slot_get_binary_changes('', '', NULL, 'proto_version', '1', 'publication_names', '');` + + Advancing to `` is what skips the transaction. Advancing to the change LSN + (``) or to the commit LSN `` itself does not: Postgres replays the transaction + and the pipeline fails on the same delete. + 6. Start the pipeline. + + **Skipping the transaction skips that change, and it skips the whole transaction, + including its other changes to captured tables.** Skipping data is your decision and the + connector cannot check it for you. + + ### Unchanged TOASTed Columns + + Postgres stores large field values (`text`, `bytea`, `jsonb`, and other + variable-length types past a size threshold) out-of-line using + [TOAST](https://www.postgresql.org/docs/current/storage-toast.html). When a row is + updated but a TOASTed column is **not** changed, Postgres omits that column's value + from the logical-replication message to avoid re-transmitting large unchanged data. + + For such columns, this connector **omits the field from the record payload** rather + than emitting it as `NULL`. Emitting `NULL` would be silent data corruption — a + downstream sink cannot distinguish "the value was set to NULL" from "the value did + not change". Omission preserves that distinction: absence means "unchanged". + + What this means for your destination: + + - **Partial-update / upsert sinks** (including this connector's own Postgres + destination, which builds its `ON CONFLICT DO UPDATE SET` clause only from the + fields present in the payload) handle omission correctly — an absent field is left + untouched in the target row. + - **Full-row / full-document replace sinks** must treat an absent field as + "unchanged" and preserve the existing target value. If a sink cannot do this, set + `REPLICA IDENTITY FULL` on the source table. Under `REPLICA IDENTITY FULL` the old + row image carries every column, and this connector backfills unchanged TOASTed + values from it so the payload is always complete. + + ### Value Representation + + Values do not depend on the time zone of the machine running the connector or on the + version of the Postgres driver. The same row gives the same record everywhere, for + snapshot and CDC. + + | Postgres type | Without an Avro schema | With an Avro schema (`logrepl.withAvroSchema`, the default) | + | --- | --- | --- | + | `timestamp` | The wall clock stored in Postgres, as a time with no offset information (`2024-01-02T03:04:05Z` for `2024-01-02 03:04:05`). | `local-timestamp-micros` of the same wall clock. | + | `timestamptz` | The instant, in UTC (`Z`). | `timestamp-micros`. | + | `date` | Midnight UTC of the date. | `date`. | + | `tsvector` | Postgres text form, `'a':1 'cat':3A,4B`. | `string`, same text. | + | `infinity`, `-infinity` | The strings `infinity` and `-infinity`. | The source halts with error code `postgres.source.temporal_not_representable`. | + | Years before 1 BC or after 9999 AD | Postgres text, for example `0044-03-15 12:00:00 BC`. | As above, up to what 64-bit microseconds hold; beyond that the same error. | + | Arrays, ranges, multiranges of these | Element and bound values as above. | Not supported by the Avro schema. | + + A range or multirange is a JSON object `{"Lower":...,"Upper":...,"LowerType":105, + "UpperType":101,"Valid":true}`, and a multirange is an array of them. `LowerType` and + `UpperType` are the ASCII codes of the bound kind: `105` (`i`) inclusive, `101` (`e`) + exclusive, `85` (`U`) unbounded (the bound is `null`), `69` (`E`) empty range. Bounds + follow the rules above. Arrays of `tsmultirange`, `tstzmultirange` and `datemultirange` + are emitted as the Postgres text of the array. + + If the source halts with `postgres.source.temporal_not_representable`, set + `logrepl.withAvroSchema` to `false`. Removing the row does not help a change that is + already in the WAL, because it is read again after every restart. + + ### Numeric Columns + + A `numeric` column with a precision and scale, `numeric(p,s)`, is an Avro + `decimal(p,s)` and its values are `big.Rat`. + + A `numeric` column **without** a precision and scale can hold any value Postgres + can store, which no Avro `decimal` can. With `logrepl.withAvroSchema: true` (the + default) it is an Avro `string` holding the exact text of the value, with its + scale (`1.50` stays `1.50`), and `NaN`, `Infinity` and + `-Infinity` as those three words: **a consumer of such a column must handle + them**, a plain decimal parser will reject them. Earlier versions used `decimal(65535, 65531)`, which could not encode a + value of 10000 or more and stopped the pipeline. The first record after an + upgrade registers a new schema version for the table; consumers must read the + field as a string. This connector's destination writes the string to a `numeric` + column exactly. + + **Known issue:** a zero with a display scale reads as `0` in the snapshot and as + `0.000` (its stored text) in change data capture. The values are numerically + equal; only the text differs. Tracked in + [#391](https://github.com/ConduitIO/conduit-connector-postgres/issues/391). + + So the Go type of an unconstrained `numeric` depends on the setting: a `string` + with `logrepl.withAvroSchema: true`, a `big.Rat` with `false`. Changing a column + between `text`/`varchar` and an unconstrained numeric is accepted by the + `evolve` schema-drift policy only when `withAvroSchema` is `true` (the Avro + shape does not change); with `false` the value's type changes, so it halts. + + Without an Avro schema, a `numeric` is a `big.Rat`; `NaN`, `Infinity` and + `-Infinity` are the strings `"NaN"`, `"Infinity"` and `"-Infinity"`, and the same + applies to the elements of `numeric[]` (finite elements are unchanged). A `NULL` + stays `NULL`. + + `NaN` in a `numeric(p,s)` column (the infinities cannot be stored there) cannot + be an Avro decimal, and the connector never replaces it with `NULL` or a number: + the source stops with the error code `postgres.source.numeric_not_representable`, + which names the table, the column, the value and the row's key. To continue: + + - **Change data capture:** restart once with `logrepl.withAvroSchema: false` + from the same position. The value is delivered as a string. **Then set it + back to `true` as soon as that record has been delivered.** Without an Avro + schema a finite `numeric` is a `big.Rat`, which a standalone connector + cannot send to Conduit (#378): the next finite numeric would stop the + pipeline. Built-in pipelines are not affected. + On a standalone connector the SDK also fails the whole read batch on the + first record it cannot send, so if the NaN/Infinity record or its batch + contains any finite numeric, it is never delivered with Avro off and this + workaround does not help (#378 is the fix). While Avro is off, + JSON-serializing destinations receive a finite numeric as the string `"3/2"`. + - **Snapshot:** `UPDATE` the row to a finite value (or `NULL`) and restart. + - Removing the row, or adding a `CHECK` constraint, does not help for a change + that is already in the WAL: it is read again after every restart. Do not drop + the replication slot to get past it, that loses the position. A constraint + only keeps new bad values out. + + ### Limitations + + - **`TRUNCATE` is not propagated.** OpenCDC has no truncate operation, so a + `TRUNCATE` on a published table produces no record: rows it removed remain in + the destination. If the publication publishes `truncate` (the one the + connector creates does by default), the connector logs a `WARN` ("TRUNCATE was NOT + propagated") naming the table and the LSN, at most once a minute per table. + A restricted or user-supplied publication (for example + `publish = 'insert, update, delete'`) never sends the message, so there is + no warning and no other signal. The `suppressed` field counts the TRUNCATEs + of that table the rate limit swallowed since the previous warning; when + another table's warning evicts an expired entry that still holds a count, a + one-line summary with that count is logged. Treat the log lines as a signal, + not a precise counter: a count still pending when the process exits is + lost. Inserts, updates and deletes around the `TRUNCATE` are delivered as + usual. A configurable policy for TRUNCATE is planned. Until then, treat the + warning as a signal to reconcile the destination by hand. + + ## Destination + + The Postgres Destination takes a Conduit record and stores it using a SQL statement. + The Destination is designed to handle different payloads and keys. Because of this, + each record is individually parsed and upserted. + + ### Handling record operations + + Based on the `Operation` field in the record, the destination will either insert, + update or delete the record in the target table. Snapshot records are always inserted. + + If the target table already contains a record with the same key as a record being + inserted, the record will be updated (upserted). This can overwrite and thus potentially + lose data, so keys should be assigned correctly from the Source. + + If the target table does not contain a record with the same key as a record being + deleted, the record will be ignored. + + If there is no key, the record will be simply appended. + + ### Keys, columns and acknowledgements + + - **Upsert key.** With a structured record key (or a raw key that is a JSON object), + a create, update or snapshot record is `INSERT ... ON CONFLICT () DO + UPDATE SET` every other field of the record. The key columns must carry a primary + key or unique constraint, otherwise Postgres rejects the statement. When a record has + no field besides its key columns (a table whose columns are all key columns), the + statement is `ON CONFLICT DO NOTHING`, so a redelivery is a no-op. + - **Columns.** Each record is written with exactly its own fields. A field the table has + no column for fails the write; the destination does not create or alter tables. A + field the record lacks is not written: an insert gets the column default, an existing + row keeps its value. A create, update or snapshot record without fields fails with + `postgres.destination.payload_empty`. + - **Key values** must be one non-null scalar per key column, for upserts and deletes. A + null value fails with `postgres.destination.key_column_missing` (a delete would match + every row with a NULL there, an upsert would append a new row on every redelivery), and + a list or object fails with `postgres.destination.key_value_not_scalar` (a delete would + match several rows or none). + - **Deletes** need a key and delete the rows matching every key column. + - **Raw JSON** keys and payloads are decoded without going through `float64`: an integer + literal is an `int64` (or `uint64`), and any other number keeps its literal, so + `numeric`, `text` and `json`/`jsonb` columns store it exactly. Integer columns take a + whole value in any form (`1e3`, `7.0`) and reject a fraction; `real` and `float8` + columns store the nearest float. Structured data that arrives over the plugin protocol + has already passed through a double (ConduitIO/conduit#3100), so integers above 2^53 + sent that way are rounded before the destination sees them. + - **Names** of the table and columns are quoted as Postgres identifiers, so reserved + words, upper case and double quotes work. A name with a NUL byte or longer than 63 bytes + fails with `postgres.destination.invalid_identifier`, because Postgres would address a + different table or column. The table name is one identifier: a dot is part of the name, + not a schema separator. + - **Acknowledgements.** A write is one implicit transaction. If any statement or the + commit fails, nothing from that write is stored and no record of it is acknowledged; + the records are redelivered (at-least-once). Keyed upserts make redelivery + idempotent; keyless inserts append again. + + ### Schema-drift markers + + When the Postgres source halts on schema drift it emits a drift marker: a create + record with no key and no payload, with the metadata `postgres.schema.drift=true`. + The destination skips and acknowledges a marker without writing a row, but only + when the record's position proves the Postgres source produced it as a marker for + the table named in `postgres.schema.drift.table`. The metadata alone is not proof, + because any processor or upstream system can set it. + + - A record that carries the metadata but not a marker's position, and has a key + or a payload, is written as an ordinary record. One WARN per batch reports it. + - A record shaped like a marker (no key, no payload) without a marker's position + stops the pipeline with `postgres.destination.drift_marker_unverified`. Nothing + in that batch is written or acknowledged. + - A proven marker that a processor added a key or fields to is still skipped as + a marker, with a WARN that names the added fields. + + A marker loses its position when it passes through another system, such as Kafka + or a file, because records read from there carry that system's position. In a + Postgres to Kafka to Postgres setup, drop drift markers before the hop with a + filter processor on the `postgres.schema.drift` metadata. + + The proof relies on the engine not letting processors change a record's + position. Conduit's classic engine refuses that; the arch-v2 engine lets a + processor rewrite positions, so there a processor could forge a marker. The + forged record would be skipped, which a processor with that power could do to + any record anyway. + ### Column type changes on a running pipeline + + The destination describes every statement before it runs it (pgx's + `describe_exec` query mode), so after `ALTER TABLE ... ALTER COLUMN ... TYPE` + the next write is typed for the new column. A value the new type cannot hold + fails the write instead of being converted. + + This costs throughput. On one dedicated machine (benchi + `destination-exec-mode`: keyed upserts into a two-column table) the + destination wrote 35% fewer rows per second than with cached statements at + `sdk.batch.size: 1`, the default, and 16% fewer at 1000. If the tables a + pipeline writes never change type while it runs, add + `default_query_exec_mode=cache_statement` to the `url` to reuse prepared + statements as before; the same benchmark measured no difference from the + previous release with it. **Risk:** with `cache_statement` (or + `cache_describe`) a statement keeps the parameter types it was prepared with, + so after an `ALTER COLUMN ... TYPE` Postgres converts values into the new + type without an error, for example 5/2 into an integer column stored as 3, + until the destination restarts. The destination logs a warning at start when + the URL selects a cache mode. `exec` and `simple_protocol`, which never cache + parameter types (PgBouncer in transaction mode needs one of them), are kept + when the URL sets them. + + ### Numeric columns + + Decimal values (`big.Rat`) are sent to Postgres as exact decimal strings, and + Postgres applies the column's precision and scale itself when it stores the + value. The destination never reads or caches a column's scale, so an + `ALTER TABLE ... ALTER COLUMN ... TYPE numeric(p,s)` takes effect on the next + write without restarting the pipeline, and the table is resolved through the + connection's `search_path` like the rest of the statement. + + Postgres rounds half away from zero when a value has more fractional digits + than the column's scale, and rejects a value that exceeds the column's + precision (`numeric field overflow`). A rejected batch is not acknowledged. + + A decimal that has no finite expansion, such as 1/3, is truncated toward zero + after 1001 fractional digits before it is sent. So is a decimal that terminates + but needs more than 16383 fractional digits, which Postgres rejects on every + numeric column. Postgres allows a column scale of at most 1000, so the + truncation never changes the result of the column's rounding. An unconstrained + `numeric` column stores all 1001 digits. + + The destination no longer checks that the target column is numeric. The general + rule: any column whose text input accepts the decimal string stores it. + Postgres converts the string, and the examples below were checked on + Postgres 17.5: + + - `integer`, `smallint`, `bigint`: stored if the value is whole. A fractional + value is rejected, also on a destination that was already running when the + column was changed to an integer type (see "Column type changes on a running + pipeline"). With `default_query_exec_mode=cache_statement` in the URL, such a + destination rounds 5/2 to 3 instead (#367). + - `real`, `float8`: stored as the nearest binary float. + - `money`: rounded to the locale's fraction digits (1/3 is stored as `$0.33`). + - `oid`: stored if the value is a whole number (7 is stored as `7`). + - `text`: stored as the decimal string. `varchar(n)` and `char(n)`: rejected + with 22001 if the string is longer than `n`. + - `boolean`: 1 is stored as `true` and 0 as `false`; other values are rejected. + - `date`: 20261012 is stored as `2026-10-12`. + - `timestamp`: 20261012 is stored as `2026-10-12 00:00:00`. + - `timestamptz`: 20261012 is stored as `2026-10-12 00:00:00+00` (session time + zone UTC). + - `interval`: 12345 is stored as `03:25:45`. + - `json`, `jsonb`: 1/4 is stored as the JSON number `0.25`. + - `bytea`: 5/2 is stored as `\x322e35`, the ASCII bytes of `2.5`. + - `bit(n)`: stored if the decimal string is exactly `n` binary digits (101 is + stored as `101` in a `bit(3)`). + - `time`, `inet`: rejected. + + Every accepted conversion to a non-numeric type is a silent coercion if the + schema maps a numeric field to the wrong column, so check the mapping. + version: v0.15.0 + author: Meroxa, Inc. + source: + parameters: + - name: tables + description: |- + Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2". + Use "*" if you'd like to listen to all tables. + type: string + default: "" + validations: + - type: required + value: "" + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + type: string + default: "" + validations: + - type: required + value: "" + - name: cdcMode + description: CDCMode determines how the connector should listen to changes. + type: string + default: auto + validations: + - type: inclusion + value: auto,logrepl + - name: logrepl.autoCleanup + description: |- + LogreplAutoCleanup determines if the replication slot and publication should be + removed when the connector is deleted. + type: bool + default: "true" + validations: [] + - name: logrepl.import.debeziumOffset + description: |- + LogreplImportDebeziumOffset starts the pipeline from a Debezium Postgres + connector's committed offset instead of a snapshot. It is written by + `conduit migrate kafka-connect --cutover`, not by hand. It applies only + while the pipeline has no position: the first acknowledged record writes + an ordinary position and the setting is ignored from then on (remove it + once the import is complete). Requires snapshotMode "never" and cdcMode + "logrepl", and an existing slot and publication, which import mode + never creates. Requires Conduit v0.19.0 or later, which acks a record + only after its position is persisted. + type: string + default: "" + validations: [] + - name: logrepl.publicationName + description: |- + LogreplPublicationName determines the publication name in case the + connector uses logical replication to listen to changes (see CDCMode). + type: string + default: conduitpub + validations: [] + - name: logrepl.schemaDrift.policy + description: |- + LogreplSchemaDriftPolicy determines how the connector handles schema + drift (a DDL that changes a table's shape during CDC; DBZ-3 B1): + - "halt" (default): emit a drift marker record, checkpoint before it, + and stop with a terminal error until the pipeline is restarted to + approve the change. + - "evolve": accept additive changes silently; incompatible (narrowing) + changes still halt. + "dlq" is reserved for a future version and rejected at validation. + type: string + default: halt + validations: [] + - name: logrepl.slotName + description: |- + LogreplSlotName determines the replication slot name in case the + connector uses logical replication to listen to changes (see CDCMode). + Can only contain lower-case letters, numbers, and the underscore character. + type: string + default: conduitslot + validations: + - type: regex + value: ^[a-z0-9_]+$ + - name: logrepl.withAvroSchema + description: |- + WithAvroSchema determines whether the connector should attach an avro schema on each + record. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Snapshot fetcher size determines the number of rows to retrieve at a time. + type: int + default: "50000" + validations: [] + - name: snapshotMode + description: SnapshotMode is whether the plugin will take a snapshot of the entire table before starting cdc mode. + type: string + default: initial + validations: + - type: inclusion + value: initial,never + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + The destination describes each statement before it runs it, so a column + type change on a running pipeline is seen by the next write. Add + `default_query_exec_mode=cache_statement` to reuse prepared statements + for speed, at the risk that after an ALTER COLUMN ... TYPE values are + converted into the new type without an error until a restart (#367). + type: string + default: "" + validations: + - type: required + value: "" + - name: table + description: Table is used as the target table into which records are inserted. + type: string + default: '{{ index .Metadata "opencdc.collection" }}' + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/.metadata.yaml new file mode 100644 index 0000000..d5c4ecd --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:18.122773709Z +commitSHA: e6ce23ceb9aeb587fc38de966b297f1f900d1fb3 diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/connector.yaml new file mode 100644 index 0000000..fa4e6a0 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-postgres@v0.17.0/connector.yaml @@ -0,0 +1,739 @@ +version: "1.0" +specification: + name: postgres + summary: Conduit connector for PostgreSQL + description: |- + ## Connection and SCRAM authentication + + Source and destination connect with the `url` setting. Over TLS with SCRAM + authentication, the connector does not use channel binding unless you ask for it: + `channel_binding=prefer`, the default of the pgx driver since 5.9, is mapped to + `disable`, which is how the driver behaved in earlier releases. Channel binding + (SCRAM-SHA-256-PLUS) fails against a server with an Ed25519 certificate and behind + a proxy that terminates TLS and passes the SCRAM exchange through. Add + `channel_binding=require` to the URL, or set `PGCHANNELBINDING=require` in the + environment of the Conduit process, to require it. The driver reads + `PGCHANNELBINDING` when the URL has no `channel_binding`. An explicit `prefer`, + in the URL or in the environment, is also mapped to `disable`; `require` and + `disable` are kept. + + ``` + postgres://user:pass@host:5432/db?sslmode=require&channel_binding=require + ``` + + ### Upgrading from v0.14.x: connection string parsing + + This version uses pgx 5.11, which parses connection strings like libpq does. + Most URLs are unaffected. Check yours if it relies on one of these (the full + list is in the pgx 5.11.0 changelog): + + - `+` in a query value is literal, no longer a space; encode a space as `%20`. + - If a query parameter is repeated, the last one wins (the first used to). + - An IPv6 host in a URL needs brackets: `postgres://[::1]/db`. + - In a `key=value` connection string a backslash escapes the next character, + so a Windows path must be written `sslcert=C:\\path\\to\\cert`. + - Default locations of the client certificate, key and `.pgpass` file are + resolved from `$HOME` instead of the OS account's home directory. + + ## Source + + The Postgres Source Connector connects to a database with the provided `url` and + starts creating records for each change detected in the provided tables. + + Upon starting, the source takes a snapshot of the provided tables in the database, + then switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events. + + ### Snapshot + + When the connector first starts, snapshot mode is enabled. The connector acquires + a read-only lock on the tables, and then reads all rows of the tables into Conduit. + Once all rows in that initial snapshot are read the connector releases its lock and + switches into CDC mode. + + This behavior is enabled by default, but can be turned off by adding + `"snapshotMode": "never"` to the Source configuration. + + ### Change Data Capture + + This connector implements Change Data Capture (CDC) features for PostgreSQL by + creating a logical replication slot and a publication that listens to changes in the + configured tables. Every detected change is converted into a record. If there are no + records available, the connector blocks until a record is available or the connector + receives a stop signal. + + #### Logical Replication Configuration + + When the connector switches to CDC mode, it attempts to run the initial setup commands + to create its logical replication slot and publication. It will connect to an existing + slot if one with the configured name exists. + + The Postgres user specified in the connection URL must have sufficient privileges to + run all of these setup commands, or it will fail. + + Example pipeline configuration that's using logical replication: + + ```yaml + version: 2.2 + pipelines: + - id: pg-to-log + status: running + connectors: + - id: pg + type: source + plugin: builtin:postgres + settings: + url: "postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable" + tables: "users" + cdcMode: "logrepl" + logrepl.publicationName: "examplepub" + logrepl.slotName: "exampleslot" + - id: log + type: destination + plugin: builtin:log + settings: + level: info + ``` + + :warning: When the connector or pipeline is deleted, the connector will automatically + attempt to delete the replication slot and publication. This is the default behaviour + and can be disabled by setting `logrepl.autoCleanup` to `false`. + + ### Key Handling + + The connector will automatically look up the primary key column for the specified tables + and use them as the key value. If that can't be determined, the connector will return + an error. + + The table's `REPLICA IDENTITY` must carry the primary key in deletes. `DEFAULT` and + `FULL` always do. `USING INDEX` does only if the index includes the primary-key + column; otherwise Postgres sends deletes without the key, and they cannot be applied + downstream. The source refuses to open on such a table with the error code + `postgres.replica_identity.excludes_key`. Fix it with + `ALTER TABLE
REPLICA IDENTITY FULL`, or use a unique index that includes the + primary key. + + ### Before Image of Updates and Deletes + + `Payload.Before` is set only when Postgres sends the whole old row, which it does + under `REPLICA IDENTITY FULL`. Under `DEFAULT` and `USING INDEX`, Postgres sends only + the identity columns of the old row, so deletes and key-changing updates carry no + `Before`. The deleted row's key is in the record `Key`. A key-changing update carries + the new key in `Key`; the old key is not carried. If you need the old row or the old + key, set `REPLICA IDENTITY FULL` on the table. + + :warning: The Postgres destination does not delete the old row when a primary key + changes. It upserts on the new key, so a primary-key-changing `UPDATE` leaves the row + under its old key in the destination next to the new one. This happens with or without + Avro and under `REPLICA IDENTITY FULL` as well. It is a known destination bug + ([#386](https://github.com/ConduitIO/conduit-connector-postgres/issues/386)). Until it + is fixed, avoid changing primary keys on tables replicated to Postgres, or reconcile + the old rows by hand. + + ### Delete With a Missing Key + + **Symptom.** The pipeline fails and keeps failing after restarts with + `postgres.replica_identity.excludes_key: delete on table "
" (WAL LSN , in the + transaction that commits at LSN ) arrived without a value for key column ...`. + The replication slot retains WAL while it is stuck. + + **Diagnosis.** The table's `REPLICA IDENTITY` was changed while the pipeline was + running, to `USING INDEX` on an index whose key columns do not include the primary key + (columns in `INCLUDE` do not count), or the identity index was dropped. Postgres then + sends the delete without the key. Check with + `SELECT relreplident FROM pg_class WHERE oid = '
'::regclass` (`i` is + `USING INDEX`). + + **Remediation.** + + 1. Fix the identity so that new deletes carry the key: + `ALTER TABLE
REPLICA IDENTITY FULL`, or `REPLICA IDENTITY DEFAULT`, or + `REPLICA IDENTITY USING INDEX ` on a unique index that has the primary key + among its key columns. This does not repair the change already in the WAL. A restart + alone fails on it again. + 2. Stop the pipeline. Postgres refuses to read or advance a slot while a connection + uses it. + 3. Find the row. The error carries no row values, because they would end up in logs. + Read the stuck transaction from the slot instead, as a superuser or a role with + `REPLICATION`. Peeking does not consume anything. `` is the commit LSN `` + from the error with its low hex part incremented by one (`0/40C5318` becomes + `0/40C5319`); the error prints it: + + ```sql + SELECT xid, lsn, encode(data, 'escape') AS message + FROM pg_logical_slot_peek_binary_changes( + '', '', NULL, + 'proto_version', '1', 'publication_names', ''); + ``` + + Use the values of `logrepl.slotName` and `logrepl.publicationName` (defaults + `conduitslot` and `conduitpub`). + The output can start with earlier transactions that the pipeline already delivered + but Postgres has not yet been told about (the connector reports its position + periodically). They can contain the same value as the stuck delete, so do not read + them. Read only the transaction (`xid`) that contains the `D` row whose `lsn` is the + WAL LSN `` in the error. + Each row is one pgoutput message. `B` and `C` begin and commit the transaction. `R` + names a table and its columns. `I`, `U` and `D` are the changes. A `K` tuple follows + the table's object id: a 2-byte column count, then one entry per column, `n` for NULL + or `t`, a 4-byte length and the text value. For this error the only non-null columns + are those of the identity index, so the readable text is the value to look for, for + example `222` in a `D` row that shows `K ... n t ... 222 n`. Postgres only accepts a + unique index as a replica identity, so that value identifies the row in the + destination. Only tables in the publication appear, which are the tables the pipeline + captures. + 4. Delete that row from the destination yourself. The change has no key, so the pipeline + cannot do it. Then work out what else step 5 would skip. It skips every change the + peek lists before the stuck one, and the other changes of the stuck transaction. + Postgres has not been told they are done, and the pipeline may have read them + without the destination writing them before it stopped. Get the source connector's + last acknowledged position from Conduit: + `GET /v1/connectors/`, field `sourceState.position`. It is + base64 of JSON, for example + `{"version":2,"type":2,"last_lsn":"0/40845E8","tx_commit_lsn":"0/40846E8","tx_seq":1}` + (decode it with `base64 -d`). Conduit sets it only when the destination acknowledges a + record, never when the source reads one, so `last_lsn` is the WAL LSN of the last + change the destination has confirmed. If Conduit was restarted since the pipeline + stopped, it reports the last position it saved, which can be slightly older; that errs + on the safe side. Find the peek row of kind `I`, `U` or `D` whose `lsn` equals + `last_lsn`. Ignore `B` and `C` rows: a `C` row can show the same number as the first + change after it. If several rows share that `lsn` (a `COPY` or another statement + that changes many rows can do that), use `tx_seq`. It numbers the transaction's + `I`, `U`, `D` and `T` (truncate) rows in the order listed, starting at 1 after its + `B` row. `R` (relation), `Y` (type) and `O` (origin) rows are not counted. The row + with that number is the one to match, and it is an `I`, `U` or `D` row. The matched row + itself and everything listed before it are in the destination. For each change + listed after it, up to the stuck one and including the rest of the stuck transaction, + check the destination for its rows and apply any that are missing by hand (`I` and + `U` carry the full new row; `D` carries the identity columns). If no row matches, + treat every listed change as unconfirmed. A position without `last_lsn` (a pipeline + still in its snapshot) matches nothing. + 5. Skip the transaction. Use the same `` as in step 3. Writing it as a literal is + deliberate: `''::pg_lsn + 1` only works from Postgres 14. + - Postgres 11 or later: + `SELECT pg_replication_slot_advance('', '');` + - Postgres 10 has no such function. Consume the changes up to `` instead. This + skips exactly the same changes and needs no new snapshot: + `SELECT count(*) FROM pg_logical_slot_get_binary_changes('', '', NULL, 'proto_version', '1', 'publication_names', '');` + + Advancing to `` is what skips the transaction. Advancing to the change LSN + (``) or to the commit LSN `` itself does not: Postgres replays the transaction + and the pipeline fails on the same delete. + 6. Start the pipeline. + + **Skipping the transaction skips that change, and it skips the whole transaction, + including its other changes to captured tables.** Skipping data is your decision and the + connector cannot check it for you. + + ### Unchanged TOASTed Columns + + Postgres stores large field values (`text`, `bytea`, `jsonb`, and other + variable-length types past a size threshold) out-of-line using + [TOAST](https://www.postgresql.org/docs/current/storage-toast.html). When a row is + updated but a TOASTed column is **not** changed, Postgres omits that column's value + from the logical-replication message to avoid re-transmitting large unchanged data. + + For such columns, this connector **omits the field from the record payload** rather + than emitting it as `NULL`. Emitting `NULL` would be silent data corruption — a + downstream sink cannot distinguish "the value was set to NULL" from "the value did + not change". Omission preserves that distinction: absence means "unchanged". + + What this means for your destination: + + - **Partial-update / upsert sinks** (including this connector's own Postgres + destination, which builds its `ON CONFLICT DO UPDATE SET` clause only from the + fields present in the payload) handle omission correctly — an absent field is left + untouched in the target row. + - **Full-row / full-document replace sinks** must treat an absent field as + "unchanged" and preserve the existing target value. If a sink cannot do this, set + `REPLICA IDENTITY FULL` on the source table. Under `REPLICA IDENTITY FULL` the old + row image carries every column, and this connector backfills unchanged TOASTed + values from it so the payload is always complete. + + ### Value Representation + + Values do not depend on the time zone of the machine running the connector or on the + version of the Postgres driver. The same row gives the same record everywhere, for + snapshot and CDC. + + | Postgres type | Without an Avro schema | With an Avro schema (`logrepl.withAvroSchema`, the default) | + | --- | --- | --- | + | `timestamp` | The wall clock stored in Postgres, as a time with no offset information (`2024-01-02T03:04:05Z` for `2024-01-02 03:04:05`). | `local-timestamp-micros` of the same wall clock. | + | `timestamptz` | The instant, in UTC (`Z`). | `timestamp-micros`. | + | `date` | Midnight UTC of the date. | `date`. | + | `tsvector` | Postgres text form, `'a':1 'cat':3A,4B`. | `string`, same text. | + | `infinity`, `-infinity` | The strings `infinity` and `-infinity`. | The source halts with error code `postgres.source.temporal_not_representable`. | + | Years before 1 BC or after 9999 AD | Postgres text, for example `0044-03-15 12:00:00 BC`. | As above, up to what 64-bit microseconds hold; beyond that the same error. | + | Arrays, ranges, multiranges of these | Element and bound values as above. | Not supported by the Avro schema. | + + A range or multirange is a JSON object `{"Lower":...,"Upper":...,"LowerType":105, + "UpperType":101,"Valid":true}`, and a multirange is an array of them. `LowerType` and + `UpperType` are the ASCII codes of the bound kind: `105` (`i`) inclusive, `101` (`e`) + exclusive, `85` (`U`) unbounded (the bound is `null`), `69` (`E`) empty range. Bounds + follow the rules above. Arrays of `tsmultirange`, `tstzmultirange` and `datemultirange` + are emitted as the Postgres text of the array. + + If the source halts with `postgres.source.temporal_not_representable`, set + `logrepl.withAvroSchema` to `false`. Removing the row does not help a change that is + already in the WAL, because it is read again after every restart. + + ### Numeric Columns + + A `numeric` column with a precision and scale, `numeric(p,s)`, is an Avro + `decimal(p,s)` and its values are `big.Rat`. + + A `numeric` column **without** a precision and scale can hold any value Postgres + can store, which no Avro `decimal` can. With `logrepl.withAvroSchema: true` (the + default) it is an Avro `string` holding the exact text of the value, with its + scale (`1.50` stays `1.50`), and `NaN`, `Infinity` and + `-Infinity` as those three words: **a consumer of such a column must handle + them**, a plain decimal parser will reject them. Earlier versions used `decimal(65535, 65531)`, which could not encode a + value of 10000 or more and stopped the pipeline. The first record after an + upgrade registers a new schema version for the table; consumers must read the + field as a string. This connector's destination writes the string to a `numeric` + column exactly. + + **Known issue:** a zero with a display scale reads as `0` in the snapshot and as + `0.000` (its stored text) in change data capture. The values are numerically + equal; only the text differs. Tracked in + [#391](https://github.com/ConduitIO/conduit-connector-postgres/issues/391). + + So the Go type of an unconstrained `numeric` depends on the setting: a `string` + with `logrepl.withAvroSchema: true`, a `big.Rat` with `false`. Changing a column + between `text`/`varchar` and an unconstrained numeric is accepted by the + `evolve` schema-drift policy only when `withAvroSchema` is `true` (the Avro + shape does not change); with `false` the value's type changes, so it halts. + + Without an Avro schema, a `numeric` is a `big.Rat`; `NaN`, `Infinity` and + `-Infinity` are the strings `"NaN"`, `"Infinity"` and `"-Infinity"`, and the same + applies to the elements of `numeric[]` (finite elements are unchanged). A `NULL` + stays `NULL`. + + `NaN` in a `numeric(p,s)` column (the infinities cannot be stored there) cannot + be an Avro decimal, and the connector never replaces it with `NULL` or a number: + the source stops with the error code `postgres.source.numeric_not_representable`, + which names the table, the column, the value and the row's key. To continue: + + - **Change data capture:** restart once with `logrepl.withAvroSchema: false` + from the same position. The value is delivered as a string. **Then set it + back to `true` as soon as that record has been delivered.** Without an Avro + schema a finite `numeric` is a `big.Rat`, which a standalone connector + cannot send to Conduit (#378): the next finite numeric would stop the + pipeline. Built-in pipelines are not affected. + On a standalone connector the SDK also fails the whole read batch on the + first record it cannot send, so if the NaN/Infinity record or its batch + contains any finite numeric, it is never delivered with Avro off and this + workaround does not help (#378 is the fix). While Avro is off, + JSON-serializing destinations receive a finite numeric as the string `"3/2"`. + - **Snapshot:** `UPDATE` the row to a finite value (or `NULL`) and restart. + - Removing the row, or adding a `CHECK` constraint, does not help for a change + that is already in the WAL: it is read again after every restart. Do not drop + the replication slot to get past it, that loses the position. A constraint + only keeps new bad values out. + + ### Limitations + + - **`TRUNCATE` is not propagated.** OpenCDC has no truncate operation, so a + `TRUNCATE` on a published table produces no record: rows it removed remain in + the destination. If the publication publishes `truncate` (the one the + connector creates does by default), the connector logs a `WARN` ("TRUNCATE was NOT + propagated") naming the table and the LSN, at most once a minute per table. + A restricted or user-supplied publication (for example + `publish = 'insert, update, delete'`) never sends the message, so there is + no warning and no other signal. The `suppressed` field counts the TRUNCATEs + of that table the rate limit swallowed since the previous warning; when + another table's warning evicts an expired entry that still holds a count, a + one-line summary with that count is logged. Treat the log lines as a signal, + not a precise counter: a count still pending when the process exits is + lost. Inserts, updates and deletes around the `TRUNCATE` are delivered as + usual. A configurable policy for TRUNCATE is planned. Until then, treat the + warning as a signal to reconcile the destination by hand. + + ## Destination + + The Postgres Destination takes a Conduit record and stores it using a SQL statement. + The Destination is designed to handle different payloads and keys. Because of this, + each record is individually parsed and upserted. + + ### Handling record operations + + Based on the `Operation` field in the record, the destination will either insert, + update or delete the record in the target table. Snapshot records are always inserted. + + If the target table already contains a record with the same key as a record being + inserted, the record will be updated (upserted). This can overwrite and thus potentially + lose data, so keys should be assigned correctly from the Source. + + If the target table does not contain a record with the same key as a record being + deleted, the record will be ignored. + + If there is no key, the record will be simply appended. + + ### Keys, columns and acknowledgements + + - **Upsert key.** With a structured record key (or a raw key that is a JSON object), + a create, update or snapshot record is `INSERT ... ON CONFLICT () DO + UPDATE SET` every other field of the record. The key columns must carry a primary + key or unique constraint, otherwise Postgres rejects the statement. When a record has + no field besides its key columns (a table whose columns are all key columns), the + statement is `ON CONFLICT DO NOTHING`, so a redelivery is a no-op. + - **Columns.** Each record is written with exactly its own fields. A field the table has + no column for fails the write; the destination does not create or alter tables. A + field the record lacks is not written: an insert gets the column default, an existing + row keeps its value. A create, update or snapshot record without fields fails with + `postgres.destination.payload_empty`. + - **Key values** must be one non-null scalar per key column, for upserts and deletes. A + null value fails with `postgres.destination.key_column_missing` (a delete would match + every row with a NULL there, an upsert would append a new row on every redelivery), and + a list or object fails with `postgres.destination.key_value_not_scalar` (a delete would + match several rows or none). + - **Deletes** need a key and delete the rows matching every key column. + - **Raw JSON** keys and payloads are decoded without going through `float64`: an integer + literal is an `int64` (or `uint64`), and any other number keeps its literal, so + `numeric`, `text` and `json`/`jsonb` columns store it exactly. Integer columns take a + whole value in any form (`1e3`, `7.0`) and reject a fraction; `real` and `float8` + columns store the nearest float. Structured data that arrives over the plugin protocol + has already passed through a double (ConduitIO/conduit#3100), so integers above 2^53 + sent that way are rounded before the destination sees them. + - **Names** of the table and columns are quoted as Postgres identifiers, so reserved + words, upper case and double quotes work. A name with a NUL byte or longer than 63 bytes + fails with `postgres.destination.invalid_identifier`, because Postgres would address a + different table or column. The table name is one identifier: a dot is part of the name, + not a schema separator. + - **Acknowledgements.** A write is one implicit transaction. If any statement or the + commit fails, nothing from that write is stored and no record of it is acknowledged; + the records are redelivered (at-least-once). Keyed upserts make redelivery + idempotent; keyless inserts append again. + + ### Schema-drift markers + + When the Postgres source halts on schema drift it emits a drift marker: a create + record with no key and no payload, with the metadata `postgres.schema.drift=true`. + The destination skips and acknowledges a marker without writing a row, but only + when the record's position proves the Postgres source produced it as a marker for + the table named in `postgres.schema.drift.table`. The metadata alone is not proof, + because any processor or upstream system can set it. + + - A record that carries the metadata but not a marker's position, and has a key + or a payload, is written as an ordinary record. One WARN per batch reports it. + - A record shaped like a marker (no key, no payload) without a marker's position + stops the pipeline with `postgres.destination.drift_marker_unverified`. Nothing + in that batch is written or acknowledged. + - A proven marker that a processor added a key or fields to is still skipped as + a marker, with a WARN that names the added fields. + + A marker loses its position when it passes through another system, such as Kafka + or a file, because records read from there carry that system's position. In a + Postgres to Kafka to Postgres setup, drop drift markers before the hop with a + filter processor on the `postgres.schema.drift` metadata. + + The proof relies on the engine not letting processors change a record's + position. Conduit's classic engine refuses that; the arch-v2 engine lets a + processor rewrite positions, so there a processor could forge a marker. The + forged record would be skipped, which a processor with that power could do to + any record anyway. + ### Column type changes on a running pipeline + + The destination reuses prepared statements (pgx's `cache_statement` query + mode, the fast mode) and checks a per-table schema fingerprint with every + batch. After `ALTER TABLE ... ALTER COLUMN ... TYPE` the fingerprint changes, + the batch is rolled back, the statement cache is cleared, and the batch is + retried with the new column types. A value the new type cannot hold exactly + fails the write instead of being converted; a widening change is written + type-exactly without an error. + + This keeps cached statements' single round trip per batch. On one dedicated + machine (benchi `destination-exec-mode`: keyed upserts into a two-column + table) describing every statement instead cost 35% fewer rows per second at + `sdk.batch.size: 1`, the default, and 16% fewer at 1000. `exec` and + `simple_protocol`, which never cache parameter types (PgBouncer in + transaction mode needs one of them), are kept when the URL sets them, as is + `describe_exec`. + + ### Numeric columns + + Decimal values (`big.Rat`) are sent to Postgres as exact decimal strings, and + Postgres applies the column's precision and scale itself when it stores the + value. The destination never reads or caches a column's scale, so an + `ALTER TABLE ... ALTER COLUMN ... TYPE numeric(p,s)` takes effect on the next + write without restarting the pipeline, and the table is resolved through the + connection's `search_path` like the rest of the statement. + + Postgres rounds half away from zero when a value has more fractional digits + than the column's scale, and rejects a value that exceeds the column's + precision (`numeric field overflow`). A rejected batch is not acknowledged. + + A decimal that has no finite expansion, such as 1/3, is truncated toward zero + after 1001 fractional digits before it is sent. So is a decimal that terminates + but needs more than 16383 fractional digits, which Postgres rejects on every + numeric column. Postgres allows a column scale of at most 1000, so the + truncation never changes the result of the column's rounding. An unconstrained + `numeric` column stores all 1001 digits. + + The destination no longer checks that the target column is numeric. The general + rule: any column whose text input accepts the decimal string stores it. + Postgres converts the string, and the examples below were checked on + Postgres 17.5: + + - `integer`, `smallint`, `bigint`: stored if the value is whole. A fractional + value is rejected, also on a destination that was already running when the + column was changed to an integer type (see "Column type changes on a running + pipeline"). + - `real`, `float8`: stored as the nearest binary float. + - `money`: rounded to the locale's fraction digits (1/3 is stored as `$0.33`). + - `oid`: stored if the value is a whole number (7 is stored as `7`). + - `text`: stored as the decimal string. `varchar(n)` and `char(n)`: rejected + with 22001 if the string is longer than `n`. + - `boolean`: 1 is stored as `true` and 0 as `false`; other values are rejected. + - `date`: 20261012 is stored as `2026-10-12`. + - `timestamp`: 20261012 is stored as `2026-10-12 00:00:00`. + - `timestamptz`: 20261012 is stored as `2026-10-12 00:00:00+00` (session time + zone UTC). + - `interval`: 12345 is stored as `03:25:45`. + - `json`, `jsonb`: 1/4 is stored as the JSON number `0.25`. + - `bytea`: 5/2 is stored as `\x322e35`, the ASCII bytes of `2.5`. + - `bit(n)`: stored if the decimal string is exactly `n` binary digits (101 is + stored as `101` in a `bit(3)`). + - `time`, `inet`: rejected. + + Every accepted conversion to a non-numeric type is a silent coercion if the + schema maps a numeric field to the wrong column, so check the mapping. + version: v0.15.0 + author: Meroxa, Inc. + source: + parameters: + - name: tables + description: |- + Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2". + Use "*" if you'd like to listen to all tables. + type: string + default: "" + validations: + - type: required + value: "" + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + type: string + default: "" + validations: + - type: required + value: "" + - name: cdcMode + description: CDCMode determines how the connector should listen to changes. + type: string + default: auto + validations: + - type: inclusion + value: auto,logrepl + - name: logrepl.autoCleanup + description: |- + LogreplAutoCleanup determines if the replication slot and publication should be + removed when the connector is deleted. + type: bool + default: "true" + validations: [] + - name: logrepl.import.debeziumOffset + description: |- + LogreplImportDebeziumOffset starts the pipeline from a Debezium Postgres + connector's committed offset instead of a snapshot. It is written by + `conduit migrate kafka-connect --cutover`, not by hand. It applies only + while the pipeline has no position: the first acknowledged record writes + an ordinary position and the setting is ignored from then on (remove it + once the import is complete). Requires snapshotMode "never" and cdcMode + "logrepl", and an existing slot and publication, which import mode + never creates. Requires Conduit v0.19.0 or later, which acks a record + only after its position is persisted. + type: string + default: "" + validations: [] + - name: logrepl.publicationName + description: |- + LogreplPublicationName determines the publication name in case the + connector uses logical replication to listen to changes (see CDCMode). + type: string + default: conduitpub + validations: [] + - name: logrepl.schemaDrift.policy + description: |- + LogreplSchemaDriftPolicy determines how the connector handles schema + drift (a DDL that changes a table's shape during CDC; DBZ-3 B1): + - "halt" (default): emit a drift marker record, checkpoint before it, + and stop with a terminal error until the pipeline is restarted to + approve the change. + - "evolve": accept additive changes silently; incompatible (narrowing) + changes still halt. + "dlq" is reserved for a future version and rejected at validation. + type: string + default: halt + validations: [] + - name: logrepl.slotName + description: |- + LogreplSlotName determines the replication slot name in case the + connector uses logical replication to listen to changes (see CDCMode). + Can only contain lower-case letters, numbers, and the underscore character. + type: string + default: conduitslot + validations: + - type: regex + value: ^[a-z0-9_]+$ + - name: logrepl.withAvroSchema + description: |- + WithAvroSchema determines whether the connector should attach an avro schema on each + record. + type: bool + default: "true" + validations: [] + - name: snapshot.fetchSize + description: Snapshot fetcher size determines the number of rows to retrieve at a time. + type: int + default: "50000" + validations: [] + - name: snapshotMode + description: SnapshotMode is whether the plugin will take a snapshot of the entire table before starting cdc mode. + type: string + default: initial + validations: + - type: inclusion + value: initial,never + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is read from the source. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets read from the source. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.schema.context.enabled + description: |- + Specifies whether to use a schema context name. If set to false, no schema context name will + be used, and schemas will be saved with the subject name specified in the connector + (not safe because of name conflicts). + type: bool + default: "true" + validations: [] + - name: sdk.schema.context.name + description: |- + Schema context name to be used. Used as a prefix for all schema subject names. + If empty, defaults to the connector ID. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and encode the record key with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.key.subject + description: |- + The subject of the key schema. If the record metadata contains the field + "opencdc.collection" it is prepended to the subject name and separated + with a dot. + type: string + default: key + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and encode the record payload with a schema. + type: bool + default: "false" + validations: [] + - name: sdk.schema.extract.payload.subject + description: |- + The subject of the payload schema. If the record metadata contains the + field "opencdc.collection" it is prepended to the subject name and + separated with a dot. + type: string + default: payload + validations: [] + - name: sdk.schema.extract.type + description: The type of the payload schema. + type: string + default: avro + validations: + - type: inclusion + value: avro + destination: + parameters: + - name: url + description: |- + URL is the connection string for the Postgres database. + With TLS, SCRAM authentication does not use channel binding by default + (`channel_binding=disable`); add `channel_binding=require` to the URL to + require SCRAM-SHA-256-PLUS. + The destination reuses prepared statements (cache_statement) and checks a + per-table schema fingerprint with every batch, so a column type change on + a running pipeline invalidates the cache and the next write sees the new + type — fast and correct. Set `default_query_exec_mode` to `describe_exec`, + `exec` or `simple_protocol` to override; `cache_statement` and + `cache_describe` are also guarded and safe. + type: string + default: "" + validations: + - type: required + value: "" + - name: table + description: Table is used as the target table into which records are inserted. + type: string + default: '{{ index .Metadata "opencdc.collection" }}' + validations: [] + - name: sdk.batch.delay + description: Maximum delay before an incomplete batch is written to the destination. + type: duration + default: "0" + validations: [] + - name: sdk.batch.size + description: Maximum size of batch before it gets written to the destination. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.burst + description: |- + Allow bursts of at most X records (0 or less means that bursts are not + limited). Only takes effect if a rate limit per second is set. Note that + if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch + size will be equal to `sdk.rate.burst`. + type: int + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.rate.perSecond + description: Maximum number of records written per second (0 means no rate limit). + type: float + default: "0" + validations: + - type: greater-than + value: "-1" + - name: sdk.record.format + description: |- + The format of the output record. See the Conduit documentation for a full + list of supported formats (https://conduitdata.io/docs/using/connectors/configuration-parameters/output-format). + type: string + default: opencdc/json + validations: [] + - name: sdk.record.format.options + description: |- + Options to configure the chosen output record format. Options are normally + key=value pairs separated with comma (e.g. opt1=val2,opt2=val2), except + for the `template` record format, where options are a Go template. + type: string + default: "" + validations: [] + - name: sdk.schema.extract.key.enabled + description: Whether to extract and decode the record key with a schema. + type: bool + default: "true" + validations: [] + - name: sdk.schema.extract.payload.enabled + description: Whether to extract and decode the record payload with a schema. + type: bool + default: "true" + validations: [] diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/.metadata.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/.metadata.yaml new file mode 100644 index 0000000..4bbda75 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/.metadata.yaml @@ -0,0 +1,2 @@ +fetchedAt: 2026-10-19T08:25:27.331631587Z +commitSHA: e83f1050221e2c27ee6fde18890a8dabfdb216b7 diff --git a/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/connector.yaml b/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/connector.yaml new file mode 100644 index 0000000..2a18988 --- /dev/null +++ b/static/connectors/github.com/ConduitIO/conduit-connector-s3@v0.10.0/connector.yaml @@ -0,0 +1,657 @@ +version: "1.0" +specification: + name: s3 + summary: An S3 source and destination plugin for Conduit, written in Go. + description: |- + The S3 connector is one of [Conduit](https://github.com/ConduitIO/conduit) + builtin plugins. It provides both, a source and a destination S3 connectors. + + ## Source + + The S3 Source Connector connects to a S3 bucket with the provided + configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and + `aws.region`. If the bucket doesn't exist, or the permissions fail, then an + error will occur. After that, the `Open` method is called to start the + connection from the provided position. + + ### Change Data Capture (CDC) + + This connector implements CDC features for S3 by scanning the bucket for changes + every `pollingPeriod` and detecting any change that happened after a certain + timestamp. These changes (update, delete, create) are then inserted into a + buffer that is checked on each Read request. + + * To capture "delete" and "update", the S3 bucket versioning must be enabled. + * To capture "create" actions, the bucket versioning doesn't matter. + + #### Position Handling + + The connector goes through two modes. + + * Snapshot mode: which loops through the S3 bucket and returns the objects that + are already in there. The _Position_ during this mode is the object key + attached to an underscore, an "s" for snapshot, and the _maxLastModifiedDate_ + found so far. As an example: "thisIsAKey_s12345", which makes the connector + know at what mode it is and what object it last read. The + _maxLastModifiedDate_ will be used when changing to CDC mode, the iterator + will capture changes that happened after that. + + * CDC mode: this mode iterates through the S3 bucket every `pollingPeriod` and + captures new actions made on the bucket. the _Position_ during this mode is + the object key attached to an underscore, a "c" for CDC, and the object's + _lastModifiedDate_ in seconds. As an example: "thisIsAKey_c1634049397". This + position is used to resume from the last record returned. + + **CDC mode is listing based and best effort, bounded by `lookbackPeriod`.** + It detects changes by listing the bucket, which cannot be lossless: an + object that becomes visible in a listing late can be missed. Lossless S3 + change capture needs bucket event notifications, which this connector does + not use. Every poll selects the objects modified since `lookbackPeriod` + (default 5m) before the newest object seen so far, and skips versions it + already delivered, so objects that become visible late, within that + window, are picked up: S3 stamps an object with the time its upload began, + a multipart upload with the time it was initiated, and lists in key order. + The object store's own timestamps decide, so the clock of the machine + running Conduit does not matter, with one exception: the newest timestamp + is capped at this machine's clock, so one bogus timestamp from the future + (a store clock that stepped forward, or a misbehaving endpoint) cannot hide + the objects written after it. The cap lowers the window's start, which + causes redelivery. If this machine's clock is behind the store's by more + than `lookbackPeriod` the source logs a warning with the code + `s3.source.host_clock_behind_store`: the window is stretched by the skew. + + Delivery is at-least-once. After a restart the objects modified within + `lookbackPeriod` of the newest object are delivered again. The snapshot + hands over to CDC from the newest object that existed when the snapshot + started (found by one extra listing of the prefix, one request per 1000 + keys, at the start of the snapshot), minus `lookbackPeriod`, so CDC + redelivers the objects written while the snapshot ran, however long it + took, plus those within `lookbackPeriod` before it started. On a bucket + that was bulk loaded recently this can be every object of the load. + + The versions remembered for deduplication are limited to about 256 MiB + (peak memory of a poll is about 1.5 times that, candidates included). If + the objects inside the window need more, the window is **shortened** + instead of stopping the source: the oldest remembered entries are + forgotten by whole seconds and the window starts just above them. The + source logs a warning with the code `s3.source.dedupe_window_shrunk` + naming the effective lookback and the number of forgotten entries. While + the window is shortened, an object that becomes visible late by more than + the effective lookback can be missed, so the tolerance for late-visible + objects is lower than `lookbackPeriod`; lower `lookbackPeriod` to avoid + the shrinking. The source stops, with the error + `s3.source.dedupe_window_overflow` (after a warning), only if the versions + sharing a single second of `LastModified` do not fit, or when entries + stamped at or after this machine's present second (a store clock that is + ahead, or bogus timestamps from the future) take the budget: a shortened + window never starts later than the present second, because that would hide + the objects written from now on, so the source stops instead (the message + names the clock skew and says to fix the clock). A store clock that is ahead + by less than `lookbackPeriod` is normal and does not stop the source as long + as the entries stamped in the present and the future fit the budget. A + future-stamped object can therefore raise the start of the window at most + up to the present second, never past it. The shortened window is not + remembered across restarts: after a restart the window is rebuilt from the + position and shortened again, so a restart delivers the window's objects + again (duplicates, never a skip). When more objects share a second than + one poll can hand over, they are delivered over several polls, oldest + first, and the newest tracked entries may be dropped from tracking to make + room; those are delivered again later (duplicates). A catch-up after + downtime is delivered in bounded batches, oldest first. A resume position + with a timestamp in the future, which only an older release could have + written, is clamped to this machine's clock with a warning. + + Known limits (listing cannot see these, or sees them too late): + * an upload that takes longer than `lookbackPeriod` to complete is missed + when newer objects were seen in the meantime (multipart uploads carry + their initiation time); + * replica buckets whose replication lag exceeds `lookbackPeriod`; + * an identical re-put of an object within the same second is not seen as a + change; + * in an unversioned bucket an overwrite is reported as a create (there is + no older version to compare with) and a delete is not reported; + * in a versioned bucket intermediate versions that appear and are + superseded within one poll are not emitted, only the latest; removing a + delete marker (undelete) and permanently deleting the latest version are + not emitted; + * a GetObject that fails for another reason than the object having been + deleted, for example an object archived to Glacier or Deep Archive, + stops the source with that error. + + Objects deleted between being listed and being read are skipped, in the + snapshot and in CDC. Values of `lookbackPeriod` above 1h log a warning and + above 24h are rejected. + + ### Record Keys + + The S3 object key uniquely identifies the objects in an Amazon S3 bucket, which + is why a record key is the key read from the S3 bucket. + + ## Destination + + The S3 Destination Connector connects to an S3 bucket with the provided + configurations, using `aws.bucket`, `aws.accessKeyId`,`aws.secretAccessKey` and + `aws.region`. If the permissions fail, the connector will not be ready for + writing to S3. + + ### Writes and delivery + + Every batch the SDK hands to the destination (`sdk.batch.size` records, or + fewer after `sdk.batch.delay`; one record per batch by default) is written + with one `PutObject` per object, and the records are acknowledged only after + `PutObject` succeeded. Delivery is at-least-once: a batch that is retried + after a failure or a restart is written again as a new object, so a reader + can see a record twice, never zero times. Object names are + `.`, not deterministic. + + ### Formats + + - `json`: one JSON object per line with `Operation`, `Position`, `Payload`, + `Key` and `Metadata`; payload and key are JSON strings. A record whose key, + payload, position or metadata is not valid UTF-8 is refused with the error + `s3.destination.invalid_utf8` instead of being stored with replaced bytes. + - `parquet`: the same fields as string columns. + - `bytearray`: the payload bytes of every record, unchanged, each followed + by `format.bytearray.separator` (default a newline). Structured payloads + are written as their JSON encoding. A record without a payload (a delete, + a Kafka tombstone) or with an empty one is refused with the error + `s3.destination.null_value`; drop such records with a `filter` processor + if they are expected. A Kafka tombstone arrives as an empty value, so an + empty Kafka value cannot be told apart from it and is refused too. The + separator is not escaped inside payloads: a payload that contains it + cannot be split back unambiguously (as with Kafka Connect's + `ByteArrayFormat`). + + `compression` is `none` or `gzip` for `json` and `bytearray` (gzip appends + `.gz` to the extension and sets the content type to `application/gzip`), and + `none`, `gzip` (default) or `snappy` for `parquet`, where it is the column + codec. + + When a record is refused, the records before it in the batch are written as + one object and acked, the refused record and the ones after it are nacked + (to the DLQ, if one is configured). + + ### Coming from the Kafka Connect S3 sink + + - `ByteArrayFormat` is `format: bytearray`; `format.bytearray.separator` and + `format.bytearray.extension` have the same meaning (Kafka Connect's + extension default is `.bin` as well). + - `JsonFormat` for topics with JSON values is `format: bytearray` with + `format.bytearray.extension: .json`: the stored bytes are the record + values. Kafka Connect re-serializes values through its converter, so + whitespace and key order can differ, and `schemas.enable=true` envelopes + are not produced. + - `s3.compression.type: gzip` is `compression: gzip`; `parquet.codec` is + `compression` with `format: parquet` (Kafka Connect defaults to snappy, + this connector to gzip). + - `behavior.on.null.values` other than `fail` has no counterpart. + + ### Server-side encryption + + By default the destination sends no server-side encryption header, so the + bucket's default encryption applies. On AWS every bucket encrypts new + objects with SSE-S3 unless it is configured otherwise, and a bucket whose + default is SSE-KMS gets SSE-KMS objects. An S3-compatible store without a + default encryption configuration stores objects unencrypted. When + `aws.serverSideEncryption` is empty, `Open` reads the bucket's default + encryption (`GetBucketEncryption`, permission + `s3:GetEncryptionConfiguration`) and logs a warning with the code + `s3.destination.bucket_unencrypted` if there is none. It never refuses to + start; without the permission the check is skipped (debug log). + + Set `aws.serverSideEncryption` to `AES256`, `aws:kms` or `aws:kms:dsse` to + request a mode explicitly, and `aws.sseKmsKeyId` to pick the KMS key for the + two KMS modes. An explicit mode overrides the bucket default. + + Upgrading from v0.9.5 or earlier: those versions sent `AES256` on every + write, whatever the bucket default was, so an SSE-KMS bucket default was + silently replaced by SSE-S3 (#1047). To keep the old behavior, set + `aws.serverSideEncryption: AES256`. + + ### Partitioning (Kafka Connect S3 sink layout) + + `partitioner` selects the directory layout under `prefix`: + + | `partitioner` | Object key | + | --- | --- | + | `none` (default) | `/.` | + | `default` | `//partition=/.` | + | `field` | `//=/.../.` | + | `time`, `hourly`, `daily` | `//