Skip to content

Fix #112: Add the sqlserver driver for xport --source sql - #135

Open
vmose wants to merge 1 commit into
cerberauth:mainfrom
vmose:fix-issue-112
Open

vmose wants to merge 1 commit into
cerberauth:mainfrom
vmose:fix-issue-112

Conversation

@vmose

@vmose vmose commented Oct 28, 2026

Copy link
Copy Markdown

Add the sqlserver driver for export --source sql

Registers Microsoft SQL Server (and Azure SQL) as an engine for iamigrate export --source sql, on top of #90.

What changed

  • --driver sqlserver through go-mssqldb, with the engine's standard DSN (URL or key=value form), handed to the driver untouched.
  • Streaming: rows are read one at a time and handed to the mapping in batches, never loaded whole.
  • TLS is whatever the DSN says (encrypt, TrustServerCertificate, certificate, hostNameInCertificate). No iamigrate-side settings.
  • varbinary columns are read as bytes. Printable ASCII (a bcrypt string stored as bytes) stays text; anything else is hex, so password_encoding / password_salt_encoding apply as for flat files. uniqueidentifier comes out as a lower-case UUID rather than SQL Server's byte order.
  • Microsoft Entra authentication: a fedauth= parameter in the DSN switches to go-mssqldb's azuread driver; without it, a SQL login uses the plain driver.
  • Snapshot isolation when the database has it on (sys.databases.snapshot_isolation_state); otherwise the default isolation, with a notice in the progress line.
  • Tests: a live suite against SQL Server in Docker (varbinary/GUID, snapshot on and off, TLS from the DSN), plus unit tests for Entra DSN detection. New sqlserver CI job using a service container; .docker/sqlserver/docker-compose.yml for running it locally.
  • Docs: docs/guides/export/sql.mdx with an example query, TLS, Entra and isolation sections; CLI reference updated.

Also in this PR (needed by the above)

If #90 is not merged yet, this PR carries its core: the sql connector and driver registry, --driver/--dsn/--query, the SQLite driver (pure Go, no cgo), and flatfile.ExportRows, a refactor of flatfile's Export so the same mapping pipeline can run per batch. Export's behaviour is unchanged.

Not included

Reviewer notes

  • azuread pulls in the Azure SDK, which noticeably enlarges the binary. If that is a concern, the Entra driver can move behind a build tag.
  • Entra auth itself isn't tested in CI (it can't be in Docker); only the DSN routing is.
  • The DSN is redacted from every error returned by the connector. The live TLS test asserts this for a refused connection.

Test plan

go test ./...                                   # SQLite + unit tests

docker compose -f .docker/sqlserver/docker-compose.yml up --wait
SQLSERVER_DSN='sqlserver://sa:Iamigrate-Passw0rd@127.0.0.1:1433?database=master&encrypt=true&TrustServerCertificate=true' \
  go test ./pkg/connector/sql/sqlserver/... -run TestLive -v
docker compose -f .docker/sqlserver/docker-compose.yml down -v

Closes #

Comment thread go.mod
require (
github.com/brianvoe/gofakeit/v7 v7.17.1
github.com/cerberauth/x v0.0.0-20260901215913-9d7750841dbb
github.com/microsoft/go-mssqldb v1.9.2

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go.sum isn’t in the diff. Please run go mod tidy and commit both files. Consider putting azuread behind a build tag so the default binary doesn’t carry the Azure SDK.

Comment thread sqlserver-driver.patch

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 1,400-line patch of the PR’s own changes looks committed by accident. Please remove it.

}
}

func bytesText(b []byte) string {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Treating all-printable-ASCII bytes as text will misread some raw salts. A 4-byte salt has about a 1.9% chance and an 8-byte one about 0.04%, so over a million users that’s hundreds of rows. Those rows get a wrong salt, so the migrated passwords won’t verify. Please always hex-encode VARBINARY/BINARY/BLOB/bytea columns, and have users CAST to text for stored bcrypt strings.

- **`--query`** is the SQL, or `@file.sql` to read it from a file. It returns one row per user.
- **`--mapping`** renames the columns into the ones iamigrate reads, as for a [flat file](../../../reference/cli/prepare/). A query that already names its columns `source_id`, `email`, `password`... needs none.

Rows are read as they arrive and handed on in batches of 1,000, so memory doesn't grow with the number of users. The query runs in a transaction that is rolled back at the end: nothing is written.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Rolled back, nothing is written” isn’t reliable. On SQL Server, …; COMMIT; DELETE … commits its own writes. On SQLite, PRAGMA query_only can be switched off by the query itself, if the driver runs more than one statement. Please reword the guide and open SQLite with mode=ro.

manifest.NonPortableMFACounts = map[cmf.MFAType]int{}
}

rows, positions, dropped, err := exportRecord(ctx, rows, opts.Mapping)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exportRecord still keys hooks with recordKey(i), where i is the index inside the batch. So an export.record handler sees record 1…1000 again in every batch, while id.source sees the shifted numbers. Please pass offset into exportRecord and use recordKey(offset+i). Also, detectHashes only catches duplicate source_ids within one batch.

Comment thread go.mod
golang.org/x/term v0.46.0
gopkg.in/dnaeon/go-vcr.v3 v3.2.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.38.2

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need this mod?

@emmanuelgautier emmanuelgautier linked an issue Nov 1, 2026 that may be closed by this pull request
8 tasks
@emmanuelgautier

Copy link
Copy Markdown
Member

Hi @vmose,

Thanks for the contribution! I’ve left some review comments. Could you please make the requested changes when you have time?

Happy to help if anything is unclear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL source: Microsoft SQL Server

2 participants