Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Sources/Core/OutWit.Database.Core.BouncyCastle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This package provides an alternative encryption algorithm when AES-NI hardware a
## Installation

```xml
<PackageReference Include="OutWit.Database.Core.BouncyCastle" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core.BouncyCastle" Version="13.1.1" />
```

---
Expand Down
2 changes: 1 addition & 1 deletion Sources/Core/OutWit.Database.Core.IndexedDb/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ This package allows WitDatabase to run entirely in the browser with data persist
## Installation

```xml
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="13.1.1" />
```

Add the JavaScript files to your `index.html`:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,10 +164,26 @@ public void ControlTheParkedWriterAloneLosesNothingTest()
/// silent change - it is the standing evidence for why the wrapper exists.
/// </summary>
/// <remarks>
/// <para>
/// The shape of the damage varies and is deliberately not pinned: over ten runs the second writer
/// threw <c>ArgumentOutOfRangeException</c> or <c>IndexOutOfRangeException</c> nine times, and
/// once nothing threw at all and three entries were simply gone - two of them the FIRST writer's,
/// already inserted and acknowledged. The exception is the lucky outcome.
/// </para>
/// <para>
/// <b>It asserted one of those shapes anyway until 2026-08-15</b> - that the SECOND writer's key
/// was among the missing ones - which contradicted the paragraph above. It reddened CI on a branch
/// that touches no engine code and passed on a re-run of the same commit: on a loaded runner the
/// second writer did not finish inside the two seconds
/// <see cref="Outcome.SecondWriterFinishedWhileFirstWasParked"/> waits, so it landed after the
/// release and its own entry survived - while <b>207 of the first writer's were lost</b>. The
/// damage was real and larger than usual, and the case failed for having said in advance which
/// entry it would be.
/// </para>
/// <para>
/// So it asserts what it can measure on a machine whose scheduling it does not control - that two
/// writers in one leaf split damage the index - and REPORTS the shape rather than pinning it.
/// </para>
/// </remarks>
[Test]
public void ProbeConcurrentAddOverABareIndexStoreTest()
Expand All @@ -185,13 +201,20 @@ public void ProbeConcurrentAddOverABareIndexStoreTest()
// PINS A DEFECT, NOT CORRECT BEHAVIOUR. A bare StoreBTree is what CreateBTreeIndexFactory
// hands every secondary index, and it has no locking of any kind: the second writer walks
// straight into the leaf the first one is halfway through splitting, snapshots it, and
// the two then rewrite it from two different snapshots. Invert both assertions when index
// stores are serialised - nothing may be lost and no writer may throw.
// the two then rewrite it from two different snapshots. Invert this when index stores are
// serialised - nothing may be lost and no writer may throw.
Assert.That(outcome.Damage, Is.Not.EqualTo(Damage.None),
"two writers were inside the same leaf split and nothing went wrong - re-measure "
+ "before believing it");
Assert.That(outcome.MissingKeys, Does.Contain(SECOND_WRITER_KEY),
"the second writer's entry survived - the damage this probe pins has moved");

// WHOSE entries went is the weather; that acknowledged work was lost, or that a writer
// threw, is the finding. Both are already what Damage classifies, so this says out loud
// what the value has to have come from.
Assert.That(outcome.MissingKeys.Count > 0
|| outcome.FirstWriterError != null
|| outcome.SecondWriterError != null, Is.True,
"nothing is missing and nobody threw, so Damage was classified from something this "
+ "probe does not measure: " + outcome.Describe());
});
}

Expand Down
140 changes: 140 additions & 0 deletions Sources/Core/OutWit.Database.Core.Tests/ShippedReadmesTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
using System.Text.RegularExpressions;

namespace OutWit.Database.Core.Tests;

/// <summary>
/// The READMEs that ship inside the packages install the version that is being shipped.
/// </summary>
/// <remarks>
/// <para>
/// A README goes into the NuGet package, so it reaches people who never open the repository or the
/// site - and the first thing they copy out of it is the <c>PackageReference</c>. Eight of them across
/// five packages pinned <c>12.8.0</c> while every package was on 13.1.1: the number was written once,
/// per file, and nothing could notice it going stale.
/// </para>
/// <para>
/// <b>The rule is over the whole surface</b> - every README under <c>Sources</c>, every pinned
/// version in it - rather than over the one file somebody noticed, and it counts what it examined so
/// that "nothing left to find" cannot read like "the folder moved".
/// </para>
/// </remarks>
[TestFixture]
public class ShippedReadmesTests
{
#region Constants

/// <summary>An install snippet: <c>&lt;PackageReference Include="X" Version="Y" /&gt;</c>.</summary>
private static readonly Regex PACKAGE_REFERENCE =
new(@"<PackageReference\s+Include=""(?<package>OutWit\.[\w.]+)""\s+Version=""(?<version>[^""]+)""",
RegexOptions.Compiled);

/// <summary>The version a project declares: <c>&lt;Version&gt;13.1.1&lt;/Version&gt;</c>.</summary>
private static readonly Regex PROJECT_VERSION =
new(@"<Version>(?<version>[^<]+)</Version>", RegexOptions.Compiled);

#endregion

#region Tests

[Test]
public void EveryInstallSnippetNamesTheVersionThatShipsTest()
{
var versions = ProjectVersions();
var stale = new List<string>();
var examined = 0;

foreach (var readme in Readmes())
{
var text = File.ReadAllText(readme);

foreach (Match match in PACKAGE_REFERENCE.Matches(text))
{
examined++;

var package = match.Groups["package"].Value;
var pinned = match.Groups["version"].Value;

if (!versions.TryGetValue(package, out var shipping))
{
stale.Add($"{Path.GetFileName(Path.GetDirectoryName(readme))}/README.md installs "
+ $"{package}, which is not a project in this repository");
continue;
}

if (pinned != shipping)
{
stale.Add($"{Path.GetFileName(Path.GetDirectoryName(readme))}/README.md installs "
+ $"{package} {pinned}; the package is {shipping}");
}
}
}

Assert.Multiple(() =>
{
// THE SURFACE. Eight snippets across five READMEs today, and a rule that read none of
// them would pass exactly as loudly as one that read all of them.
Assert.That(examined, Is.EqualTo(8),
"the shipped READMEs carry a different number of install snippets than this rule was "
+ "measured against - check the new one, then change this number");

Assert.That(versions, Has.Count.GreaterThan(5),
"CONTROL: almost no project version was read, so nothing here is being compared");

Assert.That(stale, Is.Empty,
"these install a version that is not the one being shipped:"
+ Environment.NewLine + string.Join(Environment.NewLine, stale));
});
}

#endregion

#region Tools

/// <summary>Package name -> the version its project declares.</summary>
private static Dictionary<string, string> ProjectVersions()
{
var versions = new Dictionary<string, string>(StringComparer.Ordinal);

foreach (var project in Directory.EnumerateFiles(SourcesFolder(), "OutWit.*.csproj",
SearchOption.AllDirectories))
{
var name = Path.GetFileNameWithoutExtension(project);

if (name.EndsWith(".Tests", StringComparison.Ordinal))
continue;

var match = PROJECT_VERSION.Match(File.ReadAllText(project));

if (match.Success)
versions[name] = match.Groups["version"].Value;
}

return versions;
}

private static IEnumerable<string> Readmes() =>
Directory.EnumerateFiles(SourcesFolder(), "README.md", SearchOption.AllDirectories)
.Where(path => !path.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}",
StringComparison.Ordinal)
&& !path.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}",
StringComparison.Ordinal));

private static string SourcesFolder()
{
var directory = new DirectoryInfo(AppContext.BaseDirectory);

while (directory != null)
{
var candidate = Path.Combine(directory.FullName, "Sources");

if (Directory.Exists(candidate))
return candidate;

directory = directory.Parent;
}

throw new DirectoryNotFoundException("the Sources folder was not found from " + AppContext.BaseDirectory);
}

#endregion
}
40 changes: 32 additions & 8 deletions Sources/Core/OutWit.Database.Core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ OutWit.Database.Core is a production-ready embedded database engine designed for

### Key Features

- **Storage Engines**: B+Tree (read-optimized) and LSM-Tree (write-optimized)
- **Storage Engines**: B+Tree, the default and the right choice for almost everything, and LSM-Tree
for a narrow shape of workload - see [Choosing a storage engine](#choosing-a-storage-engine)
- **MVCC**: Multi-Version Concurrency Control with snapshot isolation
- **5 Isolation Levels**: ReadUncommitted, ReadCommitted, RepeatableRead, Serializable, Snapshot
- **Row-Level Locking**: FOR UPDATE, FOR SHARE, NOWAIT, SKIP LOCKED
Expand All @@ -35,17 +36,17 @@ OutWit.Database.Core is a production-ready embedded database engine designed for
## Installation

```xml
<PackageReference Include="OutWit.Database.Core" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core" Version="13.1.1" />
```

For ChaCha20-Poly1305 encryption:
```xml
<PackageReference Include="OutWit.Database.Core.BouncyCastle" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core.BouncyCastle" Version="13.1.1" />
```

For Blazor WebAssembly (IndexedDB storage):
```xml
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="13.1.1" />
```

---
Expand Down Expand Up @@ -178,8 +179,8 @@ var db = new WitDatabaseBuilder()
// .WithStorage(customStorage) // Custom IStorage

// Engine
.WithBTree() // B+Tree (read-optimized)
// .WithLsmTree() // LSM-Tree (write-optimized)
.WithBTree() // B+Tree - the default, and what to use unless measured otherwise
// .WithLsmTree() // LSM-Tree - see "Choosing a storage engine"
// .WithLsmTree(opts => { ... }) // LSM with custom options
// .WithStore(customStore) // Custom IKeyValueStore

Expand Down Expand Up @@ -456,9 +457,32 @@ var db = new WitDatabaseBuilder()

---

## Choosing a storage engine

**`WithBTree()` is the default and is the right choice for almost everything.** "LSM is
write-optimised" was the claim here until it was measured, and the measurement retired it.

Measured 2026-08-11, 100,000 rows written in batches of 1,000 through SQL, microseconds per row:

| | B+Tree | LSM |
|---|---|---|
| **MVCC on** - what you get by default | 36.8 | **771.9** |
| `MVCC=false` | 15.1-27.4 | 16.7 |

**MVCC is on by default**, so a database that names the store and nothing else gets the first row:
the B+Tree pays roughly 1.5-2x for MVCC and the LSM pays about **50x**. It is a per-row cost rather
than a per-transaction one, so no batch size amortises it, and `SyncWrites` is not the expensive
property.

With `MVCC=false` the two stores write at the same speed, and that is the comparison the old claim
described. `Docs/WitSQL.md` § 14.9 has the full numbers and the shape of workload where the LSM store
is the better answer.

---

## Performance Tips

1. **Choose the right engine**: B+Tree for reads, LSM-Tree for writes
1. **Start with B+Tree**, and read *Choosing a storage engine* above before changing it
2. **Tune cache size**: More cache = fewer disk reads
3. **Use appropriate page size**: 4KB default, 8KB-16KB for large values
4. **Batch operations**: Use transactions for multiple writes
Expand All @@ -474,7 +498,7 @@ WitDatabase can run entirely in the browser using IndexedDB as the storage backe
### Installation

```xml
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Core.IndexedDb" Version="13.1.1" />
```

Add JavaScript files to `index.html`:
Expand Down
2 changes: 1 addition & 1 deletion Sources/Engine/OutWit.Database.Parser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ OutWit.Database.Parser is a high-performance SQL parser built on [ANTLR4](https:
## Installation

```xml
<PackageReference Include="OutWit.Database.Parser" Version="12.8.0" />
<PackageReference Include="OutWit.Database.Parser" Version="13.1.1" />
```

---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using System.Reflection;
using OutWit.Database.Expressions;
using OutWit.Database.Parser.Expressions;
using OutWit.Database.Parser.Schema.Types;
Expand Down Expand Up @@ -808,6 +809,16 @@ public void EvaluateDatabaseTest()
Assert.That(result.AsString(), Is.EqualTo("WitDB"));
}

/// <summary>
/// <c>VERSION()</c> answers the version of the engine that is running.
/// </summary>
/// <remarks>
/// It answered the literal <c>"1.0.0"</c> until 2026-08-15, and this case pinned it there while
/// the engine was on 13.1.1 - `SELECT VERSION()` being the obvious thing for a user to run. The
/// expectation is read from the ASSEMBLY here, independently of the property under test: a case
/// carrying <c>"13.1.1"</c> in its own text would be the same defect one layer out, stale at the
/// next release.
/// </remarks>
[Test]
public void EvaluateVersionTest()
{
Expand All @@ -816,7 +827,20 @@ public void EvaluateVersionTest()

var result = evaluator.Evaluate(func, CreateEmptyRow());

Assert.That(result.AsString(), Is.EqualTo("1.0.0"));
var informational = typeof(ExpressionEvaluator).Assembly
.GetCustomAttribute<AssemblyInformationalVersionAttribute>()!.InformationalVersion;

var expected = informational.Split('+')[0];

Assert.Multiple(() =>
{
Assert.That(result.AsString(), Is.EqualTo(expected));

// CONTROL: the assembly is not itself answering 1.0.0, which would make the comparison
// above pass for the wrong reason.
Assert.That(expected, Is.Not.EqualTo("1.0.0"),
"the engine assembly reports 1.0.0, so this case cannot tell the fix from the defect");
});
}

[Test]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
using OutWit.Database.Parser.Expressions;
using OutWit.Database.Parser.Expressions;
using OutWit.Database.Sql;
using OutWit.Database.Types;
using OutWit.Database.Values;
Expand Down Expand Up @@ -186,7 +186,7 @@ private WitSqlValue EvaluateFunction(WitSqlExpressionFunctionCall func, WitSqlRo

// System Functions
"DATABASE" => WitSqlValue.FromText("WitDB"),
"VERSION" => WitSqlValue.FromText("1.0.0"),
"VERSION" => WitSqlValue.FromText(WitDatabaseVersion.Text),

// Metadata Functions
"CHANGES" => WitSqlValue.FromInt(m_context.LastChangesCount),
Expand Down
2 changes: 1 addition & 1 deletion Sources/Engine/OutWit.Database/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ OutWit.Database is the SQL execution engine built on top of OutWit.Database.Core
## Installation

```xml
<PackageReference Include="OutWit.Database" Version="12.8.0" />
<PackageReference Include="OutWit.Database" Version="13.1.1" />
```

---
Expand Down
Loading
Loading