Skip to content

development-guide.md

Latest commit

 

History

History
666 lines (488 loc) · 18.5 KB

development-guide.md

File metadata and controls

666 lines (488 loc) · 18.5 KB
Raw

Development Guide

Comprehensive guide for setting up, developing, testing, and deploying the ERE Prototype Template.

For architecture overview, see architecture.md. For API flows, see sequence-diagrams.md.

Prerequisites

Tool Version Purpose
Python 3.12+ Runtime
Poetry 1.7+ Dependency management
Docker 24+ Container runtime
Docker Compose 2.20+ Multi-service orchestration
Redis 7.x Messaging transport (or Docker)
Make any Build automation

Quick Start

# Clone and enter the project
git clone <repository-url>
cd ere-prototype-template

# Install dependencies (ERE service)
make install

# Install client dependencies
cd ere-client && make install && cd ..

# Run tests (no external services needed)
make test

# Start with Docker (Redis + ERE)
make docker-up

Dependency Management

ERE Service (root pyproject.toml)

# Install base dependencies
poetry install

# Install with MongoDB support
poetry install --extras mongodb

# Install with PostgreSQL support
poetry install --extras postgres

# Install with embeddings support (large download: PyTorch/ONNX)
poetry install --extras embeddings

# Install everything
poetry install --all-extras

ERE Client (ere-client/pyproject.toml)

cd ere-client
poetry install

The client has minimal dependencies: click, redis, rich, pydantic, PyYAML.

Configuration

ERE Service Configuration

Configuration is loaded from YAML with environment variable overrides:

# config/ere_config.yaml
service:
  name: ere-prototype
  version: "0.1.0"
  supported_entity_types:
    - "ORGANIZATION"
    - "PERSON"

messaging:
  backend: redis          # "in_memory" or "redis"
  redis:
    host: localhost
    port: 6379
    request_channel: "ere_requests"
    response_channel: "ere_responses"

storage:
  backend: in_memory      # "in_memory", "mongodb", or "postgres"

clustering:
  similarity_algorithm: cosine   # "cosine", "jaccard", or "levenshtein"
  similarity_threshold: 0.75
  confidence_threshold: 0.5
  max_candidates: 5
  # Quality enhancements
  identifier_exact_match_boost: 0.30
  identifier_mismatch_penalty: 0.02
  representative_update_threshold: 0.90
  blocking_enabled: false

embedding:
  enabled: true
  model_name: "sentence-transformers/all-MiniLM-L6-v2"

limits:
  workers: 4              # Concurrent worker threads (1 = sequential)

logging:
  level: INFO
  traffic_log_payloads: true

Environment Variable Overrides

Variable Overrides Example
ERE_CONFIG Config file path config/ere_config_docker.yaml
ERE_REDIS_HOST messaging.redis.host redis
ERE_REDIS_PORT messaging.redis.port 6379
ERE_REDIS_DB messaging.redis.db 0
ERE_REDIS_PASSWORD messaging.redis.password secretpass
ERE_MONGODB_URI storage.mongodb.uri mongodb://mongo:27017
ERE_MONGODB_DATABASE storage.mongodb.database ere_prod

Client Configuration

# ere-client/config/client_config.yaml
redis:
  host: localhost
  port: 6379
  request_channel: "ere_requests"
  response_channel: "ere_responses"

defaults:
  source_id: "ere-client"
  entity_type: "ORGANIZATION"
  content_type: "text/turtle"

response:
  timeout: 30
  poll_interval: 1

Storage Backend Configuration

InMemory (Default)

No configuration needed. Data is lost on restart. Best for:

  • Unit and integration testing
  • Quick experimentation
  • Development without Docker

MongoDB

storage:
  backend: mongodb
  mongodb:
    uri: "mongodb://ere:ere_dev_password@mongodb:27017/ere_prototype?authSource=admin&directConnection=true"
    database: "ere_prototype"
    entities_collection: "entities"
    clusters_collection: "clusters"

Requirements:

  • MongoDB 8.x running (or Docker: make docker-up-mongodb)
  • pymongo installed (poetry install --extras mongodb)
  • For vector search: MongoDB Atlas or self-managed with Atlas Search (mongot)

PostgreSQL + pgvector

storage:
  backend: postgres
  postgres:
    host: localhost
    port: 5432
    database: ere_prototype
    user: ere
    password: ""
    vector_dimension: 384

Requirements:

  • PostgreSQL 15+ with pgvector extension
  • Docker: make docker-up-postgres
  • psycopg[binary] installed (poetry install --extras postgres)

Testing Workflows

Unit Tests

make test              # Run all unit tests
make test-verbose      # Verbose output with test names
make coverage          # Run with coverage report

Unit tests use InMemory backends — no external services required.

BDD Tests

make test-bdd          # Run Behave BDD scenarios

BDD tests describe resolution behaviour in Gherkin syntax.

Docker-based Tests

make test-docker       # Run tests against Docker services

Starts Docker Compose services, runs integration tests, then stops services.

All Tests

make test-all          # Unit + BDD + Docker tests

Code Quality

make lint              # Run ruff linter
make format            # Auto-format with ruff
make lint-fix          # Auto-fix linting issues

Docker Deployment Patterns

Start Specific Profiles

# Redis only (messaging)
docker compose --profile redis up -d

# Redis + MongoDB
docker compose --profile redis --profile mongodb up -d

# Redis + PostgreSQL
docker compose --profile redis --profile postgres up -d

# Full stack with monitoring
docker compose --profile redis --profile mongodb --profile postgres --profile redis-insight up -d

Using Makefile Shortcuts

make docker-up              # Core + Redis (default)
make docker-up-mongodb      # Core + Redis + MongoDB + Mongo Express
make docker-up-postgres     # Core + Redis + PostgreSQL + pgAdmin
make docker-down            # Stop all services
make docker-logs            # Follow service logs
make docker-build           # Rebuild ERE service image

Accessing Admin UIs

UI URL Credentials
Mongo Express http://localhost:8081 (none)
pgAdmin http://localhost:5050 admin@admin.com / admin
Redis Insight http://localhost:5540 (none)

Running the ERE Service

Local (Poetry)

# With defaults (InMemory storage, InMemory messaging)
python -m ere.main

# With specific config
python -m ere.main --config config/ere_config.yaml

# With Docker Redis
ERE_REDIS_HOST=localhost python -m ere.main --config config/ere_config.yaml

Docker

make docker-up      # Starts ERE + Redis
make docker-logs    # View logs

Using the Client

cd ere-client

# Send a resolution request
poetry run ere-client resolve "ACME Corporation, London, UK"

# Send from a JSON file
poetry run ere-client send-file samples/sample_request.json

# Monitor responses
poetry run ere-client listen

# Check connection status
poetry run ere-client status

Troubleshooting

Redis Connection Refused

redis.exceptions.ConnectionError: Error connecting to localhost:6379

Solution: Start Redis via Docker (make docker-up) or install Redis locally.

MongoDB Connection Failed

pymongo.errors.ServerSelectionTimeoutError: localhost:27017

Solution: Start MongoDB (make docker-up-mongodb) and verify port 27017 is accessible.

pgvector Extension Not Found

psycopg.errors.UndefinedFile: could not open extension control file "vector"

Solution: Use the pgvector/pgvector:pg15 Docker image which includes the extension pre-installed.

sentence-transformers Import Error

ImportError: No module named 'sentence_transformers'

Solution: Install the embeddings extra: poetry install --extras embeddings. Note: this downloads ~2GB of PyTorch dependencies.

Embedding Dimension Mismatch

ValueError: Embedding dimension 768 doesn't match configured dimension 384

Solution: Ensure embedding.model_name in config matches the vector_dimension in storage config. The default model (all-MiniLM-L6-v2) produces 384-dimensional vectors.

BLPOP Timeout (No Response)

⏱ No response received within timeout.

Solution: Verify the ERE service is running and consuming from the same Redis channel. Check ere-client status to confirm connectivity and queue lengths.

Property Extraction

The Property Extraction feature enables structured RDF/TTL property extraction for entity resolution. Instead of embedding an entity's entire serialized content as a single string, the system can parse RDF/TTL content, extract individual properties, apply text preprocessing, embed each property separately, and combine them into a weighted composite vector.

This improves resolution accuracy by allowing administrators to control which properties matter most and how they are normalized before comparison.

Configuration Examples

Property extraction is configured in config/ere_config.yaml under the property_extraction key. The structure maps entity type URIs to lists of property definitions.

Example: Organization with Legal Name, Alt Name, and Identifier

property_extraction:
  "http://www.w3.org/ns/org#Organization":
    - name: "legal_name"
      xpath: "org:Organization/org:legalName"
      weight: 2.0
      filter: "lowercase"
    - name: "alternative_name"
      xpath: "org:Organization/skos:altLabel"
      weight: 1.0
      filter: "trim"
    - name: "identifier"
      xpath: "org:Organization/org:identifier"
      weight: 3.0
      filter: "remove_special_characters"

Example: Person with FOAF Properties

property_extraction:
  "http://xmlns.com/foaf/0.1/Person":
    - name: "full_name"
      xpath: "foaf:Person/foaf:name"
      weight: 3.0
      filter: "lowercase"
    - name: "email"
      xpath: "foaf:Person/foaf:mbox"
      weight: 2.5
      filter: "lowercase"
    - name: "family_name"
      xpath: "foaf:Person/foaf:familyName"
      weight: 2.0
      filter: "trim"
    - name: "given_name"
      xpath: "foaf:Person/foaf:givenName"
      weight: 1.5
      filter: "trim"

Example: SKOS Concept

property_extraction:
  "http://www.w3.org/2004/02/skos/core#Concept":
    - name: "preferred_label"
      xpath: "skos:Concept/skos:prefLabel"
      weight: 3.0
      filter: "lowercase"
    - name: "alt_labels"
      xpath: "skos:Concept/skos:altLabel"
      weight: 1.5
      filter: "trim"
    - name: "definition"
      xpath: "skos:Concept/skos:definition"
      weight: 1.0
      filter: "trim"

Example: Multiple Entity Types

You can configure different entity types in the same file:

property_extraction:
  "http://www.w3.org/ns/org#Organization":
    - name: "legal_name"
      xpath: "org:Organization/org:legalName"
      weight: 2.0
      filter: "lowercase"

  "http://xmlns.com/foaf/0.1/Person":
    - name: "full_name"
      xpath: "foaf:Person/foaf:name"
      weight: 3.0
      filter: "lowercase"
    - name: "email"
      xpath: "foaf:Person/foaf:mbox"
      weight: 2.0
      filter: "lowercase"

Example: Using Full URIs (no prefix)

If you need to reference an ontology not covered by built-in prefixes, use full URIs in angle brackets:

property_extraction:
  "<http://example.org/ontology#CustomEntity>":
    - name: "custom_label"
      xpath: "<http://example.org/ontology#CustomEntity>/<http://example.org/ontology#label>"
      weight: 2.0
      filter: "trim"

Configuration Field Reference

Field Type Constraints Description
name string Non-empty, max 128 chars, unique within entity type Human-readable property identifier
xpath string Non-empty, max 512 chars XPath selector to locate the property in RDF
weight float 0.0 ≤ weight ≤ 10.0 Importance multiplier for the composite vector (0 = extract-only, no embedding)
filter string or list One or more of the supported filters Preprocessing transformation(s) applied in sequence

Supported filters: uppercase, lowercase, trim, remove_special_characters, strip_company_suffixes, strip_telephone_chars, numeric, strip_dots, strip_spaces, alphanumeric, last_part_of_url

XPath Selector Format

The xpath field uses a simplified path notation that maps RDF types and properties to SPARQL queries internally. The format locates subjects of a specific rdf:type and extracts a specific property value from those subjects.

Syntax

<type_selector>/<property_selector>

Where each part is either a prefixed name or a full URI in angle brackets.

Prefixed Name Format

prefix:Type/prefix:property

This translates internally to the SPARQL query:

SELECT ?value WHERE {
    ?s a <expanded_type_uri> .
    ?s <expanded_property_uri> ?value .
}

Example: org:Organization/org:legalName queries for all subjects with rdf:type org:Organization and extracts their org:legalName values.

Full URI Format

<http://full-type-uri>/<http://full-property-uri>

Example: <http://example.org/ns#Company>/<http://example.org/ns#tradeName> queries for subjects typed as http://example.org/ns#Company with the property http://example.org/ns#tradeName.

Supported Prefixes

Prefix Namespace URI
org http://www.w3.org/ns/org#
foaf http://xmlns.com/foaf/0.1/
skos http://www.w3.org/2004/02/skos/core#
rdfs http://www.w3.org/2000/01/rdf-schema#
rdf http://www.w3.org/1999/02/22-rdf-syntax-ns#
dc http://purl.org/dc/elements/1.1/
dct http://purl.org/dc/terms/
schema http://schema.org/
vcard http://www.w3.org/2006/vcard/ns#
owl http://www.w3.org/2002/07/owl#
xsd http://www.w3.org/2001/XMLSchema#

XPath Examples

XPath Selector What it Extracts
org:Organization/org:legalName Legal name of organizations
org:Organization/skos:altLabel Alternative labels of organizations
foaf:Person/foaf:name Full name of persons
foaf:Person/foaf:mbox Email addresses of persons
skos:Concept/skos:prefLabel Preferred labels of SKOS concepts
rdfs:Resource/rdfs:label Labels of any RDFS resource
schema:Organization/schema:name Schema.org organization names
<http://example.org/ns#Entity>/<http://example.org/ns#id> Custom property via full URIs

Multi-Value Handling

When a property has multiple values (e.g., multiple skos:altLabel entries), all matching values are:

  1. Sorted in lexicographic (Unicode code point) ascending order
  2. Concatenated with a single space separator
  3. Truncated to 10,000 characters if the concatenated result exceeds that limit

Preprocessing Filters

Each property definition includes a filter that normalizes the extracted text before embedding. Filters are applied after property extraction and multi-value concatenation.

Filter: uppercase

Converts all characters to uppercase using Python's str.upper().

Input Output
"Acme Corporation" "ACME CORPORATION"
"café résumé" "CAFÉ RÉSUMÉ"
"hello world 123" "HELLO WORLD 123"
"Already UPPER" "ALREADY UPPER"

Filter: lowercase

Converts all characters to lowercase using Python's str.lower().

Input Output
"ACME Corporation" "acme corporation"
"John DOE" "john doe"
"Mixed Case 123" "mixed case 123"
"already lower" "already lower"

Filter: trim

Removes all leading and trailing Unicode whitespace characters (spaces, tabs, newlines, and all Unicode whitespace categories).

Input Output
" hello world " "hello world"
"\t\n Acme Corp \n" "Acme Corp"
" multiple spaces inside " "multiple spaces inside"
"no_whitespace" "no_whitespace"

Note: trim only removes leading/trailing whitespace. Interior whitespace is preserved.

Filter: remove_special_characters

Applies a three-step normalization process:

  1. NFKD Unicode normalization — decomposes accented characters (e.g., é → e + combining accent)
  2. Discard non-ASCII characters — removes anything that isn't in the 0–127 ASCII range
  3. Remove non-alphanumeric non-whitespace — strips remaining punctuation, symbols, etc.
Input Output
"Café Résumé" "Cafe Resume"
"O'Brien & Associates, Ltd." "OBrien Associates Ltd"
"user@example.com" "userexamplecom"
"hello-world_123!" "helloworld123"
"naïve coöperation" "naive cooperation"
"100% complete™" "100 completeTM"
"日本語テスト" "" (empty — all non-ASCII)

Filter Behaviour Notes

  • Filters are applied in sequence as specified in the filter list.
  • If the value is empty after filtering, that property is excluded from the composite vector computation entirely.
  • Filters operate on the already-concatenated multi-value string (after join with space).
  • The last_part_of_url filter is a stop filter: if it transforms the value (detects a URL), subsequent filters in the chain are skipped.
  • Properties with weight: 0.0 are extracted for the parsed_representation JSON but are not embedded or included in the composite vector.

Pipeline Behaviour Summary

The full property extraction pipeline operates in this order for each entity mention with content_type = "text/turtle":

  1. Config lookup — Find property definitions matching the entity's entity_type URI
  2. Parse — Parse the TTL content string into an RDF graph using rdflib
  3. Extract — For each configured property, query the graph using the XPath selector
  4. Concatenate — Sort multi-value results lexicographically, join with single space
  5. Truncate — If concatenated value exceeds 10,000 characters, truncate
  6. Preprocess — Apply the configured filter to normalize the text
  7. Embed — Pass non-empty preprocessed values to the HuggingFace embedder
  8. Weight — Multiply each embedding vector by its configured weight
  9. Combine — Sum all weighted vectors and L2-normalize to unit length

Fallback conditions — The system falls back to full-content embedding when:

  • No property_extraction config exists for the entity type
  • TTL content cannot be parsed (malformed Turtle)
  • All properties yield empty values after extraction and preprocessing
  • The embedder is disabled in configuration