Repository navigation
Python: Add Ollama text to image service - #14437
Shoumik Chakravarty (shoumikchakravarty-dev) wants to merge 3 commits into
Conversation
Adds OllamaTextToImage, a TextToImageClientBase implementation backed by the Ollama generate endpoint, which returns a base64 image payload for image generation models such as x/z-image-turbo. The service follows the existing per-modality layout of the Ollama connector: a new OLLAMA_IMAGE_MODEL_ID setting, an OllamaTextToImagePromptExecutionSettings class carrying width, height and steps, and exports from the connector package. The deprecated width and height arguments of generate_image are forwarded to the endpoint when the settings do not already specify them. A response without image data raises ServiceInvalidResponseError rather than returning empty bytes. Adds unit tests covering initialization, custom host and client, base64 decoding, get_image_content, size settings, the deprecated size arguments and their precedence, and the missing-image error path. Closes microsoft#13938
There was a problem hiding this comment.
🟢 Approval recommended
The implementation and tests are complete; only a minor documentation nit remains.
Pull request overview
Adds Ollama text-to-image support with configurable image generation settings and base64-decoded output.
Changes:
- Adds
OllamaTextToImageand public exports. - Adds image model configuration and generation settings.
- Adds comprehensive unit tests.
File summaries
| File | Summary |
|---|---|
python/tests/unit/connectors/ai/ollama/services/test_ollama_text_to_image.py |
Tests image generation behavior and error handling. |
python/tests/unit/connectors/ai/ollama/conftest.py |
Adds image model test configuration. |
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py |
Implements Ollama image generation. |
python/semantic_kernel/connectors/ai/ollama/ollama_settings.py |
Adds the image model setting. |
python/semantic_kernel/connectors/ai/ollama/ollama_prompt_execution_settings.py |
Adds image size and steps settings. |
python/semantic_kernel/connectors/ai/ollama/__init__.py |
Exports the new service. |
Review details
Suppressed comments (1)
python/semantic_kernel/connectors/ai/ollama/init.py:13
- The connector index in
python/semantic_kernel/connectors/ai/README.mdlists each provider's public modality service, including the existing OpenAI and Azure text-to-image clients, but does not list this new Ollama client. AddOllamaTextToImagethere so the exported service is discoverable and the index stays complete.
from semantic_kernel.connectors.ai.ollama.services.ollama_text_to_image import OllamaTextToImage
- Files reviewed: 6/6 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Adds the new service to the modality table in connectors/ai/README.md, next to the other Ollama services, so the exported client is discoverable alongside the OpenAI and Azure text to image clients.
There was a problem hiding this comment.
🔵 Needs a closer look
Fix typed-settings handling and keyword collisions, update the central settings reference, and add SDK-shaped response coverage.
Review details
Suppressed comments (5)
Previously missed (1) — in code that hasn't changed since the last review.
python/semantic_kernel/connectors/ai/ollama/ollama_settings.py:34
- The new
OLLAMA_IMAGE_MODEL_IDsetting is not documented in the repository's settings reference (python/samples/concepts/setup/ALL_SETTINGS.md), which currently lists the three existing Ollama services but not this new service. Please add the corresponding service/model-id/host row so users following the central setup guide can discover and configure the connector.
python/semantic_kernel/connectors/ai/README.md:51
- The settings reference at
python/samples/concepts/setup/ALL_SETTINGS.mdis linked from the setup documentation and lists each Ollama service, but this new service and itsOLLAMA_IMAGE_MODEL_IDenvironment variable are missing there. Add the service'sai_model_id/hostentry so users can discover the required configuration.
| | [`OllamaTextToImage`](./ollama/services/ollama_text_to_image.py) |
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py:104
- This unconditionally calls
from_prompt_execution_settings, so an already-typed settings object is repacked intoextension_databefore each request. If a caller clears a previously set field (for example,settings.width = None),pack_extension_data()leaves the old value inextension_dataand this conversion restores it, meaning the request ignores the caller's current settings. Preserve typed settings (or use the service helper) instead.
else OllamaTextToImagePromptExecutionSettings.from_prompt_execution_settings(settings)
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py:127
- Merging
kwargsintooptionsand then supplyingmodel,prompt, andstreamexplicitly makes any of those validAsyncClient.generatearguments fail withTypeError: multiple values for keyword argument(for example,generate_image(prompt, stream=False)). Since the method documentskwargsas endpoint arguments, reserve these request-control keys or build one dictionary and apply the fixed values before calling the client.
options = image_settings.prepare_settings_dict()
options.update(kwargs)
response_object = await self.client.generate(
python/tests/unit/connectors/ai/ollama/services/test_ollama_text_to_image.py:58
- All success and missing-image mocks here are mappings, but the Ollama SDK returns a
GenerateResponse, so the primarygetattr(response_object, "image")path in the new service is never exercised. Add a typed/SDK-shaped response case to verify the integration with the actual response object.
mock_generate.return_value = {"image": ENCODED_IMAGE}
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Lite
Preserve already-typed execution settings instead of always converting them. The unconditional from_prompt_execution_settings call packed the settings into extension_data on every request, so a field the caller had cleared was restored from the stale packed value and the request ignored the current settings. Build a single request dict instead of merging kwargs into the settings dict and passing model, prompt and stream separately, which raised TypeError on duplicate keyword arguments when a caller passed one of those as a documented endpoint argument. The service-controlled keys now override settings and kwargs. Apply the deprecated width and height arguments to the request rather than to the settings object, so a caller's settings are no longer mutated, keeping the same precedence: settings win when they carry a size. List OllamaTextToImage and OLLAMA_IMAGE_MODEL_ID in ALL_SETTINGS.md next to the other Ollama services. Adds tests for a GenerateResponse-shaped result (with and without image data), which exercises the attribute path the SDK actually returns, plus regression tests for the settings repacking, the keyword collision, and settings mutation.
|
Thank you for the contribution. We are not adding new features to Semantic Kernel (SK) at the moment, so I am closing this PR. Please consider creating a feature request issue in microsoft/agent-framework. |
Motivation and Context
Closes #13938
Ollama supports image generation https://ollama.com/blog/image-generation through
POST /api/generate, which returns a base64-encodedimagefield for image models. Semantic Kernel has no Ollama text-to-image service, so these models cannot be used through theTextToImageClientBaseabstraction.Description
Adds
OllamaTextToImage, following the connector's existing one-service-per-modalityollama_text_to_image.pyimplementinggenerate_image()viaAsyncClient.generate()and returning thedecoded image bytes
OLLAMA_IMAGE_MODEL_IDsetting onOllamaSettings, matching the existing chat, text and embedding model IDs.OllamaTextToImagePromptExecutionSettingscarryingwidth,heightandsteps.connectors/ai/ollama/__init__.py.No new dependencies: the
ollamapackage SK already depends on exposesimageonGenerateResponse, andwidth,heightandstepsongenerate().Contribution Checklist