Enhance functionality and usability across multiple scripts

[TiagoDeMatosDias]

• Updated _shared.py to support UTF-8 encoding for Windows consoles and added configuration loading for different model shapes.
• Modified sources.py to ensure text files are read with UTF-8 encoding.
• Improved model.py to allow models to be built with different data types (bf16/fp16) based on configuration.
• Introduced continue_train.py script to facilitate resuming training from saved checkpoints with additional data.
• Added export_transcript_corpus.py to convert Claude Code transcripts into a plain-text corpus for training.
• Implemented serve_openai.py to serve the model over an OpenAI-compatible HTTP API, enabling various endpoints for model interaction.
• Created a simple HTML test bench (ui.html) for easy interaction with the served model.

[TiagoDeMatosDias]

It's a good project!

I just added a few things to make it easier to train as well as serve the model.

I had some issues with UTF-8 encodings when training on some types of datasets, so i enforced encoding to fix it.

Additionally i didn't like the fact that I couldn't resume training a model, so i added that too.

The export transcript corpus lets you train on your own (or other peoples) agent transcripts. So far the training i didn't wasn't that helpful in getting a coherent setup with that, but from what i can tell its mostly a matter of giving it some training time and more data.

The serve_openai.py and its related html file make it easier to test things (and potentially let you attach agents to it, though i haven't tested it yet on that use case).

It basically just gives you a webpage you can use to run inference on:

I don't have much data, and didn't run it for long, so it's not that coherent, but i can tell a big difference from my first training run

Hope it helps!