Repository navigation
Feature/fetch - #44
Merged
Merged
Conversation
New utility das_fetch takes a das catalog path or URI, walks the federated catalog, translates interface settings (coords, data, formats, options, examples) into the HTTP query of a source and streams the response to stdout. Operating modes: roots, node (ls), json, url, data (get). Query keys are dotted member paths, bare members are the shortcut, and no keys at all runs example 0 for ease of testing. User files live under ~/.das: cache/nodes for fetched catalog nodes, servers for credentials. node.c: - catalog nodes load from local files as well as URLs - fetched nodes are cached with an age limit (DasNode_setCache) - new_RootNode_seed() names a node by URI out of band, either as the only root (-r URI=URL) or as a patch over the federated roots (-g URI=URL). - Nodes remember the credential manager and agent they were loaded with, and DasNode_subNode() inherits them when given NULL. - Lookup misses now log at info level. util.c: - das_rename() replaces an existing destination on Windows too http.c: - URLs with an explicit port now work New offline tests were added with 53 individual checks on Linux and MacOS, and a single check for Windows (node to URL). Developed-with: Claude Fable 5.1
* New function das_http_copyBody, a counter part for das_http_readUrl bug fix: New close helper frees the SSL context bug fix: always-true limit warning in das_http_readUrl Developed-with: Claude Fable 5.1
Lift the root policy, credentials and node memory that grew
up inside das_fetch into das3/catalog.{c,h} so every client
of libdas3 resolves catalog paths the same way. Catalog
sub-trees may be mounted at points inside another catalog
tree.
* Doxygen category "catalog"
* DasNode goes opaque
* das3_node.c retired
* Tests added
Developed-with: Claude Fable 5.1
URL parts that do not fit are refused, never truncated, Path and query limits raised to 255 and 3071. Developed-with: Claude Fable 5.1
Everything das2C keeps for a user lives under ~/.das on every platform, or $DAS_USERDIR, or whatever a program gives das_setUserDir(). A new DasCat caches catalog nodes there for an hour unless told otherwise. das_fetch, das3_cdf and das3_info updated to use the same area. Developed-with: Claude Fable 5.1
A cached node keeps the ETag and Last-Modified it came with. Past the trust age the session asks the server with If-None-Match and If-Modified-Since, and a 304 renews the copy for no body bytes. An unreachable server leaves the old copy in use with a warning. http.h gains das_http_getBody_ex and das_http_readUrl_ex for conditional requests and DasHttpResp_header for reading response headers. Developed-with: Claude Fable 5.1
DasSettable describes one changeable interface property of a source node: its key, type, units, default, allowed values and the protocol parameter that carries it. TestSettable adds in-tree validation tests Developed-with: Claude Fable 5.1
Data sources implemented as standalone objects that provide input validation and input-to-protocol translation facilities. Requisitions added as intentionally simple structured data requests, strings only. Catalog node (and thus data source) ID's clarified as URIs only, all loaded nodes must have some URI, the tag:das2.org,2012:local: prefix is added as a URI of last resort. das_fetch no longer accepts a bare location as -r; a root must have declared URI. settable.h is absorbed into source.h. Developed-with: Claude Fable 5.1
The word root now just refers to navigation trees, DasNode constructor functions renamed for consistency Developed-with: Claude Fable 5.1
New library features: * dassrc_loadable() lists which (if any) sub-nodes of a catalog node are usable by the library. * node.c is able to upgrade some older nodes until they can be replaced. This is a general feature that may be needed even after the current deprecated set is retired * das_fetch gained the ability to select among data source nodes of a catalog for it's preferred MIME types * the library now validates that URI's actually look like uri's Developed-with: Claude Fable 5.1
This is in preparation to adapt catalog code for clients that supply thier own networking such as javascript. Developed-with: Claude Fable 5.1
* All I/O code removed from DasNode to DasCat * New state machine class provided to run URI resolution using only DasNode objects, useful for emscripten builds * Vocabulary change: root->seed. Root now is means a seed node with no URI, that's it. See doxygen docs * das_fetch command line changes. See help text. * das_fetch can mount a local catalog on a local catalog * bug fixs: Memory leak in DasHttpResp_freeFields plugged Developed-with: Claude Fable 5.1
- das_mkdirsto() now handlesd '/' and '\' it also skips the leading leading drive spec (ex "C:") - Cast const char**, and wrap pragmas to quiet MSC builds Developed-with: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change revives the long dormant Catalog code and makes it suitable for use in emscripten environments, as well as for native C development. It provides the das_fetch program which is likely to become a commonly used utility, by me at least.
The point is to get a decent catalog API into the upcoming v3.0.0 tag.