Add OCI Logging: log groups, logs, ingestion, search

[arunesh-j]

Summary

• Implements OCI Logging against the existing portable logging driver.
• Covers all three of OCI's API surfaces — control plane, ingestion, and search — collapsed onto one listener.
• Nothing added to services/logging/driver; OCI-only behaviour is a consumer-side Extras interface, per Move OCI-only capabilities out of shared driver packages #393.

Closes #414. Part of #376.

Changes

• providers/oci/logging/ — Mock over memstore implementing driver.Logging, guarded by a sync.RWMutex, plus a search-query parser.
• server/oci/logging/ — the three-prefix wire surface.
• Wiring is one line each in providers/oci/oci.go and server/oci/oci.go.

Operations

Control plane: Create/List/Get/Update/Delete LogGroup, ChangeLogGroupCompartment, Create/List/Get/Update/Delete Log. Data planes: PutLogs, SearchLogs. All 14 portable driver methods implemented; metric filters return Unimplemented naming Service Connectors as OCI's answer.

The three prefixes — the main design risk

parsePath splits /{version}/{collection}[/{id}[/{sub}[/{subId}]]], then a switch on version claims exactly one collection set: 20200531 → logGroups, unifiedAgentConfigurations, logSavedSearches; 20200601 → logs; 20190909 → search.

The load-bearing detail: a top-level /logs collection exists only on the ingestion prefix — the control plane nests logs under their group — so /20200531/logs is deliberately unclaimed. TestMatches has 25 cases covering every positive shape, each collection asserted not claimed under the two wrong prefixes, other services' traffic, and malformed paths.

Search: supported vs rejected

Supported: search "compartmentId[/logGroupId[/logId]]" (comma-separated targets) | where <field> = | != '<value>' (* wildcard, joined by and) | sort by datetime [asc|desc]. Fields resolve over logContent.* and data.<key> of a JSON payload.

Rejected by name with a 400: summarize/stats/topN/extract/unknown operators; or, not, parenthesized where clauses; >, <, >=, <=, =~, !~; unresolvable fields or nested payload paths; sorting on anything but datetime; a target written as a name where OCI takes an OCID; a missing search clause or time range.

A real silent-empty bug was found and fixed mid-implementation: field resolution originally happened per-entry, so an unknown field on a log with no entries returned 200 []. Fields now resolve at parse time, before any entry is walked.

Judgement calls

• ListLogs takes no compartmentId — real OCI derives it from the log group in the path, so the group OCID is what is required. Noted in services.md and the handler comment.
• Log group retention is CloudEmu's own — real OCI carries retention on the log; the group holds the default its logs inherit, so the portable RetentionDays has somewhere to live.
• Log group display names are unique emulator-wide, not per compartment — that is what lets the portable driver address a group by name.
• DeleteLogGroup cascades to its logs rather than refusing, matching the other portable drivers.

Provider Coverage

• AWS
• Azure
• GCP
• OCI

Checklist

• All tests pass (go test ./...) — exit 0, 272 packages
• Linter passes (golangci-lint run --timeout=9m) — 0 issues
• Every provider the change applies to implements the same behavior — OCI-only, additive
• Integration tests added to cloudemu_test.go — driver + handler tests instead
• Unit tests added to provider test files

Test Plan

go build ./...                                              clean
go test ./...                                               exit 0, 272 packages
go test -race ./providers/oci/... ./server/oci/...          11/11 ok
golangci-lint run --timeout=9m ./providers/oci/... ./server/oci/...   0 issues
go generate ./...                                           docs/coverage committed

Coverage leak check clean: no OCI operation in docs/coverage/{aws,azure,gcp}/*.md; git diff development -- services/ empty.

End-to-end on a running server (port 4615):

create log group                              -> 202 + Opc-Work-Request-Id
poll work request                             -> ocid1.loggroup.oc1.iad...
create log                                    -> 202
put 3 entries                                 -> 200
compartment-wide search                       -> 3 results
| where data.level = 'ERROR' | sort desc      -> 2 results, newest first, payload decoded
| summarize count()                           -> 400, names "summarize"
where data.count > 3                          -> 400, names ">"
list without compartmentId                    -> 400
unifiedAgentConfigurations                    -> 501
delete log, delete group                      -> 202 each
get after delete                              -> 404 NotAuthorizedOrNotFound

Left out

No oci-go-sdk compat test — the three-client split made the e2e transcript stronger evidence for the effort. oracle.tenantid is omitted from search records; adding it would pull config identity into the handler for no behavioural gain.

[NitinKumar004]

Review notes

Real data-plane engine (per #427): N/A.

CI: CodeQL fails — 2× high go/uncontrolled-allocation-size at providers/oci/logging/portable.go:225 and :263. Guard the size before allocating.

Findings

Medium · structure — STRUCTURE.md filename parity broken: provider group.go/log.go/ingestion.go+search.go vs wire groups.go/logs.go/dataplane.go
server/oci/logging/groups.go:1
If a maintainer navigates by the STRUCTURE.md filename convention -> they cannot map a feature's mock to its wire handler by a single filename, because logging pluralizes and merges wire filenames against the singular, split provider files (server/oci/logging/groups.go, logs.go, dataplane.go).

STRUCTURE.md §3 hard rule ('a feature uses the same filename across all three layers') is broken for all three feature areas: provider group.go/log.go/ingestion.go/search.go vs wire groups.go(plural)/logs.go(plural)/dataplane.go(ingestion+search merged). The vcn reference keeps identical names across layers (dhcp.go/dhcp.go, subnet.go/subnet.go), so logging is the outlier. Naming-only, no runtime impact.

Low · correctness — Log-group displayName uniqueness is global, not per-compartment — diverges from real OCI
providers/oci/logging/group.go:24
If a user creates identically-named log groups in two different compartments (normal in real OCI) -> the second create fails with AlreadyExists, because providers/oci/logging/group.go:24 enforces a global name index rather than a per-compartment one.

createGroup calls groupByName(spec.DisplayName), which scans ALL compartments and returns AlreadyExists on any name collision (UpdateGroup rename too). Real OCI scopes log-group displayName uniqueness per compartment. The 'already exists' test (logging_test.go:83) only covers a same-compartment duplicate; documented as a deliberate tradeoff in docs/services.md to let the portable driver key groups by name.

Low · coverage — Search sort-ordering and oracle. provenance where-fields have no positive test*
server/oci/logging/handler_test.go:467
If someone later refactors sortEntries or provenanceValue and flips the desc branch or mis-maps an oracle.* field -> the bug ships green, because no test exercises sort-desc ordering or a where oracle.logid = ... comparison.

TestSearchLogs covers whole-compartment, narrowed-to-log, where-on-JSON-field, wildcards, time-range miss, field-info; the rejection table covers bad sorts/operators/fields. Neither asserts a successful 'sort by datetime asc|desc' RESULT ORDER nor a where-clause on oracle.compartmentid/loggroupid/logid/ingestedtime. provider sortEntries(desc) and provenanceValue() thus have no positive assertion; provider pkg coverage is 66.8%.

[arunesh-j]

Rebased onto current development (58787e9a) and addressed the findings across 1048ec83 and 86a227b0.

CI — CodeQL go/uncontrolled-allocation-size (alerts 17, 18)

Added maxLogLimit = 10000 and resolveLimit, called before the make at both flagged sites and at SearchLogs. Negative and over-max are rejected with InvalidArgument; 0 still means default, matching the AWS/GCP/Azure siblings.

Reverting both guards is not a subtle failure:

signal: killed
FAIL  github.com/stackshy/cloudemu/v2/providers/oci/logging  174.769s

The process is OOM-killed on limit: 1 << 40. Restored → PASS in 0.00s. Covered by TestPortableReadLimitIsBounded.

Worth recording for scope: the HTTP path was never exposed — ocirest.Limit already clamps at MaxLimit = 1000. The reachable surface was the portable driver called directly, and wire behaviour is unchanged since 1000 < 10000.

Medium — STRUCTURE.md §3/§4 filename parity

Both layers now carry the identical feature set: group.go / ingestion.go / log.go / search.go. Provider keeps logging.go (the §4 <service>.go), portable.go, query.go; wire keeps handler.go + types.go — the §4 template, and the same shape as the vcn reference. parseTime/timeError, shared by ingestion and search, moved to types.go. Renames survived the rebase as renames (groups.go => group.go and logs.go => log.go at 0 changes).

Low — per-compartment displayName: scoped, and I'd argue against keeping the tradeoff

groupByName now takes a compartment; createGroup, the UpdateGroup rename, MoveGroup and the portable compartment-move all check within one compartment. MoveGroup had the same latent bug and was not in the review.

On how the portable driver still addresses a group by name: portableGroupByName resolves across compartments but rejects an ambiguous name with InvalidArgument naming both compartments, rather than silently picking one. Single-compartment use — every existing portable test and the cross-cloud conformance suite — is unchanged; the ambiguous case is the one that used to be impossible and would otherwise resolve arbitrarily.

That keeps the repo's reject-don't-guess discipline instead of trading correctness for driver convenience. #424 (table names) and #425 (secret names) have the identical finding, and this is the answer I'd apply to both.

Low — positive tests for sort order and oracle.* provenance

Added at both layers: result order for sort by datetime asc|desc, sort by time desc, logContent.datetime desc, default, and the id tiebreak on equal timestamps; plus where on all four oracle.* fields including negation. Entries are seeded out of time order so a sorted result cannot pass by accident.

Coverage: provider 66.3% → 96.6%, wire 82.9% → 94.2%. The provider jump is mostly query.go/search.go, which had zero provider-level tests — the parser was only ever exercised through the wire.

Two things not in the review

1. Build break from upstream. The second rebase surfaced that services/logging/driver gained Put/Delete/DescribeSubscriptionFilter, so *Mock no longer satisfied driver.Logging. Implemented as Unimplemented naming the Service Connector, consistent with the existing metric-filter treatment — not a silent no-op.
2. TestSnapshotCompleteness failed: oci: field Logging (*logging.Mock) holds a memstore.Store but is not Snapshottable. Added snapshot.go modelled on vcn. This required exporting logRecord's fields (log→Log, entries→Entries) — an unexported type, so no public API change, but without it the JSON snapshot would have silently written {} and dropped every log and entry. Round-trip tests cover OCIDs, compartments, the log→group cross-reference, entry fields, searchability after restore, malformed input and the empty case.

One scope note for your call

I edited the shared logging section of docs/services.md, not just the OCI subsection. The driver gained those three operations upstream but the table and the **Total: 13 operations** line were never updated — stale before this branch. I added the rows and corrected the total to 17 to match go generate. Easy to drop if you want this PR strictly OCI-scoped, but the count would then contradict the generated coverage doc.

Verification

go build ./...                                     clean
go test ./...                                      exit 0, 342 packages
go test -race ./providers/oci/... ./server/oci/... 11/11 ok
golangci-lint (both logging packages)              0 issues
go test -cover   provider 96.6%   wire 94.2%
go generate ./...                                  docs/coverage committed
no OCI op in docs/coverage/{aws,azure,gcp}/*.md    confirmed
git diff origin/development -- services/logging/   empty (driver untouched)

golangci-lint also reports 7 gocritic issues across providers/oci/identity and {providers,server}/oci/vcn — confirmed pre-existing by running the same command against clean development, which produces the identical 7.

E2E, port 4615

create log group                          -> 202  ocid1.loggroup...0002
same name, DIFFERENT compartment          -> 202  ocid1.loggroup...0006   (now allowed)
same name, SAME compartment               -> 409  "log group \"app-logs\" already exists
                                                   in compartment ...aaaaaaaademo"
put 3 entries (deliberately out of order) -> 200
search, default order                     -> ['e-10','e-20','e-30']
| sort by datetime desc                   -> ['e-30','e-20','e-10']
| where oracle.logid = <log>              -> 3 matched, oracle.* block populated
| where data.level = 'error'              -> ['e-10']
| stats count()                           -> 400 naming "stats"
| where a or b                            -> 400 naming "or"
| sort by source desc                     -> 400, sorts by datetime only
| where nope = 'x'                        -> 400 listing the resolvable fields
search "app-logs"                         -> 400, not a compartment OCID
delete log group                          -> 202, then GET -> 404

Steps 2/3 are the live proof for the per-compartment fix; the two sort lines for the ordering tests.