Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: yarn
- name: install dependencies
run: yarn install --frozen-lockfile --network-concurrency 1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests-kmip.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: 'yarn'

- name: Install socat (tunnel to help create disconnect and reconnect to pykmip)
Expand Down
11 changes: 9 additions & 2 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ on:
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# 24 is the target runtime; 22 is what engines.node still allows and
# what cloudserver/backbeat/scuba run until they move over.
node-version: ['22', '24']
Comment thread
francoisferrand marked this conversation as resolved.
services:
# Label used to access the service container
redis:
Expand All @@ -28,7 +34,7 @@ jobs:
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
node-version: ${{ matrix.node-version }}
cache: 'yarn'
- name: install dependencies
run: yarn install --frozen-lockfile --prefer-offline --network-concurrency 1
Expand All @@ -46,6 +52,7 @@ jobs:
- name: run functional tests
run: yarn ft_test
- uses: codecov/codecov-action@v4
if: matrix.node-version == '24'
with:
token: ${{ secrets.CODECOV_TOKEN }}
- name: run executables tests
Expand All @@ -62,7 +69,7 @@ jobs:
- name: Install NodeJS
uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: yarn
- name: Install dependencies
run: yarn install --frozen-lockfile --prefer-offline
Expand Down
49 changes: 25 additions & 24 deletions documentation/BucketInfoModelVersion.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,30 +4,31 @@

- v7 to v11 can mean different features between S3C & Zenko.
- Avoid using model version to check features
- used by model < v2 (for splitter)
- used in some tests (v5 and v10)

| Version | S3C (dev/7.X) | Zenko (dev/8.X) | Status |
|---------|-------------------------|--------------------------|-------------|
| v2 - v6 | ✅ | ✅ | ✅ Common |
| v7 | ObjectLock | UID | ❌ Collision|
| v8 | BucketNotification | ReadLocationConstraints | ❌ Collision|
| v9 | SSEConfiguredMasterKey | isNFS | ❌ Collision|
| v10 | UID | Ingestion | ❌ Collision|
| v11 | Tags | AzureInfo | ❌ Collision|
| v12 | - | ObjectLock | ↪️ Forward |
| v13 | - | BucketNotification | ↪️ Forward |
| v14 | - | SSEConfiguredMasterKey | ↪️ Forward |
| v15 | - | Tags | ↪️ Forward |
| v16 | - | Capabilities(VeeamSOSApi)| ✅ New |
| v17 | - | QuotaMax | ✅ New |
| **Unified at v17** |
- used by model < v2 (for splitter)
- used in some tests (v5 and v10)

| Version | S3C (dev/7.X) | Zenko (dev/8.X) | Status |
| ------- | ---------------------- | ------------------------- | ------------ |
| v2 - v6 | ✅ | ✅ | ✅ Common |
| v7 | ObjectLock | UID | ❌ Collision |
| v8 | BucketNotification | ReadLocationConstraints | ❌ Collision |
| v9 | SSEConfiguredMasterKey | isNFS | ❌ Collision |
| v10 | UID | Ingestion | ❌ Collision |
| v11 | Tags | AzureInfo | ❌ Collision |
| v12 | - | ObjectLock | ↪️ Forward |
| v13 | - | BucketNotification | ↪️ Forward |
| v14 | - | SSEConfiguredMasterKey | ↪️ Forward |
| v15 | - | Tags | ↪️ Forward |
| v16 | - | Capabilities(VeeamSOSApi) | ✅ New |
| v17 | - | QuotaMax | ✅ New |

Model versions are unified across S3C and Zenko starting at v17.

## Model Version 0/1

### Properties

``` javascript
```javascript
this._acl = aclInstance;
this._name = name;
this._owner = owner;
Expand All @@ -44,8 +45,8 @@ property not added until Model Version 2

### Properties Added

``` javascript
this._mdBucketModelVersion = mdBucketModelVersion || 0
```javascript
this._mdBucketModelVersion = mdBucketModelVersion || 0;
this._transient = transient || false;
this._deleted = deleted || false;
```
Expand Down Expand Up @@ -216,9 +217,9 @@ The Tag Set of a bucket is an array of objects with Key and Value:
[
{
Key: 'something',
Value: 'some_data'
}
]
Value: 'some_data',
},
];
```

### Usage
Expand Down
1 change: 1 addition & 0 deletions index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ export * as s3routes from './lib/s3routes';
export * as versioning from './lib/versioning';
export * as stream from './lib/stream';
export * as jsutil from './lib/jsutil';
export * as requestUrl from './lib/utils/requestUrl';
export { default as stringHash } from './lib/stringHash';
export * as db from './lib/db';
export * as errorUtils from './lib/errorUtils';
Expand Down
4 changes: 2 additions & 2 deletions lib/auth/v2/getCanonicalizedResource.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import * as url from 'url';
import { parseRequestTarget } from '../../utils/requestUrl';
import { type ArsenalRequest } from '../../types/ArsenalRequest';

const gcpSubresources = new Set([
Expand Down Expand Up @@ -50,7 +50,7 @@ export default function getCanonicalizedResource(request: ArsenalRequest, client
// If bucket specified in hostname, add to resourceString
let resourceString = request.gotBucketNameFromHost ? `/${request.bucketName}` : '';
// Add the path to the resourceString
resourceString += url.parse(request.url || '').pathname || '';
resourceString += parseRequestTarget(request.url || '').pathname;

/*
If request includes a specified subresource,
Expand Down
60 changes: 16 additions & 44 deletions lib/network/rest/RESTServer.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import assert from 'assert';
import * as url from 'url';
import * as werelogs from 'werelogs';
import * as http from 'http';
import httpServer from '../http/server';
import * as constants from '../../constants';
import { parseURL } from './utils';
import { parseRequestTarget } from '../../utils/requestUrl';
import * as httpUtils from '../http/utils';
import errors, { ArsenalError, errorInstances } from '../../errors';

Expand All @@ -19,23 +19,14 @@ function setContentRange(
) {
const [start, end] = byteRange;
assert(start !== undefined && end !== undefined);
response.setHeader('Content-Range',
`bytes ${start}-${end}/${objectSize}`);
response.setHeader('Content-Range', `bytes ${start}-${end}/${objectSize}`);
}

function sendError(
res: http.ServerResponse,
log: werelogs.RequestLogger,
error: ArsenalError,
optMessage?: string,
) {
function sendError(res: http.ServerResponse, log: werelogs.RequestLogger, error: ArsenalError, optMessage?: string) {
res.writeHead(error.code);
const message = optMessage ?? error.description ?? '';
log.debug('sending back error response', { httpCode: error.code,
errorType: error.message,
error: message });
res.end(`${JSON.stringify({ errorType: error.message,
errorMessage: message })}\n`);
log.debug('sending back error response', { httpCode: error.code, errorType: error.message, error: message });
res.end(`${JSON.stringify({ errorType: error.message, errorMessage: message })}\n`);
}

/**
Expand Down Expand Up @@ -64,7 +55,7 @@ export default class RESTServer extends httpServer {
port: number;
dataStore: any;
bindAddress?: string;
log: { logLevel: any; dumpLevel: any; };
log: { logLevel: any; dumpLevel: any };
}) {
assert(params.port);

Expand Down Expand Up @@ -105,9 +96,7 @@ export default class RESTServer extends httpServer {
* @return new request logger
*/
createLogger(reqUids?: string) {
return reqUids ?
this.logging.newRequestLoggerFromSerializedUids(reqUids) :
this.logging.newRequestLogger();
return reqUids ? this.logging.newRequestLoggerFromSerializedUids(reqUids) : this.logging.newRequestLogger();
}

/**
Expand All @@ -120,8 +109,7 @@ export default class RESTServer extends httpServer {
_onRequest(req: http.IncomingMessage, res: http.ServerResponse) {
const reqUids = req.headers['x-scal-request-uids'];
const log = this.createLogger(reqUids as string);
log.debug('request received', { method: req.method,
url: req.url });
log.debug('request received', { method: req.method, url: req.url });
const method = req.method ?? '';
if (method in this.reqMethods) {
this.reqMethods[method](req, res, log);
Expand All @@ -138,11 +126,7 @@ export default class RESTServer extends httpServer {
* @param res - HTTP response object
* @param log - logger object
*/
_onPut(
req: http.IncomingMessage,
res: http.ServerResponse,
log: werelogs.RequestLogger,
) {
_onPut(req: http.IncomingMessage, res: http.ServerResponse, log: werelogs.RequestLogger) {
let size: number;
try {
parseURL(req.url ?? '', false);
Expand All @@ -152,8 +136,7 @@ export default class RESTServer extends httpServer {
}
size = Number.parseInt(contentLength, 10);
if (Number.isNaN(size)) {
throw errorInstances.InvalidInput.customizeDescription(
'bad Content-Length');
throw errorInstances.InvalidInput.customizeDescription('bad Content-Length');
}
} catch (err: any) {
return sendError(res, log, err);
Expand All @@ -180,18 +163,13 @@ export default class RESTServer extends httpServer {
* @param res - HTTP response object
* @param log - logger object
*/
_onGet(
req: http.IncomingMessage,
res: http.ServerResponse,
log: werelogs.RequestLogger,
) {
_onGet(req: http.IncomingMessage, res: http.ServerResponse, log: werelogs.RequestLogger) {
let pathInfo: ReturnType<typeof parseURL>;
let rangeSpec: ReturnType<typeof httpUtils.parseRangeSpec> | undefined =
undefined;
let rangeSpec: ReturnType<typeof httpUtils.parseRangeSpec> | undefined = undefined;

// Get request on the toplevel endpoint with ?action
if (req.url?.startsWith(`${constants.dataFileURL}?`)) {
const queryParam = url.parse(req.url).query;
const queryParam = parseRequestTarget(req.url).query;
if (queryParam === 'diskUsage') {
return this.dataStore.getDiskUsage((err, result) => {
if (err) {
Expand Down Expand Up @@ -237,8 +215,7 @@ export default class RESTServer extends httpServer {
if (err) {
return sendError(res, log, err);
}
log.debug('sending back 200/206 response with contents',
{ key: pathInfo.key });
log.debug('sending back 200/206 response with contents', { key: pathInfo.key });
setContentLength(res, contentLength);
res.setHeader('Accept-Ranges', 'bytes');
if (byteRange) {
Expand All @@ -263,11 +240,7 @@ export default class RESTServer extends httpServer {
* @param res - HTTP response object
* @param log - logger object
*/
_onDelete(
req: http.IncomingMessage,
res: http.ServerResponse,
log: werelogs.RequestLogger,
) {
_onDelete(req: http.IncomingMessage, res: http.ServerResponse, log: werelogs.RequestLogger) {
let pathInfo: ReturnType<typeof parseURL>;
try {
pathInfo = parseURL(req.url ?? '', true);
Expand All @@ -278,8 +251,7 @@ export default class RESTServer extends httpServer {
if (err) {
return sendError(res, log, err);
}
log.debug('sending back 204 response to DELETE',
{ key: pathInfo.key });
log.debug('sending back 204 response to DELETE', { key: pathInfo.key });
res.writeHead(204);
return res.end(() => {
log.debug('DELETE response sent', { key: pathInfo.key });
Expand Down
20 changes: 7 additions & 13 deletions lib/network/rest/utils.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { errorInstances } from '../../errors';
import * as constants from '../../constants';
import * as url from 'url';
import { parseRequestTarget } from '../../utils/requestUrl';
const passthroughPrefixLength = constants.passthroughFileURL.length;

export function explodePath(path: string) {
Expand All @@ -15,8 +15,7 @@ export function explodePath(path: string) {
if (pathMatch) {
return {
service: pathMatch[1],
key: (pathMatch[3] !== undefined && pathMatch[3].length > 0 ?
pathMatch[3] : undefined),
key: pathMatch[3] !== undefined && pathMatch[3].length > 0 ? pathMatch[3] : undefined,
};
}
throw errorInstances.InvalidURI.customizeDescription('malformed URI');
Expand All @@ -35,16 +34,12 @@ export function explodePath(path: string) {
* - pathInfo.key {String} - The requested key
*/
export function parseURL(urlStr: string, expectKey: boolean) {
const urlObj = url.parse(urlStr);
const pathInfo = explodePath(decodeURI(urlObj.path!));
if ((pathInfo.service !== constants.dataFileURL)
&& (pathInfo.service !== constants.passthroughFileURL)) {
throw errorInstances.InvalidAction.customizeDescription(
`unsupported service '${pathInfo.service}'`);
const pathInfo = explodePath(decodeURI(parseRequestTarget(urlStr).path));
if (pathInfo.service !== constants.dataFileURL && pathInfo.service !== constants.passthroughFileURL) {
throw errorInstances.InvalidAction.customizeDescription(`unsupported service '${pathInfo.service}'`);
}
if (expectKey && pathInfo.key === undefined) {
throw errorInstances.MissingParameter.customizeDescription(
'URL is missing key');
throw errorInstances.MissingParameter.customizeDescription('URL is missing key');
}
if (!expectKey && pathInfo.key !== undefined) {
// note: we may implement rewrite functionality by allowing a
Expand All @@ -53,8 +48,7 @@ export function parseURL(urlStr: string, expectKey: boolean) {
// atomicity of the update (we would just remove the old
// object when the new one has been written entirely in this
// case, saving a request over an equivalent PUT + DELETE).
throw errorInstances.InvalidURI.customizeDescription(
'PUT url cannot contain a key');
throw errorInstances.InvalidURI.customizeDescription('PUT url cannot contain a key');
}
return pathInfo;
}
9 changes: 5 additions & 4 deletions lib/s3routes/routesUtils.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import * as url from 'url';
import * as querystring from 'querystring';
import { parseRequestTarget } from '../utils/requestUrl';
import * as http from 'http';
import { eachSeries } from 'async';

Expand Down Expand Up @@ -1119,12 +1120,12 @@ export function getBucketNameFromHost(request: http.IncomingMessage, validHosts:
* @return request object with additional attributes
*/
export function normalizeRequest(request: ArsenalRequest, validHosts: string[]) {
const parsedUrl = url.parse(request.url!, true);
request.query = parsedUrl.query as Record<string, string>;
const parsedUrl = parseRequestTarget(request.url!);
request.query = querystring.parse(parsedUrl.query ?? '') as Record<string, string>;
// TODO: make the namespace come from a config variable.
request.namespace = 'default';
// Parse bucket and/or object names from request
const resources = getResourceNames(request, parsedUrl.pathname!, validHosts);
const resources = getResourceNames(request, parsedUrl.pathname, validHosts);
request.gotBucketNameFromHost = resources.gotBucketNameFromHost ?? false;
request.bucketName = resources.bucket;
request.objectKey = resources.object;
Expand Down
Loading
Loading