added log summary, reduction, and document

[TonyB9000]

Description

• Closes #<ISSUE_NUMBER_HERE>

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated (if needed)
• All tests pass (locally and CI/CD)
• Documentation/comments updated (if needed)
• Breaking change noted (if applicable)

Deployment Notes (if any)

[chengzhuzhang]

@TonyB9000 could you try fixing the pre-comitting issue that failed CI/CD tests? Also somehow, I won't assign a Copilot review on this PR. Maybe @tomvothecoder can assign before your Copilot Seat is ready.

[TonyB9000]

@chengzhuzhang Jill, this (draft) PR lists the three files I've added:

Log_Management.md (an overview)
summarize_collection.sh (a brief way to reduce 96+1 verbose daily logs to a single summary
log_redux.sh (reduces up to 700 verbose logs to a single "Sunday to Saturday" tar.gz file.

Very Brief Overview: Every day, 96 2.5MB (15 minute) "staging" collection logs and one daily archive log are produced. Logs originally accumulate forever in a "raw_logs" directory.

Whenever "summarize_collection.sh" is run, every set of raw logs for any full day (should be 96+1 logs) is summarized in a daily "Collection_Report", and the source files are moved to a directory "history_logs".

Whenever "log_redux" is run, any logs found in "history_logs", are identified according to which week-block (UTC Sunday 00:00 to Saturday 23:59) they fall into, and either a new tar_gz file is created for them, or an existing tar_gz is opened for them and they are added.

To avoid adding the same file over and over to the same tar_gx, the source files would ordinarily be deleted. With fry_run enabled for log_redux, they are simply moved to "history_logs/deleted/"

The log_redux has been successfully tested, each archiving up to 7*(96+1) = 679 logs, with a consistent reduction in volume to just 4% or original, and reduction of inodes used from 679 to 1.

[TonyB9000]

Hi Jill, No problem. I used sed to remove all traces of trailing whitespace from all lines (why this causes issues is beyond me) and did another push. (It fails on the “Log_Management.md text file – can you believe that?) If it fails again, I’ll need Copilot to figure it out. Cheers! ____tony____

[tomvothecoder]

Thanks for opening this PR Tony. No need to review with Copilot yet. Let's iterate some more on the implementation details.

I think we should adapt your bash scripts into Python modules that have unit tests, similar to the ingestion scripts. Then we can just hook those Python modules to the crontab and have them run like the other jobs. I see you're experimenting with GitHub Copilot. This is a great task to assign to Copilot to do.

We also discussed expanding the scope of this feature to being database-backed to make it easier to manage operations in a dashboard. I'll work on this separately in #217.

[TonyB9000]

@tomvothecoder Conversion of the "log_redux" script to python should be straightforward. The "summarize" script is just a cheap shot, and will wholly depend upon how logging itself is revamped (syntax, etc).

[tomvothecoder]

Noting that this PR is dependent on #365