Data pipeline architecture

[selvarajuv]

Validate how the data will flow through our app

As of now here are the major steps in our pipeline:

1. Ingestion: getting raw data from our sources
2. Raw Storage: storing our raw records
3. Parsing: turning raw records into a structured format
4. Parsed Storage: storing our parsed records
5. Feature engineering: preparing the data for model training
6. Modeling & Inference: producing model output
7. Inference Storage: storing model outputs
8. Retraining & Human Feedback: updating model performance based on human feedback

Are these the correct steps? is anything missing? Is anything redundant?

Cloud consideration:

• Depending on what cloud provider we choose, which services will we need?
• Mechanically how will the data flow (see ticket Cloud Options Research #2)? (In the context of AWS: Lambdas, SQS/SNS, Cron jobs, RDS, Sagemaker)

Acceptance criteria:

• Document how data will flow through our app
• Document what services will be needed for this flow to work
• Be ready to present at next meeting (09/30)