Security, compliance, and governance
Best Practices for Secure Data Engineering
Who controls which data (by Scoping Matrix scope)
| Data | What it is | Controlled by |
|---|---|---|
| User data | Inputs from end users | The customer, in every scope |
| Fine-tuning data | Data used to adapt a pre-trained model | The provider in Scopes 1–2, the customer in Scope 4 |
| Training data | The large dataset used to pre-train the model | The provider in Scopes 1–4, the customer in Scope 5 |
Data engineering lifecycle
An iterative cycle of collecting, preparing, and analyzing data to train, evaluate, and improve models. Collection tools on AWS include Amazon Kinesis, AWS Database Migration Service, and AWS Glue.
Secure data engineering practices
- Assess data quality
- Use privacy-enhancing technologies
- Control data access
- Ensure data integrity: validation checks (schema, referential, business rules), backup and recovery, transaction atomicity, lineage and audit trails, and regular testing of these controls
Cleaning, deduplicating, and splitting data are data preparation steps, not security practices.
The AWS Privacy Reference Architecture (AWS PRA) gives guidelines for designing privacy controls on AWS.