Best Practices for Secure Data Engineering

Review of data usage in generative AICopy Anchor Link

A generative AI application typically includes customer data, fine-tuning data, and training data. You learned about the Generative AI Security Scoping Matrix in the Approaches for Implementing Governance Strategies lesson. Depending on the scope of the application, the ownership and control of the data will vary between the customer and the application provider. The following is a review of the common data usage in generative AI.

User dataCopy Anchor Link

User data represents the specific inputs or requirements provided by the customers or end-users. This data is used to generate or personalize the output of the generative AI model.

For all application scopes, the customer controls their data.

Fine-tuning dataCopy Anchor Link

This data is used to adapt or fine-tune the pre-trained a generative AI model to the specific needs or preferences of the customers or the application domain. Here are some more details about the fine-tuning data:

• The fine-tuning data is typically a subset of the training data or additional data collected from the application domain.
• The fine-tuning process adjusts the model's parameters and weights to better fit the fine-tuning data, allowing the model to generate more relevant and personalized outputs.

For application Scopes 1 and 2, the application provider controls the fine-tuning data.

For application Scope 4, the customer controls the fine-tuning data.

Training dataCopy Anchor Link

Training data is the comprehensive dataset used to train the initial pre-trained generative AI model. Here are some more details about the training-data:

• The training data is typically a large and diverse collection of data, such as text, images, or audio, depending on the specific application.
• The training data is used to build the fundamental knowledge and capabilities of the generative AI model.

For application Scopes 1, 2, 3, and 4, the application provider controls the training data.

For application Scope 5, the customer controls the training data.

Data flows in a generative AI application

The following graphic shows an example data flow for a generic Scope 1 and Scope 2 application.

2: App queries custom data

2: App queries custom dataCopy Anchor Link

The application might optionally query data from custom data sources using plugins.

Securing Generative AI: Applying Relevant Security Controls

To learn about data flows for other application scopes, choose the following link.

AWS Blog

What is the data engineering lifecycle?Copy Anchor Link

The data engineering lifecycle is an iterative process where the data is collected, prepared, and analyzed. This data is then used to train, evaluate, and continuously improve the AI or generative AI models. This lifecycle ensures that the underlying data is of high quality, representative, and optimized for the specific AI or generative AI use case. Ultimately, this process contributes to the success and performance of the AI or generative AI systems.

2: Data collection

2: Data collectionCopy Anchor Link

This diagram shows how the data collection stage fits into the data engineering automation and access control lifecycle.

AWS provides various data collection tools. Some of these tools include:

• Amazon Kinesis
• AWS Database Migration Service
• AWS Glue

Secure data engineeringCopy Anchor Link

Secure data engineering practices are essential for ensuring the safety and reliability of AI and generative AI systems. The following are some best practices to consider.

• Assessing data quality
• Implementing privacy-enhancing technologies
• Data access control
• Data integrity

In the context of AI and generative AI, data integrity refers to the quality, accuracy, and reliability of the data used to train the AI models. It ensures that the data used for model development, training, and deployment is complete, consistent, and free from errors or inconsistencies. The following practices will ensure data integrity:

• Implement data validation and integrity checks at various stages of the data pipeline, such as schema validation, referential integrity checks, and business rule validations.
• Maintain a robust data backup and recovery strategy to ensure data can be restored in case of errors, system failures, or natural disasters.
• Employ transaction management and atomicity principles to ensure data consistency and reliability during data processing and transformation.
• Maintain detailed data lineage and audit trails to track the origin, transformations, and changes made to the data.
• Regularly monitor and test the data integrity controls to ensure their effectiveness and make necessary adjustments.

AWS Privacy Reference Architecture

The AWS Privacy Reference Architecture (AWS PRA) offers a set of guidelines to assist in the design and implementation of privacy-supporting controls within AWS services. This guide can help you make informed decisions regarding the people, processes, and technology that are necessary to ensure privacy in the AWS Cloud environment.

AWS PRA

To learn more about the AWS Prescriptive Guidance for AWS PRA, choose the following link.

Documentation

Additional resourcesCopy Anchor Link

Data Engineering

The AWS Cloud Adoption Framework provides prescriptive guidance for data engineering. To explore the documentation, choose the following link.

AWS Documentation

Data Engineering Principles

To learn more about best practices for designing and implementing modern data-centric architecture use cases, choose the following link.

AWS Documentation

Data Lifecycle

To learn more about the stages in the data lifecycle process, choose the following link.

AWS Documentation

Finally, you will answer some questions to check your knowledge.