Responsible Considerations to Select a Model
Selecting a model is one of the first and most critical steps to developing an AI system. Model selection has strategic implications for how the AI system will perform. Everything from user experience and go-to-market to hiring and profitability can be affected by selecting the right model for your use case.
Remember that you can use Model evaluation on Amazon Bedrock or SageMaker AI Clarify to evaluate models for accuracy, robustness, toxicity, or nuanced content that requires human judgement.
Define application use case narrowly
When selecting a model for your AI application, you must narrowly define your use case. This is important because you can tune your model for that specific use case.
Example: Defining application use case narrowly for traditional AI
In this example, you might have an AI application that uses face recognition. Face recognition is not a use case; it is a technology. The way your model applies that technology is a use case.
Gallery retrieval
Celebrity recognition
Virtual proctoring
For example, a gallery retrieval application might be used to help find missing persons. In this case, you would need a model that can be tuned for favor recall or precision. Favor recall would bring up many results that could be beneficial to the use case of the AI application used in finding missing persons.
However, if your AI application is being used for celebrity recognition or virtual proctoring, the model would only need to favor precision. This is because the favor recall tuning would provide too many results to be beneficial to the use case of the application.
Example: Defining application use case narrowly for generative AI
In this example, you might have an AI application to assist customers in shopping on your online store. The use case might be to provide a product catalog or to persuade customers to buy products. An appropriate model would need to be selected based on the narrowly defined use case.
| Features | Catalog a product | Persuade to buy |
|---|---|---|
| Target audience | Broad demographic | Narrow demographic |
| Possible issues | Veracity | Veracity, unwanted bias, toxicity, detail |
| Consequences | Brand damage, lost sales, and returns | Representative harm, brand damage, lost sales, and returns |
| Tuning | Favors neutrality, clarity, and completeness | Focuses on highest interest problem and benefit to group |
In an AI application to catalog a product, you would want a broad demographic target audience so that it is available for all of your customers.
In an AI application to persuade to buy, you would want a narrow target audience to capture a specific group of people. For example, you might want to target an audience that lives on the coast to buy accessories for docking boats.
Choosing a model based on performance
Model performance varies across a number of factors, including the following:
- Level of customization – The ability to change a model’s output with new data ranging from prompt-based approaches to full model retraining
- Model size – The amount of information the model has learned as defined by parameter count
- Inference options – From self-managed deployment to API calls
- Licensing agreements – Some agreements can restrict or prohibit commercial use
- Context windows – The amount of information that can fit in a single prompt
- Latency – The amount of time it takes for a model to generate an output
Consider a model based on performance with test datasets
A common mistake when choosing a model is to assume that the model, in and of itself, is either good or bad. This is not the case. Performance is a function of the model and a test dataset, not just the model. So, when you are assessing a model, you need to determine how well a model performs on a particular dataset.
Model that performs differently on different datasets
For example, a model might perform well on test dataset A over a period of time. The model might perform even better on test dataset B. However, the model might progressively get worst on test dataset C.
This means that you need to consider two development trajectories: the development trajectory of the model and the development trajectory of the datasets. Remember the dataset is not necessarily constant. It is often evolving.
Choosing a model based on sustainability concerns
Sustainability in the context of responsible AI refers to the ability of AI systems to be developed and deployed in a way that is socially, environmentally, and economically sustainable over the long term.
Responsible agency considerations for selecting a model
Responsible agency in responsible AI refers to an AI system's capacity to make good judgments and act in a socially responsible manner. The following are key aspects of moral agency for AI.
Value alignment
Value alignment is being able to understand, evaluate, and make decisions based on moral principles rather than pure utility maximization. This requires value alignment between the AI system's goals and values and the responsible human values.
Responsible reasoning skills
Responsible reasoning skills is being able to logically think through moral dilemmas and weigh various responsible considerations when making decisions. The AI needs logic and reasoning capabilities to apply responsible principles to novel situations.
The AI system should have the capacity to engage in responsible reasoning and understand moral concepts, principles, and frameworks. It should be able to apply them in context to specific situations.
Appropriate level of autonomy
The AI system should have the appropriate level of autonomy, with clear boundaries and mechanisms for human oversight and intervention, particularly in high-stakes or sensitive domains.
Transparency and accountability
The AI system should be transparent about its decision-making process. It should allow external oversight and accountability to ensure its actions are responsibly justified.
Overall, responsible agency requires AI to have sophisticated intelligence on par with human-level cognition to properly apply ethical reasoning in the real world. This remains an immense challenge for current AI.
Environmental considerations for selecting a model
When you are developing and deploying AI systems, use environmental considerations as you implement responsible AI.
The following are key environmental challenges and solutions to consider when choosing a model.
Energy consumption
| Challenge | Solution |
|---|---|
| Training large AI models and running them at scale can consume significant amounts of energy and contribute to greenhouse gas emissions and environmental impact. | The solution is to optimize energy efficiency in AI systems, use renewable energy sources where possible, and consider the overall carbon footprint of AI operations. |
Resources utilization
| Challenge | Solution |
|---|---|
| AI systems often require substantial computational resources, including specialized hardware, such as GPUs and TPUs, and data center infrastructure. The manufacturing and disposal of these resources can have environmental impacts. | Responsible AI should aim to maximize resource efficiency, promote hardware reusability and recyclability, and minimize electronic waste. |
Environmental impact assessment
| Challenge | Solution |
|---|---|
| Before deploying AI systems, it is important to assess their potential environmental impacts, both direct (for example, energy consumption and resource usage) and indirect (for example, enabling or promoting environmentally harmful activities). | Environmental impact assessments should be conducted, and mitigation strategies should be implemented if necessary. |
Economic considerations for selecting a model
Economic considerations in responsible AI include the potential benefits and costs of AI technologies and the impact on jobs and the economy.
For example, AI can automate certain tasks and improve efficiency, but it can also lead to job displacement and inequality. Additionally, there are concerns about the concentration of power and data in the hands of a few companies, which could lead to monopolies and further inequality.
In the next lesson, you will learn about about responsible preparations for datasets.