Security and Privacy Considerations for AI Systems

Five security tasks for AICopy Anchor Link

• Threat detection: watch for attackers exploiting AI, or using generative AI to generate fake content, manipulate data, or automate attacks.
• Vulnerability management: find software bugs and AI model weaknesses through assessments, penetration tests, code reviews, and patching.
• Infrastructure protection: secure cloud platforms, edge devices, and data stores with access controls, network segmentation, and encryption.
• Prompt injection: attackers craft inputs to manipulate the model. Mitigate with prompt filtering, sanitization, and validation, and with robust training.
• Data encryption: encrypt data at rest and in transit, and protect the keys.

OWASP Top 10 for LLMsCopy Anchor Link

1. Prompt injection: malicious input manipulates the model
2. Insecure output handling: outputs aren't validated
3. Training data poisoning: malicious data in the training set
4. Model denial of service: attacks on availability
5. Supply chain vulnerabilities: weak third-party components
6. Sensitive information disclosure: data leaks through outputs
7. Insecure plugin design: exploitable extensions
8. Excessive agency: too much autonomy for the model
9. Overreliance: trusting outputs without checking them
10. Model theft: copying the model's weights or architecture

MITRE ATLAS is a knowledge base of adversary tactics against AI systems.

Phishing, DDoS, and ransomware are general IT threats, not AI-specific considerations.