CoreQ Academy1.5 Summarize the information security challenges associated with artificial intelligence (AI) adoption.
Adopting AI introduces legal, ethical and technical risks. Architects must govern AI use, defend models and apps, and plan for AI-enabled attackers.
Legal and privacy implications
- Potential misuse
- AI being used for purposes it was not approved for, or to harm people.
- Explainable vs. non-explainable models
- Some models can justify decisions; others are black boxes, which is a problem for regulated decisions.
- Organizational policies on the use of AI
- Rules on approved tools, data allowed and review requirements.
- Ethical governance
- Oversight ensuring AI use is fair, transparent and accountable.
Threats to the model
- Prompt injection
- Inputs that override model instructions.
- Insecure output handling
- Using model output unvalidated in downstream systems.
- Training data poisoning
- Corrupting training data to alter behaviour.
- Model denial of service (DoS)
- Expensive requests that exhaust resources or budget.
- Supply chain vulnerabilities
- Compromised models, datasets or libraries.
- Model theft
- Stealing weights or replicating the model through queries.
- Model inversion
- Reconstructing training data from model outputs.
AI-enabled attacks
- Insecure plug-in design
- Plug-ins with weak authorisation or input validation.
- Deepfake
- Synthetic media impersonating real people.
- Digital media
- Fake images, audio or video.
- Interactivity
- Real-time deepfakes in live calls.
- AI pipeline injections
- Injecting malicious content into the data or prompts feeding AI pipelines.
- Social engineering
- AI-generated, personalised lures at scale.
- Automated exploit generation
- AI helping attackers build exploits faster.
Risks of AI usage
- Overreliance
- Trusting output without verification.
- Sensitive information disclosure
- Leakage of confidential data.
- To the model
- Users sending sensitive data to models or providers.
- From the model
- Models revealing training data, retrieved documents or system prompts.
- Excessive agency of the AI
- AI with more permissions or autonomy than needed.
AI-enabled assistants/digital workers
- Access/permissions
- Scope assistants' access to the minimum, per user.
- Guardrails
- Controls on what assistants may do and output.
- Data loss prevention (DLP)
- Inspect prompts and outputs for sensitive data.
- Disclosure of AI usage
- Telling users and customers when they are interacting with AI or AI-generated content.
Check your understanding
Loading quiz…
Original CoreQ Academy material, mapped to objective 1.5 of the publicly available CompTIA SecurityX CAS-005 exam objectives (Exam objectives V5, document version 7.0). Not affiliated with or endorsed by CompTIA. Contains no exam questions.