The rise of artificial intelligence (AI) in information systems is transforming how organisations make decisions, but it also introduces a new category of incidents: those directly concerning the models. Analysing these malfunctions requires a tailored forensic approach, capable of capturing and preserving artefacts that differ significantly from those usually collected in traditional cybersecurity. This article describes the regulatory framework, the typology of traces to be preserved, best practices for acquisition and analysis, as well as the technical and legal limitations that teams must prepare for.
Introduction – An AI incident and its forensic implications
Imagine a bank using a machine learning-based credit scoring model to approve or reject applications in real time. During the peak mortgage season, the acceptance rate plummets: 30% of files are rejected, whereas historical data shows stability at over 85%. The investigation reveals that the model was 'poisoned' via the injection of manipulated data into the training pipeline, thereby altering its generalisation capability.
This scenario, which is no longer hypothetical according to the AI Incident Database (see [1]), demonstrates that AI incidents are not limited to simple software failures or unauthorised access: they affect the quality and integrity of the model, as well as its execution environment. Forensic response must therefore extend far beyond traditional system logs to include:
- training datasets and their metadata,
- hyperparameters and CI/CD pipeline configuration,
- the weights and architecture of the deployed model,
- containerisation or virtualisation artefacts,
- inference API access logs.
Together, these traces help reconstruct when and how the compromise happened and why, identify the actors (internal, external, or automated), and guide corrective measures. This article maps the traces to preserve, outlines a reliable acquisition process, and sets out an analysis framework aligned with international standards.
1. Conceptual Framework: From Traditional Forensics to AI Forensics
| Dimension | Traditional Digital Forensics | Forensic AI (new) |
|---|---|---|
| Object | Systems, networks, binary files, access logs | Models, datasets, training pipelines, deployment artefacts |
| Time scope | Primarily post-incident (retroactive analysis) | Requires continuous preservation from the data ingestion phase |
| Evidence sources | System logs, disk images, network captures | Dataset snapshots, versioning metadata, configuration files, container artefacts |
| Reference standards | ISO/IEC 27043 (Incident management), NIST SP 800-86 (Forensics guide) | ISO/IEC 27035-2 (AI Incident Response), NIST AI RMF, AI Act (art. 14) |
Digital forensics standards (ISO/IEC 27043, NIST SP 800-86) remain relevant to collecting infrastructure logs and preserving evidence integrity. However, the NIST Artificial Intelligence Risk Management Framework (AI RMF) and the annexes of the European AI Act (§ 14) explicitly introduce the concept of 'model traces' as elements to be protected during an AI incident [2][3].
2. Typology of artefacts to be collected
2.1 Input and training data
- Raw datasets – CSV files, images, event logs, NoSQL databases.
- Provenance metadata – collection timestamps, source (API, sensor), schema version.
- Annotations and labels – label generation scripts, validation policies.
2.2 Training pipeline artefacts
| Item | Description | Reason for preservation |
|---|---|---|
| Source code (Python/Scala scripts, notebooks) | Represents the pre-processing and training logic. | Enables exact reproduction of the transformations applied to the data. |
| Job configuration – YAML/JSON files (e.g., mlflow.yaml, kubeflow pipeline) | Declares hyperparameters, resources, and library versions. | Facilitates the reconstruction of the training context and the identification of unauthorised changes. |
| Execution environment – Docker images, dependencies (requirements.txt), framework versions (TensorFlow 2.8, PyTorch 1.12) | Ensures software reproducibility. | Crucial for detecting version discrepancies that could introduce vulnerabilities or biases. |
| Model weights – .pt, .h5 files | The functional core of the model. | Any alteration (e.g., back-door injection) is directly observable via hashing and checksum comparison. |
2.3 Deployment and inference artefacts
- Containers or VMs – immutable images, Kubernetes manifests (Deployment, Service), Helm charts.
- API Endpoints – OpenAPI/Swagger specifications, call logs (payloads, responses).
- Observability metrics – distributed traces (Jaeger, Zipkin), Prometheus metrics (latency, error_rate), security alerts.
2.4 Infrastructure and security logs
- System logs – syslog, journald.
- IAM/RBAC access audits – who deployed or modified the model? (e.g., AWS CloudTrail, Azure Activity Log).
- IDS/IPS alerts – network anomaly detections around the inference service.
3. Chain of custody – from entry point to secure storage
3.1 Strategic capture points
| Phase | Capture point | Associated artefacts |
|---|---|---|
| Data Ingestion | Prior to any transformation (raw bucket, data lake) | Raw datasets, acquisition metadata |
| Pre-processing / Feature engineering | After each critical stage (e.g., normalisation) | Scripts, transformation logs |
| Training | At the end of the job (snapshot) | Model weights, hyperparameters, CI/CD artefacts |
| Deployment | During docker push or kubectl apply | Docker image, manifests, digital signatures |
| Inference | At every critical call (e.g., financial decision) | Input/output payloads, log traces |
3.2 Reliable acquisition methods
- Immutable snapshot – Use of append-only file systems (e.g., Amazon S3 Object Lock, Azure Immutable Blob Storage) to guarantee immutability.
- Cryptographic hashing – SHA-256 or BLAKE2b calculation on each artefact at the time of the snapshot; storage of hashes in a proof registry (private blockchain or time-stamping service such as OpenTimestamps).
- Secure timestamping – Use of authenticated NTP services and electronic signatures (X.509) to attest chronology.
- Encryption at rest – AES-256 GCM, with key management via HSM or KMS to reconcile preservation and confidentiality requirements.
3.3 Long-term retention
NIST SP 800-101 recommends a minimum retention period of three years for artefacts related to critical incidents [4]. However, organisations must align this period with GDPR obligations (Article 30) and, where applicable, the AI Act, which mandates the retention of audit data for at least five years for high-risk systems.
4. Forensic analysis – reconstructing the AI incident
4.1 Pipeline reconstruction
By comparing successive snapshots (pre- and post-incident), the following can be identified:
- Dataset differences: addition or removal of records, modification of labels (detected via diff on hashes).
- Hyperparameter modification: unexpected variation in learning rate or number of epochs – often an indicator of unauthorised retraining.
- Weight alteration: binary comparison or use of similarity metrics (cosine similarity) between the suspect model and the signed version.
4.2 Detection of specific attacks
| Attack type | Telltale signs |
|---|---|
| Data poisoning | Concentration of malicious samples within a batch, outliers detected via clustering (t-SNE) on embeddings. |
| Back-door / Trigger | Model activation only for inputs containing a specific sequence (e.g., pixel pattern). Analysis of the decision function via Feature Attribution (SHAP, LIME). |
| Model extraction | Abnormally high volume of inference calls from a single IP, API logs showing structured requests aimed at model reconstruction. |
| Adversarial evasion | Sudden increase in the false negative rate within production evaluation metrics; inspection of adversarial gradients via FGSM or PGD. |
4.3 Attribution and chain of responsibility
By cross-referencing:
- IAM logs (who triggered the training job),
- Git commits associated with pre-processing scripts,
- JIRA/ServiceNow tickets related to deployment,
a timeline of actions can be established to identify potential actors (developer, data scientist, CI/CD automation). Traceability must be documented in an Incident Report compliant with ISO 27043.
5. Use Case – Poisoning incident on a credit scoring model
Context
A French bank uses CreditScoreNet, a neural network trained weekly on new credit applications. The CI/CD pipeline relies on GitLab, Docker and Kubeflow.
Incident Timeline
- Day D-2: A new training dataset is ingested from an S3 bucket shared with a marketing partner.
- Day D-1: The training job starts automatically, creating a Docker image containing the weights creditscorenet_v20230715.pt.
- Day D: Following deployment, the rejection rate increases from 15% to 45%. Analysts detect a spike in errors within the 'low income' category.
- Investigation: Snapshot analysis reveals that 2% of the dataset contains falsified records (artificially inflated incomes) originating from the shared directory /partner/data/.
- Conclusion: The partner accidentally injected a data subset containing manipulated values, resulting in label flipping that biases the model.
Preserved Evidence
| Artefact | Capture Method |
|---|---|
| Raw Dataset (S3) | S3 Versioning with Object Lock + SHA-256 hash |
| Ingestion Script (ingest.py) | GPG-signed, timestamped Git commit |
| Model snapshot (creditscorenet_v20230715.pt) | Immutable storage in Glacier Deep Archive, RSA signature |
| Kubeflow and Docker logs | Forwarded to SIEM (Elastic) with 2-year retention |
Lessons learned
- Data source segregation – Implement data contracts and integrity checks (checksum) prior to ingestion.
- Automated pipeline audit – Use robustness tests (e.g., Data Validation via Deequ or Great Expectations).
- IAM access management – Restrict write permissions on the shared bucket to authorised teams only.
6. AI forensic risks and limitations
| Risk | Description | Mitigation |
|---|---|---|
| Massive volume of artefacts (TB datasets, GB models) | Costly storage, long collection times. | Lossless compression + selective archiving (e.g., retaining only version diffs). |
| Data confidentiality – datasets containing PII. | Risk of leakage during preservation or sharing with forensic teams. | Encryption before transfer, partial anonymisation, access based on the principle of least privilege. |
| Cloud volatility – ephemeral resources (serverless). | Difficulty in capturing consistent snapshots. | Utilise native immutable storage services and snapshot APIs (e.g., AWS EBS Snapshots, Azure Disk Snapshot) immediately upon incident trigger. |
| Jurisdiction & sovereignty – data stored across multiple regions. | Conflicts with GDPR or the AI Act regarding evidence localisation. | Define a data residency policy and rely on providers offering dedicated geographical zones. |
| Malicious tampering of evidence. | An attacker may attempt to modify logs to conceal their actions. | Chains of trust based on digital signatures, third-party timestamping, and write-once storage. |
7. Operational Recommendations – Decision Makers' Guide
7.1 Governance
- Create an 'AI Forensics Playbook' aligned with ISO/IEC 27035-2, including roles (Data Custodian, AI Incident Responder) and escalation procedures.
- Integrate AI forensics into the SOC – model metric deviation alerts should automatically trigger an artefact acquisition workflow.
7.2 Technical Architecture
| Action | Concrete Implementation |
|---|---|
| Immutable versioning of datasets | S3 Object Lock + AWS Glacier Deep Archive, or Azure Immutable Blob Storage. |
| Model signing | Use mlflow with model signatures plugin (SHA-256 hash + X.509 certificate). |
| Automated pipeline capture | Deploy a side-car container that exports CI/CD metadata to the SIEM for every run. |
| Trusted timestamping | RFC 3161 timestamping service (e.g., DigiCert Timestamp), integrated into acquisition scripts. |
7.3 Investigation process
- Triggering – Metric alerts or manual request.
- Isolation – Pause the affected pipeline (e.g., kubectl scale deployment creditscorenet --replicas=0).
- Acquisition – Immutable snapshots of datasets, models, containers and logs.
- Analysis – Hash comparison, training graph reconstruction, attack pattern search.
- Reporting – Documenting the chain of causality, business impacts, and recommendations.
- Remediation – Restoration from signed versions, updating quality controls.
7.4 Training and culture
*
- Train data scientists in data provenance best practices (e.g., using Data Version Control – DVC).