The Customer Is the One Left Holding the Risk
When companies lose control of personal data, who is accountable for the harm that comes years later?
A point-of-view article by Chirag Dani.
A deeper exploration of customer accountability, autonomous systems and the next generation of cyber regulation.
The breach ends. The customer’s risk does not.
I have been thinking about a question that rarely receives the same attention as the breach itself.
When a large organisation is attacked and customer information is stolen, we usually discuss three things:
How did the attacker get in?
How much data was stolen?
How much will the organisation be fined?
I think we are asking the wrong final question.
The question I increasingly want boards, regulators, technology leaders and legislators to answer is: Who carries the risk after the organisation has lost control of my identity?
A company can rebuild its systems.
It can restore its backups.
It can replace compromised infrastructure.
It can appoint a new security leader.
It can issue a public statement.
It can refuse to pay – or negotiate – an extortion demand.
It can eventually close the incident.
But if my passport information, driver’s licence, date of birth, address, financial information or other identity attributes have entered a criminal ecosystem, my incident may have only just begun.
It may surface three weeks later.
Three months later.
Three years later.
Perhaps even longer.
And when it does, I may be asked to prove something extraordinarily difficult: Prove that this fraud happened because of that particular breach.
That is where I believe our current cybersecurity and privacy frameworks have a significant structural weakness.
The customer did not create the risk
Consider a hypothetical but entirely realistic scenario.
A major financial or telecommunications organisation suffers a sophisticated cyberattack.
The attacker obtains:
- names;
- dates of birth;
- addresses;
- telephone numbers;
- email addresses;
- account identifiers;
- identity-document information;
- customer records.
The organisation informs affected customers.
The public statement says: “We take the protection of customer information extremely seriously.”
Another sentence follows: “At this stage, we have no evidence that the information has been misused.”
The company offers a monitoring service for twelve months.
The incident gradually disappears from the news.
But the stolen information doesn’t necessarily disappear.
Criminal groups can sell, combine, enrich and reuse data.
A stolen identity attribute can be combined with information obtained from another breach.
An email address can be connected with a telephone number.
A date of birth can be combined with a historical address.
An identity document can be paired with a photograph.
AI can increasingly automate the process of creating convincing social-engineering messages and fraudulent documents.
The result is not necessarily one large attack.
It can be a series of small attacks against the individual.
The three-year problem
This is the scenario I believe regulators need to think much more seriously about.
2023: Your identity information was compromised.
2023–2024: Nothing apparently happens.
2025: Your information appears in another criminal dataset.
2026: Someone applies for a financial product using your identity.
2029: The financial institution asks you: “How do you know this information came from the original breach?”
That question is reasonable from the institution’s perspective.
But it exposes a fundamental asymmetry.
The organisation that originally lost the information possesses:
- security logs;
- incident-response records;
- affected-data inventories;
- forensic reports;
- attacker indicators;
- timestamps;
- database access records;
- evidence of exfiltration.
The individual usually possesses none of that.
The customer simply knows: “Your organisation told me my identity information was stolen.”
This is why I believe data-breach liability needs to evolve from incident liability toward lifecycle liability.
A breach should not be measured only by the day it was discovered
Traditional incident management has a relatively straightforward lifecycle: Attack → Detection → Containment → Investigation → Notification → Remediation → Closure
But personal-data compromise follows a different lifecycle: Collection → Exposure → Theft → Criminal retention → Distribution → Enrichment → Dormancy → Reuse → Individual harm
Those timelines are fundamentally different.
The first can be measured in days or months.
The second can extend for years.
That difference matters enormously.
The economic value of stolen data is changing
The criminal value of personal data is no longer simply the value of the original record.
The value comes from combination.
One breach might provide: name + email + telephone number
Another might provide: date of birth + address
A third might provide: identity-document information.
A criminal actor can combine these datasets.
Generative AI adds another layer.
AI can help automate:
- highly personalised phishing;
- impersonation;
- synthetic identity construction;
- fraudulent communications;
- document generation;
- social-engineering conversations;
- credential attacks;
- targeting and prioritisation.
This changes the risk equation.
The question is no longer simply: “What did the attacker steal?”
It becomes: “What future capabilities does the stolen information give an attacker?”
That is a much more difficult regulatory problem.
And now we are introducing autonomous systems into this equation
This is where my concern becomes even greater.
We have spent years discussing whether a human attacker should be punished for misusing stolen customer data.
Now imagine a different architecture.
An organisation deploys an agentic AI system.
The system has access to:
- customer databases;
- CRM;
- email;
- internal documents;
- APIs;
- transaction systems;
- identity services;
- external tools.
The organisation gives the agent authority because it wants automation.
Then something goes wrong.
Perhaps an attacker manipulates an external document.
Perhaps an indirect prompt injection changes the agent’s behaviour.
Perhaps a compromised tool is called.
Perhaps the agent’s credentials are stolen.
Perhaps the agent itself behaves outside its intended operating boundary.
The agent retrieves customer information and sends it somewhere it should not.
Now ask the uncomfortable question: Who is responsible?
The AI model provider?
The company that deployed the agent?
The developer who configured its permissions?
The cloud provider?
The tool provider?
The operator of the compromised external system?
The person who designed the workflow?
The agent itself?
Or nobody?
At present, responsibility is generally going to be allocated through existing legal concepts rather than through a mature, universally accepted agentic accountability model.
That is not sufficient for the systems we are beginning to deploy.
Autonomous vehicles and IoT expose the same weakness
Consider another example.
A connected vehicle contains:
- location history;
- driver behaviour;
- contact information;
- vehicle telemetry;
- potentially payment information;
- connected-device credentials.
Now imagine the vehicle’s connected ecosystem is compromised.
The attacker doesn’t merely steal information.
They manipulate the vehicle’s digital environment and cause physical consequences.
Or consider an IoT ecosystem in a home.
A connected device has access to:
- household information;
- cameras;
- microphones;
- location;
- schedules;
- other connected devices.
If that ecosystem is compromised and personal data is misused, the traditional question: “Was there a data breach?”
may actually be too narrow.
The event could involve simultaneously: privacy + cybersecurity + consumer protection + product safety + physical safety.
Yet our regulatory structures are frequently divided along exactly those boundaries.
We have created a responsibility gap
This is how I see the emerging problem.
Traditional data breach: Organisation → customer data → attacker
Responsibility is relatively straightforward.
Connected-device incident: Manufacturer → software → device → cloud → user → attacker
Responsibility becomes distributed.
Agentic AI incident: Model provider → AI platform → enterprise → agent → tools → APIs → customer data → external world
Responsibility becomes even more distributed.
And the more distributed the architecture becomes, the easier it becomes for every participant to argue: “The failure occurred somewhere else.”
That is dangerous.
The “responsibility gap”
I would define it as: The responsibility gap is the condition in which multiple technology, service and operational participants collectively enable an AI-, cyber- or data-related harm, while no single participant has a clearly defined legal obligation to prevent, absorb, evidence or compensate the resulting individual harm.
This is not simply a cybersecurity problem.
It is a governance architecture problem.
We need to stop treating personal data as an organisational asset only
This is another fundamental shift I believe is necessary.
An organisation may store my information.
But that doesn’t mean the information has become economically or morally equivalent to inventory.
My identity remains mine.
The organisation is a custodian.
That distinction should have consequences.
If I give an organisation my identity information because the organisation requires it to provide a service, I am effectively entering into a trust relationship.
The organisation is saying: “Give us this information. We will protect it.”
If it fails catastrophically, I don’t believe: “We notified you”
should be the end of that relationship.
What should replace the current model?
I believe we need to move toward an inclusion of proposed governance constructs, I call: Customer Data Harm Accountability Framework – CDHAF
The objective would be simple: Shift data-breach regulation from notification-centric accountability toward customer-risk accountability.
I would structure it around seven principles.
1. Data Exposure Severity – not just number of records
A breach involving 10 million email addresses should not automatically be treated the same as a breach involving 1 million identity documents.
The organisation should classify:
What was exposed?
How reusable is it?
Can it be changed?
Can it enable impersonation?
Can it enable financial or physical harm?
Can it create long-term identity risk?
I would therefore introduce a: Customer Identity Risk Score
For example: CIRS = Sensitivity × Reusability × Immutability × Exploitability × Potential Harm
This would not replace existing risk assessments.
It would create a common regulatory language for determining the level of customer protection required after a breach.
2. Mandatory customer protection should scale with risk
Today the response can effectively become: “Your information may have been compromised. Please remain vigilant.”
That is insufficient for highly sensitive data.
If an organisation loses low-risk information, the response may be relatively limited.
But if it loses: passport + driver’s licence + financial information + date of birth + identity credentials
the organisation should have significantly stronger obligations.
For high-risk breaches I would consider mandatory:
- multi-year identity monitoring;
- credit monitoring;
- identity restoration;
- fraud-resolution services;
- replacement of compromised credentials;
- dedicated customer support;
- legal assistance where appropriate;
- documented risk assessments;
- continued notification if new intelligence emerges.
The duration should be based on data persistence and exploitability, not simply a standard 12-month package.
3. Create a legal presumption for delayed harm
This could be controversial.
I believe it is also worth discussing.
Suppose:
- Organisation A confirms that your identity information was compromised.
- You are demonstrably part of the affected population.
- A later incident involves those same identity attributes.
- The later incident occurs within a defined statutory period.
- There is no stronger competing explanation.
The law could create a rebuttable presumption of causation.
That doesn’t mean: “The company is automatically guilty.”
It means: The evidentiary burden should not automatically fall entirely on the individual who never possessed the forensic evidence in the first place.
The organisation could rebut the presumption.
But the customer would not begin the process from an almost impossible evidentiary position.
4. Create a Data Breach Chain of Custody
Digital forensics already understands chain of custody.
We should extend the concept to compromised personal information.
For every material breach, organisations should preserve a structured: Personal Data Exposure Record
Including:
- data categories exposed;
- affected population;
- systems involved;
- exposure timestamp;
- exfiltration evidence;
- suspected attacker access;
- copies known to have been obtained;
- known downstream disclosures;
- security controls that failed;
- remediation performed;
- customer notification date;
- intelligence discovered after notification.
The customer should receive a meaningful incident identifier.
If harm appears years later, that identifier becomes part of the evidentiary chain.
5. Introduce long-tail liability
This is perhaps the most important recommendation.
The liability period should reflect the persistence of the compromised data.
If a password is stolen, it can be changed.
If a payment card is stolen, it can be replaced.
But some identity attributes are effectively persistent.
A date of birth cannot simply be replaced.
A historical address cannot necessarily be erased.
A government-issued identifier may be extremely difficult to change.
Therefore: The legal lifetime of the risk should be related to the ability of the individual to recover from the exposure.
I would call this: Data Persistence Liability
The more immutable the data, the longer the organisation’s customer-protection obligation should potentially remain.
6. Agentic AI needs an explicit accountability chain
I would apply a similar model to autonomous systems.
For every production AI agent with access to sensitive information, organisations should maintain an:
Agent Accountability Chain
Model
↓
Agent
↓
Identity
↓
Permissions
↓
Tools
↓
Data
↓
Actions
↓
Human owner
↓
Business owner
↓
Responsible organisation
This should be auditable.
If an AI agent sends customer information outside its authorised environment, investigators should be able to reconstruct:
Which model was running?
Which agent configuration?
Which identity?
Which permissions?
Which tool?
Which data?
Which instruction?
Which decision?
Which action?
Who authorised the deployment?
Who owned the system?
That is the equivalent of a flight data recorder for enterprise AI.
7. The organisation deploying the AI should not be able to outsource responsibility
This is critical.
Imagine: “The model provider caused it.”
The model provider says: “The customer configured the agent.”
The customer says: “The agent autonomously made the decision.”
The tool provider says: “We only provided the API.”
The cloud provider says: “We only provided infrastructure.”
Eventually: Nobody owns the harm.
I believe the law needs a principle of: Accountability follows authority.
If an organisation gives an AI system the authority to access customer data and act upon it, that organisation should retain primary responsibility for the system’s safe deployment.
It should not be able to transfer that responsibility merely by saying: “An AI vendor supplied the technology.”
What should change internationally?
I don’t believe every country needs identical legislation.
But I do believe there should be a common international baseline.
Here is where I would start.
| Jurisdiction | What I would add |
|---|---|
| EU / GDPR | Explicit long-tail breach liability; stronger recognition of future identity harm; mandatory high-risk customer protection; agentic-system accountability records |
| EU AI Act | Explicit requirements for agent identity, delegated authority, action logging, human intervention, incident causality and post-deployment monitoring for high-risk autonomous systems |
| Australia | Strengthen Privacy Act/NDB framework with customer harm classification, long-tail obligations, mandatory protection for high-risk identity exposure and clearer private compensation mechanisms |
| United States | Establish a stronger federal baseline for breach notification, customer protection, delayed identity fraud and AI-agent accountability rather than relying heavily on state-by-state variation |
| India | Build stronger individual remediation and compensation mechanisms around the DPDP framework, particularly for identity data and automated/agentic processing |
| Singapore | Extend the PDPA model toward explicit long-tail identity protection and agentic AI accountability, while integrating cyber and privacy incident management |
| United Kingdom | Strengthen UK GDPR/Data Protection Act mechanisms around delayed harm and autonomous processing, alongside existing cyber-resilience obligations |
| UAE | Develop interoperable data-breach and AI-accountability principles across major digital economies, particularly for financial services, government and critical infrastructure |
| Japan / South Korea | Strengthen cross-border incident evidence and long-term identity protection mechanisms alongside mature privacy and cybersecurity regimes |
| Canada | Strengthen PIPEDA/CPPA-style individual remedies and breach-risk obligations with clearer mechanisms for future harm |
| Overall | Establish an internationally recognised Customer Data Harm Standard that defines severity, evidence, customer protection and long-tail liability |
These are policy recommendations rather than statements of existing law, and each would require detailed legislative drafting.
But I think the direction is important.
We already have pieces of the solution
This is not a case of starting from zero.
We already have sophisticated frameworks:
- NIST Cybersecurity Framework;
- NIST AI Risk Management Framework;
- NIST Privacy Framework;
- ISO/IEC 27001;
- ISO/IEC 42001;
- GDPR;
- EU AI Act;
- Australia’s Privacy Act;
- Singapore PDPA;
- India’s DPDP framework;
- sector-specific cyber regulations;
- product safety regulation;
- financial-sector operational resilience requirements.
The problem is that these frameworks were largely designed around organisational compliance, risk management and system safety.
What is missing is a stronger layer focused on: The lifetime of the individual harm.
A new regulatory equation
I would therefore propose that breach severity should no longer be determined primarily by: Number of records × sensitivity
Instead: Customer Harm Exposure
could consider: CHE = Data Sensitivity × Immutability × Reusability × Access Persistence × Automation Potential × Potential Harm
This is intentionally conceptual.
It is not intended to become another arbitrary compliance score.
Its purpose is to force regulators and organisations to ask better questions.
For example:
Email address = High reuse, but relatively recoverable.
Password = High sensitivity, but replaceable.
Payment card = High sensitivity, but replaceable.
Passport information = High sensitivity + high identity value + lower replaceability.
Government identifier = Potentially extremely high persistence.
Combined identity profile = Potentially catastrophic because multiple attributes can be correlated and weaponised.
The regulation should reflect these differences.
Agentic AI makes this even more urgent
The research community is already demonstrating why this cannot be treated as science fiction.
Research such as AgentDojo and InjecAgent has demonstrated practical security challenges around tool-using language-model agents and indirect prompt injection.
The underlying issue is profound.
A traditional application receives: input → process → output
An agentic system increasingly operates as: observe → reason → select tool → retrieve data → reason again → execute action → observe result → continue
Every additional step creates another opportunity for:
- manipulation;
- privilege escalation;
- data leakage;
- unintended action;
- instruction conflict;
- tool compromise;
- cascading failure.
This is why I don’t think traditional “AI output safety” is sufficient.
We need to govern AI action authority.
An AI model should not be treated like an employee
This distinction matters.
An employee normally has:
- an identity;
- a manager;
- a role;
- permissions;
- organisational accountability.
An autonomous AI agent should have exactly the same governance properties – and potentially stronger ones.
I would require:
Every production agent must have:
1. Unique identity – No shared credentials.
2. Explicit authority – What is it allowed to do?
3. Permission boundary – What can it access?
4. Action boundary – What can it change?
5. Human escalation boundary – When must it stop?
6. Kill switch – Can it actually be stopped?
7. Immutable audit trail – Can investigators reconstruct what happened?
8. Data boundary – What customer information can it access?
9. Time boundary – How long does its authority remain valid?
10. Accountable human owner – Who answers when something goes wrong?
This is where cybersecurity, AI governance and privacy governance finally need to converge.
The same principle applies to IoT and autonomous vehicles
I don’t think the answer is simply: “Regulate AI.”
The broader issue is autonomous digital authority.
A connected car can act.
An industrial IoT system can act.
A smart building can act.
An AI agent can act.
A robotic system can act.
A financial trading system can act.
The common characteristic is: Software has been given authority to produce consequences in the real world.
That is the regulatory boundary I believe deserves far more attention.
From data protection to consequence protection
This is the shift I would like to see.
Traditional privacy regulation asks: Was personal information collected lawfully?
Cybersecurity regulation asks: Was it adequately protected?
AI regulation asks: Was the AI system designed and deployed responsibly?
I believe we need another question: What happens to the individual if the system fails?
That question cuts across all three domains.
The customer should have a “right to remediation”
We already talk extensively about:
- right to access;
- right to correction;
- right to deletion;
- right to object;
- rights around automated decision-making.
I believe the next generation of privacy regulation should seriously consider a: Right to Meaningful Cyber Remediation
When an organisation materially compromises an individual’s high-risk personal information, the individual should have a legally enforceable pathway to:
- know what was exposed;
- understand the risk;
- obtain appropriate protection;
- receive assistance if fraud occurs;
- obtain evidence linking the incident to the organisation;
- seek compensation for qualifying harm;
- receive support even when the harm materialises later.
That would fundamentally change the relationship between organisations and customers after a breach.
What should executives do now?
I would ask every board and executive team five questions.
1. If we lose customer data today, how long could our customer remain exposed?
Not: “How quickly can we restore the system?”
But: “How long does the customer’s risk survive?”
2. Which customer information cannot realistically be changed?
Those datasets deserve the highest protection.
3. If an AI agent misuses customer information, can we prove exactly what happened?
If the answer is no, the organisation has an accountability problem.
4. If the customer suffers harm three years later, can we provide evidence from today’s incident?
If not, today’s incident-response process is probably incomplete.
5. Who is accountable when an autonomous system crosses its authority boundary?
If the answer contains five different vendors and no accountable executive, governance has failed before the incident has even happened.
The metric I would like boards to start measuring
Cybersecurity has hundreds of metrics.
I would add one more: Mean Time to Customer Safety – MTCS
Not: Mean Time to Detect.
Not: Mean Time to Respond.
Not: Mean Time to Recover.
But:
How long after a material data breach does it take before the organisation can reasonably demonstrate that affected customers have been brought into a controlled state of residual risk?
For highly sensitive identity breaches, the answer may be measured in months or years.
That is precisely the point.
The future will require a new social contract around data
I don’t believe customers should be told: “Your data was stolen, but there is currently no evidence of misuse.”
and then effectively be left alone.
Nor do I believe companies should be punished indefinitely for every criminal act committed using information that once passed through their systems.
There has to be proportionality.
There has to be causation.
There has to be due process.
But there also has to be fair allocation of evidence and risk.
The organisation that collected the information, secured it, monitored it and possessed the forensic evidence is in a fundamentally different position from the individual whose identity was exposed.
The law should recognise that asymmetry.
Conclusion
The most important change I would like to see in cybersecurity thinking is this:
Stop treating a data breach as an event.
Treat it as a transfer of risk.
When an organisation loses control of customer data, some of that risk moves from the organisation to millions of individuals.
The breach may be contained.
The infrastructure may be rebuilt.
The attacker may disappear.
The regulator may impose a fine.
The CEO may issue an apology.
But the customer’s identity may remain exposed.
And now we are entering an even more complicated world where autonomous agents, connected vehicles, IoT systems and AI-powered workflows can access and act upon that information.
That creates a new question for regulators: When technology is given the authority to act, who carries the consequences when that authority is abused?
I don’t think the answer can be: the customer.
And I don’t think the answer can be: nobody.
We need a system in which responsibility follows custody, authority and capability – and where the person who ultimately bears the harm has a realistic mechanism for protection, evidence and remediation.
Because there is something fundamentally wrong with a cybersecurity model in which: the organisation can close the incident while the customer continues living inside it.
That, to me, is the next frontier of cyber accountability.
References and further reading
I would use primary sources wherever possible when publishing this article, and I would keep the policy proposals clearly identified as recommendations rather than existing law.
- NIST – Cybersecurity Framework (CSF) 2.0
https://www.nist.gov/cyberframework - NIST – AI Risk Management Framework 1.0
https://www.nist.gov/itl/ai-risk-management-framework - NIST – Artificial Intelligence Risk Management Framework: Generative AI Profile (AI 600-1)
https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence - European Union – General Data Protection Regulation, Regulation (EU) 2016/679
https://eur-lex.europa.eu/eli/reg/2016/679/oj - European Union – Regulation (EU) 2024/1689 – Artificial Intelligence Act
https://eur-lex.europa.eu/eli/reg/2024/1689/oj - European Data Protection Board – Guidelines and opinions on personal-data breaches
https://www.edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en - Australian Government – Privacy Act 1988
https://www.legislation.gov.au/C2004A03712/latest - Australian Government – Notifiable Data Breaches scheme
https://www.oaic.gov.au/privacy/notifiable-data-breaches - Singapore Personal Data Protection Commission – Personal Data Protection Act
https://www.pdpc.gov.sg/overview-of-pdpa/the-legislation - India – Digital Personal Data Protection Act, 2023
https://www.meity.gov.in/data-protection-framework - US Federal Trade Commission – Data Security / Breach enforcement and guidance
https://www.ftc.gov/business-guidance/privacy-security/data-security - IBM – Cost of a Data Breach Report
https://www.ibm.com/reports/data-breach - Verizon – Data Breach Investigations Report (DBIR)
https://www.verizon.com/business/resources/reports/dbir/ - ENISA – Threat Landscape
https://www.enisa.europa.eu/topics/cyber-threats-and-trends - Zheng et al. – AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
https://arxiv.org/abs/2406.13352 - Zhan et al. – InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
https://arxiv.org/abs/2403.02691 - Wallace et al. – The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
https://arxiv.org/abs/2404.13208 - Weidinger et al. – Taxonomy of Risks Posed by Language Models
https://arxiv.org/abs/2212.03551 - Ji et al. – Survey of Hallucination in Natural Language Generation
https://dl.acm.org/doi/10.1145/3571730 - ISO/IEC 42001:2023 – Artificial Intelligence Management System
https://www.iso.org/standard/81230.html - ISO/IEC 27001 – Information Security Management Systems
https://www.iso.org/isoiec-27001-information-security.html