Your AI Agent Can Leak the Deal Without Leaking the Data

An AI agent does not have to print a secret to reveal it.

Imagine a procurement agent negotiating a software renewal. Its system prompt contains an approved budget ceiling of $40,000, a preferred close date, and instructions to avoid a service interruption. The agent never states the ceiling. It encrypts traffic, redacts logs, and refuses direct questions about internal policy.

But it concedes quickly whenever the price approaches $38,000. It waits longer before rejecting offers below that range. It becomes unusually flexible near the renewal deadline. After several rounds, a patient counterparty can infer the hidden boundary from the agent's behavior.

The secret stayed out of the text. The strategy disclosed it anyway.

This is behavioral privacy leakage: private preferences, constraints, or urgency become inferable from observable actions such as concessions, timing, retries, tool choices, escalation patterns, or willingness to walk away. It is a useful launch concern for founders building negotiation agents, customer-support agents, purchasing assistants, pricing tools, scheduling agents, claims workflows, or any product that acts repeatedly on a user's behalf.

A July 2026 paper, Behavioral Privacy Leakage in Agentic Negotiation, formalizes this problem for multi-round bargaining. The authors evaluate an adaptive randomized policy on 3,000 synthetic bilateral negotiations and report lower inference accuracy while retaining high negotiation utility. That is an interesting research result, not a production benchmark. The experiments are synthetic, the setting is narrow, and most early products should not attempt to claim formal differential privacy from a prompt tweak.

The more durable product lesson is simpler: if an outside party can observe a sequence of agent actions, inspect the sequence as a data product of its own.

This guide turns that lesson into a practical red-team method.

Privacy Has an Output Plane and a Behavior Plane

Most AI privacy reviews focus on the output plane:

• Does the agent reveal personal data in its answer?
• Can prompt injection extract system instructions or memory?
• Do logs contain credentials, documents, or conversation text?
• Does one tenant's context appear in another tenant's response?
• Are prompts retained by a model or observability provider?

Those checks are necessary. They are not sufficient for an agent that takes repeated actions.

The behavior plane includes everything a counterparty can learn without seeing private context:

• The size and direction of each concession.
• How response latency changes around particular offers.
• Which terms trigger an immediate rejection, tool call, or human escalation.
• How many retries occur before the agent accepts a fallback.
• Whether the agent becomes more flexible near a deadline.
• Which options it presents first, hides, or repeatedly recommends.
• When it stops searching and commits.

This distinction matters because conventional controls can protect one plane while leaving the other exposed. Encryption protects data in transit, not the meaning of a concession curve. Redaction protects a log field, not a predictable escalation threshold. A model refusal can stop a direct request for the reservation price while the next ten offers reveal approximately the same information.

Research on privacy side channels supports the broader systems view. A 2024 USENIX Security paper showed that filtering, preprocessing, and monitoring components can create privacy leakage even when the model is considered protected in isolation. The exact attacks differ from negotiation behavior, but the engineering implication aligns: privacy is an end-to-end property of the system, not a feature of one model call.

Decide Whether the Risk Is Material Before Building a Lab

Not every agent needs a behavioral privacy program. A restaurant-booking assistant may reveal that the user prefers 7 p.m.; the user may consider that obvious and harmless. A public FAQ agent has few private preferences to infer.

The risk becomes material when four conditions overlap:

1. The agent has private constraints. These might be a maximum price, claim threshold, urgency, health limitation, staffing shortage, churn risk, credit tolerance, inventory pressure, or internal approval boundary.
2. An outside party can observe repeated behavior. One response reveals less than a multi-round interaction, especially when the observer can choose the next input adaptively.
3. The observer benefits from inference. A seller may raise the price, a fraudster may stay just below a review threshold, or a counterparty may delay until urgency increases.
4. The product grants meaningful autonomy. A suggestion shown to the user is different from an agent that sends offers, schedules appointments, issues credits, or approves transactions.

Use these conditions to scope the review. If only one is present and the possible harm is small, basic disclosure and monitoring may be enough. If all four are present, do not treat the agent as a normal chatbot with extra tools.

The UK's Information Commissioner's Office makes a related product-level point in its agentic AI guidance: organizations should define a clear purpose for information used by an agent, communicate that purpose, and limit personal information to what is necessary. Behavioral traces deserve the same discipline. Collecting every turn “in case it helps later” can create a second privacy problem in the evaluation system itself.

Build a Behavioral Threat Model

Start with a one-page table, not a complex attack simulator.

For each private variable, write down five things:

Repeat this for deadlines, acceptable alternatives, approval levels, health or accessibility needs, preferred suppliers, and any internal risk score that can influence action.

Then define the attacker's access. Can the counterparty submit unlimited offers? Can it create multiple accounts? Does it see exact timestamps? Can it restart sessions while the private policy remains constant? Can it coordinate observations across users? Can it deliberately trigger tools or escalation?

Do not default to an all-powerful adversary. A useful threat model states realistic capabilities and exclusions. For example: “The vendor sees messages and timestamps within one negotiation, may choose offers adaptively, but cannot access internal logs or link different customer accounts.” This makes the test repeatable and the resulting controls proportionate.

Finally, specify what would count as a privacy failure. “No leakage” is not measurable. Better criteria include:

• An evaluator predicts a budget band substantially better than a simple prior.
• A single offer can reliably trigger the internal approval boundary.
• Timing separates urgent from non-urgent users with unacceptable accuracy.
• Repeated sessions narrow a private constraint beyond the precision needed to complete the task.
• A change intended to improve success rate increases inference accuracy beyond an agreed limit.

NIST's AI Risk Management Framework uses the recurring functions govern, map, measure, and manage. That sequence fits here: name the accountable owner, map private variables and observers, measure leakage and utility, then choose and monitor controls.

Run the Paired-Scenario Test

The most accessible behavioral privacy test compares scenarios that are operationally similar but differ in one private attribute.

Suppose an agent negotiates a subscription renewal. Create two profiles:

• Profile A has a $30,000 ceiling.
• Profile B has a $40,000 ceiling.

Keep product requirements, vendor, deadline, starting offer, model version, prompt, tools, and sampling settings as consistent as possible. Run each profile against the same scripted counterparty strategies many times. Include an adaptive strategy that chooses the next offer based on the previous response.

Record only the observations the real counterparty would have: proposed prices, accepted terms, rejection language, turn count, response timing in realistic buckets, and whether a human review occurs. Do not give the evaluator hidden chain-of-thought or internal logs it would never possess.

Now ask a separate evaluator to predict which profile generated each trace. Start with a simple rule or logistic regression before using a powerful model. If a basic classifier separates the profiles, the product has a clear signal worth investigating. Compare against a baseline that guesses from the prior distribution; raw accuracy alone can mislead when one profile is more common.

Measure task utility at the same time:

• Agreement or completion rate.
• Price or outcome quality relative to the user's objective.
• Time and number of turns to completion.
• Human escalation rate.
• Invalid or policy-violating actions.
• User-visible inconsistency.

Privacy and utility must be reviewed together. A mitigation that makes the agent random but incompetent is not a product success. A highly effective agent that broadcasts the user's ceiling is not acceptable either.

Run the test again after changes to prompts, model providers, memory, tools, approval thresholds, or negotiation logic. OWASP's AI Agent Security Cheat Sheet recommends structured testing after material changes to those components and calls for repeatable abuse cases and release evidence. Behavioral inference should be one row in that test matrix when the product acts on private preferences.

Controls That Founders Can Apply Now

Formal randomized policies are a research direction, but most teams need layered product controls before advanced privacy engineering.

Reduce unnecessary precision

Do not give the agent a precise private value when a band or policy decision will do. A policy service can answer “permitted,” “requires approval,” or “not permitted” without placing the exact ceiling in the model context. This does not eliminate inference—the decision boundary can still be probed—but it reduces direct exposure and simplifies control.

Separate strategy from language generation

Let a deterministic, reviewable policy component choose allowed action ranges while the model drafts natural language. This prevents a prompt or model change from silently rewriting business constraints. It also creates a place to enforce rate limits, minimum concession intervals, and approval requirements.

Coarsen observable signals

Exact timestamps, highly granular status updates, and distinct error messages may expose more than the task requires. Use sensible response-time buckets, uniform public states, and generic escalation messages where precision has no user benefit. Do not add artificial delays blindly: they harm accessibility and can still form a pattern. Test the chosen interface.

Limit adaptive probing

Apply session limits, cooldowns, abuse detection, and identity or account controls when repeated queries could map a private boundary. Decide whether a restarted conversation should retain the same sensitive policy, and detect attempts to run many near-identical negotiations in parallel.

Add bounded variability only with evaluation

Some predictable behavior can be softened by choosing among several policy-compliant actions rather than always taking the same one. The July 2026 negotiation paper studies a mathematically specified randomized policy, not casual prompt randomness. Telling a model to “be unpredictable” is not a privacy guarantee. It can create unfair outcomes, unstable prices, or policy violations.

If variability is appropriate, define the allowed action set, seed and record tests, measure utility, and verify hard constraints outside the model. Never randomize safety, legal, eligibility, or authorization boundaries merely to hide them.

Put humans at consequential boundaries

Human approval is useful when an agent approaches a confidential ceiling, makes an irreversible commitment, or encounters adversarial probing. The reviewer should see why the case was escalated without exposing unnecessary private details to the counterparty. Human review is a control, not a privacy proof; reviewers can also behave predictably or leak information.

Minimize and protect behavioral traces

Trace data is valuable for debugging and red teaming, but it may contain inferred urgency, preferences, risk labels, and decision boundaries even after message text is removed. Define purpose, access, retention, and deletion for traces. Keep production identifiers out of research datasets where possible. NIST's Privacy Framework is a useful organizational reference for identifying and managing privacy risk rather than treating privacy as a one-time launch check.

What This Research Does Not Prove

The new paper deserves careful limits.

It evaluates 3,000 synthetic bilateral negotiations under a defined mathematical model. Its reported 43–50% reduction in adversarial inference accuracy and above-90% success and utility results belong to that setup. They do not show that adding noise to any LLM agent will produce the same result. They do not establish performance in procurement, insurance, employment, healthcare, or consumer credit. They do not replace legal review.

Differential privacy also has a precise technical meaning. NIST SP 800-226 warns practitioners to evaluate the complete guarantee and common implementation hazards. A founder should not market an agent as “differentially private” because its temperature is nonzero, its wording varies, or a prompt requests random behavior.

There are also cases where hiding a decision rule is the wrong goal. Eligibility, moderation, insurance, lending, employment, and public-sector decisions may require explanations, consistency, contestability, or disclosure. Privacy controls must not become a pretext for opaque or discriminatory decisions. Consult domain experts before deploying autonomous decisions in high-impact settings.

Finally, some information must be revealed to complete a task. Negotiation works because parties communicate preferences. The practical objective is not zero information flow. It is to prevent unnecessary, surprising, or exploitable inference while preserving enough signal to reach a legitimate outcome.

A Launch Gate for Behavioral Privacy

Before shipping an agent that acts repeatedly on private constraints, require evidence for these questions:

• Have we listed the private attributes that influence actions?
• Do we know exactly what a counterparty can observe across turns and sessions?
• Have we defined a realistic adaptive attacker?
• Did we run paired scenarios that differ in one sensitive attribute?
• Can a simple evaluator infer that attribute better than the prior baseline?
• Are privacy and task utility measured together?
• Are exact sensitive values kept out of model context when a policy decision is enough?
• Are hard constraints enforced outside free-form model generation?
• Are probing, restarts, and parallel sessions bounded where necessary?
• Are logs and traces treated as sensitive derived data?
• Do consequential boundaries require valid human approval?
• Will this test rerun after changes to models, prompts, tools, memory, or policies?
• Are privacy claims reviewed by someone who understands their technical meaning?
• Have we documented where this framework does not apply?

If the team cannot answer these questions, narrow the agent's autonomy. Launch it as an advisor that proposes options for a user to approve, or restrict it to low-stakes scenarios while collecting consented, minimized evidence. Autonomy can expand after the product earns it.

The deeper point is that an agent's behavior is part of its interface. Users entrust agents with goals, limits, and preferences precisely because those agents need context to act. Founders should review not only what the agent says, stores, and sends, but also what an observant counterparty can learn from the pattern of its choices.

The absence of a leaked secret in the transcript is not proof that the secret stayed private.

References

• Barkha Rani, Behavioral Privacy Leakage in Agentic Negotiation: Formalizing and Mitigating Inference Attacks via Randomized Policies, arXiv, July 2026.
• Apple Machine Learning Research, Behavioral Privacy Leakage in Agentic Negotiation, research index entry.
• Edoardo Debenedetti et al., Privacy Side Channels in Machine Learning Systems, USENIX Security 2024.
• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1.
• NIST, Privacy Framework, privacy risk management guidance.
• NIST, Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226.
• OWASP, AI Agent Security Cheat Sheet, agent architecture, privacy, and adversarial testing guidance.
• UK Information Commissioner's Office, Data Protection and Privacy Risks in Agentic AI, purpose limitation and data minimization guidance.