Adversarial prompting. Current systems prefix, suffix, or otherwise template user prompts into an instruction prompt before sending it to the model. That might be ChatGPT giving the model instructions "Assistant is a large language model trained by OpenAI. Knowledge cutoff: ..." or Bing's Sydney.
On Prompt Injection
On Prompt Injection
On Prompt Injection
Adversarial prompting. Current systems prefix, suffix, or otherwise template user prompts into an instruction prompt before sending it to the model. That might be ChatGPT giving the model instructions "Assistant is a large language model trained by OpenAI. Knowledge cutoff: ..." or Bing's Sydney.