AI Language Models: Threats and Safeguards

July 17, 2023 / Artificial Intelligence

Introduction

Although Language Models, which are one of the most important Artificial Intelligence technologies, have been developed and used in various applications for years, they have recently gained significant attention from the general public worldwide, particularly since the release of OpenAI‘s ChatGPT on November 30, 2022.

The release of ChatGPT has sparked a revolution in the relationship between human society and artificial intelligence technology, specifically language models. Firstly, it was the first application that allowed ordinary users to interact with such an advanced technology. Secondly, it opened the door to countless uses of language models for infinite purposes. Lastly, its release ignited intense competition among major technology companies, which had been hesitant to release similar AI applications due to various concerns, with Google being at the forefront. As a result, these companies raced to develop more powerful technologies in a remarkably short period of time. Today, in addition to the release of the fourth generation of GPT (GPT-4, built on GPT-3.5 technology), there are applications such as Microsoft Bing-AI (based on a modified version of GPT-4), Google Bard, and Anthropic Claude.

What are Language Models?

Language models are a type of Artificial Intelligence that focuses on understanding, generating, and Natural Language Processing. These models are trained on massive amounts of textual data, analyzing it to learn statistical patterns and structural characteristics of language, enabling them to predict and produce texts based on given contexts. Language models are developed using Language Modelling techniques, which utilize statistical and probabilistic methods to determine the likelihood of a specific sequence of words occurring in a sentence.

There are numerous applications for language models, including text translation between different languages, including real-time translation. They can also be used for sentiment analysis through written text, which can be applied to gauge the public opinion, customer satisfaction with services and products, and prevailing trends on social media. Other applications include text summarization, answering questions based on a given knowledge base, conversational robots and virtual assistants, content generation, and producing human-like texts for various purposes such as article writing, marketing messages, and personal communication. Additionally, they serve as the foundation for multiple Natural Language Processing tasks and applications.

The responsible development that led to the historical turning point we are experiencing today began with the development of Large language models. These models rely on the use of Artificial Neural Networks, which helped overcome some of the obstacles that hindered the generation of accurate predictions. The breakthrough occurred in 2017 when Google researchers developed the Transformer architecture or model. This new technology facilitated the development of large language models such as BERT and XLNet, which could be used for various purposes, including question answering, natural language understanding, sentiment analysis through written text, and document classification. However, these models were not Generative in nature, meaning they were capable of content classification and distinguishing details but unable to produce original new content. This step was accomplished by OpenAI researchers in 2018 when they developed the Generative Pre-trained Transformer (GPT) technology. In addition to being built on the Transformer technology, GPT models are pre-trained on massive datasets and can produce content similar to what humans produce. These characteristics distinguish all subsequent large language models produced by OpenAI and other companies.

The next evolution of language models specifically concerns their size, measured by a key value: the number of variables that can improve the accuracy of predicting the next word (or elements of images, sound, or video) during the generation process. While the number of variables in OpenAI’s first model, GPT-1, was 117 million, it increased to 1.5 billion in the subsequent model, GPT-2, in 2019. Turing NLG, developed by Microsoft in 2020, reached 17 billion variables. In the same year, OpenAI developed its third model, GPT-3, with 175 billion variables. This year, OpenAI released the fourth generation, GPT-4, with 170 billion variables.

What is meant by the threats of language models?

While language models open the door to an infinite number of applications that can revolutionize productivity and improve the daily lives of millions of people, they also entail various threats that can cause harm to individuals, communities, and institutions, resulting in significant losses. This is due to the way these models are developed, with numerous considerations that can be overlooked, as well as the flaws in the data used for their training. Moreover, their openness to various forms of use, which can be abusive in one way or another. Language models can produce biased or discriminatory content, promote hate speech, and pose a threat to privacy, data security, and sensitive personal information.

The dangers of language models can increase based on several conditions related to the policies of the companies producing them. These conditions include the lack of transparency and insufficient explanations about the limits of these models’ capabilities and their functioning, the failure to ensure that language models are sufficiently reliable, the neglect of data filtering and evaluation during training, the inadequate role of human supervision in the training process, and the insufficient commitment to legal and ethical considerations.

Based on the above, the implementation of protection and security procedures at different stages of the development, release, and use of language models becomes highly important. While technology companies developing these models bear the primary responsibility for developing and implementing protection and security measures, there is also an important responsibility on the part of legislative and executive institutions of countries to ensure the compliance of these companies. Additionally, users, whether individuals or organizations, have a responsibility to prevent unintentional misuse of these models and to protect personal and sensitive data. Above all, there is a general responsibility to recognize both the potential dangers of using language models and the necessary protection and security measures to avoid or minimize these risks. Given the significant importance that language models are acquiring in our daily lives, it is essential not to underestimate their real and tangible risks or to neglect the use of all available means to mitigate these dangers.

What is in this paper?

This paper provides information about language models and the threats associated with their use as a means of raising awareness about these threats to enable users to be aware of the necessary standards that should be present in the language models they use in order to use them as safely as possible.

The paper also aims to provide a concise yet comprehensive overview of the major threats associated with using language models, especially since some of the most powerful models have become publicly available and are likely to be incorporated into numerous products used by billions of users worldwide on a daily basis. To accomplish this, the paper begins with an introduction that defines language models and highlights the key historical milestones in their development. It then presents the threats of language models through three main areas: discrimination and bias threats, privacy threats, and risks arising from inadequate model construction. The paper concludes by presenting the most important approaches to address these risks in the same order.

Risks of Language Models

There are numerous risks inherent in the use of language models. According to a study conducted by researchers at DeepMind, these risks, which have already been identified and studied or are expected to arise, can be classified into six main areas:

Discrimination, Exclusion, and Toxicity: This refers to language models producing content that discriminates against societal groups based on gender, race, origin, religion, political ideology, sexual orientation, or other factors, leading to the exclusion of individuals from these groups or the propagation of a toxic climate.
Information-related risks: This includes breaches of information confidentiality, leaks, and the consequent threats to privacy rights.
Misinformation-related harms: Language models can generate false information, and they can be directed to produce misleading content that harms the reputation of individuals, or organizations, or is used for political propaganda or other purposes.
Misuse: In addition to the misuse of language models in producing false content as mentioned earlier, they can be used for fraud, executing cyber attacks, and more.
Human-computer interaction-related harms: Language models can establish psychological connections with users that blur their awareness of them being mere computer programs, which may lead users to engage in actions that harm themselves or others.
Automation, accessibility, and environmental harms: Language models have the potential to automate a vast number of tasks currently performed by human employees, which is expected to result in the loss of millions of jobs in the coming years. Due to the enormous resources required for the development, training, and operation of language models, there is a significant possibility of depriving a large number of people of access to them, widening existing gaps based on geography, economy, or society. Lastly, language models consume massive amounts of energy, contributing to a notable percentage of greenhouse gas emissions and environmental damage.

The paper focuses its analysis on three main areas of risks resulting from the use of language models, namely bias and discrimination, privacy-related risks, and risks arising from design and implementation flaws.

Bias and Discrimination

Language models rely on the data they are trained on, and therefore, the bias and discrimination present in the training data have a significant impact on how language model outputs express them. Bias in language model outputs can result from several sources, including the nature of the data used for training, the technical specifications of the model, the constraints imposed by the algorithms used in the model, the design of the product through which the model is deployed (programming language, APIs used, etc.), and the policies and decisions of the producing companies.

Representational Bias: When certain societal groups or categories are underrepresented or overrepresented in the training data, language models can produce outputs that inadequately reflect the diversity of perspectives and opinions. Instead, they may exhibit bias and discrimination against some of these perspectives and opinions.

Amplification Bias: Training data may contain predominantly information that supports specific viewpoints or beliefs. As a result, language models tend to confirm and reinforce certain perspectives or views, even when presented with alternative or contradictory information, thus limiting the diversity of ideas and hindering critical thinking.

Confirmation Bias: Training data can contain discriminatory and prejudiced content, which language models learn and reproduce in their generated content. This leads to the promotion and perpetuation of harmful stereotypes, discriminatory language, and offensive content.

Temporal Bias: Training data that predominantly represents a specific time period can lead language models to learn language usage, customs, and beliefs that are no longer relevant or acceptable in the current society. This results in the generation of content that is incompatible with current values, customs, and prevailing language.

Measurement Bias: This refers to the collection or definition of training data in a biased manner, causing language models to prioritize certain features and patterns that do not accurately represent the true distribution in the data. This can lead to outputs that are biased towards specific characteristics or attributes, resulting in biased expectations and recommendations.

Given the significant role that language models are expected to play in the near future as crucial sources of information for everyday users through countless applications, any bias or discrimination in their outputs will have a significant social impact by supporting specific biases and increasing discrimination against certain categories. This can have consequences, including increased rates of hate crimes, racist or sectarian violence, or violence against women, minorities, and immigrants, among others.

Privacy Threats in Language Model Training and Usage

Language models can pose several concerns regarding user data and the model training process. Some of the key threats include:

Data leakage: During the training process, language models are exposed to a vast amount of textual data that may contain sensitive information or data that can be used to identify individuals. If the model unintentionally retains this information, it can lead to the disclosure of personal information when generating text during usage.
Data inference attacks: Attackers can utilize the outputs of a language model to infer sensitive information about the training data. For example, they can input specific queries to the model and analyze its responses to deduce information about the data it was trained on.
Unauthorized access: If the API or application through which a language model is accessed lacks sufficient security measures, unauthorized users may gain access to the model and exploit it for malicious purposes, such as generating harmful content or extracting sensitive information.
Bias and discrimination: While these risks are independent in nature, they intersect with privacy risks as inferring personal information about users can lead to the dissemination of biased, discriminatory, and abusive content.
Misuse of generated content: Language models’ generated content can be misused for malicious purposes, such as producing false content, spreading misinformation, or facilitating social engineering attacks. This can be achieved by exploiting personal information obtained or inferred from user inputs, compromising their privacy.

Reverse Engineering Attacks

Reverse engineering attacks on language models are a type of attack that compromises privacy, where the attacker attempts to reconstruct or infer sensitive information in the training data of the language model through its variables or outputs. These attacks represent a potential threat to the privacy of individuals whose data was used during the training process.

Privacy threats associated with reverse engineering attacks on language models include:

Successful attacks can result in the disclosure of sensitive information or personally identifiable information (PII) present in the training data, leading to a breach of privacy.
Privacy breaches resulting from reverse engineering attacks on language models can have legal and regulatory consequences for organizations, including fines and damage to their reputations.

Risks of Weaknesses in Language Models

Language models are highly complex software, and the development of each model goes through multiple stages, making them vulnerable to potential weaknesses that can have a significant impact on the model’s functioning and the potentially catastrophic outcomes it can produce in certain cases. Specifically, there are two areas of concern within this context: adversarial attacks and over-specialization of the language model.

Adversarial attacks refer to designing input to the language model in a way that seeks to deceive or confuse it, resulting in inaccurate outputs. These attacks can exploit vulnerabilities in the language model, particularly by bypassing the security measures incorporated into its design, thereby leading to misuse for criminal purposes such as obtaining information for manufacturing explosives, drugs, chemical weapons, carrying out large-scale cyberattacks, and more.

Over-specialization occurs when a language model becomes exceptionally proficient in performing specific tasks based on its training data but fails to generalize that knowledge to unseen data. This includes instances where the model retains some training data, potentially breaching privacy when incorporating such data into its outputs. It also involves the model producing outputs that appear correct but are, in reality, inaccurate and irrational, commonly referred to as artificial intelligence hallucinations. This happens because the model attempts to impose data it has memorized into an unrelated context.

Dealing With Language Model Risks to Mitigate Them

Ways to detect and mitigate bias during the training and evaluation of language models.

Dealing with biases in language models requires implementing measures to both detect and mitigate biases during the training and evaluation stages of language model development. There are several approaches to achieve this, including:

Ensuring diverse training data that represent different groups, perspectives, and language usage patterns, as well as actively seeking out underrepresented data sources.
Identifying and removing or reducing the intensity of biased, offensive, or discriminatory content in the training data. Tactics such as keyword filtering or modeling closer topics that have less biased content can be used to detect and address potential biases.
Using fairness-aware algorithms or processes that counteract biased inferences during training, such as counterfactual training or sample re-selecting, to minimize the impact of biases in the training data on the language model.
Measuring and monitoring equity indicators during the model evaluation phase, including demographic parity and equal opportunities for diverse outputs. Additionally, comparing the model’s performance across different demographic groups can help detect potential biases.
Applying post-processing techniques to adjust the model’s outputs and mitigate biases, including using indicators to surpass certain thresholds or reordering different options. Employing techniques that highlight biases for detection can also be beneficial.
Utilizing various means to interpret the inner workings of the language model, aiding in understanding the factors contributing to the emergence of biases in its outputs. Additionally, using explanatory tools to provide understandable insights into the model’s outputs can assist in detecting and addressing potential biases.
Expanding human intervention by involving experts in the development and evaluation stages of the model to identify biases that may not be easily detected through automated methods. Incorporating user feedback can also help improve the model and align it with societal values and expectations.
Regularly monitoring and evaluating performance and fairness indicators of the model in real-world usage conditions, updating the model and training data as needed to address emerging biases or changes in societal language habits and usage patterns.

Dealing with Privacy-Related Risks

To mitigate these privacy concerns, language model developers can take several security measures, including:

Data anonymization: Sensitive information and data that can be linked to individuals should be removed or anonymized in language model training data to reduce the risks of data leakage.
Differential privacy tactics: By using this tactic, sensitive data can be obfuscated by mixing it with unrelated data, making it harder to extract sensitive information and increasing the difficulty of accessing it.
Access control: The API interface of the language model and any applications built on it should be secured with appropriate access control mechanisms to prevent unauthorized data access.
Regular review and updates: Language models should be continuously monitored and updated to address any urgent privacy concerns, biases, or weaknesses.
Transparency and user consent: Users should be informed of potential privacy risks associated with the use of language models, and their consent should be obtained before processing their personal data.

Dealing with Reverse Engineering Attacks

Countermeasures against reverse engineering attacks on language models include the following:

Data anonymization: Sensitive information or data that can reveal the identity of individuals should be removed or anonymized in the training data to mitigate the risks of data leakage.
Differential privacy technique: This technique involves adding misleading data to the model’s outputs, making it difficult for attackers to reconstruct the original data.
Model generalization: Training the model on unseen data instead of preserving specific cases in the training data can be achieved through tactics such as data augmentation, early stopping of the training process, and data fusion.
Model compression: Using model compression techniques to create a smaller and less complex model that retains the performance of the original model while being less susceptible to adversarial attacks due to its reduced complexity.
Regular review and updates, along with model monitoring, to address any privacy concerns, weaknesses, or biases.

Dealing with the risks of building language models

Dealing with the risks associated with building language models is of great importance, as their outcomes can be highly dangerous depending on the context in which the model is used. Among the ways to address these risks are the following:

Adversarial training: This involves using examples of inputs similar to those used in adversarial attacks and training the model to not be deceived by them.
Regularization techniques: These techniques include deleting some training data after a period of time, reducing the weights of input data during training, or early stopping of the training process to avoid overfitting and help improve generalization.
Fine-tuning for specific tasks: This involves adjusting the model’s performance using task-specific data to improve its performance and compatibility with new domains.
Monitoring and evaluation: This includes continuous monitoring of the model’s performance on real-world tasks and evaluating its robustness against potential attacks.

Conclusion

Language models and their various applications are among the most important forms of software available for general use, given their capabilities and potential achievements. Millions of people worldwide have already started using these advanced software tools for research, learning, improving work tasks, producing creative works, entertainment, time-passing, or satisfying curiosity. However, some individuals have also begun using these models for criminal purposes. It is certain that the use of these models for all these purposes and more will increase in the near future, allowing billions of people to use them on a daily basis, consciously or even unconsciously, as they become integrated into various applications such as search engines, text processing programs, spreadsheets, graphic and photo editing programs, and countless other applications and software. All of this highlights the utmost importance of understanding the risks associated with using these software tools, especially considering their vastness and the numerous challenges they still face, resulting in significant potential risks.