Discover more from The Cyber Solicitor

Covering the intersection between law, technology and society around the world.

Privacy Engineering

Notes on E2EE and Client-Side Scanning

A look at the risks of scanning messages on devices in E2EE environments

Mahdi Assan

Aug 5, 2023

Intro

These are my notes on end-to-end encryption (E2EE) and Bugs in Our Pockets, a paper on the risks of client-side scanning (CSS). The authors of this paper include some well-known experts and researchers in the cryptography space; one of them, for example, is Whitfield Diffie, who is one of the creators of the Diffie-Hellman key exchange protocol (along with Martin Hellman) that is used to achieve E2EE for Signal Messenger, WhatsApp and a number of other instant messaging platforms.

This post is inspired by the recent debates concerning E2EE and the threat it faces from proposed pieces of legislation. In the UK, the Online Safety Bill has been criticised for potentially having the effect of requiring online platforms to break E2EE.

Also in the EU, E2EE has become a key issue for the European Commission's proposed legislation for combatting online child sexual abuse. Wired reported back in May 2023 that Spain is one EU Member State in favour of banning E2EE to enable the detection of child sexual abuse online.

The structure for this post is as follows:

A short primer on CSS - This explains what CSS is and how it works.
A short primer on E2EE - This explains what E2EE is and how it works.
CSS in E2EE environments - This contains my notes on the paper and explores the (in)compatibility of CSS and E2EE.
The Signal Protocol - These are my notes explaining how the Signal Protocol works, which is a popular method for implementing E2EE for instant messaging services.

Leave any comments below if you have any, and consider subscribing if you enjoy the content here.

1. A short primer on CSS

CSS is the analysis/scanning/processing of data in the contents of a communication on the device of the sender of that communication before it is sent to the recipient of the communication. CSS can thus form part of a service provider's content moderation system to detect content on its service that violates its terms of service (ToS) or is otherwise illegal.

The term 'client-side' comes from a type of system architecture typically used for communication services that uses 'clients' and 'servers':

The 'client' is any device that is used to access the service. This could include desktop and laptop computers, smartphones or tablets.
The 'server' is essentially the computer(s) operated by the service provider that allows users to access the service through their device.

The diagram below provides a simple illustration of this architecture:

Thus, CSS is about scanning communications at the client-level rather than at the server-level. So before a communication is sent from a users' device to the server to then be distributed to the recipient, the communication is scanned to detect for illicit content. If illicit content is detected, then it is flagged to the service provider who will address the content in accordance with its policies/ToS or the applicable law, and the communication will not continue its normal transmission journey. If no illicit content is detected, then the communication is permitted to continue on its normal journey to the recipient.

There are several ways to implement this form of CSS:

App-level: this is where the CSS mechanism is built into the app itself. The mechanism is therefore built and operated by the app developer, who will therefore also be responsible for any data processing that takes place.
OS-level: this is where the CSS mechanism is applied via the operating system of the device.
Middleware: aka 'software glue', this is where the CSS mechanism is applied to applications on the device using a service that is independent of the operating system.

2. A short primer on E2EE

Probably the most important thing to understand about E2EE is that it does not refer to one particular algorithm or protocol. Rather, E2EE is a concept; it is a technical method whereby the goal is to ensure the confidentiality of communications between two (or more) users such that they can send communications to each other that only they can access, thus preventing access by any other third party. There are different protocols that seek to achieve this (e.g., the Signal Protocol which is used in Signal Messenger, WhatsApp and other services).

E2EE uses public key encryption algorithms to allow a user to send an encrypted message that only the recipient can decrypt and read. This requires the use of both public and private keys; these are pieces of data that are used to trigger the encryption algorithm to transform the data into an unintelligible form, known as ciphertext. The public key, as its name suggests, is shared with everyone else. Conversely, the private key remains private.

There is some complicated maths that explains how these keys can be used to create a communication that only the sender and the recipient can read, but to set out the process in a simple way:

Both Alice and Bob sign up to HomingPigeon, a (fictional) messaging app for smartphones that uses E2EE. In doing so, the apps on their respective devices generate a set of public and private keys. A copy of both Alice and Bob's public keys are stored on the central server for HomingPigeon (remember, public keys are meant to be shared with other users)
Alice wants to send a message to Bob. To initiate this process, the app on Alice's device retrieves a copy of Bob's public key from HomingPigeon's central server.
Alice creates her message for Bob on her device, which reads: "Hi Bob, how are you?". The HomingPigeon app on her device then uses both her own private key (which must be kept secret from everyone else, including Bob) and Bob's public key it retrieved to encrypt the message.
The HomingPigeon app transmits the encrypted message from Alice's device to the HomingPigeon central server. At this point it is important to note that a communication is made up of two main parts; the header and the body. The header contains the metadata of the communication, which is essentially consists of descriptive information about the communication, such as the time it was sent, the size of the communication as well as information about the intended destination of the communication (i.e., information that indicates to the server who the communication is for, such as the recipient's phone number or IP address). The body is the part of the communication that contains the actual message (in this case, the message that reads: "Hi Bob, how are you?"). So when the HomingPigeon app encrypts the communication on Alice's device before it is sent, it will encrypt the body (and perhaps even some metadata depending on how E2EE has been implemented), but it will not encrypt the metadata needed by the server to determine who to forward the message to. However, the server will not have a copy of the requisite keys needed to decrypt the body and read the message that Alice has created. Thus, HomingPigeon's central server will see that a message is intended for Bob, and will forward the message from Alice to Bob's device without being able to read what that message says. This is also the case for any other third party that attempts to intercept the message in-transit; without the requisite keys, the message cannot be read by that third party and all they will be able to see is ciphertext and at least some of the metadata.
Bob's device receives the encrypted message via the HomingPigeon app. The app then applies Bob's private key to decrypt the body and access the message created by Alice.

There are many other nuanced aspects of this process that are not detailed in the above summary. This includes for example cloud backups, cryptographic key storage mechanisms, digital certificate protocols and other components. But the above is sufficient to illustrate, at a basic level, what E2EE is meant to achieve; ensuring the confidentiality of communications between two (or more) users such that they can send communications to each other that only they can access, thus preventing access by any other third party.

See Section 4 of this post for an explanation of the Signal Protocol which sets out in more detail an example of an E2EE approach used for messaging apps like WhatsApp.

3. CSS in E2EE Environments

The central argument advocated in Bugs in Our Pockets regarding the use of CSS can be found in its executive summary:

...we argue that CSS neither guarantees efficacious crime prevention nor prevents surveillance. Indeed, the effect is the opposite. CSS by its nature creates serious security and privacy risks for all society while the assistance it can provide for law enforcement is at best problematic. There are multiple ways in which client-side scanning can fail, can be evaded, and can be abused. (Abelson et al 2021, 2)

So is CSS compatible with E2EE? In other words, does CSS 'break' E2EE encryption? There seems to be two important issues when discussing this question. The first concerns the exact definition/perception of what E2EE is and therefore where the opportunities for intervention could potentially be identified (sometimes called 'exceptional access'). The second is about, if an opportunity of intervention can be found and is as privacy-preserving as possible, how can function creep be avoided.

E2EE and 'exceptional access'

As described already, E2EE is not a specific algorithm or protocol. If it were, then answering the question of whether CSS breaks that algorithm/protocol would be relatively straightforward; if the CSS mechanism prevents the full functionality of the algorithm/protocol (e.g., the Signal Protocol), then we could say that the CSS mechanism 'breaks' that algorithm/protocol. However, since E2EE is more of a concept of which can be achieved through a variety of technical means, then it becomes more tricky. Even if it is about establishing secure communications between two (or more) users, can there be exceptions for communications that contain illicit content? Or should a strict interpretation be applied in which secure communications should be maintained at all times and no exceptions can be made for any purpose?

Another way to think about this is look at E2EE through the prism of two broad POVs:

'The privacy advocate POV': if using E2EE, the whole system architecture must be treated as the users' private space and thus any interference with that breaches that private space and is not acceptable.
'The law enforcement POV': if using E2EE, only some parts of the system architecture should be considered a users' private space, and limited access to those parts that are not part of that private space should be allowed.

The law enforcement POV - which is essentially the idea that nothing should be off limits and that there should always be a way for governments to gain access to data when needed - is one that has been maintained by state agencies and governments regarding encryption for a long time:

Since strong encryption became available to the public nearly half a century ago, intelligence and law enforcement agencies around the world have sought to limit its use. The first attempts were to restrict access to encryption devices. When the invention of the PC made encryption widely available in software, attempts were made to require backdoors that would governmental access to decryption. These proposals were effectively abandoned in most democratic countries by the end of the twentieth century. Since then, several national intelligence and law enforcement agencies have instead worked to enlist vendors as surrogates to provide access to encrypted traffic, whether using technical vulnerabilities or covert court-ordered access. (Emphasis added) (Abelson et al 2021, 3)

An example of covert and (somewhat) court-ordered access can be found in the UK's Investigatory Powers Act 2016:

Under s.253 IPA 2016, the government can issue technical capability notices (TCNs) on telecommunications operators:
- Who are telecommunications operators? - Under the 2016 Act, this has a wide definition. Section 261(10) defines 'telecommunications operators' as "a person who offers or provides a telecommunications service or controls or provides a telecommunication system whether that system is in the UK or controlled from the UK (either wholly or partly)." In turn, s.261(11) defines a 'telecommunications service' as "any service that consists in the provision of access to, and of facilities for making use of, any telecommunication system." Furthermore, s.261(13) a 'telecommunication system' is defined as "a system for the purposes of facilitating the transmission of communications by means involving the use of electrical or electromagnetic energy." In aggregate, these definitions cover a wider range of entities and is not just limited to public networks or carriers. Private networks and online storage providers may also fall within the scope of entities who are within scope of the IPA 2016. Thus, providers of messaging applications like Signal and WhatsApp are within scope.
A TCN can be served on telecommunications operators to aid the surveillance operations of public authorities when they exercise their surveillance powers under the IPA 2016; it is effectively a notice to telecommunications operators to help public authorities exercise their surveillance powers. This would include interception warrants, under which UK public authorities can intercept communications while they are in transit and collect both content and metadata. The public authorities who may apply for such warrants include GCHQ, MI5, MI6, the National Crime Agency, the police, HMRC and Defence Intelligence.
Thus, one of the relevant public authorities could apply for an interception warrant and serve this on a telecommunications operators; once served, the operator must take all the steps required to give effect to the warrant as notified by the public authority (and keep this secret from the public). In addition, the relevant public authority could apply for and serve a TCN to supplement the interception warrant and obligate the operator to remove "electronic protection applied by or on behalf of that operator to any communications or data" (see s.253(5)(c) IPA 2016).
Both TCNs and interception warrants are subject to the double lock regime under the IPA, which requires a judicial commissioner (essentially a judge) to assess the application and ensure that it is both necessary and proportionate (and thus in compliance with applicable human rights law). But assuming that the application passes this step, it would seem like a public authority could, under the IPA, demand a company like Signal or WhatsApp to remove or modify E2EE for the purposes of national security, preventing or detecting serious crime, or in the interests of the economic well-being of the UK so far as those interests are also relevant to the interests of national security.

The apparent premise for the law enforcement POV is that access to the content of communications (i.e., the message body) is necessary to detect illicit content/illegal activity, but is this really true? Since communications are made up of the header and the message body, there seems to be two ways to carry out CSS:

Content scanning (scanning the message body): This would be the most straight forward, and potentially more effective, method for CSS. The scanning tool would take in as the input the message body and any attachments. This would allow the tool to directly process the content that may be illegal or otherwise illicit. This could include text, images or videos. If this content is encrypted, and there is no access to the correct cryptographic keys, then the tool can only ever access the ciphertext which is unintelligible and therefore cannot be scanned to determine if it is illegal or illicit.
Metadata scanning (scanning the message header): An alternative to scanning the content of a communication to detect potential illegality is to use the metadata as an input for the scanning tool. The metadata essentially consists of the header of a message, which would include information like the address of the sender and recipient and the date and time that the message was sent (the image below shows metadata that can typically be collected from an email). Microsoft has demonstrated the possibility of building neural networks trained to detect potential child sexual abuse material (CSAM) using file paths (i.e., C:\Users\pictures\hidden\filename.png). Meta has carried out research on CSA offenders on its platforms to deduce common patterns that can be used to train behavioural ML classifiers for CSA detection (not exactly metadata scanning but nevertheless an approach that does not necessarily rely on message content as such). However, the feasibility of metadata scanning depends on the nature of the metadata available in an E2EE service. Signal Messenger for instance implements the Signal protocol in a manner the encrypts both the message body and some elements in the header, leaving very little metadata to process (as it explained in response to a subpoena from the Eastern District of Virginia in 2016). Where this is the case, metadata scanning is basically futile.

Much of the conversation around breaking E2EE to detect illegal activity might hinge on how E2EE environments are being used for illegal activity and therefore whether law enforcement having access to such environments would actually help with the detection and prevention of illegal activity. In other words, would CSS in E2EE environments be an effective means for detecting and combatting crime? (This question is important because it is a factor to consider when evaluating the necessity and proportionality of any proposed CSS mechanism):

A report published in June 2023 from the Stanford Internet Observatory (SIO) on the distribution of CSAM on platforms like Instagram and Twitter (now 'X') suggests that breaking E2EE to detect CSAM is probably putting the cart before the horse; some of its key findings were the following:
- "Large networks of accounts that appear to be operated by minors are openly advertising self-generated child sexual abuse material (SG-CSAM) for sale."
- "Instagram is currently the most important platform for these networks with features like recommendation algorithms and direct messaging that help connect buyers and sellers."
- "Twitter had an apparent and now resolved regression allowing CSAM to be posted to public profiles despite hashes of these images being available to platforms and researchers for automated detection and removal."
It would therefore seem that, at least when it comes to CSAM, much of this activity happens in the open and, at least according to SIO, is not being adequately addressed.
If this is true, then detection efforts should be focused on these low-hanging fruits before attempting to infiltrate activity taking place in E2EE environments, as the SIO paper points out:
- "Better proactive investigations and heavier enforcement on keywords and hashtags: The proliferation of these accounts, particularly on platforms such as Twitter and Instagram, and the recurring patterns common to many accounts in the network (i.e., mentions of “menu”, certain emoji in bio, obvious hashtags involving variations on “pedo”), indicate a general lack of resources devoted to detecting SG-CSAM and associated commercial activity. SG-CSAM distribution rings could easily have been detected and acted upon by internal investigation teams, and there are existing policies in place that cover this type of content, but accounts often stay active for months. Reactive enforcement based on user reports is inadequate: investigative reporters regularly find activity that platforms have missed, simply by actively looking rather than relying on reports. Platforms should have a well-staffed internal team responsible for the detection and content enforcement of SG-CSAM, with proactive discovery mechanisms at their disposal." (Thiel et al 2023, 14)

Plus, one needs to also look at how E2EE-based services are used for CSA purposes:

Typically, perpetrators of CSA use tactics of encouragement, deception and coercion to record children engaging in sexual activities and (sometimes) distribute that CSAM (which could include photos or videos of the acts). The Centre of Expertise on Child Sexual Abuse, which was launched by the UK Home Office in 2017, produced a report in March 2020 detailing how perpetrators may seek to solicit CSAM from victims online:

Source: Centre of Expertise on Child Sexual Abuse, *A new typology of child sexual abuse offending* (March 2020), p.15.

As shown above, perpetrators eventually try to move their interactions with the victim to a 'private platform', which could be an E2EE-based messaging service like WhatsApp or Signal. This finding is also supported by the work of Rachel O'Connell who details in her paper the different stages of CSA and identifies the utility of 'private spaces' in the context of cybersexploitation and grooming conversations:
- "Throughout each of the stages there are clear and easily identifiable differences in the patterns of behaviour of the individuals, and these appear to relate closely to their motivations...But, it is important to note that whilst the stages outlined here provide a summary of the possible stages of cybersexploitation and grooming conversations, not all users will progress through the stages in the conversations sequentially, i.e. some adults will remain in one stage for longer periods than other adults and some will skip one or more stages entirely. The order and number of stages will vary and these variations provide clues of the user with ill-intent’s motivations. Furthermore, whereas some stages, for example the risk assessment stage, have specific and identifiable goals, the goals of other stages are psychological and relate closely to both the aims of the adult and his perceptions of, for example, how malleable a child is in terms of meetings his requirements. Very early in the initial friendship-forming stage the adult will suggest moving from the public sphere of the chat room into a private chat room in which rather than the one-to-many facility of a public arena, an exclusive one-to-one conversation can be conducted." (Emphasis added) (O'Connell 2003, 8)
So it is likely the case that E2EE-based messaging services are used for CSA, but how prevalent is such activity on these platforms?
- This is probably hard to measure given that those service providers implementing true E2EE do not have the requisite visibility to accurately assess this.
- When in 2019 Facebook proposed implementing E2EE across all of its messaging services, US, UK and Australian government ministers warned that this would hinder "law enforcement agencies’ ability to stop criminals and abusers in their tracks." The ministers further stated that, if E2EE were implemented, NCMEC estimates that 70% of Facebook's reporting would be lost.
- Between 2017 and 2020 in the UK, 10,019 offences of sexual communication with a child were recorded. "There were 5,784 offences where the means of communication was recorded. Facebook, Messenger, Instagram or WhatsApp was used in 3,203 instances, 55% of the total where means of communication is known, while Snapchat was used 1,060 times." (The Guardian)
- In a report published by the UK Council for Child Internet Safety in 2016, some findings were presented regarding children's exposure to sexual content online:
  - "Just over one in ten (12%) of online 9-16s say they have seen sexual images online." (p.10)
  - 4% said they have received "sexting" images
- Another report from the former Children's Commissioner Anne Longfield in December 2020 found that "9 in 10 children in England (aged 8-17) use a messaging app or website" with the most popular platform being WhatsApp. Furthermore, "around one in ten children are messaging people they don’t know" and "Over a third (38%) of children aged 8-17 report having received content on a messaging platform which worried them or made them feel uncomfortable in the four weeks prior to our survey. This material was most likely to have been sent by friends (16%) but children also reported receiving content that made them feel uncomfortable from acquaintances (12%), family members (9%) and strangers (10%) in the previous four weeks." (pp.9-11)

Function creep

Apart from the issues of CSS breaking E2EE, there is also the issue of function creep:

Function creep is the idea that a system is built for one purpose, but over time it is gradually reconfigured to be used for other purposes that are often outside the initial expectations for what the system would be used for.
In other words; a detection tool is built to detect one type of illicit content (CSAM, for example); the government comes along and says 'we order you, the service provider, to reconfigure your detection tool to detect other content we deem illicit/illegal' (as could be the case with interception warrants and TCNs under the IPA 2016); the service provider cannot argue that this is technically impossible to do and so either has to either comply with the order, legally challenge the order (if that option is even available) or, as a last resort, leave the market.
Function creep cannot be dealt with through technical solutions, as while it might be difficult to reconfigure a detection tool to detect other types of content even where its been built to detect only a certain type of content, it is not impossible.
Function creep can only be addressed through policy mechanisms, i.e., the service provider stating in its privacy notice/TOS: 'We will challenge unlawful government orders to collect data from your device'. But if an order comes that is not technically unlawful, what then?
- "Many critics of CSS have pointed out that governments have the power to order the service provider to search for material beyond the initial, publicly acknowledged target. The European Union has been pushing for CSS since July 2020, with concerns not just about child abuse, but also terrorism. Australian police have raided journalists for publishing images of war crimes; and the UK seeks to ban a range of online content, including some forms of speech that are perfectly legal face-to-face. All three already have laws enabling them to mandate tech companies to repurpose or retarget existing scanning techniques. Once we move beyond the OECD countries, the legal constraints against government action become weaker and the range of targeted content becomes larger, typically including LGBTQ+ content, political activists, and domestic rivals of authoritarian regimes. In such places, CSS will provide a means of repression and political manipulation. Augmenting the scope of CSS may not only refer to topic, but also to algorithmic changes. For instance, given that NCMEC’s mission includes missing children, it would not be surprising if there were pressure for CSS to be augmented with a face recognition capability." (Abelson et al 2021, 12-13)
An important principle from the world of data protection designed to stop function creep from happening is purpose limitation. Article 5(1)(b) of the GDPR provides that personal data must be "collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes."
Think also Helen Nissenbaum's theory about contextual integrity (i.e., meeting the expectations of the data subject which derive from the relevant context of the data processing). For example, most people would probably be happy to share their health data with a doctor for a medical diagnosis, but sharing that data with a pharmaceutical company for drug research might be outside the scope of reasonable expectations, thus requiring the patient to at least be notified of such data sharing if not given the opportunity to object:
- "Systems with security or safety requirements are also designed to limit the harm that they can do, including harm to third parties. In the case of surveillance systems, one strategy is to limit them to specific purposes, such as speed cameras that only detect vehicles breaking the speed limit at one location, and the software in scanners and copiers that prevents the copying of banknotes. The other is to limit them to specific targets, as with the law enforcement interfaces for wiretapping mobile phone systems that support only a limited number of wiretaps, which are supposed to be used only following an appropriate judicial order, and whose use can be audited. CSS must not be an exception to the principle of purpose limitation. Scanning systems should be designed so as to limit their scope in terms of what material can be scanned (topic and format), as well as where (which components of memory are accessible to the scanner), and by whom (who sets targets and receives alerts)." (Abelson et al 2021, 17-18)
If CSS is introduced, it can be argued that it introduces the risk of expansive surveillance activities that go beyond what the initial purpose of the capability might be:
- "When scanning for targeted material takes place at a central server, it affects only content the user uploads for sharing with others, such as photos and emails. The same communications monitoring capabilities would also have been provided by previous proposals for banning encryption or introducing government backdoors in encryption algorithms or protocols...The deployment of CSS changes the game completely by giving access to data stored on users’ devices. First, it facilitates global surveillance by offering economies of scale. Second, while proposals are typically phrased as being targeted to specific content, such as CSAM, or content shared by users, such as text messages used for grooming or terrorist recruitment, it would be a minimal change to reconfigure the scanner on the device to report any targeted content, regardless of any intent to share it or even back it up to a cloud service. That would enable global searches of personal devices for arbitrary content in the absence of warrant or suspicion. Come the next terrorist scare, a little push will be all that is needed to curtail or remove the current protections. For example, a scanner that looks only for CSAM images might have its target list extended to scan for other images and its software “upgraded” to scan for text as well, at population scale." (Abelson et al 2021, 21)

Hash-based tools for content moderation

Mahdi Assan

Apr 21

This post explores how hash-based detection tools work and how they are used for content moderation on internet platforms.

Read full story

When Apple first proposed NeuralHash in 2021, a copy of model could be found in a software update containing the tool and was extracted for independent use. This then allowed engineers to test the model and identify certain weaknesses, from which methods could be devised for evading detection (see here on GitHub).

Security

Typically, apps on mobile devices are compartmentalised. This means that apps cannot access the data contained/generated by other apps. Depending on exactly how CSS is implemented, mobile devices may need to move away from this method. This may be the case if the CSS developer is seeking to address users evading the mechanism by simply moving to a different messaging platform (for example if CSS is introduced in WhatsApp, users may simply move to Signal Messenger or another service that does not have CSS).

Additionally, CSS, like any other computer system/algorithm, is made up of lines of code, and code can come with bugs:

Updates are therefore necessary, but are also a powerful way to scale attacks—an example being the Russian attacks on US agencies through SolarWinds. A criminal adversary who can subvert the update mechanism could use it to install ransomware; the new models of ransomware-as-a-service enable such attacks to be monetized quickly with the help of others. (Abelson et al 2021, 24)

Accuracy

Both distributors and consumers of targeted material may seek to defeat a CSS system by making it useless for enforcement. This can be done in broadly two ways: first, by ensuring that targeted material of interest to them evades detection (i.e., by increasing the rate of false negatives), and second, by tricking the CSS system into flagging innocuous content, thereby flooding it with false alarms (i.e., by increasing the rate of false positives). (Abelson et al 2021, 24)

False Negative Attacks
This is where the attacker is trying to evade the scanning tool. It essentially means using techniques to trigger a false negative (i.e., the scanning tool determines that the content is not illegal/illicit when it is actually illegal/illicit).

Such attacks have been shown to work on CSS systems:

Jain et al. studied evasion in the context of CSS. Their results confirmed Hao et al.’s findings: for a large number of images (99.9% of the images in their study), it is possible to find nearly imperceptible changes to an image that cause it to not be detected any more. For a detector to avoid these false negatives, the number of images flagged would be orders of magnitudes larger, rendering manual review infeasible. Their experiments also showed that effective perturbations span a wide range of modifications, so building a robust defense that blocks all of them appears to be a wicked problem. (Abelson et al 2021, 27)

(The Shubham Jain paper can be found here).

If the CSS mechanism is using a hash-based detection tool, this could also be subject to poisoning attacks:

In a poisoning attack, the adversary deliberately influences the training dataset or process to manipulate a model and cause misclassifications...In the context of CSS, poisoning could involve an adversary altering the feed of data to the agency that curates the target list, so that some class of material that should be targeted is passed as innocuous instead. (Abelson et al 2021, 28)

False Positive Attacks
This is where an attacker sends innocuous content to a target recipient that triggers a false positive by the CSS mechanism (i.e., the scanning tool determines that the content is illegal/illicit when it is not actually illegal/illicit). These kinds of false alarms are a common attack method used against computer systems.

In the context of hash-based detection tools, attackers can uses false positive attacks by trying to find 'collisions', which is where two different images produce the same perceptual hash value. This can happen even though hash-based detection tools are designed to reduce the probability of this occurring (since of course the probability is never 0):

To create a collision with an existing target image, the adversary has to find what cryptographers call a second preimage of the hash (the target image is the first preimage). Generally, finding second preimages is a harder problem than finding collisions, but it still turns out to be easy for many of the known perceptual hash functions...The ease of finding second preimages of illicit content opens the door to adversarial collisions, where an attacker creates images with hashes that “match” those of targeted images. A user who downloads such content would trigger a match in their CSS system, which could result in a notification to their provider and possibly to law enforcement. (Abelson et al 2021, 29-30)

The Abelson paper shows an example of a collision from Apple's NeuralHash:

4. The Signal Protocol

Overview

The Signal Protocol is one of the most secure protocols for implementing E2EE for instant messaging applications. It is quickly becoming an industry standard; it is used by Signal Messenger, WhatsApp, Facebook Messenger and Skype (however some applications do not use it by default).

The protocol uses a number of different algorithms and protocols to implement E2EE for instant messaging applications. These include:

Extended Triple Diffie-Hellman key agreement protocol (X3DH)
XEdDSA and VXEdDSA signature schemes
Double Ratchet algorithm

The Diffie Hellman key agreement protocol is an algorithm developed by Whitfield Diffie and Martin E. Hellman in 1976; it comes from their paper entitled "New Directions in Cryptography." This algorithm is used in the Signal Protocol to produce three different keys:

Identity keys, which are long-term keys that represent the user
One-time pre-keys, which are ephemeral (i.e., time-limited) public keys that are deleted after use
Signed pre-keys, which are medium-term public keys which can be used if a users' one time pre-keys have expired

The XEdDSA and VXEdDSA signature schemes are used to sign and verify the Diffie Hellman keys used in the Signal Protocol.

The Double Ratchet algorithm is used to achieve what is called forward and backward secrecy. This means that different keys are generated with each message sent between users so that, if an attacker obtains one set of keys, those keys cannot be used to decrypt future or past messages.

How the Signal protocol implements E2EE

Using the various algorithms, the Signal Protocol implements E2EE by executing three main operations: (i) registration of public keys, (ii) X3DH key handshake and (iii) exchanging encrypted messages with forward and backward secrecy.

1. Registration of public keys

When Alice and Bob register with the instant messaging service, they submit a public key bundle to the central server. This bundle consists of the following Diffie Hellman public keys:

An identity key
One signed pre-key and its signature
A defined number of one-time signed pre-keys

These keys are submitted to the central server directing messages to the correct recipients. In this case of Signal Messenger, for example, users will submit their key bundles (and exchange their encrypted messages via) central servers operated by Signal.

2. X3DH key handshake

If Bob wants to send a message to Alice, he will need to retrieve her public key bundle containing the three public keys mentioned above (only one of the signed pre-keys are needed here). Bob will then need to use the XEdDSA and VXEdDSA signature schemes to verify that the signature over the signed pre-key is correct (i.e., that it actually belongs to Alice).

Bob will then use the Diffie Hellman protocol to generate a pair of ephemeral (i.e., time-limited) private and public keys to start a new session with Alice (after which these keys are discarded). Bob uses his ephemeral private key and his identity key with Alice's public keys retrieved in the bundle to produce a shared secret. This shared secret is used to produce a sending key and a receiving key; Bob will use the sending key to encrypt the message intended for Alice, and Alice will use the receiving key to decrypt the message from Bob.

3. Exchanging encrypted messages with forward and backward secrecy

As mentioned beforehand, the Double Ratchet algorithm is applied to the shared secret to ensure that, if it is obtained by an attacker, that attacker cannot view the future or past messages.

To achieve forward secrecy, a symmetric ratchet is applied to the sending key which turns it into a sending chain key. This means that when Bob or Alice are sending encrypted messages to each other, the sender will pass the sending key into a one-way function that produces a sending chain key (used for the next message to the recipient) and the sending key for the current message. The recipient will do the same with the receiving key. With this ratcheting, the compromise of a sending or receiving key by an attacker does not allow that attacker to recover previous messages.

To achieve backward secrecy, the Signal Protocol applies a Diffie Hellman ratchet. This works by including a ratchet public key in every message sent. This ratchet public key changes regularly, and whenever it does, the recipient will need to perform a key exchange with the new ratchet key and their ratchet key. The output of this exchange is then used to decrypt the message. The recipient will then also generate a new random ratchet key used to encrypt messages sent back to the sender. This prevents an attacker that manages to obtain one set of keys from being able to see future messages exchanged between users.

The combination of the symmetric ratchet and the Diffie Hellman ratchet is what makes up the Double Ratchet algorithm.

Summary of the Signal Protocol

Putting these different features together, this is how the Signal Protocol can be used implement E2EE for an instant messaging service:

Alice and Bob publish their Diffie Hellman public key bundles to the central server
Bob wants to start a session with Alice and send her encrypted messages and so he retrieves Alice's public key bundle from the central server to his device
Bob uses the XEdDSA and VXEdDSA signatures schemes to verify Alice's signature over her pre-key to ensure that the keys come from Alice
Bob uses the Diffie Hellman protocol to generate his ephemeral private key and public key
Bob combines his ephemeral private key and his identity key with Alice's public keys to generate a shared secret
The shared secret is used to produce a sending key and a retrieving key
The sending key is passed into a one-way function (the symmetric ratchet) that produces a sending chain key (for future messages) and a sending key for the current message created by Bob
Bob also uses the Diffie Hellman ratchet to generate a ratchet public key
Bob encrypts the message to Alice using the sending key
Bob then sends his encrypted message, a shared secret and a ratchet public key to Alice's device through the central server
Alice receives on her device Bob's encrypted message and the corresponding keys
Alice uses her ephemeral private key and her identity key to decrypt the shared secret
Alice uses the shared secret to produce a sending and retrieving key
Alice uses the retrieving key to decrypt the message from Bob and establish the session with Bob (i.e., start a new chat session)
From then on, the conversation between Alice and Bob uses the Double Ratchet algorithm to regularly generate different keys for the encrypted messages exchanged between the two users

The privacy value of the Signal Protocol

E2EE, and the Signal Protocol in particular, achieves two important objectives that together ensure a high level of privacy for users:
- Low identifiability: the communications are encrypted, so unintelligible to third parties, including the service provider, without the requisite keys, of which are only in the hands of the user.
- Low centricity: the keys are not stored on the server, and instead are stored on the devices of the users.

The more centralised and identifiable the data, the more policy controls needed to ensure privacy (i.e., the more trust that needs be given to those working for the service provider to not access or misuse the data). Conversely, the less centralised and identifiable the data, the less policy controls needed to ensure that the data are protected.

This a method for measuring the strength of systems designed to protect personal data and ensure privacy (in this case the confidentiality of communications) that comes from the work of Lorrie Cranor (Carnegie Mellon University) and Sarah Spiekermann(Business Informatics Professor). The graph below shows the degree of privacy friendliness using the variables of identifiability and centricity; the closer to the upper right part of the graph the system is (i.e., the lower the identifiability and centricity of the data processed), the more privacy-preserving the system: