Key takeaways of this interview:
- Trust in AI is not limited to the quality and lawfulness of training data. Above all, it requires effective capabilities for evaluating, testing and monitoring systems and their uses.
- The availability and quality of data are becoming a major issue, with tensions arising from intellectual property rights, personal data protection and the concentration of actors capable of purchasing or licensing large-scale corpora.
- Synthetic data and multimodal training (text, image, audio, video) can help alleviate data scarcity. However, excessive reliance on synthetic data at the pre-training stage may impair model performance, with a risk of so-called model collapse.
- To improve the reliability of the non-deterministic outputs of generative AI, large-scale observability and source traceability (in particular through RAG mechanisms) are decisive in measuring error rates, limiting bias and strengthening transparency.
- Regulation and governance frameworks (the AI Act, the Council of Europe, AI Safety Institutes, and Chinese and US approaches) seek to reaffirm the primacy of human oversight, to contain risks (discrimination, fundamental rights, information pluralism) and to establish clear lines of responsibility.
Sylvie Rozenfeld: One initial aspect of the question of trust concerns training data. The quality of AI depends on the data it uses. The higher the quality of the data, the more reliable and effective the AI. However, the volume of text written by humans is limited and is renewed more slowly than the rate at which AI systems consume it. Some estimate that only two or three years of data remain available, while others anticipate a depletion of this stock between 2026 and 2032.
Can we move towards trustworthy AI, and how might this be achieved?
Murielle Popa-Fabre: Most studies focus on the exhaustion of written content available on the web for training large-scale algorithms such as language models. This use is expanding more rapidly than the renewal of human-produced content. Several factors nevertheless allow these forecasts to be placed in context. First, web crawling techniques are now more sophisticated. They involve a more thorough data-cleaning stage, which makes higher-quality data far more accessible.
There is also increasing experimentation with solutions that generate data automatically from examples of human-produced content in order to reach the required volumes, without building entire corpora solely from synthetic data. Training is divided into two main phases: what is known as pre-training, during which the neural network is shaped through learning on the training corpus, and post-training, where the model is refined and optimised. It is during this second phase that high-quality data is most needed, as the algorithm must be instructed to perform specific tasks using sets of instruction–response pairs. Synthetic data is particularly useful here, as it replaces human annotation and therefore reduces costs. Several studies show that pre-training based primarily on synthetic data leads to a degradation of outputs, referred to as model collapse.
However, it should be added that generative AI has become multimodal. Using non-textual data such as images, audio or video also helps models learn about the world (probability distributions) through modalities other than text. This enriches the model with information that is often not present in written form and reduces the relative importance of text. Recent studies by the start-up EpochAI indicate that multimodal training could potentially triple the volume of available training data.
Beyond data-related issues, the key element for trustworthy AI lies in having the capacity to evaluate, test and monitor outputs and uses. This is particularly important for generative AI solutions which, unlike traditional software, produce non-deterministic responses: the same input may correspond to multiple outputs, some correct and others incorrect.
Does intellectual property, and more specifically the fact that rights holders prohibit the use of their data, not reduce the volume of data available?
Murielle Popa-Fabre: There are indeed significant tensions surrounding data availability, but I would not, as a general matter, set the protection of intellectual property or personal data against innovation. I would instead look at how the past year has led to major data acquisition campaigns involving large publishers, which have been widely reported in the press. The way data and generative AI disputes reached US courts as early as 2023 prompted a number of major players, such as OpenAI and Amazon, to launch data purchase or licensing initiatives (Springer, Reddit, etc.) in order to move beyond this deadlock.
Pressure on data availability was also evident over the summer, when we successively saw changes to the terms of use of social media platforms aimed at allowing AI training on user data by default, initiatives that were blocked by European law for users based in the EU. More recently, we have seen a surge of messages on LinkedIn from US influencers over the past week warning non-European users that a default opt-in mechanism is now in place, granting the platform and its affiliates rights to use personal data and published content to train generative AI models. American users are fully aware that European citizens benefit from stronger legal protection.
Without seeking to predict future developments, I would add that data quality is becoming even more decisive, as technology and adoption are moving towards increasingly smaller and more specialised models, which allow companies to better control production costs.
Are we heading towards increased commodification of data through licensing agreements, resulting in access being reserved for the wealthiest actors and a growing concentration of AI companies, potentially leading to oligopolies or even monopolies?
Murielle Popa-Fabre: This reflects one of the underlying issues of artificial intelligence: data and computing capacity are as significant as the concentration of talent. The quality of an algorithm depends on the quality of the data to which a developer has access. Today, algorithms are no longer simple language models but composite systems that require a number of stages to be optimised and to simulate what is described as a “good conversation”, referred to in Anglo-Saxon literature as alignment. Achieving this requires data derived from user interactions with generative AI systems, for example to determine what retains user engagement. Large platforms already hold this type of data and can reintegrate it into the optimisation of algorithmic outputs, for instance on the basis of a user’s retention history. The concentration of power is therefore felt at both ends of the data value chain for training generative AI: pre-training and the final optimisation of products.
Will we see the same situation as with search engines, with a single highly dominant player?
Murielle Popa-Fabre: That will depend on many factors: whether European actors manage to hold their own in the competition to bring sovereign products to market (their main advantage at present lies in product ergonomics), whether smaller-scale models succeed in gaining lasting traction, whether European investment in computing infrastructure and cloud services keeps pace, and whether national and global regulation, together with competition authorities, are able to address this concentration of power. Ultimately, it will also depend on awareness among users and society at large, and on how the United States positions itself in relation to China. Geopolitics plays a central role in this matter.
Generative AI systems such as ChatGPT also face the issue of personal data contained in training datasets. How can this be addressed? Are we able to identify personal data?
Murielle Popa-Fabre: Yes, it is possible to identify personal data both upstream, within training datasets, and downstream, in outputs, although this comes at a cost. Language models can also be tested for what they memorise and are therefore liable to reproduce, using a technique known since 2019 as extraction attacks. In addition, a growing number of tools referred to as observability tools are being developed. These implement output monitoring and make it possible to apply real-time filters, although these are costly. The ability to identify or filter personal data at output stage will depend on the technical testing capabilities and expertise that European or international competent authorities choose to develop, as well as on the governance frameworks adopted in Brussels.
There is, however, some positive news. OpenAI and its competitor Anthropic (Claude) have recently agreed to submit the algorithms they commercialise to testing by the US AI Safety Institute. This body was established following an executive order by President Joe Biden, with the aim of assessing the safety of generative AI. The United Kingdom was the first to establish such an institute, following the summit organised last November by the Prime Minister, in order to bring together the expertise required to evaluate these systems. However, a House of Commons report published last May revealed that no actor had submitted its algorithms to that body. While the UK authorities were unable to persuade generative AI providers to lodge their models for testing, the United States has now succeeded in doing so. Whether these models will be tested primarily against privacy and intellectual property criteria remains to be seen. It should also be noted that these forms of testing are not yet standardised.
Sylvie Rozenfeld: What would these tests focus on?
Murielle Popa-Fabre: Primarily on the reliability of outputs, then on their discriminatory character and on the impact on fundamental rights arising from uses and generated content. They would also address the widely publicised issue of AI autonomy risks, following statements by certain researchers claiming that AI will become autonomous. I remain sceptical about this transatlantic narrative and consider that alarmist discourse tends to obscure the real issues currently at stake, such as the plurality of knowledge and access to information. If information retrieval were to result in the presentation of a single response option, how could the pluralism that underpins democracy remain unaffected? This is all the more significant given that several empirical studies show that interacting with a conversational agent alters users’ opinions.
Sylvie Rozenfeld: One of the reasons for the shortage also lies in the fact that most high-quality content is protected by intellectual property rights.
Murielle Popa-Fabre: There is increasingly limited transparency regarding training data. Trade secret protection is often invoked in this respect, as the correlation between the quality of training data and model performance is now better understood. Traditional textual training corpora such as Book3 (The Pile or C3) contain a large majority of works protected by copyright, which has given rise to a significant number of legal actions before US courts.
There are several ways of detecting whether a work has been used to train an algorithm, using relatively straightforward tests. Solutions also exist to filter recurring citations from a language model at output stage. This offers partial protection for rights holders against the extraction of memorised data by a language model, but it does not resolve the issue of remuneration. A study published in September reports that, in 2023, 40% of AI-generated content detected on the web (which represents only a small fraction of automatically generated content, as most of it is not detectable) did not comply with intellectual property rules.
Would ChatGPT have been as performant if it had been trained solely on royalty-free data?
Murielle Popa-Fabre: The performance of the first version of ChatGPT (November 2023) is largely attributable to click-workers in Kenya (Samasource), who painstakingly annotated toxic, violent and discriminatory outputs from GPT-3 in order to enable optimisation and turn it into a tool and product capable of being commercialised. Pure training, without successive phases of refinement and optimisation, is not sufficient to produce a final product.
This then raises the question of how to remunerate intellectual property rights. A metaphor can help to illustrate the complexity of the issue: a language model is not like a fruit salad, as the web is, where individual fruits can be identified just as sources of information can be identified. It is more akin to an informational smoothie. One can sense the presence of banana or strawberry, but it is not possible to identify that a particular response or sentence was generated on the basis of a specific protected text.
It is therefore not possible to define the contribution of Harry Potter to the linguistic statistics underlying the calculations used to answer a given question, unless verbatim excerpts from Harry Potter appear in the machine’s responses. For example, it was possible to extract quotations from Donald Trump’s tweets from versions predating ChatGPT, because they circulated widely in the media sphere and were frequently reproduced online.
What about the use of RAG (Retrieval Augmented Generation), which makes it possible to optimise generative AI outputs by relying on an organisation’s own data?
Murielle Popa-Fabre: RAG is a way of adding information drawn from a defined database to a language model. Within these composite systems, models are used primarily for their ability to rephrase and deliver responses in natural language, rather than as standalone sources of knowledge. The database may contain private data, corporate data, or news content in informational RAG systems that are now emerging. This is currently the most widely used approach, as it allows certainty as to sources and, above all, enables their provenance to be traced. What enters the database is known, and traceable within the responses produced.
Being able to identify which information was used to generate an answer opens the way to new mechanisms for remunerating rights holders.
Sylvie Rozenfeld: And what about trust in the outputs?
Murielle Popa-Fabre: I work on the specification and evaluation of these types of composite systems in real-world settings within companies and institutions, and I believe it is important not to give in to the impression that individual users can assess these systems on their own. Only observation of their operation at scale makes it possible to determine error rates in responses. In other words, a generative AI system cannot be tested solely at the level of the individual user because, unlike traditional software, it is not deterministic. The same question may yield several correct and incorrect answers in an unpredictable manner. As outputs are not stable, the issue must be considered at scale, with appropriate tools for observability (a term that has become established, albeit in a distorted form, in the field of generative AI) and for tracing responses, as demonstrated on a large scale by the start-up AIForensic during the European elections.
You referred to a study published in May as marking a turning point in how we can understand how AI produces an output. What is this about?
Murielle Popa-Fabre: This is a study by Anthropic, the company behind the Claude algorithm. The study shows how, in a commercial language model, we are beginning to glimpse the way information is compressed within the LLM, and the probability distributions it learns. From a more technical standpoint, it detects and classifies patterns of neuronal activation in the middle layers of the LLM. This study moves that neural-network architecture beyond the “black box” paradigm.
Accordingly, if we can observe the kinds of information clusters inside a neural network, we may be able to modify it. This not only provides transparency, but also enables the development of more sophisticated governance techniques for these generative AI models. Until now, attempts to control these systems have focused on shaping their outputs through what I have referred to as post-training, and through filtering. For example, by training them to answer questions, to adopt politically acceptable language using different methodologies, or by preventing them from saying certain things, all with the aim of controlling what the algorithm generates.
If we continue to understand what is happening, what is being aggregated, and how information is organised and compressed across billions of texts, images and so on, we can move towards more advanced governance. We already knew, in principle, that this was possible, and this study demonstrates it in practice with a medium-sized model of 70 billion parameters. Until now, the literature largely offered proofs of concept; here, we have concrete results that allow us to observe more closely biases, or simply the kinds of informational representations at work.
Is open source a factor of trust?
Murielle Popa-Fabre: The question is simple, and the answer complex, because levels of transparency within the open-source community are not uniform. Today, we distinguish three or four degrees of openness for language models, extending as far as the publication of scripts or training data. Licences describe rights relating to publication and commercial re-use, but not necessarily the degrees of transparency that may matter for impact assessments or regulatory authorities. The issue of open source remains fundamental from an economic perspective, because it creates a base upon which new application solutions can be built. It is, of course, correlated with the ability to test an algorithm. However, questions of trade secrets and technological sovereignty re-emerge.
The positive development is that transparency can also be achieved while preserving trade secrets through various governance tools, as is currently the case in the United States with inspection by the US AI Safety Institute of OpenAI and Anthropic’s algorithms, or in China. For example, since its first law on recommendation algorithms, China has established a national repository where all algorithms and their training corpora are deposited; filing is mandatory before market deployment. The coming months will indicate which European governance structures are selected for a growing body of legislation regulating uses that involve highly complex computational objects in real-world settings.
It should nevertheless be noted that, in both configurations, there remains a need to achieve a level of transparency towards civil society, which is typically included within the principle of transparency invoked by various international recommendations on responsible AI.
Sylvie Rozenfeld: You work for the Council of Europe on AI and human rights. The Council of Europe produced the first international standard, approved by ministers of the 56 member states and signed on 5 September. In parallel, the EU has just published the AI Act; China, for its part, regulated AI at an early stage; and in the United States the sector calls for flexibility, while California has adopted binding legislation. Some sixty countries already have a text. AI is global, and legal approaches differ markedly.
What do you think of these different legal approaches, particularly China’s, given that you are a Sinophone? Can regulation have beneficial effects?
Murielle Popa-Fabre: The multiplicity of regulatory initiatives emerging worldwide on AI reflects a clear determination, within societies, to place human beings above machines. The Council of Europe’s approach is highly complementary to that of the European Union, because it focuses on fundamental rights and the principles that underpin them. It has a different economic reach from the AI Act, but it is not devoid of a geopolitical dimension, which will be interesting to observe as the number of signatories increases. The convention, for example, offers African countries the possibility of adhering to this vision of fundamental rights, at a time when China is also promoting principles of AI governance through its Belt and Road Initiative. The first signatures show that it is possible to bring countries together, on a broader scale, around certain core principles of the Council of Europe.
Positions on the global stage differ substantially. The United States has decided, through a presidential executive order by Joe Biden, to act quickly and in a highly detailed manner through around a hundred targeted actions, whereas European regulation constitutes a cross-cutting framework for AI uses and risks. In this phase of the “algorithmisation” of society, where quantitative approaches increasingly displace qualitative ones, I consider that starting from the premise that human beings must govern AI reflects how we wish, in general terms, to shape the future of societies. An algorithm manages quantities; AI governance also means reintroducing qualitative considerations and freedom. It reflects a desire to locate freedom at a non-quantitative level. What interests me, in the context of fundamental rights, is how they are to be interpreted in an increasingly quantitative world.
Moreover, behind the question of freedom lies that of responsibility. The Biden administration’s first political intervention was to convene leading tech executives at an early stage and ask them to act responsibly in relation to generative AI. In the United States, the issue of responsibility predominates within the term “Responsible AI”. In Europe, and in France, the term “ethical AI” is more commonly used. This may reflect the fact that, in common law, as you said, it is often easier to operate at the level of the individual rather than at the level of overarching principles.
Can you clarify China’s approach to AI regulation?
Murielle Popa-Fabre: Chinese regulation addressed AI before the emergence of generative AI. Its approach has been pragmatic and technology-focused, adopting laws on different types of algorithms as they emerged: one on recommendation algorithms, another on “deep synthesis” algorithms, and a third on language models. These instruments regulate what an algorithm may or may not do in society, ranging from price-setting to the allocation of delivery drivers’ workloads, and voice cloning, through texts (and standards) that are highly precise but not cross-cutting in the way the AI Act is. These are very different ways of approaching the problem. Since last summer, however, China has been considering a cross-cutting text similar to the AI Act, and two proposals have already emerged.
Sylvie Rozenfeld: The Council of Europe’s Committee of Experts on the implications of generative artificial intelligence for freedom of expression (MSI-AI) held its first meeting in May. You have been appointed co-rapporteur for the draft practical guidelines intended to maximise benefits and mitigate risks arising from generative AI that affect freedom of expression. In your view, what are the risks, and what avenues exist to address them?
Murielle Popa-Fabre: We are working extensively on the state of the art concerning the benefits and risks relating, in particular, to freedom of expression, and to the technological object itself and its presence within tools and products. Publication of our work is expected by the end of 2025. One example already highlighted by various institutions is the possibility of simultaneous translation in Europe through applications based on generative AI. This is a major opportunity. Among the benefits is the ability to build multilingual debates at European level rapidly, without having to default to English. Companies working in this area are currently quantifying and analysing this use case with researchers at Sciences Po. Individual expression may therefore be supported by this type of multilingualism, while it also carries certain risks.
There are, in particular, linguistic biases. Many have noticed that, when using ChatGPT, turns of phrase are not necessarily incorrect, but they can sound not quite French. The way arguments are developed, or forms of address, may also be skewed, because most training data is Anglo-Saxon. Users therefore find themselves with tools that suggest content not fully aligned with the linguistic norms of their native language. That is a first and problematic bias, quite apart from the fact that language conveys a view of the world. This also raises a question of linguistic sovereignty. In practice, the use of these tools standardises users’ expression at scale, as several studies on the automated generation of text and images have shown. The analysis of AI’s impact always takes place on two different but complementary planes: at the level of the individual, and at that of the collective, or society as a whole.
More generally, it remains important to move beyond the “black box” paradigm in generative AI and to understand the technological layering behind products, so as to identify where, along the value chain, biases may arise, as competition authorities in the United Kingdom and France have already done in their analyses. This is a methodology that has proven effective.
Another aspect already highlighted by a number of studies concerns “over-reliance” on a result produced by a machine. For example, many mistakenly assume that AI has “read” billions of books that a human being could never read in a lifetime, whereas it is a system that calculates likely sentences. There is also, plainly, the issue of voice cloning and personification, whether stylistic or video-based. The outcome of this work at the Council of Europe will likely reflect a very broad field of study and the diversity of members participating in the committee. They come from very different backgrounds (law, social sciences, academia, NGOs, etc.) and are as committed as they are capable.
Pull quotes
The ability to identify or filter personal data at output stage will depend on the technical testing capabilities and expertise that European or international competent authorities choose to develop, and on the governance frameworks adopted in Brussels.
There are several ways of determining whether a work has been used to train an algorithm, using relatively straightforward tests. (…) The question then becomes how those intellectual property rights should be remunerated.
Only observation of their operation at scale makes it possible to determine the error rate of responses.
If we continue to understand (…) how information is organised and compressed across billions of texts, images and so on, we can move towards more advanced forms of governance.
The multiplicity of regulatory initiatives emerging worldwide in relation to AI reveals a deep-rooted societal intention to place human beings above machines.
Disclaimer
The opinions, presentations, figures and estimates set forth on the website including in the blog are for informational purposes only and should not be construed as legal advice. For legal advice you should contact a legal professional in your jurisdiction.
The use of any content on this website, including in this blog, for any commercial purposes, including resale, is prohibited, unless permission is first obtained from Evidency. Request for permission should state the purpose and the extent of the reproduction. For non-commercial purposes, all material in this publication may be freely quoted or reprinted, but acknowledgement is required, together with a link to this website.
