Enhancing AI Training: OpenAI’s New Strategy to Combine Human Insight with AI Assistance

To review this article again, go to My Profile and then click on View saved stories.

Knight Will

OpenAI Advocates for AI Assistance in Training AI Models

A significant factor contributing to ChatGPT's remarkable triumph is the multitude of human trainers that directed the AI behind the chatbot, educating it on the difference between acceptable and unacceptable responses. OpenAI believes that incorporating additional AI to support these human trainers could enhance the intelligence and dependability of AI assistants.

OpenAI led the way in implementing a unique approach called reinforcement learning with human feedback (RLHF) during the creation of ChatGPT. This method involves enhancing an AI system's responses by incorporating evaluations from human participants, aiming to improve the AI's relevance, reduce inappropriate content, and increase precision. The feedback provided by these human evaluators is used to refine the AI's underlying algorithm, influencing its performance. This strategy has been instrumental in both elevating the utility and dependability of conversational agents and curbing their potential for problematic behavior.

“RLHF demonstrates strong performance, yet it's not without its significant drawbacks,” states Nat McAleese, a member of the research team at OpenAI. One issue is the variability in human feedback. Additionally, it can pose a challenge for even experienced individuals to evaluate highly complex results, like advanced software programming. Furthermore, this method may lead the model to generate outputs that appear credible on the surface but lack true accuracy.

OpenAI has advanced its technology by enhancing its top-tier model, GPT-4, creating a specialized version designed to support human evaluators in reviewing code. This upgraded version, named CriticGPT, has demonstrated an ability to identify errors overlooked by humans, with human reviewers preferring its code analysis 63 percent of the time over their own. OpenAI plans to explore how this method can be applied in domains other than coding moving forward.

"McAleese mentions that they are beginning to incorporate this method into their RLHF chat framework. He acknowledges that the method is not without flaws, as CriticGPT can generate incorrect outputs due to hallucinations. However, he believes that this approach could enhance the precision of OpenAI’s models, including ChatGPT, by minimizing human training errors. Furthermore, he suggests this method could be pivotal in advancing AI models' intelligence, potentially enabling humans to train AI that surpasses their own capabilities. 'As the models improve incrementally, we anticipate a growing need for human assistance,' McAleese states."

The latest method is among several currently in progress aimed at enhancing big language models and amplifying their capabilities. Additionally, it contributes to the initiative to guarantee that AI maintains appropriate conduct as it advances in proficiency.

In the early days of this month, Anthropic, which competes with OpenAI and was established by former OpenAI staff, unveiled an enhanced iteration of its chatbot named Claude. This advancement was attributed to refinements in the training process and the quality of data used for training. Additionally, both Anthropic and OpenAI have introduced innovative methods for examining AI models. These techniques aim to comprehend the processes behind their responses, with the goal of reducing the likelihood of undesirable actions like deceit.

A novel approach may aid OpenAI in developing increasingly sophisticated AI systems that generate outputs more aligned with human principles and reliability, particularly if this method is applied beyond coding. OpenAI has announced it is working on its next significant AI project, clearly demonstrating its commitment to ensuring this technology behaves appropriately. This development comes after the dissolution of a key team focused on evaluating AI's long-term dangers. The team, which had been co-led by OpenAI cofounder and former board member Ilya Sutskever—who at one point challenged CEO Sam Altman's leadership before retracting and assisting in his reinstatement—has seen several of its former members openly criticize the company. They argue that OpenAI is proceeding dangerously fast in its quest to develop and market potent AI technologies.

Dylan Hadfield-Menell, an MIT professor focused on AI alignment strategies, notes that the concept of using AI to train more advanced versions has been under consideration for some time. "It's a fairly logical progression," he remarks.

Hadfield-Menell points out that the scientists behind the creation of methods for RLHF had already touched upon similar concepts a few years back. He believes it's still uncertain how widely useful and potent this approach can be. "This could result in significant improvements in specific areas, and it could also act as a precursor to more efficient feedback mechanisms in the future," he suggests.

Authored by Kelly

Authored by Jaina Grey

Authored by David

Authored by Kate Knibbs

Explore Election Period with Our WIRED Politics Lab Newsletter and Podcast

Unconvinced that breakdancing qualifies as an Olympic sport? The global champion shares your sentiments (to some extent).

Investigators unlocked a decade-old encryption to access a cryptocurrency wallet valued at $3 million.

The surprising emergence of the first-ever beauty contest judged by artificial intelligence

Ease the strain on your spine: Discover our top picks for office chairs based on our evaluations.

Caroline Haskins

Dhruv Mehrotra

Knight Will

Additional Content from WIRED

Evaluations and Instructions

© 2024 Condé Nast. Rights reserved. WIRED could receive a share of revenue from items sold through our website, thanks to our Affiliate Partnerships with retail stores. The content of this site is protected and cannot be copied, shared, transmitted, stored in a cache, or utilized in any way without explicit prior consent from Condé Nast. Advertising Options

Choose a global website