DeepSeek R-1 Model Overview and how it Ranks Versus OpenAI’s O1

DeepSeek is a Chinese AI company “committed to making AGI a truth” and open-sourcing all its models. They began in 2023, but have been making waves over the past month or so, and particularly this past week with the release of their two latest thinking designs: DeepSeek-R1-Zero and the more sophisticated DeepSeek-R1, likewise understood as DeepSeek Reasoner.

They’ve released not just the designs however also the code and evaluation triggers for public use, in addition to an in-depth paper describing their approach.

Aside from producing 2 extremely performant designs that are on par with OpenAI’s o1 design, the paper has a lot of valuable information around reinforcement knowing, chain of thought thinking, prompt engineering with reasoning designs, and more.

We’ll start by focusing on the training procedure of DeepSeek-R1-Zero, which uniquely relied entirely on support learning, rather of standard supervised knowing. We’ll then proceed to DeepSeek-R1, how it’s thinking works, and some prompt engineering finest practices for thinking designs.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current design release and comparing it with OpenAI’s thinking designs, particularly the A1 and A1 Mini designs. We’ll explore their training procedure, reasoning capabilities, and some crucial insights into prompt engineering for reasoning designs.

DeepSeek is a Chinese-based AI company devoted to open-source advancement. Their current release, the R1 reasoning design, is groundbreaking due to its open-source nature and innovative training techniques. This includes open access to the models, prompts, and research study documents.

Released on January 20th, DeepSeek’s R1 achieved excellent efficiency on various benchmarks, rivaling OpenAI’s A1 designs. Notably, they also introduced a precursor model, R10, which works as the foundation for R1.

Training Process: R10 to R1

R10: This model was trained specifically using reinforcement learning without supervised fine-tuning, making it the first open-source model to accomplish high efficiency through this technique. Training involved:

– Rewarding appropriate answers in deterministic tasks (e.g., math issues).
– Encouraging structured reasoning outputs utilizing design templates with “” and “” tags

Through countless models, R10 developed longer thinking chains, self-verification, and even reflective behaviors. For instance, during training, the design demonstrated “aha” minutes and self-correction behaviors, which are uncommon in conventional LLMs.

R1: Building on R10, R1 included numerous improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human choice alignment for sleek actions.
– Distillation into smaller sized designs (LLaMA 3.1 and 3.3 at various sizes).

Performance Benchmarks

DeepSeek’s R1 design performs on par with OpenAI’s A1 models across many thinking benchmarks:

Reasoning and Math Tasks: R1 rivals or surpasses A1 designs in accuracy and depth of reasoning.
Coding Tasks: A1 models usually carry out much better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 typically surpasses A1 in structured QA tasks (e.g., 47% precision vs. 30%).

One notable finding is that longer reasoning chains usually enhance performance. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some restrictions:

– Mixing English and Chinese responses due to an absence of supervised fine-tuning.
– Less sleek responses compared to chat models like OpenAI’s GPT.

These problems were addressed throughout R1’s refinement process, including monitored fine-tuning and human feedback.

Prompt Engineering Insights

An interesting takeaway from DeepSeek’s research study is how few-shot triggering abject R1’s efficiency compared to zero-shot or concise tailored prompts. This lines up with findings from the Med-Prompt paper and OpenAI’s suggestions to limit context in thinking models. Overcomplicating the input can overwhelm the design and reduce precision.

DeepSeek’s R1 is a significant action forward for open-source thinking designs, showing abilities that match OpenAI’s A1. It’s an exciting time to try out these models and their chat interface, which is complimentary to use.

If you have questions or desire to find out more, inspect out the resources linked listed below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only approach

DeepSeek-R1-Zero stands apart from the majority of other cutting edge models because it was trained using only reinforcement learning (RL), no supervised fine-tuning (SFT). This challenges the present standard method and opens new chances to train reasoning models with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source model to verify that advanced thinking abilities can be established purely through RL.

Without pre-labeled datasets, the model learns through experimentation, fine-tuning its habits, specifications, and weights based solely on feedback from the services it produces.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL procedure for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero included presenting the design with various reasoning tasks, ranging from mathematics problems to abstract logic difficulties. The model produced outputs and was evaluated based on its efficiency.

DeepSeek-R1-Zero received feedback through a benefit system that assisted guide its knowing process:

Accuracy rewards: Evaluates whether the output is right. Used for when there are deterministic results (math problems).

Format rewards: Encouraged the design to structure its reasoning within and tags.

Training timely design template

To train DeepSeek-R1-Zero to produce structured chain of idea sequences, the researchers utilized the following prompt training design template, replacing prompt with the reasoning question. You can access it in PromptHub here.

This template triggered the model to clearly outline its thought procedure within tags before delivering the last answer in tags.

The power of RL in thinking

With this training procedure DeepSeek-R1-Zero began to produce advanced reasoning chains.

Through thousands of training steps, DeepSeek-R1-Zero evolved to solve increasingly complex issues. It found out to:

– Generate long thinking chains that enabled deeper and more structured analytical

– Perform self-verification to cross-check its own responses (more on this later).

– Correct its own errors, showcasing emerging self-reflective habits.

DeepSeek R1-Zero efficiency

While DeepSeek-R1-Zero is mainly a precursor to DeepSeek-R1, it still accomplished high efficiency on a number of standards. Let’s dive into some of the experiments ran.

Accuracy improvements throughout training

– Pass@1 accuracy started at 15.6% and by the end of the training it enhanced to 71.0%, equivalent to OpenAI’s o1-0912 model.

– The red solid line represents efficiency with majority voting (similar to ensembling and self-consistency methods), which increased accuracy even more to 86.7%, surpassing o1-0912.

Next we’ll take a look at a table comparing DeepSeek-R1-Zero’s performance across numerous thinking datasets versus OpenAI’s thinking designs.

AIME 2024: 71.0% Pass@1, a little listed below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll look at how the action length increased throughout the RL training process.

This graph reveals the length of actions from the model as the training procedure advances. Each “step” represents one cycle of the design’s learning process, where feedback is offered based on the output’s efficiency, evaluated using the prompt design template discussed earlier.

For each concern (corresponding to one step), 16 responses were sampled, and the typical accuracy was computed to make sure stable assessment.

As training advances, the design generates longer thinking chains, permitting it to solve increasingly complex thinking jobs by leveraging more test-time compute.

While longer chains don’t constantly ensure much better results, they normally correlate with enhanced performance-a pattern also observed in the MEDPROMPT paper (check out more about it here) and in the initial o1 paper from OpenAI.

Aha moment and self-verification

Among the coolest elements of DeepSeek-R1-Zero’s advancement (which also uses to the flagship R-1 model) is just how good the design became at reasoning. There were advanced thinking behaviors that were not clearly programmed but arose through its support finding out procedure.

Over countless training actions, the model began to self-correct, reevaluate problematic logic, and validate its own solutions-all within its chain of thought

An example of this noted in the paper, described as a the “Aha minute” is below in red text.

In this instance, the design actually said, “That’s an aha minute.” Through DeepSeek’s chat function (their variation of ChatGPT) this kind of thinking generally emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and challenges in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to perform at a high level, there were some drawbacks with the model.

Language mixing and coherence problems: The design occasionally produced actions that combined languages (Chinese and English).

Reinforcement knowing trade-offs: The lack of monitored fine-tuning (SFT) implied that the design did not have the improvement needed for totally polished, human-aligned outputs.

DeepSeek-R1 was developed to attend to these concerns!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning design from the Chinese AI lab DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained entirely with support knowing. Unlike its predecessor, DeepSeek-R1 includes supervised fine-tuning, making it more fine-tuned. Notably, it outshines OpenAI’s o1 model on several benchmarks-more on that later.

What are the primary distinctions between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 constructs on the foundation of DeepSeek-R1-Zero, which serves as the base model. The two vary in their training techniques and overall efficiency.

1. Training method

DeepSeek-R1-Zero: Trained totally with support learning (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes monitored fine-tuning (SFT) initially, followed by the exact same reinforcement discovering procedure that DeepSeek-R1-Zero wet through. SFT helps enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Battled with language mixing (English and Chinese) and readability issues. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these concerns with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still a very strong thinking design, often beating OpenAI’s o1, however fell the language blending problems decreased use significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on most reasoning benchmarks, and the actions are much more polished.

In short, DeepSeek-R1-Zero was an evidence of concept, while DeepSeek-R1 is the completely optimized variation.

How DeepSeek-R1 was trained

To tackle the readability and coherence concerns of R1-Zero, the scientists integrated a cold-start fine-tuning stage and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a premium dataset of long chains of idea examples for initial supervised fine-tuning (SFT). This information was collected utilizing:- Few-shot prompting with detailed CoT .

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the same RL process as DeepSeek-R1-Zero to fine-tune its thinking capabilities further.

Human Preference Alignment:

– A secondary RL stage improved the model’s helpfulness and harmlessness, making sure better alignment with user requirements.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller sized, effective models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 standard performance

The scientists checked DeepSeek R-1 throughout a range of benchmarks and versus leading designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The benchmarks were broken down into several classifications, shown listed below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were used across all models:

Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 outshined o1, Claude 3.5 Sonnet and other models in the majority of thinking benchmarks.

o1 was the best-performing model in 4 out of the five coding-related criteria.

– DeepSeek performed well on creative and long-context job job, like AlpacaEval 2.0 and ArenaHard, exceeding all other designs.

Prompt Engineering with thinking designs

My favorite part of the short article was the scientists’ observation about DeepSeek-R1’s sensitivity to triggers:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which references Microsoft’s research on their MedPrompt framework. In their research study with OpenAI’s o1-preview model, they discovered that frustrating thinking designs with few-shot context broken down performance-a sharp contrast to non-reasoning designs.

The crucial takeaway? Zero-shot triggering with clear and concise guidelines appear to be best when utilizing reasoning designs.