Technical Review: Claude 3.7 Sonnet & Claude Code for Developers

Yusuf Ishola,

Lina Lam

· February 25, 2024

Anthropic just released Claude 3.7 Sonnet and Claude Code, showing significant advancements in reasoning and AI development assistance that could fundamentally change how developers work with AI tools.

In this blog, we dive into technical details of the new model and Claude Code. We'll explore benchmarks, real-world projects, and why developers should pay attention to these releases. Let's dive in!

Key Takeaways

• Claude 3.7 Sonnet excels in software engineering and agentic workflows, outperforming OpenAI, Grok, and DeepSeek in these categories.
• OpenAI's models still lead in multilingual Q&A, visual reasoning, and mathematical problem-solving, with o3-mini dominating in complex math tasks.
• Grok 3 showed superior performance in visual reasoning and high school mathematics than Claude 3.7 Sonnet.
• Claude 3.5 Sonnet lags behind Claude 3.7 Sonnet in most areas (unsurprisingly).

What's new about Claude 3.7 Sonnet?

1. The first hybrid reasoning model

Users have full transparency into the model's thought process. Being the first hybrid reasoning model, Claude 3.7 Sonnet can operate in two distinct modes:

• Standard mode: Quick responses for everyday tasks
• Extended thinking mode: Deep reasoning for complex problems

2. API access for reasoning models

Claude 3.7 is available via Anthropic API, AWS Bedrock, and Google Vertex AI, making it one of the few reasoning models accessible via API.

Developers can set a "thinking budget" through the API by setting a maximum token limit for reasoning (up to 128K tokens).

3. Better coding capabilities

Claude 3.7 Sonnet establishes itself as an industry leader in code generation and understanding. The model:

• Achieves state-of-the-art results on SWE-bench Verified (a benchmark for real-world software issues)
• Excels at TAU-bench (which tests AI agents handling complex workflows)
• Has been recognized as the preferred AI coding assistant by major development platforms including Cursor, Cognition, Vercel, and Replit

4. More advanced agentic abilities

Initial experimentation with Claude 3.7 Sonnet showed impressive agentic abilities:

• Supports GitHub integration for a much deeper understanding of your codebase
• Achieved an 81% success rate in online shopping tasks and 58.4% in booking flights

5. Improved safety and reduced refusals

The new model shows a 45% reduction in unnecessary refusals compared to Claude 3.5 Sonnet, while maintaining strong resistance to prompt injection attacks and other adversarial exploits.

6. Claude Code: a command-line AI assistant

Claude Code is a command-line AI assistant that integrates with development workflows, a completely new product category for Anthropic.

Unlike previous AI coding assistants, Claude Code runs in your terminal and directly modifies your local files, reducing the need for complex integrations or extra servers.

import anthropic

client = anthropic.Anthropic(
  api_key=ANTHROPIC_API_KEY,
  base_url="https://anthropic.helicone.ai/{HELICONE_API_KEY}",
)

How much does Claude 3.7 Sonnet costs?

Claude 3.7 Sonnet pricing matches that of Claude 3.5 Sonnet:

• $3 per million input tokens
• $15 per million output tokens (including thinking tokens)

You can calculate the cost of Claude 3.7 Sonnet using Helicone's LLM API pricing calculator.

Claude 3.7 Sonnet Benchmark Comparison

Anthropic released a series of benchmarks to showcase Claude 3.7 Sonnet's capabilities.

Overall, Claude 3.7 is the best LLM for software engineering and building AI agents, but isn't the best at math or visual reasoning.

Image Source: Official Claude Announcement

How to Access Claude 3.7 Sonnet and Claude Code

• Claude 3.7 Sonnet is available to all users.
• While the free tier does not include extended thinking capabilities, Pro, Team, and Enterprise users can access itvia Web or Apps.
• Developers can access the latest model via the Anthropic API - use model string claude-3-7-sonnet-20250219, Amazon Bedrock or Google Cloud Vertex AI
• Claude Code is currently in limited research preview and is available only to a few select users. You can join the waitlist here.

Real-World Projects with Claude 3.7 Sonnet

Generally speaking, Claude 3.7 Sonnet has been nothing short of remarkable.

Let's take a look at what users have been creating with Claude 3.7 Sonnet, all with a single prompt:

1. Stunning 3D City (with Live NPCs) by Ozgur Ozer

2. Animated Weather Cards by @AGI_FromWalmart

3. Ball in a Rotating Hexagon by @t3dotgg

The result below was generated by the standard Claude 3.7. Interestingly, the result from the extended thinking mode was broken.

TL;DR: Developers have generally found Claude to be a great coding companion—the best among all the currently available models.

These models often give me the same feeling I had when using ChatGPT-4 for the first time, where I am equally impressed and a little unnerved by what it can do.—Ethan Mollick

Where the future of AI development is headed

With Claude 3.7 Sonnet and Claude Code, we're seeing a shift from AI as a simple autocomplete tool to an autonomous development assistant.

Here are some trends we can expect:

1. AI-native development environments will become standard, with models that understand code at a fundamental level.
2. Continuous integration with AI agents that automatically maintain and improve codebases.
3. Specialized AI assistants for different development roles will emerge (frontend, backend, DevOps).

So, give them a try today, we can't wait to see what you build!

You might also like

• Grok 3 Technical Review: Everything You Need to Know
• Claude 3.5 Sonnet vs OpenAI o1: A Comprehensive Comparison
• GPT-4o Mini vs. Claude 3.5 Sonnet