Skip to Content
OverviewKey Concepts

Key Concepts

Reasoning models

Reasoning models like step-3.5-flash are designed for tasks requiring deep logical analysis, multi-step problem solving, and long-context reasoning.

Reasoning models excel at:

  • Complex Logic: Breaking down intricate problems into manageable steps.
  • Mathematics & Coding: Solving advanced equations and debugging software.
  • Long-context Agents: Maintaining stability and reasoning over massive datasets.

See Reasoning Model for model details.

Audio models

Audio models such as step-tts-2 convert text into natural speech and support voice cloning.

Audio models can be used for tasks including, but not limited to:

  • Voice assistants: customer service and smart speakers.
  • Audiobooks and podcasts: text narration with consistent voice.
  • Games and NPCs: character voices at scale.
  • Media production: quick voiceover drafts.

See Audio Models for model details and Generate audio for the API.

Context length

Context length is the amount of input text a model considers when generating or predicting. It limits how much information the model processes in a single request.

Why it matters

  • Quality: Context length governs how much the model can remember and use, affecting understanding and generation.
  • Performance: Larger contexts can improve accuracy but also increase compute cost.
  • Cost: Longer contexts may help in certain scenarios but raise usage costs, so balance quality and spend.

Where it applies

  • Chat systems: affects coherence and context retention across turns.
  • Creative writing: longer contexts can produce more coherent, logical narratives.
  • Research papers: helps the model digest background, data, and detailed discussion.
  • Novels and literature: captures plot progression and character relationships.

Token

A token is the basic unit of text a model processes. It can be a character, word, phrase, or sentence depending on the tokenizer and training data. In Chinese, tokenization is especially important because words are not separated by spaces.

Token length

  • Chinese characters vs. tokens: Roughly 1 token equals about 1.5–2 Chinese characters, though actual counts vary by content.

Context limits

  • Maximum context: The combined input (prompt) and output must stay within the model’s context window.
  • Why the limit matters: It keeps processing efficient and avoids errors from overly long text.

Practical considerations

  • Plan text length: Fit your text within the maximum context so the model can process everything.
  • Optimize tokens: Remove unnecessary tokens or reorganize text to stay within the limit.

Rate limits

Rate limits protect service stability and fairness by capping how many requests a user can make within a given time. Three main measures:

RPM (requests per minute): Number of requests allowed per minute. If RPM is 20, you can make at most 20 requests in any rolling one-minute window.

TPM (tokens per minute): Number of tokens you can send per minute across requests and responses. Many short requests may hit RPM before TPM.

Concurrency (simultaneous requests): Number of in-flight requests. If the limit is 20, only 20 concurrent requests are allowed; new ones are rejected until earlier ones finish.

When rate limits trigger: Any one of the above can hit first. For example:

If your limits are RPM=20 and TPM=200K, and you send 20 requests to ChatCompletions with 100 tokens each,
TPM is still under 200K, but hitting 20 requests triggers the RPM limit.
Last updated on
Quickstart