🚀 7 AI Inference Cost-Performance Optimization Metrics to Master in 2026

Video: AI Inference: The Secret to AI’s Superpowers.

Stop guessing and start optimizing: the secret to slashing your AI bill isn’t buying cheaper hardware, it’s mastering the Time to First Token (TTFT) and tokens-per-second trade-offs that define true efficiency. We’ve seen teams burn six figures on underutilized GPUs because they ignored these critical AI inference cost-performance optimization metrics, only to realize later that a simple shift to continuous batching could have saved them 60%.

The math is brutal but simple: inference now consumes up to 80% of total AI spend, yet most engineers still optimize for training accuracy instead of serving economics. Imagine running a high-end restaurant where you pay for a full kitchen staff even when only one customer walks in; that’s exactly what happens when your GPU sits idle waiting for the next request.

We once watched a startup scale to 10k users, only to hit a wall where their cloud bill doubled while their response times tripled. The culprit? They were optimizing for raw throughput while ignoring latency distributions, causing their P9 latency to spike and users to churn.

The solution wasn’t more money; it was a strategic pivot to quantization and smart caching that dropped their cost per token by half without sacrificing quality.

Key Takeaways

The “Impossible Trinity” is real: You must choose between Speed, Quality, and Cost; you cannot maximize all three simultaneously.
Latency is king: Optimizing Time to First Token (TTFT) is often more critical for user retention than raw throughput.
Quantization pays off: Moving from FP16 to FP8 or INT4 can reduce costs by 50–75% with negligible accuracy loss.
Concurrency has a sweet spot: Pushing too many simultaneous requests increases cost per token; find your model’s optimal batch size.
Monitor the tail: Always track P95 and P9 latency, not just averages, to prevent user experience degradation.

⚡️ Quick Tips and Facts
🕰️ A Brief History of AI Inference Economics: From Lab Bench to Cloud Bill
🧠 Decoding the Metrics: Latency, Throughput, and Token Economics
💸 The Real Cost of Thinking: Estimating Inference Spend
📉 7 Critical Strategies to Slash AI Inference Costs Without Killing Performance
🚀 5 Proven Techniques to Boost Inference Throughput at Scale
⚖️ The Trilemma: Balancing Speed, Quality, and Wallet Impact
🛠️ Hardware Showdown: GPU vs. TPU vs. NPU for Cost-Effective Inference
🧪 Model Optimization Deep Dive: Quantization, Pruning, and Distillation
📊 Benchmarking Real-World Scenarios: Concurrency, Batch Sizes, and Load Patterns
🚧 Common Pitfalls: Why Your Optimization Efforts Might Be Failing
🔮 Future-Proofing: Emerging Trends in Efficient AI Inference
🏁 Conclusion
🔗 Recommended Links
❓ FAQ
📚 Reference Links

⚡️ Quick Tips and Facts

Before we dive into the nitty-gritty of GPU bills and token economics, let’s hit the pause button on the hype. If you’re running an AI startup or managing an enterprise AI budget, here are the non-negotiable truths that separate the pros from the people burning cash:

The “Impossible Trinity” is Real: You cannot simultaneously maximize Model Quality, Inference Speed, and Low Cost. You must pick two. Trying to get all three is like trying to find a square circle in a flat universe. 🔄
Inference is the New Training: Remember when everyone was obsessed with training costs? Those days are gone. Inference now consumes 65–80% of total AI compute budgets. If you aren’t optimizing inference, you’re bleeding money. 🩸
Concurrency is a Double-Edged Sword: Cranking up concurrency (number of simultaneous requests) lowers the cost per token up to a point. Push it too far, and your Time to First Token (TTFT) spikes, users get angry, and your throughput tanks. There is a “sweet spot,” usually between 16 and 48 concurrent requests for most 30B+ models.
Tokenization Matters More Than You Think: A model might be “fast,” but if its tokenizer is inefficient (like some older Mistral variants), it could be churning through 2x the input tokens for the same task, doubling your bill. Always check the input/output token ratio.
Self-Hosting vs. API: If you’re processing more than 50 million tokens a month, self-hosting (or using spot instances) usually beats managed APIs like OpenAI or Anthropic. Below that threshold? The convenience of an API might still save you time, even if the per-token cost is higher.

For a deeper dive into how these metrics translate to real-world business value, check out our guide on AI Benchmarks.

🕰️ A Brief History of AI Inference Economics: From Lab Bench to Cloud Bill

It wasn’t long ago that “AI” was a buzzword reserved for academic papers and science fiction. Back then, the only metric that mattered was accuracy. If a model could beat a human at a game or diagnose a disease, the cost of running it on a cluster of GPUs was irrelevant. It was a “cool science project” expense.

Then came the Generative AI explosion. Suddenly, everyone wanted an AI assistant. The “cool science project” became a “production service.”

The Great Inversion

We witnessed a massive economic shift known as the “Great Inversion.” In the early days of Deep Learning, training was the expensive part. Once a model was trained, you could deploy it cheaply. But with Large Language Models (LLMs), the math flipped.

Training: A one-time, massive capital expenditure (CAPEX).
Inference: A recurring, compounding operational expenditure (OPEX).

As noted in recent industry analyses, once a model ships to production, serving costs accumulate every hour, every day, indefinitely. A 70B parameter model serving just 1,0 daily active users can rack up hundreds of dollars a day in compute costs alone.

From Static to Dynamic

In the early days of deployment, engineers used static batching. You’d set a batch size of 32 and run it forever. Simple, but inefficient. If only 5 users were active, you were paying for 27 empty slots.
Today, the industry has moved toward dynamic batching and continuous batching (pionered by frameworks like vLLM), where the system dynamically groups requests to keep the GPU humming at 80%+ utilization.

This evolution mirrors the shift from mainframe computing to cloud elasticity. We’ve moved from “buying the biggest GPU and hoping it’s enough” to “orchestrating a fleet of heterogeneous hardware to match demand.”

🧠 Decoding the Metrics: Latency, Throughput, and Token Economics

Video: Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou.

You can’t optimize what you can’t measure. But the world of AI metrics is full of jargon that sounds like it was invented by a robot trying to confuse a human. Let’s break down the Big Three that actually matter for your wallet and your users.

1. Latency: The User Experience Killer

Latency isn’t just one number. It’s a distribution.

Time to First Token (TTFT): This is the time from the user hitting “Enter” to seeing the first word appear. For a chatbot, < 1 second is the gold standard. Anything over 2 seconds feels like a broken connection.
Time Per Output Token (TPOT): How fast is the model “thinking” and generating subsequent words?
End-to-End Latency: The total time to generate a full response.

Pro Tip: Don’t just look at the average (mean) latency. Look at the P95 and P9. If your average is 20ms but your P9 is 5 seconds, 1% of your users are having a terrible experience, and they will leave.

2. Throughput: The Volume Game

Throughput is measured in tokens per second (tok/s).

High Throughput is critical for batch jobs (e.g., summarizing 10,0 documents overnight).
Low Latency is critical for real-time chat.
The Conflict: You often have to trade one for the other. Increasing batch size boosts throughput but increases latency.

3. The Token Economy

The fundamental unit of cost is the token. But not all tokens are created equal.

Input Tokens: The prompt you send.
Output Tokens: The response the model generates.
Cost Per Million Tokens (CPM): This is your North Star metric. It normalizes the cost across different hardware and models.

Metric	Why It Matters	Target Benchmark (Interactive)	Target Benchmark (Batch)
TTFT	User patience	< 50ms	< 5s
TPOT	Reading speed	> 20 tok/s	> 50 tok/s
GPU Utilization	Cost efficiency	40–60%	70–90%
Cost per 1k Tokens	Budget planning	Varies by model	Varies by model

💸 The Real Cost of Thinking: Estimating Inference Spend

Video: LLM Inference Engines: Optimizing Performance.

So, you have a model. You have a server. How much will it cost to run? It’s not as simple as “price per hour.” The real cost is a function of hardware depreciation, power, cooling, and utilization.

The Self-Hosted Equation

If you buy your own hardware (say, an NVIDIA A80 or H10), the hourly cost isn’t just the electricity bill. It’s a formula:
Hourly Cost = (Depreciation + Power + Maintenance) / Utilization

Depreciation: If you buy a $120k GPU and depreciate it over 3 years, that’s roughly $0.64/hour.
Power: A high-end GPU draws 40W–70W. With PUE (Power Usage Effectiveness) of 1.5 and electricity at $0.10/kWh, you’re looking at another $0.08–$0.12/hour.
Maintenance: Data center fees, network, and support usually add another 3–5%.

The Result: A self-hosted A80 might cost you $0.79/hour to run. But if your GPU is idle 50% of the time, your effective cost per inference doubles.

The Cloud Reality

Cloud providers (AWS, Azure, GCP) charge a premium for convenience.

AWS P4de (8x H10): ~$5.08/hour.
Alibaba Cloud: ~$4.80/hour.
Spot Instances: Can drop to $0.32/hour for L40S, but risk interruption.

The Break-Even Point:
If you are running a steady, high-load workload, self-hosting or using Spot Instances on platforms like RunPod or Spheron can save you 50–70% compared to on-demand cloud instances. However, if your traffic is spiky and unpredictable, the flexibility of on-demand cloud might save you from over-provisioning.

Did you know? Some models, like Mistral-Small, can have higher effective costs than larger models because their tokenizer is inefficient, requiring more input tokens to express the same idea. Always check the token efficiency before committing to a model.

📉 7 Critical Strategies to Slash AI Inference Costs Without Killing Performance

Video: The secret to cost-efficient AI inference.

You want to cut costs, but you don’t want your users to think the AI is stupid. Here are 7 battle-tested strategies we’ve seen work in production environments.

1. Embrace Quantization (FP8, INT4)

This is the low-hanging fruit. Quantization reduces the precision of the model weights from 16-bit (FP16) to 8-bit (FP8) or even 4-bit (INT4).

Impact: Reduces VRAM usage by 50–75% and can double throughput.
Trade-off: Usually less than 2% loss in accuracy for most tasks.
Tools: TensorRT-LLM, vLLM, GGUF.

2. Continuous Batching

Old-school batching waits for the whole batch to finish before starting the next. Continuous batching (or iterative batching) allows the system to start processing new requests as soon as a slot opens up in the current batch.

Impact: Increases GPU utilization from ~30% to 80%+.
Result: 3–4x throughput improvement.

3. Speculative Decoding

Use a tiny, fast “draft” model to guess the next few tokens, then have the large model verify them in parallel.

Impact: Can speed up generation by 2–4x for output-heavy tasks.
Best For: Chatbots and code generation where the draft model is often correct.

4. Smart Caching

If a user asks the same question twice, why compute it again? Implement semantic caching or exact match caching for frequent queries.

Impact: Even a 10–20% cache hit rate can slash your compute bill significantly.
Use Case: Customer support bots, search engines, and recommendation systems.

5. Model Distillation

Train a smaller “student” model to mimic a larger “teacher” model.

Impact: A 7B model distilled from a 70B model can be 10x faster and 10x cheaper while retaining 95% of the performance.
Example: Llama-3-8B distilled from Llama-3-70B.

6. Right-Sizing Hardware

Stop defaulting to H10s for everything.

< 10B Models: Run on L40S or A10G.
70B+ Models: You need H10 or A10 with high VRAM.
Strategy: Use Multi-Instance GPU (MIG) to slice a single powerful GPU into smaller instances for smaller models.

7. Dynamic Scaling

Don’t keep 10 GPUs running at 2 AM when traffic is zero. Use Kubernetes or serverless platforms to scale down to zero (or near zero) during off-peak hours.

Impact: Eliminates idle costs entirely.

🚀 5 Proven Techniques to Boost Inference Throughput at Scale

Video: Inference Optimization: Making AI Faster & Cheaper (Latency, Throughput & GPUs).

When you need to serve millions of requests, throughput is king. Here is how to squeeze every drop of performance out of your hardware.

1. Optimize the KV Cache

The Key-Value (KV) Cache stores the context of the conversation. As the conversation grows, the cache grows, eating up VRAM.

Technique: Use PagedAttention (used in vLLM) to manage memory like an operating system, preventing fragmentation.
Benefit: Supports longer contexts and higher concurrency without running out of memory.

2. Tensor Parallelism vs. Pipeline Parallelism

Tensor Parallelism: Splits a single layer of the model across multiple GPUs. Best for low latency on a single node.
Pipeline Parallelism: Splits different layers across GPUs. Best for massive models that don’t fit one card.
Hybrid Approach: Most production systems use a mix to balance latency and throughput.

3. Flash Attention

A specialized algorithm that optimizes the attention mechanism, reducing memory access and increasing speed.

Impact: Can provide 2–3x speedup for attention-heavy models.
Requirement: Must be supported by your hardware (H10, A10, etc.).

4. Request Prioritization

Not all requests are equal. A VIP user’s chat should get priority over a background batch job.

Strategy: Implement priority queues in your inference engine.
Result: Better SLA for critical users without needing more hardware.

5. Heterogeneous Inference

Route the prefill phase (processing the prompt) to a fast, high-memory GPU, and the decode phase (generating tokens) to a cheaper, high-throughput GPU.

Benefit: Optimizes hardware usage for each specific phase of the inference process.

⚖️ The Trilemma: Balancing Speed, Quality, and Cost

Video: Why AI Inference Costs Money.

We mentioned the “Impossible Trinity” earlier. Let’s explore it deeper. In the real world, you are constantly making trade-offs.

The Trade-Off Matrix

Goal	What You Sacrifice	The Risk
Maximize Speed	Quality & Cost	Hallucinations increase; you might need more GPUs to handle the load.
Maximize Quality	Speed & Cost	Slower responses; massive compute bills; users get bored.
Minimize Cost	Speed & Quality	High latency; poor user experience; lower accuracy.

Finding the “Sweet Spot”

The goal isn’t to maximize one variable; it’s to find the optimal cost-effectiveness zone.

For a Chatbot: Speed (Latency) is paramount. You might accept a slightly smaller model (lower quality) to get sub-50ms TTFT.
For Data Analysis: Quality is paramount. You can afford a 5-second delay to get a 9% accurate answer.
For Content Moderation: Cost is paramount. You can use a tiny, fast model to filter 90% of content, and only send the tricky 10% to a large, expensive model.

Insight: There is no “best model.” There is only the best model for your specific concurrency and quality requirements. A model that is “best” at 1 concurrent request might be the worst at 10 concurrent requests.

🛠️ Hardware Showdown: GPU vs. TPU vs. NPU for Cost-Effective Inference

Video: AI Inference Cost: How to Slash It (with Specialized CPU Acceleration).

Choosing the right silicon is like picking the right engine for a race car. It depends on the track.

NVIDIA GPU (The Gold Standard)

Pros: Massive ecosystem, supports almost all models, excellent software stack (CUDA, TensorRT).
Cons: Expensive, power-hungry.
Best For: General-purpose LMs, complex models, rapid protyping.
Top Picks: H10 (Performance), A10 (Value), L40S (Cost-effective).

Google TPU (The Specialized Beast)

Pros: Incredible throughput for specific models (like JAX-based LMs), often cheaper per token at scale.
Cons: Less flexible, harder to set up, limited model support outside the Google ecosystem.
Best For: Large-scale batch processing, Google-native models.

Apple NPU / AMD MI30 (The Challengers)

Apple NPU: Great for edge inference (MacBooks, iPhones). Low power, decent speed for small models.
AMD MI30: A strong competitor to NVIDIA, offering high memory bandwidth at a lower price point.
Best For: Edge devices, cost-sensitive cloud deployments.

Comparison Table: Cost-Performance per Token (Estimates)

Hardware	Relative Cost	Throughput (Tokens/s)	Best Use Case
NVIDIA H10	High	Very High	High-performance LMs
NVIDIA A10	Medium-High	High	Balanced workloads
NVIDIA L40S	Medium	Medium	Cost-effective serving
Google TPU v5	Low (at scale)	Very High	Batch processing
Apple M3 Max	Low (Edge)	Medium	Local/Edge inference

🧪 Model Optimization Deep Dive: Quantization, Pruning, and Distillation

Video: Sonic Inference Pod: A data centre built for AI inference.

Let’s get technical. How do we actually make these models smaller and faster?

Quantization: The Precision Cut

FP16 (16-bit): The standard. High precision, high memory.
FP8 (8-bit): The new standard for inference. 2x speed, 2x memory savings, negligible quality loss.
INT4 (4-bit): The extreme. Fits 70B models on a single consumer GPU. Great for edge, but can degrade quality on complex reasoning tasks.
Technique: AWQ (Activation-aware Weight Quantization) and GPTQ are popular methods.

Pruning: Cutting the Fat

Unstructured Pruning: Removes individual weights. Hard to accelerate on hardware.
Structured Pruning: Removes entire neurons or layers. Easier to accelerate, but harder to do without losing quality.
Result: Can reduce model size by 30–50% with minimal accuracy drop.

Distillation: The Student-Teacher Model

Process: A large “teacher” model generates data, which is used to train a smaller “student” model.
Result: A 7B model that behaves like a 70B model.
Example: Phi-3 by Microsoft is a prime example of a small model distilled from larger, more complex models.

📊 Benchmarking Real-World Scenarios: Concurrency, Batch Sizes, and Load Patterns

Video: What Is LLM Inference Optimization? (Why Inference Costs More Than Training).

Theory is great, but what happens when 1,0 users hit your API at once?

The Concurrency Curve

Imagine a graph where the X-axis is Concurrency and the Y-axis is Cost per Token.

Low Concurrency (1–8): High cost per token. The GPU is underutilized.
Optimal Concurrency (16–48): The curve bottoms out. You get the best cost-performance ratio.
High Concurrency (64+): The curve spikes. TTFT increases, throughput drops, and cost per token rises again due to context switching and memory pressure.

Real-World Data: WiNGPT-3.5 vs. Others

Based on recent benchmarks (WiNEval-3.0), here is how different models perform under optimal concurrency:

Model	Optimal Concurrency	Total Time (s)	Cost ($)	Quality Score
WiNGPT-3.5	48	74.1	$0.34	76.2
Qwen3-30B	64	616.7	$0.27	6.9
Mistral-Small	64	59.2	$0.25	59.8
gpt-oss-20b	64	249.17	$0.10	56.4

Note: WiNGPT-3.0 is an outlier with high cost due to “Chain of Thought” reasoning, making it expensive for simple tasks.

Load Patterns Matter

Spike Traffic: Use auto-scaling and spot instances.
Steady Traffic: Use reserved instances or self-hosted hardware.
Burst Traffic: Use a hybrid approach (base load on reserved, spikes on spot).

🚧 Common Pitfalls: Why Your Optimization Efforts Might Be Failing

Video: SBTB 2023: Chris Matteson, Cost and Performance Optimization of LLM Inferencing.

You’ve quantized, you’ve batched, you’ve scaled. Why is the bill still high?

1. Ignoring the “Tail”

You optimized the average latency, but ignored the P9. If 1% of requests take 10 seconds, your users are unhappy, and you might be over-provisioning to fix it.

Fix: Monitor distributions, not just averages.

2. Over-Quantizing

You went straight to INT4 and now the model is hallucinating.

Fix: Start with FP8. Only go lower if you’ve tested and verified the quality is acceptable.

3. Static Batching in a Dynamic World

You set a batch size of 32 and never changed it.

Fix: Implement continuous batching or dynamic batching based on queue depth.

4. Neglecting the Network

You optimized the GPU, but the data transfer between CPU and GPU is the bottleneck.

Fix: Use NVLink and high-bandwidth memory (HBM) to ensure the GPU isn’t waiting for data.

5. Forgetting the “Hidden” Costs

You calculated the GPU cost, but forgot egress fees, networking, and storage.

Fix: Use a FinOps tool to track total cost of ownership (TCO), not just compute.

🔮 Future-Proofing: Emerging Trends in Efficient AI Inference

Video: Inference Optimization | AI Engineering #9.

The landscape is changing fast. What’s next?

1. Specialized Inference Chips

Companies like Groq are building LPUs (Language Processing Units) that promise 35x more throughput per watt than GPUs by using on-chip SRAM instead of HBM.

Impact: Could revolutionize real-time inference for chatbots.

2. Mixture of Experts (MoE)

Models like Mixtral use a “sparse” architecture where only a subset of parameters are active for each token.

Impact: High quality with lower inference cost.

3. Edge AI

Running models directly on devices (phones, laptops) to reduce latency and privacy concerns.

Impact: Shifts the cost from the cloud to the device.

4. AI-Native Operating Systems

New OS layers designed specifically to manage AI workloads, optimizing scheduling and resource allocation automatically.

Question: Will the future of AI be in the cloud, or on your pocket? The answer might be a hybrid of both.

🏁 Conclusion

🔗 Recommended Links

Ready to take action? Here are the tools and resources we recommend to get started:

👉 CHECK PRICE on: NVIDIA H10 GPUs on RunPod | NVIDIA H10 on Spheron | NVIDIA Official Store
👉 CHECK PRICE on: L40S Spot Instances on DigitalOcean | L40S on Paperspace
👉 Shop AI Books on: Amazon: “Deep Learning” by Ian Goodfellow | Amazon: “Designing Machine Learning Systems” by Chip Huyen

❓ FAQ

How can AI inference metrics drive competitive advantage industry?

AI inference metrics like latency and cost per token directly impact user experience and profit margins. Companies that optimize these metrics can offer faster, cheaper services than competitors, leading to higher customer retention and lower operational costs. For example, a chatbot with a 20ms response time will retain users better than one with a 2-second delay.

What strategies improve cost-performance balance in AI deployment?

The most effective strategies include quantization (reducing model precision), continuous batching (maximizing GPU utilization), and dynamic scaling (matching resources to demand). Combining these techniques can reduce costs by 50–70% while maintaining or even improving performance.

How does model compression impact AI inference cost and performance?

Model compression techniques like quantization and pruning reduce the size of the model, leading to lower memory usage and faster inference. While there is a slight risk of quality loss (usually <2%), the trade-off is often worth it for the significant cost savings and speed improvements.

Which tools help measure AI inference efficiency and cost-effectiveness?

Tools like vLLM, TensorRT-LLM, and Prometheus/Grafana dashboards are essential for monitoring GPU utilization, latency distributions, and cost per token. Cloud providers also offer built-in monitoring tools, but third-party solutions often provide deeper insights.

What role does latency play in AI inference cost-performance optimization?

Latency is a critical user experience metric. High latency can lead to user churn. Optimizing for low latency often requires more powerful (and expensive) hardware or advanced techniques like speculative decoding. Balancing latency with cost is a key challenge in AI deployment.

What are the key metrics for optimizing AI inference cost-performance?

The key metrics are Time to First Token (TTFT), Throughput (tokens/s), GPU Utilization, and Cost per Million Tokens (CPM). Tracking these metrics helps identify bottlenecks and optimize resource allocation.

How do you calculate the cost per inference for large language models?

Cost per inference = (Hourly GPU Cost × Time per Inference) / Number of Inferences. Alternatively, use Cost per Million Tokens (CPM) = (Total Cost / Total Tokens) × 1,0,0. This normalizes costs across different models and hardware.

What are the key performance metrics for optimizing AI inference latency?

Key metrics include TTFT, Time Per Output Token (TPOT), and End-to-End Latency. Monitoring the P95 and P9 distributions is crucial to ensure consistent performance for all users.

How can businesses reduce AI inference costs without sacrificing accuracy?

Businesses can reduce costs by using quantization (FP8/INT4), model distillation (smaller models), and caching (reusing results). These techniques can significantly lower costs with minimal impact on accuracy.

What is the relationship between throughput and inference cost in AI models?

Throughput and cost are inversely related up to a point. Increasing throughput (via batching) reduces the cost per token. However, beyond a certain concurrency level, throughput drops and cost per token rises due to resource contention.

Which metrics best measure the efficiency of GPU utilization for AI inference?

GPU Utilization Rate (active compute vs. idle capacity) and Memory Bandwidth Utilization are the best metrics. High utilization (60–80%) indicates efficient resource use.

How does model quantization impact AI inference cost and performance?

Quantization reduces model size and memory usage, leading to faster inference and lower hardware requirements. It can reduce costs by 30–50% with minimal accuracy loss.

What tools are available for monitoring real-time AI inference costs?

Tools like Datadog, Grafana, and cloud-native monitoring (AWS CloudWatch, Google Cloud Monitoring) can track real-time costs. Specialized AI platforms like vLLM also provide built-in metrics for cost and performance.

📚 Reference Links

AI Inference at Scale: Cost Breakdown and Optimization Best Practices: GMICloud Blog
NVIDIA H10 Specifications: NVIDIA Official
vLLM Documentation: vLLM GitHub
Spheron Network: Spheron Blog
ArXiv Paper: AI Inference Cost-Performance Optimization Metrics: ArXiv 2510.26136v1
TensorRT-LLM: NVIDIA TensorRT-LLM
RunPod GPU Cloud: RunPod
DigitalOcean GPU Instances: DigitalOcean
Paperspace: Paperspace