Serverless functions are a great way to start a data pipeline. There is nothing to patch, you pay per invocation, and scaling is someone else’s problem. Then the pipeline grows. Jobs start hitting execution time limits, large files have to be split into awkward chunks, and the monthly bill begins to look strange for a workload that runs on the same schedule every day.
Many data teams then assume the only next step is a distributed cluster. There is often a simpler option. Modern single machines have enough cores and memory to handle workloads that once required many nodes, and for a predictable batch pipeline, one big server can be both cheaper and easier to run. This guide explains how to tell whether your pipeline fits and how to move it safely.
Signs Your Pipeline Has Outgrown Serverless
Not every pipeline should move. These signals suggest it may be time to reconsider:
- Jobs are regularly designed around time or memory limits rather than around the data.
- Most of the logic is spent on splitting, coordinating and retrying small chunks.
- Workloads are steady, with little idle time across the day.
- Data transfer between functions and object storage has become a visible cost line.
- Debugging a failure means reading logs from hundreds of separate invocations.
If your jobs are genuinely spiky, with long idle periods and unpredictable bursts, serverless probably remains the right tool. A fixed server pays off when utilisation is high and steady. Before going further, calculate how many hours per day your pipeline actually does work. That number tells you more than any benchmark.
Profile the Jobs Before Choosing Anything
The most important step is understanding what your jobs really do with resources. Run each major job on a test machine with detailed monitoring and record the following.
|
Measurement |
Why it matters |
What it tells you |
|
Peak memory |
Joins and sorts fail outright if memory runs out |
Minimum RAM and whether spilling is acceptable |
|
CPU use across cores |
Shows how well the job parallelises |
Whether more cores will help at all |
|
Disk read and write rate |
Shuffles and temporary files are often the real bottleneck |
NVMe count and layout |
|
Network throughput |
Reading from object storage can limit the whole job |
Port speed and data location |
|
Runtime at half and full data size |
Reveals non-linear behaviour early |
How the job will scale next year |
Pay close attention to the third row. Teams commonly assume they have a CPU problem and then discover that the job spends most of its time waiting on temporary files. The Apache Spark hardware provisioning guide makes a similar point: it recommends several local disks per node, because Spark keeps data that does not fit in RAM and intermediate output between stages on them.
Rewrite for Parallelism on One Machine
Code written for functions often needs changes to take advantage of a large server. Fortunately, those changes usually simplify things.
Remove the chunking. Logic that split files for function limits can often disappear, leaving one job that reads the whole input with a parallel engine.
Use tools built for a single node. Engines such as DuckDB or Polars, or Spark in local mode, use all available cores with far less operational overhead than a cluster.
Control concurrency deliberately. Running everything at once causes memory pressure and disk contention. A small scheduler with resource pools will usually finish the whole batch faster than an unlimited free-for-all.
Store temporary data on local NVMe. Keep inputs and final outputs in durable storage, and use fast local disks for intermediate results. Treat those local disks as disposable.
Choosing hardware for this pattern is mostly about balance. You need a large memory capacity for joins, enough cores to use the parallelism your profile revealed, and several NVMe drives for temporary data. A single-socket dedicated server epyc configuration is one common choice for this shape of workload, since it offers many cores and memory channels without the complexity of a multi-socket system. Size it against your measured peak with room to grow, and check that the network port can keep up with reads from wherever your source data lives.
Plan for Failure, Because It Is Now Your Job
Serverless platforms hide most hardware failures. On a single machine, that responsibility moves to you. It is manageable, but it has to be designed from the start.
- Make every job idempotent. A job that runs twice must produce the same result. Write to a temporary location and swap atomically when finished.
- Checkpoint long jobs. A failure at the end of a six-hour run should not mean starting over.
- Keep state outside the server. Durable outputs, job metadata and configuration should survive the loss of the machine.
- Automate the rebuild. Keep the whole setup in code, and test building a replacement from scratch. If you cannot rebuild in a few hours, you have a pet, not a platform.
- Monitor the basics. Disk usage, memory pressure, job duration and data freshness catch most problems before business users do.
Also be honest about the limit of this design. One machine is one failure domain, so if your pipeline has strict recovery targets, you need a second server or a tested fallback to a cloud environment.
A Safe Migration Plan
Avoid switching everything at once. A staged approach lets you find surprises while the old system still works.
The result should be a pipeline that is easier to reason about, with one place to look at logs and one set of resource limits to manage. It will not suit every team, but for steady batch workloads, one big machine is often the simplest system that does the job.