Pain points in cloud and local file transfer workflows

Reflecting on modern Data Engineering bottlenecks, I've discovered that blob storage can often become a major performance constraint — even though it isn’t the sole issue. In a recent experiment, I transferred data from cloud blob storage to local disk and processed it with an extensive Polars /DuckDB setup. The performance improvement was striking, revealing several key lessons about data infrastructure design: - While blob storage provides high durability and scalability, it typically incurs higher latency and lower throughput compared to local or directly attached disks. - Sequentially reading large files might work reasonably well on blob storage, but random access patterns or operations on small files tend to suffer more. - Modern tools like Polars and DuckDB are fine-tuned for in-memory and local disk operations, which means that using remote blob storage can exacerbate performance limitations. - Improving performance may require a comprehensive approach, including redesigning data partitioning, enhancing data locality, or adding caching layers to alleviate blob storage constraints. - Although local disks offer faster performance, they may not match the flexibility, durability, and ease of management provided by cloud blob storage.

20 years ago, I flew with a hard drive full of SAP data to a data center. Today, cloud tools do it faster. We've come a long way. But the hard part hasn’t changed. Back in the early 2000s, network speeds were so limited that we had to copy SAP system data to a physical disk, get on a plane, and hand-deliver it to the destination data center. Yes...commercial flight as a data pipeline. That sounds crazy now, but here’s the truth: Even with faster infrastructure, the hardest part of migrations hasn’t changed. You still have to plan every detail, script every step, and test everything until it breaks - and then fix it. Cloud may have accelerated the transfer speeds, but it hasn’t eliminated the complexity. Here’s what still matters — even in a cloud-native world: 1. Data volume still dictates downtime You can’t cheat physics. Whether it’s 10TB or 50TB, moving large databases still takes planning, staging, and validation. 2. Network is faster - but not always reliable Latency, throughput, and cloud ingress still cause delays. And in some regions, it’s still faster to ship a physical device. 3. Automation reduces effort, not responsibility We’ve gone from hand-crafted scripts to automated workflows - but someone still has to understand the logic underneath in case things go wrong. 4. Parallelization helps — if you can orchestrate it Moving 50,000 tables in parallel only works if you’ve segmented your data right. That’s still a technical and strategic challenge. 5. Real risk hides in the exceptions Most of the migration might run smoothly. But it’s the 5% - the slow disks, unexpected locks, or hidden job schedules - that blow your timeline. I’ve seen teams rely on shiny tools and forget the fundamentals. That’s how migrations break - not from lack of speed, but from lack of foresight. So yes, we’ve come a long way from flying with disks. But migrations still require discipline, orchestration, and real-world experience. Because when the system goes live, no one cares how fast the data moved - they care that everything works.

From Cloud = Comfort → to Local = Control   Experience with cloud collaboration, multiple people editing the same document. Real-time updates and everything synced instantly. And compared to my experiences in the early stages, meanwhile it works like magic, unless …   You travel. When you’re in a train with a patchy connection or stuck in an airport with overloaded Wi-Fi, that cloud magic disappears.   Your file won’t load. Edits won’t save. And worst of all, sometimes you’re locked out of your own work.   Yes, I know synchronization exists. Files can be synced locally. But here’s the catch: Even sync can fail. Conflicts, missing updates, files that don’t save properly and much more important for me, I’ve always been the type who prefers having control.   ✔️ I download important files. ✔️ I keep critical data on my device. ✔️ I make sure I can work offline if needed.   Some would call it old-fashioned. I call it being prepared.   Because while cloud tools are fantastic for collaboration, speed, and accessibility, they come with a trade-off: control.   For me, the best solution is a mix: Collaborate in the cloud, but keep a local copy. Sync when it works, but never rely on it blindly.   Because I’d rather have a local file and not need it, than need it and realize the sync failed. #cloud #data