This intensive three-day practical course is dedicated to constructing and refining high-efficiency data-processing workloads by leveraging PySpark, Pandas, and Polars within Kubernetes-based environments.
Learners will gain a hands-on understanding of Spark application execution on Kubernetes, focusing on how application-level configuration decisions impact performance, scalability, resource utilization, and overall cost. Key optimization topics include executor sizing, memory allocation, dynamic allocation, partitioning strategies, shuffle behavior, mitigating small-file issues, and optimized Parquet processing.
The curriculum also tackles common challenges associated with Pandas, such as memory constraints and out-of-memory errors, while introducing Polars as a high-performance alternative for specific data-processing tasks. Through practical exercises, participants will learn to diagnose performance and memory bottlenecks, evaluate various configuration strategies, and implement optimization techniques in realistic ETL and machine learning contexts.
Throughout the course, the primary focus remains on practical decision-making: identifying performance bottlenecks, selecting the most suitable tools, configuring Spark effectively, and balancing performance against infrastructure resource consumption and costs.
Read more...