Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Foundations of Big Data & Spark

• Overview of the Big Data ecosystem and Spark's role in modern data platforms
• Deep dive into Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG and execution planning
• Distinguishing between RDD and DataFrame APIs, and determining when to apply each
• Creating and configuring SparkSession, along with core application configuration principles

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats: CSV, JSON, Parquet and Delta
• Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins and aggregations
• Implementing advanced operations such as window functions, timestamp handling and nested data manipulation
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods in relation to Spark driver and executor processes — a conceptual view for cluster consumers rather than administrators
• Tracing the execution of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Distinguishing between controls managed via Spark session parameters versus those managed by platform teams through cluster provisioning — and why this boundary is critical for tuning
• Practical exercise: Examine a live Spark-on-Kubernetes job in a sandbox environment to identify driver and executor pods

Module 4: In-Depth Spark Session Configuration on Kubernetes

• Analyzing executor count versus size: balancing memory and core trade-offs and logical reasoning for selection
• Managing driver and executor memory allocation, overhead, and their translation into pod resource definitions
• Dynamic allocation: behaviour on Kubernetes, scenarios for resource savings and limitations
• Practical exercise: Execute a job under varying executor and core configurations to benchmark runtime and resource consumption

Module 5: Scaling Dynamics & Cost Optimisation

• Evaluating how node addition or removal impacts job behaviour, completion time and resource usage
• Comparing the performance and cost implications of using many small executors versus fewer large ones
• Optimising shuffle behaviour and partition tuning, including estimating the cost impact of specific configurations
• Practical exercise: Expand the sandbox cluster from five to ten nodes to observe effects on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Analyzing the small-files problem: why data split across many 1–5 MB Parquet files degrades performance and distorts partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during both read and write operations
• Optimising Parquet writes to prevent downstream small-files issues
• Practical exercise: Load a dataset composed of numerous small Parquet partitions, apply varied repartitioning strategies and assess performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes and symptoms of out-of-memory errors in Pandas
• Implementing memory-efficient patterns for conversion between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Utilising chunked processing and dtype optimisation in resource-constrained environments
• Practical exercise: Recreate a common Pandas out-of-memory scenario and resolve it through chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars against Pandas: analysing performance, lazy evaluation and memory behaviour
• Understanding Polars' role alongside PySpark and Pandas in modern cloud data stacks and migration strategies, including AWS environments
• Practical exercise: Refactor a Pandas-heavy transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning and memory-management principles across a realistic ETL pipeline
• Considering optimisation factors specific to machine-learning workloads running on shared clusters
• Implementing a structured tuning workflow to systematically diagnose cost and performance issues
• Practical exercise: Complete an end-to-end mini-project involving data loading, transformation and basic model training, while independently tuning Spark configurations