Workshop Program
Flink Ecosystem - Building Pipelines for Real-Time Data Lakes
This intensive, 2-day face-to-face program is designed for Apache Flink users with 1-2 years of experience who want to take their skills to the intermediate level. We'll delve into advanced Flink concepts and techniques, empowering you to build and deploy highly scalable and efficient real-time data processing pipelines. Leveraging Ververica technology, you'll gain a deeper understanding of Flink and explore best practices for production deployments.
Target Audience
Intermediate to Advanced; 2-4 years of experience working with data lakes, stream processing, or Apache Flink
Why this workshop?
Flink helps unlock the full potential of data lakes by providing the necessary stream and batch processing capabilities to transform raw data into actionable insights. This course will:
- Focus on the relationship between data lakes and how they are rooted in Flink's ability to process and analyze large-scale data that often resides in data lakes.
- Highlight Ververica/Alibaba expertise with the technology behind the Flink ecosystem.
What will this workshop cover?
- Data Lakes as a Source: These lakes are often built on distributed storage systems like Hadoop or cloud-based solutions (e.g., AWS S3, Azure Data Lake, etc.).
- Flink for Real-time Processing: In the context of a data lake, Flink processes and analyzes the data stored in these lakes in real-time or in batch modes.
- Data Integration: Flink can integrate with data lakes to perform complex transformations, aggregations, and analytics on data as it flows in. This enables real-time analytics, insights, and decision-making based on data that is continuously ingested into the lake.
- Long-Term Storage and Retrieval: While Flink processes and analyzes data, the results are often stored back in the data lake, making it a crucial part of the ecosystem for both storing processed data and enabling further downstream analytics.
- Unified Data Processing: Flink provides a unified framework for both stream and batch processing, which is important for managing and processing data at scale in data lakes. It helps provide consistency in how data is processed and ensures that data workflows are optimized.
How the Flink ecosystem technologies fit into the workshop?
We will use Flink ecosystem technology along with real-world use cases and hands-on exercises. These technologies (Flink CDC, Paimon, Fluss, and Streamhouse) are integral to different parts of the Flink ecosystem when dealing with large-scale data in data lakes.
- Flink CDC allows businesses to keep data in the lake up-to-date with changes from operational databases, enabling near real-time updates for continuous analytics.
- Paimon helps enhance the data lake architecture by turning it into a lakehouse, providing both efficient storage and scalable analytics that can work in both batch and streaming modes. It connects the best features of data lakes (large-scale storage) and data warehouses (structured analytics). By integrating seamlessly with Flink, it can process real-time and batch data and store the results back in the lake, supporting more structured and queryable data formats for analytics.
- Fluss supports low-latency stream processing, ideal for real-time analytics on data that is being fed into the lake or processed within the lake. It enables high-performance event processing, making it easier to handle data that is continuously ingested and stored in a lake.
- Streamhouse simplifies stream processing by providing a serverless, scalable framework that can handle real-time data processing and seamlessly integrate with data lakes. It allows businesses to process data from various sources, including data lakes, without worrying about managing infrastructure, and directly feeds processed data back into the lake for further use or analysis.
