Apache Flink Architecture

Flink Architecture

Execution of streaming applications requires efficient allocation and management of resources. In this section, we explore Flink’s architecture and how its components interact to execute applications.

Flink operates with two core processes:

• JobManager: The master process responsible for managing job execution.
• TaskManager: The worker process executing tasks assigned by the JobManager.

The client submits a job but is not part of the runtime. It prepares and sends a dataflow to the JobManager. TaskManagers connect to JobManagers to announce availability and are assigned work accordingly.

Flink Execution Architecture

JobManager

The JobManager serves as the master process and has three key components:

• ResourceManager:
  • Handles resource allocation and provisioning.
  • Manages task slots, which are the unit of resource scheduling in a Flink cluster.
• Dispatcher:
  • Provides an interface for submitting Flink applications.
  • Initiates a new JobMaster for each submitted job.
  • Includes a Web UI for job monitoring.
• JobMaster:
  • Supervises task execution within a single job.
  • Coordinates execution within the Flink application.
  • Ensures fault tolerance with High Availability (HA) support.

There is always at least one JobManager. To prevent a single point of failure, JobManager High Availability (HA) is supported.

TaskManager

TaskManagers are responsible for executing tasks and managing dataflow assignments.

• Also referred to as worker nodes.
• Each TaskManager is a JVM process and can run multiple subtasks.
• At least one TaskManager is required.
• Task Slots:
  • The smallest unit of resource scheduling in a TaskManager.
  • Determine the number of concurrent tasks a TaskManager can process.

Each task slot uses a fixed portion of a TaskManager’s resources, ensuring efficient parallel execution.

Next Steps

In the next section, we will explore the components of the Flink ecosystem in detail.

• Tweet
• Share
• Share
• Share
• Share