Overview

What is Apache NiFi?

Apache NiFi is a powerful dataflow management tool for any application that requires automated data movement between systems. It is an easy to use, powerful, and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface.

What is Apache NiFi used for?

Apache NiFi is used for automating and managing the flow of information be it between databases, applications, or devices. It is a powerful and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface.

Hands-on Lab

In this hands-on lab, we'll provision an EC2 instance on AWS and install Apache NiFi on it. Additionally, we will set it up correctly and expose port 80.

We will create an instance using the AWS UI by Web Console.

We need to select Amazon Machine Image from the list. I prefer Amazon Linux 2 AMI. It will be enough for our demo.

We will select an instance type. I prefered t2.micro.

Pass others till review page.

We need to create a new key pair or use exist one. Don't forget to download the Key Pair.

We have access to our instance now.

ssh -i "London-KP.pem" ec2-user@ec2-XX-XX-XX-XX.eu-west-2.compute.amazonaws.com

We need to install docker on our instance

Update the installed packages and package cache on your instance.

sudo yum update -y

Install the most recent Docker Engine package.

sudo amazon-linux-extras install docker

Start the Docker service.

sudo service docker start

Add the ec2-user to the docker group so you can execute Docker commands without using sudo.

sudo usermod -a -G docker ec2-user

Log out and log back in again to pick up the new docker group permissions. Verify that the ec2-user can run Docker commands without sudo.

docker info

The standalone unsecured instance runs a NiFi instance by itself. This is the minimum configuration needed to run a NiFi instance.

docker run --name nifi \
  -p 18080:8080 \
  -d \
  apache/nifi:latest

This will provide a running instance, exposing the instance UI to the host system on port 18080, viewable at http://localhost:18080/nifi.

We need to change security rules to expose NiFi port 80.

That's all. We're able to see the dashboard. 🥳