aws Verified current stable Not installed? Cloud Infrastructure

Aws / Create Kafka Cluster

Create Kafka Cluster

Creates a Kafka cluster with specified configurations in AWS.

Terminal

aws kafka create-cluster --cluster-name <cluster_name> --broker-node-group-info instanceType=<instance_type>,clientSubnets=<subnet_id1 subnet_id2 ...> --kafka-version <version> --number-of-broker-nodes <number>

aws kafka create-cluster --cluster-name <cluster_name> --broker-node-group-info instanceType=<instance_type>,clientSubnets=<subnet_id1 subnet_id2 ...> --kafka-version <version> --number-of-broker-nodes <number>

#!/bin/bash

# Create Kafka Cluster
aws kafka create-cluster --cluster-name {{cluster_name}} --broker-node-group-info instanceType={{instance_type}},clientSubnets={{subnet_id1 subnet_id2 ...}} --kafka-version {{version}} --number-of-broker-nodes {{number}}

import subprocess

# Create Kafka Cluster
# Make sure to replace <placeholders> with actual values
def run_command():
    cmd = [
        "aws",
        "kafka",
        "create-cluster",
        "--cluster-name",
        "<cluster_name>",
        "--broker-node-group-info",
        "instanceType=<instance_type>,clientSubnets={{subnet_id1",
        "subnet_id2",
        "...}}",
        "--kafka-version",
        "<version>",
        "--number-of-broker-nodes",
        "<number>"
    ]
    
    try:
        print(f"Executing: {' '.join(cmd)}")
        subprocess.run(cmd, check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
    except FileNotFoundError:
        print("Error: aws not found. Please install it first.")

if __name__ == "__main__":
    run_command()

When To Use

During initial setup of a Kafka cluster for high-throughput data processing applications.

Pro Tip

Optimize instance types based on workload; underprovisioning can lead to bottlenecks in data processing rates.

Command Builder

Tune the command before you copy it

Back to syntax

Generated Command

aws kafka create-cluster --cluster-name <cluster_name> --broker-node-group-info instanceType=<instance_type>,clientSubnets=<subnet_id1 subnet_id2 ...> --kafka-version <version> --number-of-broker-nodes <number>

Terminal Output

Expected runtime feedback

Simulated preview

Output

{\n  "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcd1234-56ef-78gh-90ij-klmnopqrstuv",\n  "State": "CREATING"\n}

Anatomy of Output

Understanding the result

ClusterArn: arn:aws:kafka:region:account-id:cluster/MyKafkaCluster/1234abcd-56ef-78gh-90ij-klmnopqrstuv Cluster ARN

The unique global identifier for the created Kafka cluster.

State: ACTIVE Cluster Status

Cluster is up and running, ready for use.

BrokerSummaryList: [{...}] Broker Summary List

Contains details of the launched broker nodes.

Troubleshooting

Common pitfalls

An error occurred (InvalidParameter) when calling the CreateCluster operation: Invalid instance type.

Solution: Ensure the provided instance type is valid and available in your region.

An error occurred (ResourceLimitExceeded) when calling the CreateCluster operation: Too many clusters.

Solution: Review existing clusters and consider deleting or scaling down prior to creating a new cluster.

An error occurred (AccessDenied) when calling the CreateCluster operation: User does not have permissions to create cluster.

Solution: Verify IAM permissions specifically related to Kafka service actions.

Command Breakdown

What each part is doing

aws: Base Command; The executable that performs this operation. Here it runs Aws before the shell applies any redirect operators.
<cluster_name>: cluster name; The value supplied for cluster name.
<instance_type>: instance type; The value supplied for instance type.
<subnet_id1 subnet_id2 ...>: subnet id1 subnet id2 ...; The value supplied for subnet id1 subnet id2 ....
<version>: version; The value supplied for version.
<number>: number; The value supplied for number.
--cluster-name: Command Option; Tool-specific option used by this command invocation.
--broker-node-group-info: Command Option; Tool-specific option used by this command invocation.
--kafka-version: Command Option; Tool-specific option used by this command invocation.
--number-of-broker-nodes: Command Option; Tool-specific option used by this command invocation.

How To Run

Execution path

Step 1
Run the command: aws kafka create-cluster --cluster-name my-cluster --broker-node-group-info instanceType=m5.large,clientSubnets=subnet-12345678 subnet-87654321 --kafka-version 2.6.0 --number-of-broker-nodes 3
Step 2
Check status with: aws kafka describe-cluster --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcd1234-56ef-78gh-90ij-klmnopqrstuv

Alternative Approaches

Comparable commands in other tools

Alternative cloud infrastructure tools for the same job.

Devpod / Start Workspace From Github devpod up <github.com/user/repo> -i <vscode> Cephadm / Deploy Specific Ceph Service sudo cephadm deploy <service_type> --name <service_name>