Upload Local File To Table Partition

Uploads a local file to a specified table and partition.

Terminal

tunnel upload <path/to/file> <table_name>/<partition_spec>;

tunnel upload <path/to/file> <table_name>/<partition_spec>;

#!/bin/bash

# Upload Local File To Table Partition
tunnel upload {{path/to/file}} {{table_name}}/{{partition_spec}};

import subprocess

# Upload Local File To Table Partition
# Make sure to replace <placeholders> with actual values
def run_command():
    cmd = [
        "tunnel",
        "upload",
        "<path/to/file>",
        "<table_name>/<partition_spec>;"
    ]
    
    try:
        print(f"Executing: {' '.join(cmd)}")
        subprocess.run(cmd, check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
    except FileNotFoundError:
        print("Error: tunnel not found. Please install it first.")

if __name__ == "__main__":
    run_command()

When To Use

During scheduled batch uploads to ensure data consistency in production environments.

Pro Tip

Using -n to skip existing rows can prevent data duplication; never use -p flag unless necessary as it may overwrite partitions.

Command Builder

Tune the command before you copy it

Back to syntax

Generated Command

tunnel upload <path/to/file> <table_name>/<partition_spec>;

Anatomy of Output

Understanding the result

Upload started for file: /path/to/file.csv File Path

Indicates the file being uploaded.

Target table: my_table/2021/05 Table/Partition

Shows the specific table and partition for upload.

Upload status: SUCCESS Status

Confirms that the upload completed successfully.

Troubleshooting

Common pitfalls

ERROR: Permission denied for table my_table

Solution: Check if you have write permissions for the target table.

ERROR: File not found: /path/to/file.csv

Solution: Verify the file path and correct if necessary.

ERROR: Invalid partition spec: 2021/05

Solution: Ensure that the partition specification matches the table's defined partitions.

Command Breakdown

What each part is doing

tunnel: Base Command; The executable that performs this operation. Here it runs Tunnel before the shell applies any redirect operators.
<path/to/file>: Input Files; The file path or paths supplied to this command.
<table_name>: table name; The value supplied for table name.
<partition_spec>: partition spec; The value supplied for partition spec.

Alternative Approaches

Comparable commands in other tools

Alternative data processing tools for the same job.

Gdown / Download With Fuzzy Id Extraction gdown --fuzzy <url> Picotool / Convert Elf Bin To Uf2 picotool uf2 convert <path/to/elf_or_bin> <path/to/output> Aws / List Glue Jobs aws glue list-jobs Aws / List Triggers aws glue list-triggers Aws / Create Dev Endpoint aws glue create-dev-endpoint --endpoint-name <name> --role-arn <role_arn_used_by_endpoint>