|
In the present day, we’re saying a easy, cost-effective path for information lake modernization. Amazon S3 Tables now assist managed migration of your Apache Hive tables to Apache Iceberg. Migration works with Parquet tables in Amazon S3 which can be registered within the AWS Glue Information Catalog and permits you to improve your information lake with no pipelines to construct or compute to provision, whereas manufacturing workloads maintain operating.
Apache Iceberg has change into the open normal for giant analytics datasets. It helps you to handle petabyte-scale tables with options like schema evolution, hidden partitioning, and time journey queries, whereas protecting your information open and queryable with any Iceberg-compatible engine. Amazon S3 Tables supply totally managed storage purpose-built to maintain Iceberg tables performant and cost-effective as they develop, with options like automated upkeep, optimization, replication, and S3 Clever-Tiering built-in.
What’s new
Beforehand, migrating Apache Hive tables to Apache Iceberg might take weeks of engineering work to construct and handle the required infrastructure. Changing information varieties and constructing rollback capabilities added operational complexity. Now, that is accessible as a managed, API-driven operation.
S3 Tables deal with your entire migration workflow, together with information validation, compatibility checks, sort conversions, file recompression, and compaction. Earlier than copying any information, S3 Tables establish each supply object in scope and generate a compatibility report. You possibly can sync as many occasions as you want with out stopping writes to your supply desk, aside from a couple of minutes throughout cutover. As soon as you’re prepared, you’ll be able to full the migration, minimizing time that tables are offline.
In comparison with self-managed approaches, S3 Tables managed migrations are cost-effective since you pay just for the info copied, with no cost for the Hive-to-Iceberg conversion, and supply objects in S3 Clever-Tiering keep of their present entry tier.
A primary take a look at the migration
Emigrate an AWS Glue Hive desk, select your vacation spot desk bucket within the S3 console, then select Create desk from migration.
Create a namespace and desk title on your vacation spot Iceberg desk. The vacation spot desk is created utilizing really useful default settings tuned for question efficiency and storage effectivity.
If you begin the migration, S3 Tables validate the supply desk, generate a compatibility report, and start copying information. The vacation spot desk is read-only in the course of the migration. When you full migration, the vacation spot desk is promoted to read-write.
You possibly can start working along with your information instantly utilizing a direct integration with Amazon Athena and Amazon SageMaker Unified Studio within the AWS console, or use any instrument suitable with the Apache Iceberg REST API to straight question the tables. To be taught extra, go to the S3 Tables migration documentation.
You can too use AWS Command Line Interface (AWS CLI), AWS SDKs, or the Agent Toolkit for AWS. For instance, arrange migration jobs and name the start-table-migration command.
cat > /tmp/start-migration.json <<'EOF'
{
"roleARN": "arn:aws:iam::123456789012:position/S3TablesMigrationRole",
"destinationTable": {
"tableBucketARN": "arn:aws:s3tables:us-east-1:123456789012:bucket/my-destination-bucket",
"namespace": "mynamespace",
"title": "my_destination_table",
"encryptionConfiguration": {
"sseAlgorithm": "aws:kms",
"kmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab"
},
"storageClassConfiguration": { "storageClass": "INTELLIGENT_TIERING" },
"tags": { "crew": "s3tables", "objective": "migration" },
"properties": {
"writeFormat": "parquet",
"icebergFormatVersion": 2,
"compressionCodec": "zstd",
"dictionaryEncoding": true
},
"schema": {
"sort": "struct",
"schemaId": 0,
"identifierFieldIds": [1],
"fields": [
{ "id": 1, "name": "id", "type": "long", "required": true, "doc": "primary key" },
{ "id": 2, "name": "payload", "type": "string", "required": false }
]
},
"icebergSortOrder": {
"orderId": 1,
"fields": [ { "sourceId": 1, "transform": "identity", "direction": "asc", "nullOrder": "nulls-first" } ]
}
},
"sourceType": "glueTable",
"supply": {
"glueTable": {
"catalogId": "123456789012",
"databaseName": "my_glue_database",
"tableName": "my_source_table"
}
},
"dataMigration": {
"maintenanceSettings": {
"icebergCompaction": { "targetFileSizeMB": 512, "technique": "binpack" },
"icebergSnapshotManagement": { "minSnapshotsToKeep": 3, "maxSnapshotAgeHours": 168 }
},
"migrationPolicies": {
"unmatchedSourceFieldsPolicy": "fail",
"incompatibleValuesPolicy": "fail"
}
},
"reviewMigrationPlan": false,
"migrationOutput": {
"bucket": "arn:aws:s3:::my-report-bucket",
"prefixes": {
"tableCompatibilityReport": "experiences/table-compatibility/",
"objectCompatibilityReport": "experiences/object-compatibility/",
"objectMigrationReport": "experiences/object-migration/"
}
}
}
EOF
aws s3tables start-table-migration
--cli-input-json file:///tmp/start-migration.jsonTo catch up the vacation spot desk with adjustments within the supply, name sync-table-migration as many occasions as you want earlier than you full the migration. Every sync copies solely what modified.
If you happen to’re utilizing coding brokers like Kiro, the migrating-hive-tables-to-s3-tables talent within the Agent Toolkit for AWS offers step-by-step migration steerage with security guardrails for brokers. The talent can:
- Analyze current IAM insurance policies and suggest equal vacation spot insurance policies
- Validate the migration plan earlier than getting began
- Execute the Begin → Sync → Full API calls
- Monitor job progress and information validation and cutover
To be taught extra, go to the GitHub README web page for the talent.
Now accessible
Amazon S3 Tables managed migration is obtainable at present in US East (N. Virginia, Ohio), US West (Oregon), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Eire, Stockholm) Areas. At launch, the aptitude helps AWS Glue-registered Hive tables saved in Parquet format throughout the identical AWS account and AWS Area. If you wish to name APIs, search documentation, discover regional availability, or troubleshoot about this new function, attempt utilizing the AWS MCP Server and plugins along with your most well-liked AI instrument.
Attempt it within the Amazon S3 console at present and ship suggestions to AWS re:Put up for Amazon S3 or by means of your standard AWS Assist contacts.
— Channy