Simplify real-time analytics with zero-ETL from Amazon DynamoDB to Amazon SageMaker Lakehouse

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "TablePolicy01",
            "Effect": "Allow",
            "Principal": {
                "Service": "glue.amazonaws.com"
            },
            "Action": [
                "dynamodb:ExportTableToPointInTime",
                "dynamodb:DescribeExport",
                "dynamodb:DescribeTable"
            ],
            "Useful resource": "*"
        }
    ]
}

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:::root"
            },
            "Action": "glue:CreateInboundIntegration",
            "Resource": "arn:aws:glue:::database/salesmarketing_XXX"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "glue.amazonaws.com"
            },
            "Action": "glue:AuthorizeInboundIntegration",
            "Resource": "arn:aws:glue:::database/salesmarketing_XXX"
        }
    ]
}

Now you’re able to create your zero-ETL integration.

Create a zero-ETL integration

Full the next steps to create a zero-ETL integration:

1. On the AWS Glue console, choose Zero-ETL integrations within the navigation pane.

2. Choose “Create zero-ETL integration” to create a brand new configuration.

3. Choose Amazon DynamoDB because the supply kind.

4. Beneath Supply particulars, choose ecommerce_customer_behavior for DynamoDB desk.

5. Beneath Goal particulars, present the next data:
   1. For AWS account, choose Use the present account.
   2. For Information warehouse or catalog, enter the account ID of your default catalog.
   3. For Goal database, enter salesmarketing_XXX.
   4. For Goal IAM position, enter datazone_usr_role_XXX.

6. Beneath Output settings, choose Unnest all fields and Use main keys from DynamoDB tables, depart Configure goal desk identify because the default worth (ecommerce_customer_behavior), then choose Subsequent.

7. Enter zetl-ecommerce-customer-behavior for Title below Integration particulars, then choose Subsequent.

8. Choose Create and launch integration to launch the combination.

The standing must be Creating after the combination is efficiently initiated.
The standing will change to Lively in roughly a minute.

Confirm that the SageMaker Lakehouse desk exists. This course of may take as much as quarter-hour to finish, as a result of the default refresh interval from DynamoDB is ready to fifteen minutes.

Validate the SageMaker Lakehouse desk

Now you can question your SageMaker Lakehouse desk, created by way of zero-ETL integration, utilizing numerous question engines. Full the next steps to confirm you may you see the desk in SageMaker Unified Studio:

1. Log in to the SageMaker Unified Studio portal utilizing the only sign-on (SSO) possibility.

2. Choose your undertaking to view its particulars web page.

3. Choose Information within the navigation pane.

4. Confirm which you could see the Iceberg desk within the SageMaker Lakehouse catalog.

Question with Athena

On this part, we present easy methods to use Athena to question the SageMaker Lakehouse desk from SageMaker Unified Studio. On the undertaking web page, find the ecommerce_customer_behavior desk within the catalog, and on the choices menu (three dots), choose Question with Athena.

This creates a SELECT question in opposition to the SageMaker Lakehouse desk in a brand new window, and it is best to see the question outcomes as proven within the following screenshot.

Question with Amazon Redshift

You can even question the SageMaker Lakehouse desk from SageMaker Unified Studio utilizing Amazon Redshift. Full the next steps:

1. Choose the connection on the highest proper.
2. Choose Redshift (Lakehouse) from the listing of connections.
3. Choose the awsdatacatalog database.
4. Choose the salesmarketing schema.
5. Choose Select button.

The outcomes shall be proven within the Amazon Redshift Question Editor.

Question with Amazon EMR Serverless

You’ll be able to question the Lakehouse desk utilizing Amazon EMR Serverless, which makes use of Apache Spark’s processing capabilities. Full the next steps:

1. On the undertaking web page, choose Compute within the navigation pane.
2. Choose Add compute on the Information processing tab to create an EMR Serverless compute related to the undertaking.

3. You’ll be able to create new compute sources or hook up with current sources. For this instance, choose Create new compute sources.

4. Choose EMR Serverless.

5. Enter a compute identify (for instance, Gross sales-Advertising), choose the newest launch of EMR Serverless, and choose Add compute.

It is going to take a while to create the compute.

You need to see the standing as Began for the compute. Now it’s prepared for use as your compute possibility for querying by way of a Jupyter pocket book.

6. Choose the Construct menu and choose JupyterLab.

It is going to take a while to arrange the workspace for operating JupyterLab.

After the Jupyter Lab house is ready up, it is best to see a web page just like the next screenshot.

7. Choose the brand new folder icon to create a brand new folder.

8. Title the folder lakehouse_zetl_lab.

9. Navigate to the folder you simply created and create a pocket book below this folder.

10. Choose the pocket book Python3 (ipykernel) on the Launcher tab, and rename the pocket book to query_lakehouse_table.

You’ll be able to observe that the pocket book is displaying native Python as default language and compute. The 2 drop down menus present the connection kind and compute for the chosen connection kind, simply above the primary cell inside the Jupyter pocket book.

11. Choose PySpark because the connection, and choose the EMR Serverless software as compute.

12. Enter the next pattern code to question the desk utilizing Spark SQL:

import sys
from pyspark.sql import SparkSession
from pyspark.sql.capabilities import *

# Set the present database
spark.catalog.setCurrentDatabase("salesmarketing_XXX")

# Execute SQL question and retailer leads to DataFrame
df = spark.sql("choose * from ecommerce_customer_behavior restrict 10")

# Show the outcomes
df.present()

You’ll be able to see the Spark DataFrame outcomes.

Clear up

To keep away from incurring future expenses, delete the SageMaker area, DynamoDB desk, AWS Glue sources, and different objects created from this submit.

Conclusion

This submit demonstrated how one can set up a zero-ETL connection from DynamoDB to SageMaker Lakehouse, making your knowledge accessible in Iceberg format with out constructing customized knowledge pipelines. We confirmed how one can analyze this DynamoDB knowledge by way of numerous compute engines inside SageMaker Unified Studio. This streamlined strategy alleviates conventional knowledge motion complexities, and permits extra environment friendly knowledge evaluation workflows instantly out of your DynamoDB tables.

Check out this answer to your personal use case, and share your suggestions within the feedback.

Concerning the authors

Narayani Ambashta is an Analytics Specialist Options Architect at AWS, specializing in the automotive and manufacturing sector, the place she guides strategic clients in creating trendy knowledge and AI methods. With over 15 years of cross-industry expertise, she makes a speciality of huge knowledge structure, real-time analytics, and AI/ML applied sciences, serving to organizations implement trendy knowledge architectures. Her experience spans throughout lakehouse, generative AI, and IoT platforms, enabling clients to drive digital transformation initiatives. When not architecting trendy options, she enjoys staying lively by way of sports activities and yoga.

Raj Ramasubbu is a Senior Analytics Specialist Options Architect centered on huge knowledge and analytics and AI/ML with AWS. He helps clients architect and construct extremely scalable, performant, and safe cloud-based options on AWS. Raj supplied technical experience and management in constructing knowledge engineering, huge knowledge analytics, enterprise intelligence, and knowledge science options for over 18 years previous to becoming a member of AWS. He helped clients in numerous {industry} verticals like healthcare, medical gadgets, life sciences, retail, asset administration, automotive insurance coverage, residential REIT, agriculture, title insurance coverage, provide chain, doc administration, and actual property.

Yadgiri Pottabhathini is a Senior Analytics Specialist Options Architect within the media and leisure sector. He makes a speciality of aiding enterprise clients with their knowledge and analytics cloud transformation initiatives, whereas offering steerage on accelerating their Generative AI adoption by way of the event of knowledge foundations and trendy knowledge methods that leverage open-source frameworks and applied sciences.

Junpei Ozono is a Sr. Go-to-market (GTM) Information & AI options architect at AWS in Japan. He drives technical market creation for knowledge and AI options whereas collaborating with international groups to develop scalable GTM motions. He guides organizations in designing and implementing modern data-driven architectures powered by AWS companies, serving to clients speed up their cloud transformation journey by way of trendy knowledge and AI options. His experience spans throughout trendy knowledge architectures together with Information Mesh, Information Lakehouse, and Generative AI, enabling clients to construct scalable and modern options on AWS.