GitHub - alphilla/Data-Preprocessing-Models: Kafka and others

Name		Name	Last commit message	Last commit date
Latest commit History 178 Commits
AWS Glue Job trigger from Lambda.JPG		AWS Glue Job trigger from Lambda.JPG
AWS MSK Connector for Snowflake.txt		AWS MSK Connector for Snowflake.txt
AWS Pinpoint using Boto3.ipynb		AWS Pinpoint using Boto3.ipynb
AWSS3ToSnowflake.zip		AWSS3ToSnowflake.zip
Access_s3_data_in_Spark_outside_AWS_Account.ipynb		Access_s3_data_in_Spark_outside_AWS_Account.ipynb
Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena.png		Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena.png
Architecture of the generic Job to ingest data from s3 to Snowflake.png		Architecture of the generic Job to ingest data from s3 to Snowflake.png
Athena_Table_as_source_for_Apache_Spark.ipynb		Athena_Table_as_source_for_Apache_Spark.ipynb
Athena_parameterized_queries_using_Python.ipynb		Athena_parameterized_queries_using_Python.ipynb
Automatic Mean Imputation.ipynb		Automatic Mean Imputation.ipynb
Automatic Mode Imputation.ipynb		Automatic Mode Imputation.ipynb
CSV_Sentiment.ipynb		CSV_Sentiment.ipynb
Cake preparation-Detail Architecture (1).jpg		Cake preparation-Detail Architecture (1).jpg
Cloudy-Snow.ipynb		Cloudy-Snow.ipynb
Consumer_File SQS.ipynb		Consumer_File SQS.ipynb
Create_,_Update_,_Delete_Athena_Prepared_Statement_using_boto3.ipynb		Create_,_Update_,_Delete_Athena_Prepared_Statement_using_boto3.ipynb
DLQ Redrive.drawio.png		DLQ Redrive.drawio.png
Data Analysis Practice.ipynb		Data Analysis Practice.ipynb
Data Warehouse in Snowflake.png		Data Warehouse in Snowflake.png
Data_Analysis_Play.tsv		Data_Analysis_Play.tsv
Data_Analysis_Practice.ipynb		Data_Analysis_Practice.ipynb
Demo Data used in Generic Framework for External Table.zip		Demo Data used in Generic Framework for External Table.zip
End-to-End Project on Location Based Service.txt		End-to-End Project on Location Based Service.txt
Error Notifications for Snowflake Tasks using SNS.png		Error Notifications for Snowflake Tasks using SNS.png
Error_Handler_Python_File SQS.ipynb		Error_Handler_Python_File SQS.ipynb
Feature Scaling.ipynb		Feature Scaling.ipynb
Feature Selection based on correlation.ipynb		Feature Selection based on correlation.ipynb
Flat Any JSON File PySpark.ipynb		Flat Any JSON File PySpark.ipynb
Flat any Json Using_Recursion,Filter & Map.ipynb		Flat any Json Using_Recursion,Filter & Map.ipynb
Flat_DataFrame.ipynb		Flat_DataFrame.ipynb
Flatten JSON & Harmonize Schema.ipynb		Flatten JSON & Harmonize Schema.ipynb
Fundamentals of Data Preprocessing using Python.ipynb		Fundamentals of Data Preprocessing using Python.ipynb
Glue Workflow Lab.txt		Glue Workflow Lab.txt
Go Deeper in Partitioning.ipynb		Go Deeper in Partitioning.ipynb
Incremental Crawling , Incremental ETL with Glue Workflow.png		Incremental Crawling , Incremental ETL with Glue Workflow.png
Kafka , Faust & Snowflake.txt		Kafka , Faust & Snowflake.txt
Kafka Consumer Internals with Auto Offset Commits and At Least Once Processing.png		Kafka Consumer Internals with Auto Offset Commits and At Least Once Processing.png
Kafka Faust Snowflake Python.PNG		Kafka Faust Snowflake Python.PNG
Kafka Producer Internals.png		Kafka Producer Internals.png
Kafka Retry Architecture.png		Kafka Retry Architecture.png
Kmeans.ipynb		Kmeans.ipynb
Limit_and_offset_concept_in_API.ipynb		Limit_and_offset_concept_in_API.ipynb
Long-Running Workflows with Human Interactions using Step Functions.txt		Long-Running Workflows with Human Interactions using Step Functions.txt
MSK Lab 1.txt		MSK Lab 1.txt
MSK To AWS Lambda.txt		MSK To AWS Lambda.txt
MSK project with serverless Producer & Consumer.txt		MSK project with serverless Producer & Consumer.txt
MSK with s3 sink connector and Snowflake.txt		MSK with s3 sink connector and Snowflake.txt
Manual Offset Commits and At-most Once Processing.png		Manual Offset Commits and At-most Once Processing.png
Manual Offset Commits and Exactly Once Processing.png		Manual Offset Commits and Exactly Once Processing.png
Multicollinearity Effect in Regression.ipynb		Multicollinearity Effect in Regression.ipynb
Multiple Linear Regression Code.ipynb		Multiple Linear Regression Code.ipynb
Null_Value_Handling_Pyspark.ipynb		Null_Value_Handling_Pyspark.ipynb
One Hot Encoding.ipynb		One Hot Encoding.ipynb
Outliers_&_Skewness_Handling.ipynb		Outliers_&_Skewness_Handling.ipynb
Partitioning in Athena using AWS Lambda.txt		Partitioning in Athena using AWS Lambda.txt
Percentage_of_Null_values_in_each_column_of_Dataframe.ipynb		Percentage_of_Null_values_in_each_column_of_Dataframe.ipynb
Pivoting.ipynb		Pivoting.ipynb
Practice Session Data Analysis.ipynb		Practice Session Data Analysis.ipynb
Publish Message in MSK Cluster from AWS Lambda.txt		Publish Message in MSK Cluster from AWS Lambda.txt
Publisher SQS.ipynb		Publisher SQS.ipynb
PySpark5.ipynb		PySpark5.ipynb
PySpark6.ipynb		PySpark6.ipynb
Pyspark1.ipynb		Pyspark1.ipynb
Pyspark2.ipynb		Pyspark2.ipynb
Pyspark3.ipynb		Pyspark3.ipynb
Pyspark4.ipynb		Pyspark4.ipynb
Real-Time Streaming Project with Smartphone Data.txt		Real-Time Streaming Project with Smartphone Data.txt
Receive_message_from_SQS_Queue.ipynb		Receive_message_from_SQS_Queue.ipynb
Recursion Pattern with AWS Step Funciton & Lambda.png		Recursion Pattern with AWS Step Funciton & Lambda.png
SNS_Message_Publish.ipynb		SNS_Message_Publish.ipynb
Send_message_SQS_Queue.ipynb		Send_message_SQS_Queue.ipynb
Serverless Manual Approval Steps in AWS Step Functions and Amazon API Gateway.png		Serverless Manual Approval Steps in AWS Step Functions and Amazon API Gateway.png
Shake detection using Accelerometer , Kafka & Python.txt		Shake detection using Accelerometer , Kafka & Python.txt
Simple Linear Regression (ML).ipynb		Simple Linear Regression (ML).ipynb
Simple OTP System using AWS Serverless.txt		Simple OTP System using AWS Serverless.txt
Snowflake Codes for generic s3 to snowflake loader.txt		Snowflake Codes for generic s3 to snowflake loader.txt
Snowflake External Table Partitioning.txt		Snowflake External Table Partitioning.txt
Snowflake Parallel Processing using Python Lab.txt		Snowflake Parallel Processing using Python Lab.txt
Snowflake Row Level Security.sql		Snowflake Row Level Security.sql
Snowflake Schema Detection.txt		Snowflake Schema Detection.txt
Snowflake Stored Porcedure Parallel execution (Part 1).txt		Snowflake Stored Porcedure Parallel execution (Part 1).txt
Snowflake Stored Porcedure Parallel execution (Part 2).txt		Snowflake Stored Porcedure Parallel execution (Part 2).txt
Snowflake code for External Table refresh framework.txt		Snowflake code for External Table refresh framework.txt
Snowflake logging (1).txt		Snowflake logging (1).txt
Sorting.ipynb		Sorting.ipynb
String Functions.ipynb		String Functions.ipynb
Success File in PySpark.txt		Success File in PySpark.txt
Talend with EMR & Snowflake.png		Talend with EMR & Snowflake.png
Time Traven in Snowflake.txt		Time Traven in Snowflake.txt
Trigger Airflow code via rest api.txt		Trigger Airflow code via rest api.txt
Untitled7.ipynb		Untitled7.ipynb
Using KMS for Client Side Encryption.ipynb		Using KMS for Client Side Encryption.ipynb
_Manual Offset Commits & At Least Once Processing in Kafka Consumer.drawio.png		_Manual Offset Commits & At Least Once Processing in Kafka Consumer.drawio.png
airflow_emr_s3_snowflake_setup.txt		airflow_emr_s3_snowflake_setup.txt
airflow_emr_spark_s3_snowflake.py		airflow_emr_spark_s3_snowflake.py
airflow_news_data_pipeline.py		airflow_news_data_pipeline.py
algolia_layer1.zip		algolia_layer1.zip
aws-eventbridge-kinesisfirehose-s3.drawio.png		aws-eventbridge-kinesisfirehose-s3.drawio.png
convert_json_to_csv_in_kinesis_firehose_transformation.ipynb		convert_json_to_csv_in_kinesis_firehose_transformation.ipynb
dbscan_visualization.txt		dbscan_visualization.txt

Folders and files

Latest commit

History

About

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages