Learning Objectives – This module introduces you to the core concepts, processes, and tools you need to know in order to get a foundational knowledge of data engineering. You will gain an understanding of the modern data ecosystem and the role Data Engineers, Data Scientists, and Data Analysts play in this ecosystem.

Topics –

• Introduction to Big Data (Classification and Characteristics)
• Drawbacks of RDBMS
• Challenges of Big Data
• Sources of Big Data
• Distributed Systems
• What is Data Engineering, Roles and Responsibilities
• Data Pipelines and types
• Hadoop 2.x and Core Components
• Hadoop Daemons
• Hadoop Architecture
• HDFS File Blocks and Architecture
• Introduction to YARN and Architecture
• YARN Application Execution Flow

Learning Objectives – In this module, you will learn different commands to work with HDFS File System, YARN Commands and how to execute and monitor jobs.
Topics –

• Exploring Important Configuration Files
• Exploring HDFS File System Commands
• Exploring Hadoop Admin Commands
• Exploring YARN Commands
  • Executing YARN Jobs
  • Monitoring YARN Jobs
  • Monitoring different -appTypes
  • Killing YARN Jobs
• Exploring Name Node and Resource Manager UI

Learning Objectives – In this module, you will understand Hive concepts, Hive Data types, loading and querying data in Hive, running hive scripts and Hive UDF.
Topics –

• Introduction to Hive and Architecture
• Exploring Hive table types
• Data loading techniques in Hive
• Hive Complex Data types (Array, Map,
• tructType)

Learning Objectives – In this module, you will learn about spark architecture in comparison with Hadoop Ecosystem and you will learn one of the fundamental building blocks of Spark – RDDs and related manipulations for implementing business logic (Transformations, Actions and Functions performed on RDD).
Topics –

• Overview of Apache Spark
• Data sharing in MapReduce vs Spark
• Exploring Spark Ecosystem
• Exploring RDD’s : Basic Building Block
• Partitions
• Starting Spark Shell
• RDD Creations
  1. Loading a file
  2. Parallelize Collections
• Exploring RDD Operations
  1. Transformations
  2. Actions
• RDD Actions
  1. count()
  2. first()
  3. take(int)
  4. saveAsTextFile(path:String)
  5. reduce(func)
  6. collect(func)
• RDD Transformations
  1. map(func)
  2. foreach(func)
  3. filter(func)
• Chaining Transformation and Actions in Spark
• Configuring Development environment
• Initializing SparkSession i.e Spark 2.x entry point
• Pair RDD
• Sorting Grouping and Aggregations

Learning Objectives – In this module, you will understand Kafka and Kafka Architecture.
Topics –

• Integration between components
• What is Kafka
• Components of Messaging System
• Understanding Kafka components in detail
  1. Producer
  2. Consumer
  3. Broker
  4. Cluster
  5. Topic
  6. Partitions
  7. Offset
  8. Consumer groups
• Message Retention in kafka
• Kafka Commit Log
• Kafka
  1. Starting Zookeeper
  2. Starting Kafka Server
  3. Topic operations: create, list, delete, describe
  4. Publishing data to a topic using console producer
  5. Publishing data to a topic using console consumer
  6. Sending and receiving messages
• Hands on – Kafka Cluster with Multiple Brokers
  1. Creating separate configuration files for brokers
  2. Launching multiple brokers
  3. Getting cluster information and broker details from Zookeeper
• Hands on – Topic with multiple partitions
  1. Creating topic with multiple partitions
  2. How messages are spread across partitions
  3. Reading messages from specific partitions
  4. Reading messages from specific offset in specific partition
• Understanding Kafka Core API’s
• Implementing Kafka Producer & Consumer

Learning Objectives – In this module, you will understand you will learn advance Kafka Core API i.e Producer API, Consumer API, Kafka Connect
Topics –

• Understanding Producer Partitioning Mechanism using Java
• Different ways to implement partitioning mechanism
  • Providing partition number
  • Using Round Robin
  • Key Hashing
• Messaging Sending
• Producer API
  • Synchronous Send
  • ASynchronous Send

Learning Objectives – In this module, you will understand you will learn advance Kafka Core API i.e Producer API, Consumer API, Kafka Connect
Topics –

• Introduction to Stream Processing
• Batch Processing vs Stream Processing
• Streaming Processing API in Spark
• Overview of Structured Streaming
• Notions of Stream Processing
• Streaming Sources, Sinks and Output Modes
• Streaming data from socket, file as input source
• Output Modes (Append, Complete, Update)
• Aggregating on Streaming Data
• Running SQL Queries on Streaming Data
• Processing JSON data using Stream processing
• Joining Batch and Streaming Data
• Triggers

Learning Objectives – In this module, you will learn the fundamentals of AWS, IAM and S3 Storage.
Topics –

• Introduction to Cloud Computing
• AWS Services, Regions and Zones
• Identity and Access Management (IAM)
• Create AWS IAM User and Download Credentials
• Install and Configure AWS CLI
• Introduction to S3 Storage
• S3 Objects, Buckets, and Key Value pairs
• Create Bucket, Upload Object and Exploring UI
• Tagging Bucket and Object
• Optimizing S3 Costs
• S3 Storage classes
• Lifecycle Management
• Retrieve Objects from Glacier
• Protecting Data
• S3 Object Lock
• S3 Versioning and Encryption
• Setup Cross Origin Replication
• AWS Budget Setup – Billing Preferences, Budgets and Alarms
• AWS S3 CLI Commands
• Event Notification, Logging and Analytics
• Billing Preferences, Budgets and Alarms

Learning Objectives – In this module, you will understand Amazon EMR Instance and how to launch spark cluster.
Topics –

• EC2 Fundamentals
• EC2 Security Groups
• EC2 Key Pair for SSH Connection
• Introduction to EMR
• EMR Architecture
• EMR Types
• EMR Cluster LIfecycle
• Optimizing Instance Types
• Hands-on : Creating EMR Cluster and Connecting via SSH
• Hands-on : Building Packaging and Deploying Spark App
• Stream Processing with EMR
• Setup Glue Catalog Integration with Hive
• Hands-on Query DynamoDB from EMR using Hive
  

Learning Objectives – In this course you learn to harness the power of Real-time streaming using Kinesis family of services Kinesis Data Streams (KDS), Kinesis Data Firehose (KDF) and Kinesis Data Analytics (KDA)  to construct high-throughput, low latency, pipelines of data across a variety of architectural components leading to scalable and loosely coupled systems.
Topics –

• Introduction to Amazon Kinesis
• Kinesis Core Services (Streams, Firehose and Analytics)
• Kinesis Data Streams Components(Producers, Shards, Consumers)
• Creating Kinesis Data Stream using AWS CLI and AWS Console
• Publish Records to Kinesis Data Stream
• Consume Records from Kinesis Data Stream
• Introduction to Kinesis Firehose
• Writing Stream Data to S3
• Connecting Kinesis Firehose with Kinesis DataStream
• Sending Data through AWS CLI
• Introduction to Kinesis Data Analytics
• Running SQL queries to Process Streaming Data
• Connecting a Destination to SQL Stream Processing
• Building Data pipeline using Kinesis
  

Learning Objectives – In this course you learn to harness the power of Real-time streaming using Kinesis family of services Kinesis Data Streams (KDS), Kinesis Data Firehose (KDF) and Kinesis Data Analytics (KDA)  to construct high-throughput, low latency, pipelines of data across a variety of architectural components leading to scalable and loosely coupled systems.
Topics –

• Introduction to ETL
• AWS Glue Introduction
• Components of AWS Glue (Crawler, Data catalog)
• Hands-on Developing Data catalog with Glue crawlers
• Querying data using Amazon Athena
• Glue Jobs
• Hands-on Developing Glue Jobs
• Running Spark transformation jobs on AWS Glue
• Creating Developer end point and running spark code
• Glue Catalog Management
• Partitioned table creation and maintenance
• Hive compatible Partitioning
  

Learning Objectives – 
Topics –

• Introduction to Azure Databricks
• Signup for the Azure Account
• Launch Azure Workspace and Databricks Cluster
• Upload Data
• Databricks Clusters
• Databricks Notebook Introduction (Create, Import and Export)
• Magic Commands
• Databricks Mounts
• Develop Spark Application using Azure Databricks
• Performing ETL Operations using Azure Databricks
• Export and Import Databricks Notebooks
• Getting Started with Azure CLI
• Create Resource Group using Azure CLI

Learning Objectives – 
Topics –

• Create and Upload data to ADLS File System or  Container
• Mount ADLS on to Azure Databricks to access files from Azure Blob Storage
• Creating and ADLS Gen2 Account
• Storage Explorer
• Accessing via Access Keys
• Accessing via SAS Token
• Mounting ADLS to DBFS Overview

Learning Objectives – 
Topics –

• Introduction to Azure Databricks
• Signup for the Azure Account
• Launch Azure Workspace and Databricks Cluster
• Upload Data
• Databricks Clusters
• Databricks Notebook Introduction (Create, Import and Export)
• Magic Commands
• Databricks Mounts
• Develop Spark Application using Azure Databricks
• Performing ETL Operations using Azure Databricks
• Export and Import Databricks Notebooks
• Getting Started with Azure CLI
• Create Resource Group using Azure CLI

Learning Objectives – 
Topics –

• Create and Upload data to ADLS File System or  Container
• Mount ADLS on to Azure Databricks to access files from Azure Blob Storage
• Creating and ADLS Gen2 Account
• Storage Explorer
• Accessing via Access Keys
• Accessing via SAS Token
• Mounting ADLS to DBFS Overview