Centralized Log Processing with Amazon Elasticsearch Service

Centralized Log Processing with Amazon Elasticsearch Service

2 hours 8 Credits

SPL-237 - Version 1.1.0

© 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon Web Services, Inc. Commercial copying, lending, or selling is prohibited.

Errors or corrections? Email us at

Other questions? Contact us at

Lab overview


Applications capture large amounts of data that provide significant insight into your applications when analyzed in real time. You can use real-time log analysis to ensure security compliance, troubleshoot operation events, identify application usage patterns, and much more.

Let's take a look at a use case: You have several AWS accounts all collecting logs and shipping them to a central account for processing. You have a team of developers who are looking for stack traces from their code, and the business and product owners want to know the events that occurred yesterday. The security team just contacted you about getting abuse notifications from AWS. Your need to quickly identify what the problem is and remedy the situation. This lab provides you with experience building a scalable centralized log processing solution and inspecting various security-related threats so you can respond as soon as possible.

The solution uses Amazon Elasticsearch Service (Amazon ES), a managed service that simplifies the deployment, operation, and scaling of Elasticsearch clusters in the AWS Cloud, as well as Kibana, an analytics and visualization platform that is integrated with Amazon ES in combination with other AWS managed services.

Topics covered

By the end of this lab, you will be able to:

  • Deploy an Amazon ES cluster
  • Create user profiles using an Amazon Cognito user pool
  • Set up an AWS Lambda function to pull logs into Amazon ES
  • Analyze the logs in the Kibana dashboard

Technical knowledge prerequisites

To successfully complete this lab, you should be familiar with basic navigation of the AWS Management Console and have basic knowledge of web, application, and operating system log formats. It is also helpful to have working knowledge of Amazon ES and Kibana for creating and customizing your own dashboards and visualizations.

Note Once you click Start Lab provisioning of the this lab will take between 20 and 25 minutes. While booting up this lab, grab a cup of coffee, relax, and browse through all the services used in this lab. You can also visit AWS Documentation for more content.

Start Lab

  1. At the top of your screen, launch your lab by clicking Start Lab

This will start the process of provisioning your lab resources. An estimated amount of time to provision your lab resources will be displayed. You must wait for your resources to be provisioned before continuing.

If you are prompted for a token, use the one distributed to you (or credits you have purchased).

  1. Open your lab by clicking Open Console

This will automatically log you into the AWS Management Console.

Please do not change the Region unless instructed.

Common login errors

Error : Federated login credentials

If you see this message:

  • Close the browser tab to return to your initial lab window
  • Wait a few seconds
  • Click Open Console again

You should now be able to access the AWS Management Console.

Error: You must first log out

If you see the message, You must first log out before logging into a different AWS account:

  • Click click here
  • Close your browser tab to return to your initial Qwiklabs window
  • Click Open Console again

Architecture of the lab

Centralized Log Analytics Architecture

Lab breakdown

  • Log generation: When the lab is provisioned, it spins off two web servers running on Amazon Elastic Compute Cloud (Amazon EC2) instances. These servers mimic real-world ecommerce application servers running with Apache web server software. These EC2 instances are installed with Amazon CloudWatch agents to collect Apache access logs and server security logs. The logs are being sent to CloudWatch Logs, which is a centralized log service. Additionally, the CloudWatch is collecting Amazon CloudTrail logs and VPC Flow Logs. All of the logs will start flowing to CloudWatch Logs once the lab is fully provisioned.

  • Log analytics: During the lab, you will verify the Amazon ES cluster that was set up during lab provisioning, configure a user pool with Amazon Cognito, set up a Lambda function to ship all the CloudWatch logs over to the Amazon ES cluster, and finally use a Kibana dashboard to see the logs in real time.

Services used in this lab

  • Amazon Elasticsearch Service: Amazon ES is a fully managed service that makes it easy for you to deploy, secure, and operate Elasticsearch at scale with zero downtime. The service offers open-source Elasticsearch APIs, managed Kibana, and integrations with Logstash and other AWS Services, enabling you to securely ingest data from any source and search, analyze, and visualize the data in real time.

  • Kibana: Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.

  • Amazon Cognito: Amazon Cognito provides authentication, authorization, and user management for your web and mobile apps. Your users can sign in directly with a user name and password, or through a third party such as Facebook, Amazon, or Google. The two main components of Amazon Cognito are user pools and identity pools. User pools are user directories that provide sign-up and sign-in options for your app users. Identity pools enable you to grant your users access to other AWS services. You can use identity pools and user pools separately or together.

  • AWS Lambda: Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume—there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service—all with zero administration.

  • Amazon CloudWatch: CloudWatch is a monitoring and management service. It provides you with data and actionable insights to monitor your applications, understand and respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers.

  • Amazon CloudTrail: CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. CloudTrail provides event history of your AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command line tools, and other AWS services. This event history simplifies security analysis, resource change tracking, and troubleshooting.

  • VPC Flow Logs: VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your virtual private cloud (VPC). Flow log data can be published to CloudWatch Logs and Amazon Simple Storage Service (Amazon S3). Flow logs can help you with a number of tasks; for example, to troubleshoot why specific traffic is not reaching an instance, which in turn helps you diagnose overly restrictive security group rules. You can also use flow logs as a security tool to monitor the traffic that is reaching your instance.

  • AWS Identity and Access Management (IAM): IAM enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.

  • Amazon Virtual Private Cloud (Amazon VPC): Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you define. You have complete control over your virtual networking environment, including selection of the IP address range, creation of subnets, and configuration of route tables and network gateways. You can leverage multiple layers of security, including security groups and network access control lists (ACLs) to help control access to Amazon EC2 instances in each subnet. Additionally, you can create a hardware virtual private network (VPN) connection between your corporate data center and your VPC and leverage the AWS Cloud as an extension of your corporate data center.

  • Amazon EC2: Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.

Join Qwiklabs to read the rest of this lab...and more!

  • Get temporary access to the Amazon Web Services Console.
  • Over 200 labs from beginner to advanced levels.
  • Bite-sized so you can learn at your own pace.
Join to Start This Lab