Centralized Log Processing with Amazon Elasticsearch Service
SPL-237 - Version 1.1.0
© 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon Web Services, Inc. Commercial copying, lending, or selling is prohibited.
Errors or corrections? Email us at firstname.lastname@example.org.
Other questions? Contact us at https://aws.amazon.com/contact-us/aws-training/
Applications capture large amounts of data that provide significant insight into your applications when analyzed in real time. You can use real-time log analysis to ensure security compliance, troubleshoot operation events, identify application usage patterns, and much more.
Let's take a look at a use case: You have several AWS accounts all collecting logs and shipping them to a central account for processing. You have a team of developers who are looking for stack traces from their code, and the business and product owners want to know the events that occurred yesterday. The security team just contacted you about getting abuse notifications from AWS. Your need to quickly identify what the problem is and remedy the situation. This lab provides you with experience building a scalable centralized log processing solution and inspecting various security-related threats so you can respond as soon as possible.
The solution uses Amazon Elasticsearch Service (Amazon ES), a managed service that simplifies the deployment, operation, and scaling of Elasticsearch clusters in the AWS Cloud, as well as Kibana, an analytics and visualization platform that is integrated with Amazon ES in combination with other AWS managed services.
By the end of this lab, you will be able to:
- Deploy an Amazon ES cluster
- Create user profiles using an Amazon Cognito user pool
- Set up an AWS Lambda function to pull logs into Amazon ES
- Analyze the logs in the Kibana dashboard
Technical knowledge prerequisites
To successfully complete this lab, you should be familiar with basic navigation of the AWS Management Console and have basic knowledge of web, application, and operating system log formats. It is also helpful to have working knowledge of Amazon ES and Kibana for creating and customizing your own dashboards and visualizations.
Note Once you clickprovisioning of the this lab will take between 20 and 25 minutes. While booting up this lab, grab a cup of coffee, relax, and browse through all the services used in this lab. You can also visit AWS Documentation for more content.
- At the top of your screen, launch your lab by clicking
This will start the process of provisioning your lab resources. An estimated amount of time to provision your lab resources will be displayed. You must wait for your resources to be provisioned before continuing.
If you are prompted for a token, use the one distributed to you (or credits you have purchased).
- Open your lab by clicking
This will automatically log you into the AWS Management Console.
Please do not change the Region unless instructed.
Common login errors
Error : Federated login credentials
If you see this message:
- Close the browser tab to return to your initial lab window
- Wait a few seconds
- Click again
You should now be able to access the AWS Management Console.
Error: You must first log out
If you see the message, You must first log out before logging into a different AWS account:
- Click click here
- Close your browser tab to return to your initial Qwiklabs window
- Click again
Architecture of the lab
Log generation: When the lab is provisioned, it spins off two web servers running on Amazon Elastic Compute Cloud (Amazon EC2) instances. These servers mimic real-world ecommerce application servers running with Apache web server software. These EC2 instances are installed with Amazon CloudWatch agents to collect Apache access logs and server security logs. The logs are being sent to CloudWatch Logs, which is a centralized log service. Additionally, the CloudWatch is collecting Amazon CloudTrail logs and VPC Flow Logs. All of the logs will start flowing to CloudWatch Logs once the lab is fully provisioned.
Log analytics: During the lab, you will verify the Amazon ES cluster that was set up during lab provisioning, configure a user pool with Amazon Cognito, set up a Lambda function to ship all the CloudWatch logs over to the Amazon ES cluster, and finally use a Kibana dashboard to see the logs in real time.
Services used in this lab
Amazon Elasticsearch Service: Amazon ES is a fully managed service that makes it easy for you to deploy, secure, and operate Elasticsearch at scale with zero downtime. The service offers open-source Elasticsearch APIs, managed Kibana, and integrations with Logstash and other AWS Services, enabling you to securely ingest data from any source and search, analyze, and visualize the data in real time.
Kibana: Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.
Amazon Cognito: Amazon Cognito provides authentication, authorization, and user management for your web and mobile apps. Your users can sign in directly with a user name and password, or through a third party such as Facebook, Amazon, or Google. The two main components of Amazon Cognito are user pools and identity pools. User pools are user directories that provide sign-up and sign-in options for your app users. Identity pools enable you to grant your users access to other AWS services. You can use identity pools and user pools separately or together.
AWS Lambda: Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume—there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service—all with zero administration.
Amazon CloudWatch: CloudWatch is a monitoring and management service. It provides you with data and actionable insights to monitor your applications, understand and respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers.
Amazon CloudTrail: CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. CloudTrail provides event history of your AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command line tools, and other AWS services. This event history simplifies security analysis, resource change tracking, and troubleshooting.
VPC Flow Logs: VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your virtual private cloud (VPC). Flow log data can be published to CloudWatch Logs and Amazon Simple Storage Service (Amazon S3). Flow logs can help you with a number of tasks; for example, to troubleshoot why specific traffic is not reaching an instance, which in turn helps you diagnose overly restrictive security group rules. You can also use flow logs as a security tool to monitor the traffic that is reaching your instance.
AWS Identity and Access Management (IAM): IAM enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.
Amazon Virtual Private Cloud (Amazon VPC): Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you define. You have complete control over your virtual networking environment, including selection of the IP address range, creation of subnets, and configuration of route tables and network gateways. You can leverage multiple layers of security, including security groups and network access control lists (ACLs) to help control access to Amazon EC2 instances in each subnet. Additionally, you can create a hardware virtual private network (VPN) connection between your corporate data center and your VPC and leverage the AWS Cloud as an extension of your corporate data center.
Amazon EC2: Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.
이 실습의 나머지 부분과 기타 사항에 대해 알아보려면 Qwiklabs에 가입하세요.
- Amazon Web Services 콘솔에 대한 임시 액세스 권한을 얻습니다.
- 초급부터 고급 수준까지 200여 개의 실습이 준비되어 있습니다.
- 자신의 학습 속도에 맞춰 학습할 수 있도록 적은 분량으로 나누어져 있습니다.