{"componentChunkName":"component---src-templates-blog-js","path":"/ctf/aws-formation/tp10-monitoring-reporting","result":{"data":{"markdownRemark":{"html":"

TP10 - Monitoring & Reporting

\n

1. Cloudwatch Introduction

\n

definition

\n

CW is a monitoring service to monitor your AWS resources, as well as the applications that you run on AWS

\n

what can we monitor with CW

\n\n

CloudWatch and EC2

\n

When CW comes on EC2, it monirtores by default :

\n\n

How long are CW metrics Stored

\n\n

CloudWatch Alarms

\n

You can create an alarm to monitor any Amazon CloudWatch metric in your account, with appropriate thresholds and with defined actiosn triggered.

\n

It includes :

\n\n

2. Monitoring EC2 With Custom Metrics

\n

edit the crontab

\n

lowest granularity is 1 minutes

\n

3. Monitoring EBS

\n

EBS = virutal disk in the cloud

\n

4 different volume types

\n\n

Uses cases

\n

\"image-20200729222515366\"

\n

IOPS & Volumes Examples

\n\n

IOPS

\n

IOPS are a unit of measure representing input/output operations per second. The operations are measured in KiB, and the underlying drive technology determines the maximum amount of data that a volume type counts as a single I/O. I/O size is capped at 256 KiB for SSD volumes and 1,024 KiB for HDD volumes because SSD volumes handle small or random I/O much more efficiently than HDD volumes.

\n

IOPS per GiB of volume size

\n

Genral Purpose SSD volumes have a base of 3 IOPS per GiB of volume size.

\n\n

So the bigger is the volume, the more performance you get, untill 10 000 IOPS (from 3 000).

\n

Thus, if we burst our 100 GiB (-> 3IOPS * 100 = 300 IOPS) volume to 3000 IOPS, the burst will be 2700 (3000 - 300 )

\n

\"image-20200729224635676\"

\n

Above 10 000 IOPS (Y axis), :

\n\n

I/O Credits

\n\n

Pre Warming EBS Volumes

\n

Les volumes EBS vides reçoivent leurs performances maximum au moment où ils sont créés et ne nécessitent pas d'initialisation (anciennement préchauffage).

\n

Concernant les volumes ayant été créés à partir d'instantanés, les blocs de stockage doivent être extraits d'Amazon S3 et écrits sur le volume pour que vous puissiez y accéder. Cette action préalable prend du temps et peut causer une hausse significative de la latence des opérations d'E/S lors du premier accès à chaque bloc.\nLes performances du volume sont obtenues une fois que tous les blocs ont été téléchargés et écrits sur le volume.

\n

Pour éviter cette baisse de performances initiale dans un environnement de production, vous pouvez utiliser l'une des solutions suivantes :

\n\n

EBS cloudwatch metrics

\n

https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/using_cloudwatch_ebs.html#ebs-volume-metrics

\n

L'espace de noms AWS/EBS inclut les métriques suivantes pour les volumes EBS. Pour obtenir des informations sur l'espace disque disponible à partir du système d'exploitation sur une instance, veuillez consulter Affichage de l'espace disque disponible.

\n

Volume Status Checks

\n

https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/monitoring-volume-status.html#monitoring-volume-checks

\n

Amazon Web Services (AWS) fournit automatiquement les données que vous pouvez utiliser pour surveiller vos volumes Amazon Elastic Block Store (Amazon EBS).

\n

Le tableau suivant répertorie les statuts des volumes Amazon EBS.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Statut du volumeStatut d'activation des E/SStatut des performances d'E/S (uniquement disponible pour les volumes IOPS provisionnés)
okActivé (E/S activées ou E/S activées automatiquement)Normal (performances du volume telles qu'attendues)
warningActivé (E/S activées ou E/S activées automatiquement)Dégradé (performances du volume inférieures aux attentes) Profondément dégradé (performances du volume bien inférieures aux attentes)
impairedActivé (E/S activées ou E/S activées automatiquement) Désactivé (volume hors connexion et récupération en attente, ou en attente d'activation par l'utilisateur des E/S)Interrompu (performances du volume profondément impactées) Non disponible (impossible de déterminer les performances d'E/S parce que les E/S sont désactivées)
insufficient-dataActivé (E/S activées ou E/S activées automatiquement) Données insuffisantesDonnées insuffisantes
\n\n

Modifying EBS volumes

\n

If you Amazon EBS Volume is attached to a current generation EC2 instance type, you can, whitout detaching it. :

\n\n

If they are detached, you can as well :

\n\n

4. Monitoring ELB

\n

Elastoc Load Balancer

\n

3 Types of ELB

\n

3 Differet Types of Elastic Load Balancers :

\n
    \n
  1. Application Load Balancer
    2. \n
  2. Network Load Balancer
    3. \n
  3. Classic Load Balancer
    4. \n
\n

4 Different ways to monitor your Load Balancer

\n
    \n
  1. CloudWatch metrics
    2. \n
  2. Access logs
    3. \n
  3. Request tracing
    4. \n
  4. CloudTrail logs
    5. \n
\n

CloudWatch vs CloudTrail

\n\n

CloudWatch Metrics

\n

ELB publishes to Amazon CloudWatch, for your load balancers and your targets :

\n\n

Access Logs

\n

Access logs = information about request :

\n\n

Access logging = optional feature of ELB disabled by default. It enables the store of access logs in a S3 bucket in compressed files.

\n

It is still possible to trace these errors using Access Logs which would be stored on S3.

\n

Access Logs can store data where the EC2 instance has been deleted.

\n

Request Tracing

\n

Available for Application Load Balancers only

\n

You can use request tracing to track HTTP request from clients to targets or other services -> adding W-AWS-Trace-Id header before sending request to the target.

\n

CloudTrail

\n

To capture calls made to the ELB API and store them as log files in Amazon S3.

\n

Used to determine the following details :

\n\n

5. Monitoring Elasticache

\n

Elasticach = 2 engines :

\n
    \n
  1. Memcached
    2. \n
  2. Redis
    3. \n
\n

4 important things to look at :

\n
    \n
  1. CPU Utilization
    2. \n
  2. Swap Usage
    3. \n
  3. Evictions
    4. \n
  4. Concurrent Connections
    5. \n
\n

Comparison

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MemcachedRedis
CPU UtilizationMulti-threaded
Loads up to 90% ; if exceeds -> add more nodes to the cluster		Mono-threaded
Threshold = 90 / number of cores
Swap UsageShould be around 0 most of the time, should not exceed 50Mb.
If it is the case -> increase the memcached connections overhead parametes		No SwapUsage metric
Use reserved-memory
EvictionsNo recommended settings
Scale OUT (add more nodes)
Scale UP (increasing the memory of existing nodes)		No recommended settings
Scale OUT (add replicas) only
Concurrent ConnectionsNo recommended settingsNo recommended settings
\n

Swap Usage

\n\n

Evictions

\n

An Evictions occurs when a new item is added and an old item must be removed due to lake of free space in the system.

\n

Concurent connections

\n

If there is a large spike in the number of concurrent connections this can either mean :

\n\n

Remember to set an alarm on number of concurrent connections for Elasticache.

\n

6. Metrics From Multiple Regions & Custom Dashboards

\n

CloudWatch - Dashboards are multi-region and can display any widget to any region.

\n

To ad the widget, change to the region that you need and then add the widget to the dashboard.

\n

7. Create A Billing Alarm

\n

\"image-20200731012254243\"

\n

8. AWS Organisations Lab

\n

Definition

\n

AWS Organisations = allows you to manage multiple AWS accounts at once.\nYou can create groups of accounts and then apply policies to those groups

\n\n

Central Management

\n

Centralization of policies management across multiple accounts

\n

Control Access

\n

Allow or Deny individuals AWS Services using Service Control Policies (SCPs).

\n

Even if IAM in an accounts allows a resource not allowed by SCP, this last will override it.

\n

Automate AWS Account Creation

\n

Automate the creation and management of new AWS Accounts

\n

Consolidated Billing

\n

Set up a single payment method for all the AWS accounts in your organization through consolidated billing.

\n

9. AWS Resource Groups & Tagging

\n

Tags definition

\n\n

Resource Groups

\n

It is a way of grouping tags

\n

Group your resource using the tafs that are assigned to them.

\n

Resource groups can contain foloowing tags :

\n\n

You can use resource groups with AWS Systems manager to automate task

\n

10. Cost Explorer and Cost Allocation Tags

\n

AWS Cost Explorer has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. You can view data up to last 13months and get recommandation.

\n\n

11. EC2 Pricing Models

\n

EC2 options :

\n\n

12. AWS Config 101

\n

With AWS Config, you are able to continuously monitor and record configuration changes of your AWS resources. Config also enables you to inventory your AWS resources, the configurations of your AWS resources, as well as software configurations within EC2 instances at any point in time. Once change from a previous state is detected, an Amazon Simple Notification Service (SNS) notification can be delivered for you to review and take action.

\n\n

\"image-20200731015230281\"

\n

13. AWS Config Lab

\n

Compliance checks

\n\n

Permission needed for config

\n\n

Restrict Access :

\n\n

Monitoring Config :

\n\n

14. AWS Config Vs AWS CloudTrail Vs CloudWatch

\n

All of these monitors somethings :

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfigCloudTrailCloudWatch
performanceAPI calls in the AWS platformstate of your AWS environment
\n

CloudWatch can notify you of changes.

\n

15. Health Dashboards

\n

2 types of Dashboards :

\n
    \n
  1. Health :\nshows the health of each AWS Service as a whole per region
    2. \n
  2. Personal :\nprovides alert and remediation guidance when AWS is experiencing events that may impact you
    3. \n
\n

Service Health Dashboard

\n

\"Service

","frontmatter":{"date":"July 31, 2020","path":"ctf/aws-formation/tp10-monitoring-reporting","title":"TP10 - Monitoring & Reporting","tags":["cloud","ec2","aws"],"categorie":"ctf","thumbnail":"/assets/alex-machado-80sv993luki-unsplash.jpg"},"fields":{"readingTime":{"text":"13 min read"}}},"file":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAMEBf/EABUBAQEAAAAAAAAAAAAAAAAAAAED/9oADAMBAAIQAxAAAAGFz11jKaIn/8QAGhAAAgMBAQAAAAAAAAAAAAAAAAECERMSMf/aAAgBAQABBQLm3gxwaJTxe1i8/8QAGBEAAgMAAAAAAAAAAAAAAAAAAAECElH/2gAIAQMBAT8BUdKn/8QAFhEBAQEAAAAAAAAAAAAAAAAAABEB/9oACAECAQE/Abqv/8QAGhAAAQUBAAAAAAAAAAAAAAAAAQACEBESIv/aAAgBAQAGPwJckOjObVBtR//EABwQAQACAgMBAAAAAAAAAAAAAAEAETFBIWFxkf/aAAgBAQABPyEVB9ih2jUSxfkFQDkWX4Ju5grU/9oADAMBAAIAAwAAABDjD//EABYRAQEBAAAAAAAAAAAAAAAAAAEAIf/aAAgBAwEBPxAA1M//xAAWEQEBAQAAAAAAAAAAAAAAAAABIRD/2gAIAQIBAT8QWYTP/8QAGhABAQEAAwEAAAAAAAAAAAAAAREAIUFxwf/aAAgBAQABPxAYkFlcFzRnCFCH3zCCggiKI5gk6hL1lpwEhs0zEABv/9k=","aspectRatio":1.4970059880239521,"src":"/static/11cdcb302d032fa2db3a46428a8f74c9/a7715/alex-machado-80sv993luki-unsplash.jpg","srcSet":"/static/11cdcb302d032fa2db3a46428a8f74c9/8f7df/alex-machado-80sv993luki-unsplash.jpg 250w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/0f3a1/alex-machado-80sv993luki-unsplash.jpg 500w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/a7715/alex-machado-80sv993luki-unsplash.jpg 1000w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/37d86/alex-machado-80sv993luki-unsplash.jpg 1500w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/a41d1/alex-machado-80sv993luki-unsplash.jpg 2000w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/6a059/alex-machado-80sv993luki-unsplash.jpg 6000w","sizes":"(max-width: 1000px) 100vw, 1000px"}}}},"pageContext":{"slug":"ctf/aws-formation/tp10-monitoring-reporting","featuredImage":"alex-machado-80sv993luki-unsplash.jpg"}}}