{"componentChunkName":"component---src-templates-blog-js","path":"/ctf/aws-formation/tp3-database-on-aws","result":{"data":{"markdownRemark":{"html":"<h2>1 - Databases 101</h2>\n<h3>RDS : Relational databases Services</h3>\n<p>Relational databases : Tables Row + Fields (Columns) </p>\n<p>on AWS : </p>\n<ul>\n<li><strong>SQL server</strong></li>\n<li><strong>Oracle</strong></li>\n<li><strong>MySQL server</strong></li>\n<li><strong>Aurora</strong></li>\n<li><strong>MariaDB</strong></li>\n</ul>\n<h3>Failover :</h3>\n<ul>\n<li>Failover is the constant capability to automatically and seamlessly switch to a highly reliable backup. This can be operated in a redundant manner or in a standby operational mode upon the failure of a primary server, application, system or other primary system component.</li>\n<li>The main purpose of failover is to eliminate, or at least reduce, the impact on users when a system failure occurs.</li>\n<li>This term is also known as fallover.</li>\n</ul>\n<h3> 2 features :</h3>\n<ol>\n<li>multi-AZ - For disaster recovery</li>\n<li>Primary and Secondary Database </li>\n<li>Fallover (turnover of Database pointment)  is automatic with Multi AZ as we point on an url, the DNS will be updated  to switch of IP </li>\n<li>read replicas - For performance</li>\n<li>Primary Data base and Read Replica (5copies)</li>\n<li>Failover not automatic, if we loose primary data base we have</li>\n</ol>\n<h3>No Relational Databases</h3>\n<p><strong>DynamoDB</strong></p>\n<ul>\n<li>Row</li>\n<li>Columns</li>\n<li>Key value pairs</li>\n</ul>\n<p>Javascript notation langage</p>\n<h3>Data Warehousing</h3>\n<p>Large and complex data set</p>\n<p>tools : Cognos, Japersoft, SQL\npurposes : Business inteliigence</p>\n<h3>OLTP vs OLAP</h3>\n<p>Differes with types of queries</p>\n<ul>\n<li>OLTP : Online Transaction Processing = SQL</li>\n<li>\n<p>OLAP : Online ANalytc Pricessing (more complex query : summarise) = NoSQL</p>\n<ul>\n<li>Redshift : Datawarehousing solutions analysis </li>\n</ul>\n</li>\n</ul>\n<h3>Elastic Cache</h3>\n<p><a href=\"https://aws.amazon.com/fr/elasticache/\">https://aws.amazon.com/fr/elasticache/</a></p>\n<p>Magasin de données en mémoire géré, compatible avec Redis ou Memcached.\nSpeeds up performance of existing databases (frequent identical queries) :</p>\n<ul>\n<li>Makes easier deployment, operat, scale in memory cache</li>\n<li>Informations relies on memory caches instead of slower disk-bases databases</li>\n</ul>\n<p>2 in-memory caching engines</p>\n<ol>\n<li>\n<p>Redis </p>\n<ul>\n<li>multi AZ</li>\n<li>you can do backu ups and restores</li>\n</ul>\n</li>\n<li>Memcached</li>\n</ol>\n<h2>2 - Create RDS Instance</h2>\n<h3>Data base creation</h3>\n<p> template available : </p>\n<ul>\n<li>production</li>\n<li>dev</li>\n<li>free tiers (not ebs optimized)</li>\n</ul>\n<p>Storage</p>\n<ul>\n<li>enabling autoscaling</li>\n</ul>\n<p>Configuration</p>\n<ul>\n<li>initial database name to create the database</li>\n<li>maintenance windows</li>\n<li>deletion protection</li>\n</ul>\n<h3>EC2 creation</h3>\n<ul>\n<li>WebDMZ (WordPress) is the web ressource which need to speak with the database</li>\n<li>the database is in rds-launch-wizard security group</li>\n</ul>\n<p>so we have to allow mysql queries in security group from the web ressource to the  wizard to communicate </p>\n<p>note the connectivity endpoint &#x26; port of the database Amazon RDS</p>\n<p>create the wp-config.php file</p>\n<h3>remember</h3>\n<ul>\n<li>\n<p>RDS runs on virtual machines </p>\n<ul>\n<li>do not have virtual access to it (can not ssh it)</li>\n<li>can not log into these operating systems </li>\n<li>amazon responsibility</li>\n</ul>\n</li>\n<li>RDS is NOT serverless</li>\n<li>Aurora serverless IS serverless</li>\n</ul>\n<h3>Serverless : Informatique sans serveur</h3>\n<p>L'informatique sans serveur ou serverless computing est un paradigme de cloud computing dans lequel le fournisseur de serveur gère dynamiquement les ressources allouées au service client. Le prix dépend des ressources effectivement consommées et non des capacités d'un serveur acheté à l'avance1. Mais le terme 'sans serveur' ne signifie pas qu'il n'y a pas de serveurs impliqués. Cela signifie qu'ils sont gérés par les fournisseurs et non par les consommateurs. Sans trop penser à leur maintenance, les ressources informatiques sont utilisées comme des services. Les principaux avantages de sans serveur sont le modèle de tarification à l'utilisation, une évolutivité élevée, la disponibilité et un temps réduit pour développer et livrer les fonctionnalités du produit.</p>\n<h2>3 - RDS Backuo, Multi AZ &#x26; Read Replicas</h2>\n<h3>Back up with RDS</h3>\n<p>2 types </p>\n<ol>\n<li>Automated Backups</li>\n<li>Database Snapshot</li>\n</ol>\n<h4>Automated Backups</h4>\n<p>Full save within a retention perdiod (1 to 35 days)\ntime recovery to a second\nstored in S3\nfree Automated Backups of the same amount of your S3</p>\n<p>during the recovery the data may be beacked up with elevetad latency</p>\n<h4>Database snapshot</h4>\n<p>Manually </p>\n<p>the restored version will be a new RDS instance with a new end point</p>\n<h3>Encryption at rest</h3>\n<p>KMS : Key Management Service\nwhen the RDS instance is encrypted, the data as rest is encrypted too, as well as its automated bacckup, read replicas and snapshots</p>\n<h4>At Rest vs. In Motion</h4>\n<p><a href=\"https://brightlineit.com/encryption-at-rest-important-business/\">https://brightlineit.com/encryption-at-rest-important-business/</a></p>\n<p>De manière générale, il existe deux types de données : les données en mouvement et les données au repos. Les données en mouvement (ou \"données actives\") sont des données que vous utilisez très probablement au quotidien. Elles sont généralement stockées dans une base de données à laquelle on accède par des applications ou des programmes. Les données au repos sont stockées et sont généralement protégées par un pare-feu ou un logiciel anti-virus. Bien que ces méthodes de protection des données au repos soient bonnes, une sécurité totale nécessite l'ajout d'une couche de défense supplémentaire. C'est là que le cryptage au repos entre en jeu.</p>\n<p>Dès que RDS instance est encrypté, ses sauvegardes le sont aussi.</p>\n<h3>Multi AZ</h3>\n<p>Easteic Balancer - 3 EC2 instance - 1 primary data base + X  replicates</p>\n<p>Synchronization to the stand database</p>\n<p>Is for desaster covery (DR) only (not for performance increasing)\nallow to have an exact copy in another Availabitlity Zone\nDurent a maintenance, mutli az is automatically fail over</p>\n<p>you van force a failover from one availability region to another by rebooting the RDS instances</p>\n<p>Following database available with multi AZ :</p>\n<ul>\n<li>SQL server</li>\n<li>Oracle</li>\n<li>MySql Server</li>\n<li>PG SLQ</li>\n<li>Maria DB</li>\n</ul>\n<h3>Read Replica</h3>\n<p>Easteic Balancer - 3 EC2 instance - 1 primary data base +  2 replicates</p>\n<p>2 replicates replicate asynchonously thus they can commmunicate directly with one of the 3 EC2 instances</p>\n<p>use read replicas primarily for very-heavy database workloads</p>\n<p>Following database available with multi AZ :</p>\n<ul>\n<li>Aurora</li>\n<li>Oracle</li>\n<li>MySql Server</li>\n<li>PG SLQ</li>\n<li>Maria DB</li>\n</ul>\n<p>Things to know about Read Replicas</p>\n<ul>\n<li>used for scaling or increasing performance</li>\n<li>must have automatic backups  turned on in order to deploy a read replica</li>\n<li>up to 5 read replica copie in any databse</li>\n<li>read replicas of read replicas possibility but watch out to latency</li>\n<li>each read replica, will have its own DNS point</li>\n<li>read replicas with Multi AZ source database</li>\n<li>have a replica in a second region</li>\n<li>read replicas can be promoted to be their own database (master) -> breaks the replication</li>\n</ul>\n<h2>4 - RDS backup Lab</h2>\n<p>To create a read replicas you have to have backup turn on</p>\n<p>you can apply cahnge in another time</p>\n<p>you can have you readreplica be encrypted or not, be public or not, in the same region or not, mutli az deployment or not</p>\n<p>we can promote a read replica to a primary data base</p>\n<h2>5 - DynamoDB</h2>\n<p>Opposite of RDS</p>\n<p>is a fast and lfexible NoSQL database service for all application that need consistenc, single digit millisend lantency at any scale</p>\n<p>support both document and key pair value relation</p>\n<h3>basis of dynamoDB</h3>\n<ul>\n<li>stored on SSD storage</li>\n<li>spread accross 3 geographically distinct data centers</li>\n<li>eventual consistent read (default)</li>\n<li>strongly consistent reads</li>\n</ul>\n<h4>eventual consistent reads</h4>\n<p>Best read performance (accès aux données constant)</p>\n<h4>strongly  consistent reads</h4>\n<p>Be sure to have the good data</p>\n<h4>eventual vs strong consistency</h4>\n<p><strong>Eventual Consistency</strong> Conflicts can arise, but nodes communicate each other their changes to solve those conflicts, so in time they agree upon the definitive value. Thus, if no more changes are applied to the data for a certain period, then all nodes will agree in the data value (i.e. they will eventually agree) so readers of data will eventually see the same value</p>\n<p><img src=\"/assets/TP3-Databases-Eventual.png\" alt=\"img1\" title=\"eventual\"></p>\n<p><strong>Strong consistency</strong> Quite different to the other two. Here it is a requirement that upon update operations all nodes agree on the new value before making the new value visible to clients. That way updates are visible to all clients 'at the same time', so they will read the same value at all times. Now this introduces the requirement for some blocking in update operations. Both in EC and SEC an update operation was over as soon as the local copy was updated (then the operation was broadcasted to the other nodes). Here a client update does not return until all nodes have agreed upon the data value, and while this is done all accesses to any copy of that data are 'locked' (so other clients reads are blocked). In our example for EC, if cliA runs write(\"Frank\"), cliA will be blocked until the update is agreed by both nA and nB, and then it will made visible for both cliA and cliB at the same time, i.e. the read() operation should return the same value from then on.</p>\n<p><img src=\"/assets/TP3-Databases-Strong.png\" alt=\"img\" title=\"strong\"></p>\n<h2>6 - Redshift</h2>\n<p>Redshift : Amazon Datawarehousing solutions analysis </p>\n<h3>redshift configuration</h3>\n<ul>\n<li>single node : 160 Gb</li>\n<li>\n<p>mutlinode :</p>\n<ul>\n<li>leader node (manages client connection and receives queris)</li>\n<li>compute node (store data and perform queries and computation) ; up to 128 compute nodes by leader </li>\n</ul>\n</li>\n<li>available in only one AZ</li>\n</ul>\n<h4>advanced compression</h4>\n<p>multiple compression techinuqes to enhance the level of compression in comarison to relational database system cause there is not visual relation</p>\n<h4>PMM : Massively Parallel Procession</h4>\n<p>make it easier to add node to our data warehousing enable fast query performance as our data warehousing grow</p>\n<h3>redshift Backups</h3>\n<ul>\n<li>\n<p>**enables bBackups are Enalbe by default : **</p>\n<ul>\n<li>1 day of retention min</li>\n<li>35 max</li>\n</ul>\n</li>\n<li><strong>maintains at least 3 copies of our data (origninal + 1 replicate on compute node + 1 replicate one a AWS S3 backup)</strong></li>\n<li><strong>can also replicate asynchonously our snapshot to S3 in another region for DR (distater recovery)</strong></li>\n</ul>\n<h3>price</h3>\n<ul>\n<li>by node by hour</li>\n<li>charge only for the compute node (not the leader)</li>\n<li>chrage for the data transfert</li>\n<li>charge for the backup</li>\n</ul>\n<h3>encryption</h3>\n<ul>\n<li>for in transit : using SSL</li>\n<li>for in rest : AES-256</li>\n<li>\n<p>taking care of key management :</p>\n<ul>\n<li>manage your own key with HSM</li>\n<li>AWS key management service</li>\n</ul>\n</li>\n</ul>\n<h3>availability</h3>\n<ul>\n<li>only available in 1 AZ</li>\n<li>can restore snapshot to a new AZ in case of outage</li>\n</ul>\n<h2>Aurora</h2>\n<p>Amazon unpropriety Database engine : </p>\n<ul>\n<li>MySQL + PGSQL compatibel </li>\n<li>speed and availabitity</li>\n<li>low cost and simplicity</li>\n<li><strong>Open Source databases</strong></li>\n</ul>\n<p>5 fois plus performant que MySQL</p>\n<ol>\n<li>from 10 Gb to 64Gb by 10</li>\n<li>compute ressource can scale up to 32vCPU and 244 Gb</li>\n<li>2 copie of your ddata is container in each AZ with at leat 3 AZ -> 6 copies in sum</li>\n</ol>\n<h3>scaling aurora</h3>\n<ul>\n<li>handle to loss of up to two copies of data without affecting database write and read  availability</li>\n<li>storage is self healing</li>\n</ul>\n<h3>three type of aurora replicas</h3>\n<ul>\n<li>aurora replica (15)</li>\n<li>mysql read replica (5)</li>\n<li>pgslq (1) </li>\n</ul>\n<p><img src=\"/assets/TP3-Databases-AuroraComparison-MySQL.png\" alt=\"database-aurora\" title=\"aurora\"></p>\n<h3>backups</h3>\n<ul>\n<li>\n<p>automated backups are always enables on Amazon Aurora DB instance</p>\n<ul>\n<li>Backups do not impact data base performance</li>\n</ul>\n</li>\n<li>\n<p>take snapchot with aurora</p>\n<ul>\n<li>snapshots  do not impact data base performance</li>\n</ul>\n</li>\n<li>share aurora snapshots with other aws account</li>\n</ul>\n<h3>Aurora serverless</h3>\n<ul>\n<li>on demand</li>\n<li>autoscaling</li>\n<li>MySQL and PGSQL compatible</li>\n<li>\n<p>automatically start up, shuts down and scale capacity according the application needs</p>\n<ul>\n<li>thus it s cheap option for infrequent, intermittent unpredictable worklaods </li>\n</ul>\n</li>\n</ul>\n<h3>remember</h3>\n<ul>\n<li>2 copie of your ddata is container in each AZ with at leat 3 AZ -> 6 copies in sum</li>\n<li>3 types of replicas available : Aurora, MySQL , PGSQL</li>\n<li>Aurora has automated backups turned on ; can take snaphot, share this snap shot with other aws account</li>\n<li>automatically start up, shuts down and scale capacity according the application needs</li>\n</ul>\n<h2>ElastiCache</h2>\n<p><img src=\"/assets/TP3-Databases-ElastiCache-RedisMemcached.png\" alt=\"elc\" title=\"elc\"></p>\n<p><em>web application overloaded, how to make this application perform better</em> ?\nOptions : </p>\n<ol>\n<li>add replicas and point your reads to your read replicats</li>\n<li>or use elasticache</li>\n</ol>","frontmatter":{"date":"July 31, 2020","path":"ctf/aws-formation/tp3-database-on-aws","title":"TP3 - Database on AWS","tags":["cloud","ec2","aws"],"categorie":"ctf","thumbnail":"/assets/alex-machado-80sv993luki-unsplash.jpg"},"fields":{"readingTime":{"text":"10 min read"}}},"file":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAMEBf/EABUBAQEAAAAAAAAAAAAAAAAAAAED/9oADAMBAAIQAxAAAAGFz11jKaIn/8QAGhAAAgMBAQAAAAAAAAAAAAAAAAECERMSMf/aAAgBAQABBQLm3gxwaJTxe1i8/8QAGBEAAgMAAAAAAAAAAAAAAAAAAAECElH/2gAIAQMBAT8BUdKn/8QAFhEBAQEAAAAAAAAAAAAAAAAAABEB/9oACAECAQE/Abqv/8QAGhAAAQUBAAAAAAAAAAAAAAAAAQACEBESIv/aAAgBAQAGPwJckOjObVBtR//EABwQAQACAgMBAAAAAAAAAAAAAAEAETFBIWFxkf/aAAgBAQABPyEVB9ih2jUSxfkFQDkWX4Ju5grU/9oADAMBAAIAAwAAABDjD//EABYRAQEBAAAAAAAAAAAAAAAAAAEAIf/aAAgBAwEBPxAA1M//xAAWEQEBAQAAAAAAAAAAAAAAAAABIRD/2gAIAQIBAT8QWYTP/8QAGhABAQEAAwEAAAAAAAAAAAAAAREAIUFxwf/aAAgBAQABPxAYkFlcFzRnCFCH3zCCggiKI5gk6hL1lpwEhs0zEABv/9k=","aspectRatio":1.4970059880239521,"src":"/static/11cdcb302d032fa2db3a46428a8f74c9/a7715/alex-machado-80sv993luki-unsplash.jpg","srcSet":"/static/11cdcb302d032fa2db3a46428a8f74c9/8f7df/alex-machado-80sv993luki-unsplash.jpg 250w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/0f3a1/alex-machado-80sv993luki-unsplash.jpg 500w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/a7715/alex-machado-80sv993luki-unsplash.jpg 1000w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/37d86/alex-machado-80sv993luki-unsplash.jpg 1500w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/a41d1/alex-machado-80sv993luki-unsplash.jpg 2000w,\n/static/11cdcb302d032fa2db3a46428a8f74c9/6a059/alex-machado-80sv993luki-unsplash.jpg 6000w","sizes":"(max-width: 1000px) 100vw, 1000px"}}}},"pageContext":{"slug":"ctf/aws-formation/tp3-database-on-aws","featuredImage":"alex-machado-80sv993luki-unsplash.jpg"}}}