{"id":3596,"date":"2023-12-30T23:07:12","date_gmt":"2023-12-30T22:07:12","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3596"},"modified":"2023-12-30T23:07:22","modified_gmt":"2023-12-30T22:07:22","slug":"podcast-about-evolution-of-mysql-at-booking-com","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2023\/12\/podcast-about-evolution-of-mysql-at-booking-com\/","title":{"rendered":"podcast about evolution of mysql at booking.com"},"content":{"rendered":"\n<p><\/p>\n\n\n\n<p>very interesting, content-heavy, interview with <a href=\"https:\/\/www.linkedin.com\/in\/isotopp\/\">Kristian K\u00f6hntopp<\/a> about his work at booking.com: <a href=\"https:\/\/severalnines.com\/podcast\/running-data-infrastructure-at-an-enterprise-scale\/\" target=\"_blank\" rel=\"noreferrer noopener\">1<\/a>, <a href=\"https:\/\/severalnines.com\/podcast\/how-booking-com-built-their-own-sovereign-dbaas-at-scale\/\" target=\"_blank\" rel=\"noreferrer noopener\">2<\/a> [ alternatively on youtube: <a href=\"https:\/\/www.youtube.com\/watch?v=C1-o8ubZlyE\" target=\"_blank\" rel=\"noreferrer noopener\">1<\/a>, <a href=\"https:\/\/www.youtube.com\/watch?v=ZD8L2z74zL4\" target=\"_blank\" rel=\"noreferrer noopener\">2<\/a> ].<\/p>\n\n\n\n<!--more-->\n\n\n\n<p>topics discussed:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>scale of operations:\n<ul class=\"wp-block-list\">\n<li>thousands of relatively small database servers,<\/li>\n\n\n\n<li>using database replicas instead of caches to present simpler infrastructure to developers,<\/li>\n\n\n\n<li>mysqls managed by team of &lt; 20 dba devops automatizing away routine tasks, without that business would not be able to grow over x400 in the past decade,<\/li>\n\n\n\n<li>at their scale manual interventions done via SSH are ticket-worthy incidents,<\/li>\n\n\n\n<li>node provisioning, adding replicas, failovers are automated [ although the last one came relatively late ],<\/li>\n\n\n\n<li>challenge has always been to stay alive, and not die from data or traffic growth,<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>security aspects:\n<ul class=\"wp-block-list\">\n<li>automated rotation of all db-related credentials every 30 days,<\/li>\n\n\n\n<li>upgrading version of database engine by provisioning new server, promoting it to master or slave, phasing out its predecessor,<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>dealing with data \/ load growth:\n<ul class=\"wp-block-list\">\n<li>loose \/ JSON-based data representation is common in the early stage, as products catch up it&#8217;s eventually converted to more normalized form,<\/li>\n\n\n\n<li>with every ~10x data grow for given subsystem it&#8217;s commonly re-designed to handle the growth,<\/li>\n\n\n\n<li>booking went from being PostgreSQL-based, to MySQL only to database-polyglot where data is persisted in MySQL, Cassandra, Elastic,<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>self hosting vs using public cloud:\n<ul class=\"wp-block-list\">\n<li>6-8x cost saving when running on self-managed hardware compared to using raw storage\/computer from public cloud &#8211; like AWS&#8217;s ec2,<\/li>\n\n\n\n<li>surprising limitations of Amazon&#8217;s RDS services:\n<ul class=\"wp-block-list\">\n<li>it solves easy problems but lacks more advanced features including observability &amp; functionalities needed at a large scale with high availability requirements,<\/li>\n\n\n\n<li>lack of incentives for Amazon to provide better insights about utilization level of provided service which will lead to over-provisioning,<\/li>\n\n\n\n<li>random maintenance windows that can lead to outages of infrastructure relying on the the DBaaS,<\/li>\n\n\n\n<li>using cloud for any non-trivial workloads requires DB admins regardless if it&#8217;s on-prem or in cloud,<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>dealing with new versions of MySQL where optimizer improvements lead to unintended performance degradations\/changes in query plans,<\/li>\n\n\n\n<li>staffing is hard &#8211; finding competent DBas is difficult, finding ones who can write code to automate reliable database management &#8211; much more so,<\/li>\n\n\n\n<li>organizational maturity:\n<ul class=\"wp-block-list\">\n<li>creeping compliance requirements,<\/li>\n\n\n\n<li>formal tracking of dependencies, to the point that applications are blocked, on the firewall level, from talking to other systems if those systems are not formally defined as needed,<\/li>\n\n\n\n<li>forcing organization to mature by demanding documentation, pulling cables at random till everyone is comfortable with and has trust in failover mechanisms,<\/li>\n\n\n\n<li>SLAs and outage budget dictating which automation efforts must be prioritized,<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>management:\n<ul class=\"wp-block-list\">\n<li>need for c-level to have understanding of the technology and its implications on the business; lack of it leads to bad decissions,<\/li>\n\n\n\n<li>idea of <a href=\"https:\/\/mcfunley.com\/choose-boring-technology#:~:text=gets%20about%20three-,innovation%20tokens,-.%20You%20can%20spend\" rel=\"noreferrer noopener\" target=\"_blank\">innovation tokens<\/a> that every startup has and needs to spend wisely.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p><a href=\"https:\/\/severalnines.com\/\">serveralnines<\/a> &#8211; host of the interview &#8211; provides database management products and consulting services similar to <a href=\"https:\/\/percona.com\">Percona<\/a>, they are competing with database offers from the cloud behemoths like AWS, GCP and Azure.<\/p>\n\n\n\n<p><a href=\"https:\/\/jfg-mysql.blogspot.com\/\">Jean-Fran\u00e7ois Gagn\u00e9<\/a> is another fame of booking.com who shared a lot of insights about MySQL on his blog and in <a href=\"https:\/\/www.slideshare.net\/JeanFranoisGagn\/presentations\">numerous presentations<\/a>.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>very interesting, content-heavy, interview with Kristian K\u00f6hntopp about his work at booking.com: 1, 2 [ alternatively on youtube: 1, 2 ].<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,51],"tags":[],"class_list":["post-3596","post","type-post","status-publish","format-standard","hentry","category-uncategorized","category-unimportant"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3596","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3596"}],"version-history":[{"count":1,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3596\/revisions"}],"predecessor-version":[{"id":3597,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3596\/revisions\/3597"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3596"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3596"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3596"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}