{"id":743,"date":"2011-06-16T13:15:35","date_gmt":"2011-06-16T12:15:35","guid":{"rendered":"http:\/\/kudzia.eu\/b\/?p=743"},"modified":"2013-02-08T09:06:31","modified_gmt":"2013-02-08T08:06:31","slug":"mysql-curcular-master-master-replication","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2011\/06\/mysql-curcular-master-master-replication\/","title":{"rendered":"mysql circular \/ master-master replication"},"content":{"rendered":"<p>i&#8217;ve been playing for a while with mysql circular replication; here are some notes from setting it up.<br \/>\n<!--more--><br \/>\nmy configuration:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/mysqlcircularreplication-e1308997592444.png\" alt=\"\" title=\"mysqlcircularreplication\" width=\"528\" height=\"273\" class=\"alignnone size-full wp-image-777\" srcset=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/mysqlcircularreplication-e1308997592444.png 528w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/mysqlcircularreplication-e1308997592444-300x155.png 300w\" sizes=\"auto, (max-width: 528px) 100vw, 528px\" \/><br \/>\ni have two sites; asql0 and bsql0 are <i>write<\/i> servers to which backend saves information, while aslq0b is read-only slave that in case of emergency can be promoted to the master role.<\/p>\n<p>my config &#8211; my.cnf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\r\n&#x5B;client]\r\nport            = 3306\r\nsocket          = \/var\/run\/mysqld\/mysqld.sock\r\n\r\n&#x5B;mysqld_safe]\r\nsocket          = \/var\/run\/mysqld\/mysqld.sock\r\nnice            = 0\r\n\r\n&#x5B;mysqld]\r\n\r\nslave_compressed_protocol=true\r\nslave-net-timeout = 15\r\nmaster-connect-retry = 15\r\nmaster-retry-count = 0\r\n\r\nserver-id=102  ### every server involved in the replication needs to have unique id\r\n\r\n\r\nauto_increment_increment=2 ### it all depends on your application logic, but in general this should be at least equal to the number of servers invovled in the circular replication\r\nauto_increment_offset=2  ### if we have just 2 servers one of them should have it set to 2, another to 3 to guarantee unique generation of autoincrement values\r\nlog-slave-updates\r\nlog_bin= \/var\/log\/mysql\/mysql-bin.log\r\nrelay-log=mysqld-relay-bin\r\n\r\n\r\nskip_name_resolve\r\ncharacter-set-server = utf8\r\ninnodb_additional_mem_pool_size = 16M\r\ninnodb_buffer_pool_size = 128M\r\ninnodb_file_io_threads = 4\r\ninnodb_thread_concurrency = 8\r\ninnodb_flush_log_at_trx_commit = 2\r\ninnodb_log_buffer_size = 2M\r\ninnodb_log_file_size = 64M\r\ninnodb_log_files_in_group = 2\r\ninnodb_file_per_table\r\ninnodb_flush_method=O_DIRECT\r\n\r\nmax_connections        = 1000\r\nwait_timeout = 10800\r\ninteractive_timeout = 10800\r\n\r\ninnodb_lock_wait_timeout = 120\r\ninnodb_file_per_table\r\nignore_builtin_innodb\r\nplugin-load=innodb=ha_innodb_plugin.so;innodb_trx=ha_innodb_plugin.so;innodb_locks=ha_innodb_plugin.so;innodb_lock_waits=ha_innodb_plugin.so;innodb_cmp=ha_innodb_plugin.so;innodb_cmp_reset=ha_innodb_plugin.so;innodb_cmpmem=ha_innodb_plugin.so;innodb_cmpmem_reset=ha_innodb_plugin.so\r\ninnodb_file_format=barracuda\r\ninnodb_strict_mode=1\r\n\r\nthread_stack            = 256K\r\ngeneral_log= \/var\/log\/mysql\/mysql.log\r\n\r\nuser            = mysql\r\npid-file        = \/var\/run\/mysqld\/mysqld.pid\r\nsocket          = \/var\/run\/mysqld\/mysqld.sock\r\nport            = 3306\r\nbasedir         = \/usr\r\ndatadir         = \/var\/lib\/mysql\r\ntmpdir          = \/tmp\r\nlanguage        = \/usr\/share\/mysql\/english\r\nskip-external-locking\r\n\r\nkey_buffer              = 16M\r\nmax_allowed_packet      = 16M\r\nthread_cache_size       = 8\r\n\r\nmyisam-recover          = BACKUP\r\n\r\nquery_cache_limit       = 1M\r\nquery_cache_size        = 16M\r\n\r\n\r\n\r\n&#x5B;mysqldump]\r\nquick\r\nquote-names\r\nmax_allowed_packet      = 16M\r\n\r\n\r\n&#x5B;isamchk]\r\nkey_buffer              = 16M\r\n\r\n<\/pre>\n<p>securely or not in my replication setup i use debian-sys-maint account; one needed change is to allow access to it from any hosts:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;update user set Host ='%' where User='debian-sys-maint'; flush privileges;&quot; |mysql --defaults-file=\/etc\/mysql\/debian.cnf  mysql\r\n<\/pre>\n<p>to start the replication flowing between asql0 and asql0b [one way &#8216;classical&#8217; replication] i<\/p>\n<p>check master position on asql0:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;show master status\\G&quot;|mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n*************************** 1. row ***************************\r\n            File: mysql-bin.000006\r\n        Position: 494\r\n    Binlog_Do_DB:\r\nBinlog_Ignore_DB:\r\n<\/pre>\n<p>tell asql0b to follow asql0:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;SLAVE STOP; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='asql0',MASTER_USER='debian-sys-maint',MASTER_PASSWORD='whatever',MASTER_LOG_FILE='mysql-bin.000006',MASTER_LOG_POS=494,MASTER_CONNECT_RETRY=10; SLAVE START;&quot; | mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n<\/pre>\n<p>to start replication from asql0 to bsql0 i again check master status of asql0:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;show master status\\G&quot;|mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n*************************** 1. row ***************************\r\n            File: mysql-bin.000006\r\n        Position: 494\r\n    Binlog_Do_DB:\r\nBinlog_Ignore_DB:\r\n<\/pre>\n<p>and on bsql0 i run the same command as on bsql0:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;SLAVE STOP; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='asql0',MASTER_USER='debian-sys-maint',MASTER_PASSWORD='whatever',MASTER_LOG_FILE='mysql-bin.000006',MASTER_LOG_POS=494,MASTER_CONNECT_RETRY=10; SLAVE START;&quot; | mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n<\/pre>\n<p>to start replication flowing &#8216;back&#8217; from bsql0 to asql0 on bsql0 i check master log position:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;show master status\\G&quot;|mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n*************************** 1. row ***************************\r\n            File: mysql-bin.000008\r\n        Position: 285\r\n    Binlog_Do_DB:\r\nBinlog_Ignore_DB:\r\n<\/pre>\n<p>and on asql0 i run:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\necho &quot;slave stop; reset slave; CHANGE MASTER TO MASTER_HOST='bsql0',MASTER_USER='debian-sys-maint',MASTER_PASSWORD='whatever',MASTER_LOG_FILE='mysql-bin.000008',MASTER_LOG_POS=285,MASTER_CONNECT_RETRY=10; slave start;&quot; | mysql --defaults-file=\/etc\/mysql\/debian.cnf\r\n<\/pre>\n<p>from now on any sql command updating data on asql0\/bsql0 should be replicated to all other involved machines. we can check it by executing for instance:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nCREATE DATABASE test;\r\nCREATE TABLE test.t(id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, data varchar(10));\r\nINSERT INTO test.t(data) values('a'), ('b'),('c');\r\nSELECT * FROM test.t;\r\n<\/pre>\n<p>we can check status of every slave by running:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nSHOW SLAVE STATUS\\G\r\n<\/pre>\n<p>i&#8217;m aware of possible consequences of a split brain scenario when both sites are partitioned and some client first submits change to existing entity to one of site A and then fails over and sends another change to the site B. i can think of following ways to solve it<\/p>\n<ul>\n<li>save all previous versions of the entity in the database and change application logic to always pick the newest one<\/li>\n<li>have automated consistency check between sites and fix &#8211; manually or automatically &#8211; found problems based on the timestamp<\/li>\n<\/ul>\n<p>both solutions require good time synchronisation between sites. for now i&#8217;ve set up hourly comparison between all involved servers using <a href=\"http:\/\/www.maatkit.org\/doc\/mk-table-checksum.html\">mk-table-checksum<\/a> from <a href=\"http:\/\/maatkit.org\">maatkit<\/a>:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nmk-table-checksum h=acds0,u=debian-sys-maint,p=whatever h=bcds0,u=debian-sys-maint,p=whatever --databases mysql,test| mk-checksum-filter\r\n<\/pre>\n<p>few links:<\/p>\n<p>http:\/\/www.cwik.ch\/2011\/03\/setting-up-multi-master-circular-replication-with-mysql\/<\/p>\n<p>http:\/\/dev.mysql.com\/doc\/refman\/5.1\/ja\/replication-topology-circular.html<\/p>\n<p>http:\/\/onlamp.com\/pub\/a\/onlamp\/2006\/04\/20\/advanced-mysql-replication.html<\/p>\n<p>http:\/\/capttofu.livejournal.com\/1752.html<\/p>\n<p>http:\/\/freenac.net\/fr\/techguide\/multimaster<\/p>\n<p>http:\/\/www.slideshare.net\/datacharmer\/advanced-mysql-replication-for-the-masses<\/p>\n<p>handy commands:<\/p>\n<p>disable logging of the queries to the binlog &#8211; handy when synchronising back servers out of sync:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nSET sql_log_bin = 0;\r\n<\/pre>\n<p>telling the replication mechanism to skip one statement:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nSET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; slave stop; slave start;\r\n<\/pre>\n<p>apparently <a href=\"http:\/\/dev.mysql.com\/doc\/refman\/5.1\/en\/replication-options-slave.html#option_mysqld_master-connect-retry\">master-connect-retry<\/a> does not work the way i would expect, at least not in debian&#8217;s 5.1.49-3-log &#8211; i was getting &#8220;error reconnecting to master &#8216;debian-sys-maint@tprxy1:3306&#8242; &#8211; retry-time: 180  retries: 0&#8217; with the default 180sec value. but i solved it by using MASTER_CONNECT_RETRY option in the CHANGE MASTER TO command:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nCHANGE MASTER TO MASTER_HOST='tprxy0',MASTER_USER='debian-sys-maint',MASTER_PASSWORD='WHATEVER',\r\nMASTER_LOG_FILE='mysql-bin.000025',MASTER_LOG_POS=106,MASTER_CONNECT_RETRY=15;\r\n<\/pre>\n<p>sidenote: in all my configuration i use actually ip addreses not dns names. i compare the content of the servers every 60 minutes to make sure they did not run out of sync; it&#8217;s done with <a href=\"http:\/\/www.percona.com\/doc\/percona-toolkit\/2.1\/pt-table-checksum.html\">table-checksum<\/a> from maatkit\/percona toolkit. i also monitor the slave lag [in both directions]. if there are inconsistencies\/high slave lag &#8211; i get a notification and have ready scripts for semi-manual synchronization of the content [scripts are very much data specific and cannot be used in generic setups].<\/p>\n","protected":false},"excerpt":{"rendered":"<p>i&#8217;ve been playing for a while with mysql circular replication; here are some notes from setting it up.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[48,12],"class_list":["post-743","post","type-post","status-publish","format-standard","hentry","category-tech","tag-high-availability","tag-mysql"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/743","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=743"}],"version-history":[{"count":34,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/743\/revisions"}],"predecessor-version":[{"id":1575,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/743\/revisions\/1575"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=743"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=743"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=743"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}