Networking
Troubleshooting MySQL InnoDB Cluster pada OpenStack Charm
Tutorial troubleshooting MySQL InnoDB Cluster pada OpenStack Charm ketika cluster tidak sehat, mulai dari identifikasi error, reboot cluster dari complete outage, pengecekan status cluster, backup menggunakan mysqldump, pengambilan leader data, instalasi MySQL Shell, akses cluster menggunakan mysqlsh, remove instance bermasalah, membersihkan metadata group replication, menambahkan node kembali ke cluster, menjalankan command SQL pengecekan replication, hingga pengecekan ulang Vault jika ikut terdampak setelah incident MySQL.
Tutorial ini troubleshooting incident MySQL InnoDB Cluster yang tidak sehat pada OpenStack Charm.
1. Gejala
Contoh error:
TEXT
Error : MySQL InnoDB Cluster not healthy: None
2. Coba Reboot Cluster dari Complete Outage
BASH
juju run-action mysql-innodb-cluster/0 reboot-cluster-from-complete-outage
juju run-action mysql-innodb-cluster/1 reboot-cluster-from-complete-outage
juju run-action mysql-innodb-cluster/2 reboot-cluster-from-complete-outage
3. Cek Status Cluster
BASH
juju run-action mysql-innodb-cluster/0 cluster-status
juju run-action mysql-innodb-cluster/1 cluster-status
juju run-action mysql-innodb-cluster/leader --wait cluster-status
juju show-action-output XX
Backup/dump jika diperlukan:
BASH
juju run-action mysql-innodb-cluster/0 mysqldump
4. Ambil Leader Data
BASH
juju run --unit mysql-innodb-cluster/1 leader-get
Data penting yang dicari:
TEXT
cluster-password: <CLUSTER_PASSWORD>
mysql.passwd: <MYSQLROOTPASSWORD>
5. Install MySQL Shell
BASH
wget https://dev.mysql.com/get/Downloads/MySQL-Shell/mysql-shell8.0.34-1ubuntu22.04amd64.deb
sudo dpkg -i mysql-shell8.0.34-1ubuntu22.04amd64.deb
6. Masuk ke Node Leader dengan mysqlsh
BASH
mysqlsh clusteruser@<MYSQLNODEIP> --js
Setelah login:
JAVASCRIPT
var cluster = dba.getCluster("jujuCluster")
7. Remove Instance Bermasalah dari Cluster
Pastikan node mana yang bermasalah, lalu remove dengan force:
JAVASCRIPT
cluster.removeInstance('root@<MYSQLNODEIP_1>:3306',{force: true})
cluster.removeInstance('root@<MYSQLNODEIP_2>:3306',{force: true})
cluster.removeInstance('root@<MYSQLNODEIP_3>:3306',{force: true})
8. Bersihkan Node Bermasalah
Masuk ke node bermasalah, login MySQL:
BASH
mysql -u root -p
Jalankan SQL:
SQL
set GLOBAL superreadonly = OFF;
STOP GROUP_REPLICATION;
RESET SLAVE ALL;
DROP DATABASE mysqlinnodbcluster_metadata;
9. Tambahkan Instance Kembali ke Cluster
Masuk ke MySQL Shell lagi, lalu tambahkan node:
JAVASCRIPT
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP1>", port: 3306, password: "<CLUSTERPASSWORD>"});
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP2>", port: 3306, password: "<CLUSTERPASSWORD>"});
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP3>", port: 3306, password: "<CLUSTERPASSWORD>"});
10. Command SQL Tambahan
SQL
select * from performanceschema.replicationgroup_members;
show master status;
SHOW REPLICAS;
Jika perlu bootstrap group replication manual:
SQL
dba.configureInstance();
SET GLOBAL groupreplicationbootstrap_group = ON;
START GROUP_REPLICATION;
SET GLOBAL groupreplicationbootstrap_group = OFF;
11. Jika Vault Ikut Error
Setelah MySQL sehat, biasanya Vault perlu dicek/unseal ulang:
BASH
juju ssh vault/0
export VAULTTOKEN=<VAULTTOKEN>
vault operator unseal <UNSEALKEY1>
vault operator unseal <UNSEALKEY2>
vault operator unseal <UNSEALKEY3>
vault status
mysqlmysql innodb clusteropenstackopenstack charmjujumysql-innodb-clustercluster outagegroup replicationmysqlshcluster-statusmysqldumpleader-getvaultunseal vaultdatabase recoverycloud databasehigh availabilitytroubleshootingincident handlingopenstack deployment