Kembali ke Beranda
Networking

Troubleshooting MySQL InnoDB Cluster pada OpenStack Charm

Tutorial troubleshooting MySQL InnoDB Cluster pada OpenStack Charm ketika cluster tidak sehat, mulai dari identifikasi error, reboot cluster dari complete outage, pengecekan status cluster, backup menggunakan mysqldump, pengambilan leader data, instalasi MySQL Shell, akses cluster menggunakan mysqlsh, remove instance bermasalah, membersihkan metadata group replication, menambahkan node kembali ke cluster, menjalankan command SQL pengecekan replication, hingga pengecekan ulang Vault jika ikut terdampak setelah incident MySQL.

Tutorial ini troubleshooting incident MySQL InnoDB Cluster yang tidak sehat pada OpenStack Charm.

1. Gejala

Contoh error:

TEXT
Error : MySQL InnoDB Cluster not healthy: None

2. Coba Reboot Cluster dari Complete Outage

BASH
juju run-action mysql-innodb-cluster/0 reboot-cluster-from-complete-outage
juju run-action mysql-innodb-cluster/1 reboot-cluster-from-complete-outage
juju run-action mysql-innodb-cluster/2 reboot-cluster-from-complete-outage

3. Cek Status Cluster

BASH
juju run-action mysql-innodb-cluster/0 cluster-status
juju run-action mysql-innodb-cluster/1 cluster-status
juju run-action mysql-innodb-cluster/leader --wait cluster-status
juju show-action-output XX

Backup/dump jika diperlukan:

BASH
juju run-action mysql-innodb-cluster/0 mysqldump

4. Ambil Leader Data

BASH
juju run --unit mysql-innodb-cluster/1 leader-get

Data penting yang dicari:

TEXT
cluster-password: <CLUSTER_PASSWORD>
mysql.passwd: <MYSQLROOTPASSWORD>

5. Install MySQL Shell

BASH
wget https://dev.mysql.com/get/Downloads/MySQL-Shell/mysql-shell8.0.34-1ubuntu22.04amd64.deb
sudo dpkg -i mysql-shell8.0.34-1ubuntu22.04amd64.deb

6. Masuk ke Node Leader dengan mysqlsh

BASH
mysqlsh clusteruser@<MYSQLNODEIP> --js

Setelah login:

JAVASCRIPT
var cluster = dba.getCluster("jujuCluster")

7. Remove Instance Bermasalah dari Cluster

Pastikan node mana yang bermasalah, lalu remove dengan force:

JAVASCRIPT
cluster.removeInstance('root@<MYSQLNODEIP_1>:3306',{force: true})
cluster.removeInstance('root@<MYSQLNODEIP_2>:3306',{force: true})
cluster.removeInstance('root@<MYSQLNODEIP_3>:3306',{force: true})

8. Bersihkan Node Bermasalah

Masuk ke node bermasalah, login MySQL:

BASH
mysql -u root -p

Jalankan SQL:

SQL
set GLOBAL superreadonly = OFF;
STOP GROUP_REPLICATION;
RESET SLAVE ALL;
DROP DATABASE mysqlinnodbcluster_metadata;

9. Tambahkan Instance Kembali ke Cluster

Masuk ke MySQL Shell lagi, lalu tambahkan node:

JAVASCRIPT
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP1>", port: 3306, password: "<CLUSTERPASSWORD>"});
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP2>", port: 3306, password: "<CLUSTERPASSWORD>"});
cluster.addInstance({user: "clusteruser", host: "<MYSQLNODEIP3>", port: 3306, password: "<CLUSTERPASSWORD>"});

10. Command SQL Tambahan

SQL
select * from performanceschema.replicationgroup_members;
show master status;
SHOW REPLICAS;

Jika perlu bootstrap group replication manual:

SQL
dba.configureInstance();
SET GLOBAL groupreplicationbootstrap_group = ON;
START GROUP_REPLICATION;
SET GLOBAL groupreplicationbootstrap_group = OFF;

11. Jika Vault Ikut Error

Setelah MySQL sehat, biasanya Vault perlu dicek/unseal ulang:

BASH
juju ssh vault/0
export VAULTTOKEN=<VAULTTOKEN>
vault operator unseal <UNSEALKEY1>
vault operator unseal <UNSEALKEY2>
vault operator unseal <UNSEALKEY3>
vault status
mysqlmysql innodb clusteropenstackopenstack charmjujumysql-innodb-clustercluster outagegroup replicationmysqlshcluster-statusmysqldumpleader-getvaultunseal vaultdatabase recoverycloud databasehigh availabilitytroubleshootingincident handlingopenstack deployment