[root@ms-sbc2-0 ~] 2023-07-24 11:43:10 $ /tmp/convert_cluster_to_mbc.sh -a -f Option -a will allow to run the script with active alerts Option -f will allow script to run event if backup resource file exists LOG FILE WRITABLE VERIFYING WEKA AGENT VERIFYING WEKA VERSION Supported Weka version 4.1.0.77 VERIFY USER IS LOGGED IN Weka user login successful. has clusterAdmin role WEKA IDENTIFIED Working on CLUSTER: ms-sbc2 UUID: (2c2fc1ea-1994-4e52-8454-d6448e252330) STATUS: OK (11 backend containers UP, 77 drives UP) STARTED 1 hour ago (154 io-nodes UP, 396 Buckets UP). VERIFYING WEKA LOCAL CONTAINER STATUS Weka local container is running. CHECKING FOR ANY ALERTS There are no active alerts No Weka alerts present. CHECKING REBUILD STATUS No rebuild in progress. VERIFYING WEKA BACKEND HOST STATUS Verified all backend host's are UP. VERIFYING WEKA NODES STATUS Weka Nodes Status OK. SSH connectivity between backend hosts [SSH PASSWORDLESS CONNECTIVITY CHECK] SSH connectivity test completed STARTING CLUSTER WIDE conversion VERIFYING /tmp/mbc_divider_script.py PERMISSIONS MAKING /tmp/mbc_divider_script.py EXECUTABLE VERIFYING /tmp/resources_generator.py PERMISSIONS MAKING /tmp/resources_generator.py EXECUTABLE DISTRIBUTING FILE TO HOST 10.108.123.20 scp -p /tmp/mbc_divider_script.py 10.108.123.20:/tmp > /dev/null Warning: Permanently added '10.108.123.20' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.123.20 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:48:28 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 0 -J" 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:06:3e:92/10.108.175.243/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 00:00:17:02:48:73/10.108.28.118/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:02:fe:5d/10.108.102.107/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:03:78:c0/10.108.191.71/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:06:3a:96/10.108.83.141/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:02:f6:78/10.108.190.2/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:06:61:cd/10.108.171.7/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:06:19:52/10.108.200.192/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:09:f7:21/10.108.221.10/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:06:88:10/10.108.167.199/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:03:ab:cf/10.108.132.181/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:de:a3/10.108.197.168/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 00:00:17:02:52:10/10.108.211.80/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - DEBUG: Adding net devices: 00:00:17:02:64:ce/10.108.5.109/16/10.108.0.1 2023-07-24 11:48:28 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 11:48:28 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:48:29 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 11:48:29 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:48:29 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 11:48:29 mbc divider - INFO: Removing host from nfs interface group Ifgrp 2023-07-24 11:48:29 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port delete Ifgrp 0 ens3 -f" 2023-07-24 11:48:29 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:48:29 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 11:48:34 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:48:34 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 11:48:34 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 2023-07-24 11:48:34 mbc divider - INFO: Management IPs selected: --management-ips=10.108.123.20 2023-07-24 11:48:34 mbc divider - INFO: Stopping old container 2023-07-24 11:48:34 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 11:48:43 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:3e:92/10.108.175.243/16/10.108.0.1 00:00:17:02:48:73/10.108.28.118/16/10.108.0.1 02:00:17:02:fe:5d/10.108.102.107/16/10.108.0.1 02:00:17:03:78:c0/10.108.191.71/16/10.108.0.1 02:00:17:06:3a:96/10.108.83.141/16/10.108.0.1 02:00:17:02:f6:78/10.108.190.2/16/10.108.0.1 02:00:17:06:61:cd/10.108.171.7/16/10.108.0.1 02:00:17:06:19:52/10.108.200.192/16/10.108.0.1 02:00:17:09:f7:21/10.108.221.10/16/10.108.0.1 02:00:17:06:88:10/10.108.167.199/16/10.108.0.1 02:00:17:03:ab:cf/10.108.132.181/16/10.108.0.1 02:00:17:0a:de:a3/10.108.197.168/16/10.108.0.1 00:00:17:02:52:10/10.108.211.80/16/10.108.0.1 00:00:17:02:64:ce/10.108.5.109/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 11:48:43 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:3e:92/10.108.175.243/16/10.108.0.1 00:00:17:02:48:73/10.108.28.118/16/10.108.0.1 02:00:17:02:fe:5d/10.108.102.107/16/10.108.0.1 02:00:17:03:78:c0/10.108.191.71/16/10.108.0.1 02:00:17:06:3a:96/10.108.83.141/16/10.108.0.1 02:00:17:02:f6:78/10.108.190.2/16/10.108.0.1 02:00:17:06:61:cd/10.108.171.7/16/10.108.0.1 02:00:17:06:19:52/10.108.200.192/16/10.108.0.1 02:00:17:09:f7:21/10.108.221.10/16/10.108.0.1 02:00:17:06:88:10/10.108.167.199/16/10.108.0.1 02:00:17:03:ab:cf/10.108.132.181/16/10.108.0.1 02:00:17:0a:de:a3/10.108.197.168/16/10.108.0.1 00:00:17:02:52:10/10.108.211.80/16/10.108.0.1 00:00:17:02:64:ce/10.108.5.109/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [12, 10, 24, 26, 20, 4, 16, 8, 6, 28, 18, 22, 14, 2], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 11:48:43 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 11:48:43 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 11:48:43 mbc divider - INFO: Starting new containers 2023-07-24 11:48:43 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" 2023-07-24 11:48:43 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up drives0: Allocated network device "0000:00:07.0" (with identifier "0000:00:07.0") to slots [1] on "ms-sbc2-0":"drives0" (1/1) drives0: Allocated core 14 to slot 1 on "ms-sbc2-0":"drives0" (1/1) drives0: Starting hugepages allocation for "ms-sbc2-0":"drives0" drives0: Container "ms-sbc2-0":"drives0" allocated 712 out of 712 required hugepages after 1 retries drives0: Allocated 1424MB hugepages memory from 1 NUMA nodes for "ms-sbc2-0":"drives0" drives0: Bandwidth of "ms-sbc2-0":"drives0" set to unlimited drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-0":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-0":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-0":"drives0" Container "drives0" is ready (pid = 64307) 2023-07-24 11:49:11 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:49:16 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 11:49:17 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:49:22 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:49:26 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:49:27 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 0 -J" 2023-07-24 11:49:27 mbc divider - DEBUG: Waiting for nodes ['NodeId<0>', 'NodeId<1>', 'NodeId<2>', 'NodeId<3>', 'NodeId<4>', 'NodeId<5>', 'NodeId<6>', 'NodeId<7>', 'NodeId<8>', 'NodeId<9>', 'NodeId<10>', 'NodeId<11>', 'NodeId<12>', 'NodeId<13>', 'NodeId<14>'] to reach DOWN state 2023-07-24 11:49:27 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 0 -J" 2023-07-24 11:49:27 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 11:49:27 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 0 -J" 2023-07-24 11:49:27 mbc divider - INFO: The following drives will be moved to the new container: ['f4be38eb-b375-4c71-bc1f-7c15ad195d3c', '38253905-1a98-42a6-8ddc-62767f146a21', '38c8b856-17c1-4c0f-aa36-4f57d6a8de19', 'fe41af65-4ba9-4632-ac05-ffa7772f29df', '53e0cb98-913e-4f49-aebb-92d418ede5bd', 'e2d4d6e9-37f9-46ed-ae6e-6c836373b33b', '2fb56b30-4a7a-42a4-9221-8fe3667ab8d9'] 2023-07-24 11:49:27 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 13" 2023-07-24 11:49:29 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 0 -J" 2023-07-24 11:49:29 mbc divider - INFO: Done scanning drives 2023-07-24 11:49:29 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" 2023-07-24 11:49:29 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 65775) 2023-07-24 11:49:40 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" 2023-07-24 11:49:40 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-000 --management-ips=10.108.123.20 --join-ips=10.108.63.220,10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 67103) 2023-07-24 11:50:09 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:50:13 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 11:50:14 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 11:50:19 mbc divider - INFO: Adding NFS protocol to frontend container 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port add Ifgrp 15 ens3" 2023-07-24 11:50:19 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers successfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 11:50:19 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 11:50:19 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 11:50:22 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 0" 2023-07-24 11:50:22 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 11:50:22 mbc divider - INFO: running: /bin/sh -c "weka cluster host 0 -J" 2023-07-24 11:50:22 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 0 --no-unimprint" 2023-07-24 11:50:22 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 11:50:22 mbc divider - INFO: Finished cleanup 2023-07-24 11:50:22 mbc divider - INFO: running: /bin/sh -c "weka local status -v" Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 ganesha, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 65775 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (44 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 64307 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 67103 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (33 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND ganesha (ganesha): Status: RUNNING PID: 69002 Version: 4.1.0.77 Last Message: API Port: 14000 2023-07-24 11:50:28 mbc divider - INFO: Finished moving server to MBC architecture HOST 10.108.123.20 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.123.20 to rejoin cluster All Nodes belonging to 10.108.123.20 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.63.220 scp -p /tmp/mbc_divider_script.py 10.108.63.220:/tmp > /dev/null Warning: Permanently added '10.108.63.220' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.63.220 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:50:29 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 1 -J" 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:09:12:98/10.108.77.230/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:02:69:8a/10.108.0.239/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:03:3b:02/10.108.13.216/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:06:e5:b1/10.108.206.12/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:09:a6:0b/10.108.221.41/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:09:76:ef/10.108.96.77/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:93:d2/10.108.39.174/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 00:00:17:02:b3:f6/10.108.47.24/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:06:0a:0c/10.108.11.106/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:09:d2/10.108.82.184/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:99:81/10.108.57.252/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:02:1f:ca/10.108.9.134/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:03:34:87/10.108.31.213/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - DEBUG: Adding net devices: 02:00:17:09:db:84/10.108.204.179/16/10.108.0.1 2023-07-24 11:50:29 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 11:50:29 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:50:30 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 11:50:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:50:30 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 11:50:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:50:30 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 11:50:30 mbc divider - INFO: Removing host from nfs interface group Ifgrp 2023-07-24 11:50:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port delete Ifgrp 1 ens3 -f" 2023-07-24 11:50:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:50:30 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 11:50:35 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:36 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:37 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:38 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:40 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:41 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:42 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:43 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:44 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:45 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:50:47 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 11:50:47 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 2023-07-24 11:50:47 mbc divider - INFO: Management IPs selected: --management-ips=10.108.63.220 2023-07-24 11:50:47 mbc divider - INFO: Stopping old container 2023-07-24 11:50:47 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Disabling managed container ganesha Stopping containers 2023-07-24 11:51:08 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:12:98/10.108.77.230/16/10.108.0.1 02:00:17:02:69:8a/10.108.0.239/16/10.108.0.1 02:00:17:03:3b:02/10.108.13.216/16/10.108.0.1 02:00:17:06:e5:b1/10.108.206.12/16/10.108.0.1 02:00:17:09:a6:0b/10.108.221.41/16/10.108.0.1 02:00:17:09:76:ef/10.108.96.77/16/10.108.0.1 02:00:17:0b:93:d2/10.108.39.174/16/10.108.0.1 00:00:17:02:b3:f6/10.108.47.24/16/10.108.0.1 02:00:17:06:0a:0c/10.108.11.106/16/10.108.0.1 02:00:17:0a:09:d2/10.108.82.184/16/10.108.0.1 02:00:17:0a:99:81/10.108.57.252/16/10.108.0.1 02:00:17:02:1f:ca/10.108.9.134/16/10.108.0.1 02:00:17:03:34:87/10.108.31.213/16/10.108.0.1 02:00:17:09:db:84/10.108.204.179/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 11:51:08 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:12:98/10.108.77.230/16/10.108.0.1 02:00:17:02:69:8a/10.108.0.239/16/10.108.0.1 02:00:17:03:3b:02/10.108.13.216/16/10.108.0.1 02:00:17:06:e5:b1/10.108.206.12/16/10.108.0.1 02:00:17:09:a6:0b/10.108.221.41/16/10.108.0.1 02:00:17:09:76:ef/10.108.96.77/16/10.108.0.1 02:00:17:0b:93:d2/10.108.39.174/16/10.108.0.1 00:00:17:02:b3:f6/10.108.47.24/16/10.108.0.1 02:00:17:06:0a:0c/10.108.11.106/16/10.108.0.1 02:00:17:0a:09:d2/10.108.82.184/16/10.108.0.1 02:00:17:0a:99:81/10.108.57.252/16/10.108.0.1 02:00:17:02:1f:ca/10.108.9.134/16/10.108.0.1 02:00:17:03:34:87/10.108.31.213/16/10.108.0.1 02:00:17:09:db:84/10.108.204.179/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [12, 4, 8, 22, 2, 14, 6, 24, 28, 16, 26, 20, 18, 10], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 11:51:08 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 11:51:08 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 11:51:08 mbc divider - INFO: Starting new containers 2023-07-24 11:51:08 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:51:08 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 64017) 2023-07-24 11:51:30 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:51:34 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 11:51:35 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:51:41 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:51:45 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:51:45 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 1 -J" 2023-07-24 11:51:45 mbc divider - DEBUG: Waiting for nodes ['NodeId<20>', 'NodeId<21>', 'NodeId<22>', 'NodeId<23>', 'NodeId<24>', 'NodeId<25>', 'NodeId<26>', 'NodeId<27>', 'NodeId<28>', 'NodeId<29>', 'NodeId<30>', 'NodeId<31>', 'NodeId<32>', 'NodeId<33>', 'NodeId<34>'] to reach DOWN state 2023-07-24 11:51:45 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 1 -J" 2023-07-24 11:51:46 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 11:51:46 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 1 -J" 2023-07-24 11:51:46 mbc divider - INFO: The following drives will be moved to the new container: ['309aad98-4077-4625-a765-8c8268616cbc', '340aa7ed-b024-4a8a-86fc-b6339b81236e', '8dffd29d-9ade-4ffd-a5b5-a7b179da502d', '650cee45-7153-481d-9397-f7b38c6940d7', '08bc09c1-3f70-4ff6-aa70-13b918e8703b', 'de1ae3e3-f4e3-4fbe-b5cf-8bac0a18148d', '9a3c2937-b36b-4fca-b1f0-62e654857727'] 2023-07-24 11:51:46 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 16" 2023-07-24 11:51:48 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 1 -J" 2023-07-24 11:51:48 mbc divider - INFO: Done scanning drives 2023-07-24 11:51:48 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:51:48 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 65532) 2023-07-24 11:51:59 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:51:59 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-001 --management-ips=10.108.63.220 --join-ips=10.108.72.27,10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 66914) 2023-07-24 11:52:24 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:52:29 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 11:52:30 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 11:52:35 mbc divider - INFO: Adding NFS protocol to frontend container 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port add Ifgrp 18 ens3" 2023-07-24 11:52:35 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 11:52:35 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 11:52:35 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 11:52:38 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 1" 2023-07-24 11:52:38 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 11:52:38 mbc divider - INFO: running: /bin/sh -c "weka cluster host 1 -J" 2023-07-24 11:52:38 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 1 --no-unimprint" 2023-07-24 11:52:38 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 11:52:38 mbc divider - INFO: Finished cleanup 2023-07-24 11:52:38 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 11:52:44 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 ganesha, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 65532 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (44 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 64017 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 66914 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Removed from cluster: Not reachable by the cluster (26 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND ganesha (ganesha): Status: RUNNING PID: 68678 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.63.220 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.63.220 to rejoin cluster All Nodes belonging to 10.108.63.220 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.72.27 scp -p /tmp/mbc_divider_script.py 10.108.72.27:/tmp > /dev/null Warning: Permanently added '10.108.72.27' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.72.27 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:52:45 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 2 -J" 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:06:18:58/10.108.178.205/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:02:a3:1e/10.108.101.85/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:03:b0:09/10.108.217.83/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:16:b9/10.108.77.229/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:02:d8:eb/10.108.49.248/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:02:95:8f/10.108.90.27/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:06:72:50/10.108.189.69/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:03:37:f3/10.108.177.220/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:b0:d7/10.108.28.230/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 00:00:17:02:48:ef/10.108.96.22/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:06:43:d1/10.108.229.105/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:a8:3c/10.108.76.143/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:c7:4c/10.108.226.103/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - DEBUG: Adding net devices: 00:00:17:02:68:29/10.108.20.222/16/10.108.0.1 2023-07-24 11:52:45 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 11:52:45 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:52:46 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 11:52:46 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:52:46 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 11:52:46 mbc divider - INFO: Removing host from nfs interface group Ifgrp 2023-07-24 11:52:46 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port delete Ifgrp 2 ens3 -f" 2023-07-24 11:52:46 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:52:46 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 11:52:51 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:52:51 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 11:52:51 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 2023-07-24 11:52:51 mbc divider - INFO: Management IPs selected: --management-ips=10.108.72.27 2023-07-24 11:52:51 mbc divider - INFO: Stopping old container 2023-07-24 11:52:51 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 11:53:00 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:18:58/10.108.178.205/16/10.108.0.1 02:00:17:02:a3:1e/10.108.101.85/16/10.108.0.1 02:00:17:03:b0:09/10.108.217.83/16/10.108.0.1 02:00:17:0b:16:b9/10.108.77.229/16/10.108.0.1 02:00:17:02:d8:eb/10.108.49.248/16/10.108.0.1 02:00:17:02:95:8f/10.108.90.27/16/10.108.0.1 02:00:17:06:72:50/10.108.189.69/16/10.108.0.1 02:00:17:03:37:f3/10.108.177.220/16/10.108.0.1 02:00:17:0a:b0:d7/10.108.28.230/16/10.108.0.1 00:00:17:02:48:ef/10.108.96.22/16/10.108.0.1 02:00:17:06:43:d1/10.108.229.105/16/10.108.0.1 02:00:17:0a:a8:3c/10.108.76.143/16/10.108.0.1 02:00:17:0a:c7:4c/10.108.226.103/16/10.108.0.1 00:00:17:02:68:29/10.108.20.222/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 11:53:00 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:18:58/10.108.178.205/16/10.108.0.1 02:00:17:02:a3:1e/10.108.101.85/16/10.108.0.1 02:00:17:03:b0:09/10.108.217.83/16/10.108.0.1 02:00:17:0b:16:b9/10.108.77.229/16/10.108.0.1 02:00:17:02:d8:eb/10.108.49.248/16/10.108.0.1 02:00:17:02:95:8f/10.108.90.27/16/10.108.0.1 02:00:17:06:72:50/10.108.189.69/16/10.108.0.1 02:00:17:03:37:f3/10.108.177.220/16/10.108.0.1 02:00:17:0a:b0:d7/10.108.28.230/16/10.108.0.1 00:00:17:02:48:ef/10.108.96.22/16/10.108.0.1 02:00:17:06:43:d1/10.108.229.105/16/10.108.0.1 02:00:17:0a:a8:3c/10.108.76.143/16/10.108.0.1 02:00:17:0a:c7:4c/10.108.226.103/16/10.108.0.1 00:00:17:02:68:29/10.108.20.222/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [12, 10, 20, 22, 24, 8, 14, 26, 2, 6, 4, 28, 18, 16], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 11:53:00 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 11:53:00 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 11:53:00 mbc divider - INFO: Starting new containers 2023-07-24 11:53:00 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:53:00 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 65400) 2023-07-24 11:53:22 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:53:26 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 11:53:27 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:53:33 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:53:37 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:53:37 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 2 -J" 2023-07-24 11:53:37 mbc divider - DEBUG: Waiting for nodes ['NodeId<40>', 'NodeId<41>', 'NodeId<42>', 'NodeId<43>', 'NodeId<44>', 'NodeId<45>', 'NodeId<46>', 'NodeId<47>', 'NodeId<48>', 'NodeId<49>', 'NodeId<50>', 'NodeId<51>', 'NodeId<52>', 'NodeId<53>', 'NodeId<54>'] to reach DOWN state 2023-07-24 11:53:37 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 2 -J" 2023-07-24 11:53:37 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 11:53:37 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 2 -J" 2023-07-24 11:53:37 mbc divider - INFO: The following drives will be moved to the new container: ['9da09c25-e220-4ee7-a9f8-91ed5a1123fd', '9fa9ba84-c8c7-4f3e-b9cb-c794930775e5', '40f1a365-a5c0-43bc-bf0d-9fee5e1cf030', '36561611-1502-46b2-a403-76323caff9ef', '4c773c45-e58b-4462-a5ef-367f3161b660', '124b8ab3-dcc0-4c7b-ad40-5b7c497d9f37', '3de32a43-f01a-48f2-a3e6-fad83a78cefa'] 2023-07-24 11:53:37 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 19" 2023-07-24 11:53:40 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 2 -J" 2023-07-24 11:53:40 mbc divider - INFO: Done scanning drives 2023-07-24 11:53:40 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:53:40 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 66823) 2023-07-24 11:53:52 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" 2023-07-24 11:53:52 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-002 --management-ips=10.108.72.27 --join-ips=10.108.216.52,10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 68243) 2023-07-24 11:54:14 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:54:20 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 11:54:21 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:54:25 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 11:54:25 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 11:54:25 mbc divider - INFO: Adding NFS protocol to frontend container 2023-07-24 11:54:25 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port add Ifgrp 21 ens3" 2023-07-24 11:54:25 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 11:54:25 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 11:54:25 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 11:54:26 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 11:54:26 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 11:54:28 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 2" 2023-07-24 11:54:28 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 11:54:28 mbc divider - INFO: running: /bin/sh -c "weka cluster host 2 -J" 2023-07-24 11:54:29 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 2 --no-unimprint" 2023-07-24 11:54:29 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 11:54:29 mbc divider - INFO: Finished cleanup 2023-07-24 11:54:29 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 11:54:34 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 ganesha, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 66823 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Removed from cluster: Not reachable by the cluster (36 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 65400 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 68243 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (30 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND ganesha (ganesha): Status: RUNNING PID: 70119 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.72.27 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.72.27 to rejoin cluster All Nodes belonging to 10.108.72.27 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.216.52 scp -p /tmp/mbc_divider_script.py 10.108.216.52:/tmp > /dev/null Warning: Permanently added '10.108.216.52' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.216.52 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:54:35 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 3 -J" 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:09:42:de/10.108.53.72/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 00:00:17:02:3a:6e/10.108.85.151/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:03:81:71/10.108.59.253/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:30:78/10.108.230.199/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:aa:e6/10.108.147.55/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 00:00:17:02:54:75/10.108.86.39/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:06:44:4d/10.108.42.74/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:03:e9:b9/10.108.36.58/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:02:22:98/10.108.234.84/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 00:00:17:02:f4:a3/10.108.109.243/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 00:00:17:02:b5:22/10.108.155.162/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:02:2e:4d/10.108.242.110/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:03:d8:4a/10.108.62.92/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:91:54/10.108.246.14/16/10.108.0.1 2023-07-24 11:54:35 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 11:54:35 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:54:36 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 11:54:36 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:54:36 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 11:54:36 mbc divider - INFO: Removing host from nfs interface group Ifgrp 2023-07-24 11:54:36 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port delete Ifgrp 3 ens3 -f" 2023-07-24 11:54:36 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:54:36 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 11:54:41 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:54:41 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 11:54:41 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 2023-07-24 11:54:41 mbc divider - INFO: Management IPs selected: --management-ips=10.108.216.52 2023-07-24 11:54:41 mbc divider - INFO: Stopping old container 2023-07-24 11:54:41 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 11:54:50 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:42:de/10.108.53.72/16/10.108.0.1 00:00:17:02:3a:6e/10.108.85.151/16/10.108.0.1 02:00:17:03:81:71/10.108.59.253/16/10.108.0.1 02:00:17:0a:30:78/10.108.230.199/16/10.108.0.1 02:00:17:0a:aa:e6/10.108.147.55/16/10.108.0.1 00:00:17:02:54:75/10.108.86.39/16/10.108.0.1 02:00:17:06:44:4d/10.108.42.74/16/10.108.0.1 02:00:17:03:e9:b9/10.108.36.58/16/10.108.0.1 02:00:17:02:22:98/10.108.234.84/16/10.108.0.1 00:00:17:02:f4:a3/10.108.109.243/16/10.108.0.1 00:00:17:02:b5:22/10.108.155.162/16/10.108.0.1 02:00:17:02:2e:4d/10.108.242.110/16/10.108.0.1 02:00:17:03:d8:4a/10.108.62.92/16/10.108.0.1 02:00:17:0a:91:54/10.108.246.14/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 11:54:50 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:42:de/10.108.53.72/16/10.108.0.1 00:00:17:02:3a:6e/10.108.85.151/16/10.108.0.1 02:00:17:03:81:71/10.108.59.253/16/10.108.0.1 02:00:17:0a:30:78/10.108.230.199/16/10.108.0.1 02:00:17:0a:aa:e6/10.108.147.55/16/10.108.0.1 00:00:17:02:54:75/10.108.86.39/16/10.108.0.1 02:00:17:06:44:4d/10.108.42.74/16/10.108.0.1 02:00:17:03:e9:b9/10.108.36.58/16/10.108.0.1 02:00:17:02:22:98/10.108.234.84/16/10.108.0.1 00:00:17:02:f4:a3/10.108.109.243/16/10.108.0.1 00:00:17:02:b5:22/10.108.155.162/16/10.108.0.1 02:00:17:02:2e:4d/10.108.242.110/16/10.108.0.1 02:00:17:03:d8:4a/10.108.62.92/16/10.108.0.1 02:00:17:0a:91:54/10.108.246.14/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [14, 18, 2, 6, 16, 8, 26, 28, 20, 24, 10, 4, 12, 22], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 11:54:50 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 11:54:50 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 11:54:50 mbc divider - INFO: Starting new containers 2023-07-24 11:54:50 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" 2023-07-24 11:54:50 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up drives0: Allocated network device "0000:00:04.0" (with identifier "0000:00:04.0") to slots [1] on "ms-sbc2-3":"drives0" (1/1) drives0: Allocated core 12 to slot 1 on "ms-sbc2-3":"drives0" (1/1) drives0: Starting hugepages allocation for "ms-sbc2-3":"drives0" drives0: Container "ms-sbc2-3":"drives0" allocated 712 out of 712 required hugepages after 1 retries drives0: Allocated 1424MB hugepages memory from 1 NUMA nodes for "ms-sbc2-3":"drives0" drives0: Bandwidth of "ms-sbc2-3":"drives0" set to unlimited drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-3":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-3":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-3":"drives0" Container "drives0" is ready (pid = 66402) 2023-07-24 11:55:18 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:55:22 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 11:55:23 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:55:29 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:55:33 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:55:33 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 3 -J" 2023-07-24 11:55:33 mbc divider - DEBUG: Waiting for nodes ['NodeId<60>', 'NodeId<61>', 'NodeId<62>', 'NodeId<63>', 'NodeId<64>', 'NodeId<65>', 'NodeId<66>', 'NodeId<67>', 'NodeId<68>', 'NodeId<69>', 'NodeId<70>', 'NodeId<71>', 'NodeId<72>', 'NodeId<73>', 'NodeId<74>'] to reach DOWN state 2023-07-24 11:55:33 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 3 -J" 2023-07-24 11:55:33 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 11:55:33 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 3 -J" 2023-07-24 11:55:33 mbc divider - INFO: The following drives will be moved to the new container: ['18fe8078-4f6c-4201-a828-2b823b14fca7', '8c9cd2e0-e977-47a9-993c-ce7700d079c5', 'c83fe2a5-2c29-4980-9e97-4a0c6b680dbc', '4e318100-73a2-4f82-a146-a0aa9e145635', '5f6879df-e8f5-4176-a47b-6b0d3ad2dc9b', '801093c4-052a-4443-b205-b33017e76cbb', 'b07a9bbf-af7a-4e37-bd77-5d2ab77028c2'] 2023-07-24 11:55:33 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 22" 2023-07-24 11:55:36 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 3 -J" 2023-07-24 11:55:36 mbc divider - INFO: Done scanning drives 2023-07-24 11:55:36 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" 2023-07-24 11:55:36 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 67853) 2023-07-24 11:55:45 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" 2023-07-24 11:55:45 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-003 --management-ips=10.108.216.52 --join-ips=10.108.119.146,10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 69189) 2023-07-24 11:56:06 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:56:11 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 11:56:12 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 11:56:17 mbc divider - INFO: Adding NFS protocol to frontend container 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group port add Ifgrp 24 ens3" 2023-07-24 11:56:17 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 11:56:17 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 11:56:17 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 11:56:20 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 3" 2023-07-24 11:56:20 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 11:56:20 mbc divider - INFO: running: /bin/sh -c "weka cluster host 3 -J" 2023-07-24 11:56:20 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 3 --no-unimprint" 2023-07-24 11:56:20 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 11:56:20 mbc divider - INFO: Finished cleanup 2023-07-24 11:56:20 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 11:56:25 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 ganesha, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 67853 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (37 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 66402 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 69189 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (26 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND ganesha (ganesha): Status: RUNNING PID: 70938 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.216.52 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.216.52 to rejoin cluster All Nodes belonging to 10.108.216.52 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.119.146 scp -p /tmp/mbc_divider_script.py 10.108.119.146:/tmp > /dev/null Warning: Permanently added '10.108.119.146' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.119.146 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 11:56:26 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:56:26 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:56:27 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 4 -J" 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:df:5b/10.108.18.98/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:03:d5:89/10.108.157.189/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:06:fa:1f/10.108.163.104/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:02:14:0f/10.108.154.111/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:09:10:30/10.108.6.116/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:03:7b:a8/10.108.74.92/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:41:cc/10.108.252.56/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:49:8e/10.108.102.105/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:48:08/10.108.103.76/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 00:00:17:02:bb:5e/10.108.220.41/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:09:3a:f7/10.108.224.76/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:09:94:4f/10.108.191.158/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:09:61:cc/10.108.171.46/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - DEBUG: Adding net devices: 02:00:17:03:4a:82/10.108.170.12/16/10.108.0.1 2023-07-24 11:56:27 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:56:27 mbc divider - INFO: Checking etcd Health 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:56:27 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 11:56:27 mbc divider - WARNING: Draining S3 container 2023-07-24 11:56:27 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster drain 4" 2023-07-24 11:57:47 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:48 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:48 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:48 mbc divider - INFO: Got 1 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:49 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:49 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:49 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:49 mbc divider - INFO: Got 2 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:50 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:50 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:51 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:51 mbc divider - INFO: Got 3 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:52 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:52 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:52 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:52 mbc divider - INFO: Got 4 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:53 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:53 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:53 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:53 mbc divider - INFO: Got 5 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:54 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:55 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:55 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:55 mbc divider - INFO: Got 6 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:56 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:56 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:56 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:56 mbc divider - INFO: Got 7 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:57 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:57 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:57 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:57 mbc divider - INFO: Got 8 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:57:58 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:57:59 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:57:59 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:57:59 mbc divider - INFO: Got 9 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=4" 2023-07-24 11:58:00 mbc divider - INFO: Got 10 successful drain checks in a row on host ms-sbc2-4; We need 10 2023-07-24 11:58:00 mbc divider - WARNING: Removing container from S3 cluster 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 5,6,7" 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 11:58:00 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 11:58:01 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 11:58:01 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 11:58:06 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:58:06 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 11:58:06 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 2023-07-24 11:58:06 mbc divider - INFO: Management IPs selected: --management-ips=10.108.119.146 2023-07-24 11:58:06 mbc divider - INFO: Stopping old container 2023-07-24 11:58:06 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 11:58:14 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:0b:df:5b/10.108.18.98/16/10.108.0.1 02:00:17:03:d5:89/10.108.157.189/16/10.108.0.1 02:00:17:06:fa:1f/10.108.163.104/16/10.108.0.1 02:00:17:02:14:0f/10.108.154.111/16/10.108.0.1 02:00:17:09:10:30/10.108.6.116/16/10.108.0.1 02:00:17:03:7b:a8/10.108.74.92/16/10.108.0.1 02:00:17:0a:41:cc/10.108.252.56/16/10.108.0.1 02:00:17:0b:49:8e/10.108.102.105/16/10.108.0.1 02:00:17:0a:48:08/10.108.103.76/16/10.108.0.1 00:00:17:02:bb:5e/10.108.220.41/16/10.108.0.1 02:00:17:09:3a:f7/10.108.224.76/16/10.108.0.1 02:00:17:09:94:4f/10.108.191.158/16/10.108.0.1 02:00:17:09:61:cc/10.108.171.46/16/10.108.0.1 02:00:17:03:4a:82/10.108.170.12/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 11:58:14 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:0b:df:5b/10.108.18.98/16/10.108.0.1 02:00:17:03:d5:89/10.108.157.189/16/10.108.0.1 02:00:17:06:fa:1f/10.108.163.104/16/10.108.0.1 02:00:17:02:14:0f/10.108.154.111/16/10.108.0.1 02:00:17:09:10:30/10.108.6.116/16/10.108.0.1 02:00:17:03:7b:a8/10.108.74.92/16/10.108.0.1 02:00:17:0a:41:cc/10.108.252.56/16/10.108.0.1 02:00:17:0b:49:8e/10.108.102.105/16/10.108.0.1 02:00:17:0a:48:08/10.108.103.76/16/10.108.0.1 00:00:17:02:bb:5e/10.108.220.41/16/10.108.0.1 02:00:17:09:3a:f7/10.108.224.76/16/10.108.0.1 02:00:17:09:94:4f/10.108.191.158/16/10.108.0.1 02:00:17:09:61:cc/10.108.171.46/16/10.108.0.1 02:00:17:03:4a:82/10.108.170.12/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [24, 8, 22, 26, 12, 4, 14, 6, 10, 18, 28, 2, 20, 16], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 11:58:15 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 11:58:15 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 11:58:15 mbc divider - INFO: Starting new containers 2023-07-24 11:58:15 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" 2023-07-24 11:58:15 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 69091) 2023-07-24 11:58:37 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:58:41 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 11:58:42 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:58:47 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:58:52 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 11:58:52 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 4 -J" 2023-07-24 11:58:52 mbc divider - DEBUG: Waiting for nodes ['NodeId<80>', 'NodeId<81>', 'NodeId<82>', 'NodeId<83>', 'NodeId<84>', 'NodeId<85>', 'NodeId<86>', 'NodeId<87>', 'NodeId<88>', 'NodeId<89>', 'NodeId<90>', 'NodeId<91>', 'NodeId<92>', 'NodeId<93>', 'NodeId<94>'] to reach DOWN state 2023-07-24 11:58:52 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 4 -J" 2023-07-24 11:58:52 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 11:58:52 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 4 -J" 2023-07-24 11:58:52 mbc divider - INFO: The following drives will be moved to the new container: ['f51aa679-7fa9-4f95-a930-ad7c12dfada6', 'bf091e60-2791-4d24-b06a-45c5434f1471', '84e92f4c-0d89-43f0-8641-64340da07aa9', '7aed7714-f5e0-4d26-8056-dcaf49277ff3', '30b3a6bc-f6db-4683-ab73-b8c3406087b2', '6f6d3ba7-ed1d-4bfb-8cee-dfbbdbe4a75d', '152eb661-bd0e-47f9-b37b-7d3f8b2a17db'] 2023-07-24 11:58:52 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 25" 2023-07-24 11:58:55 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 4 -J" 2023-07-24 11:58:55 mbc divider - INFO: Done scanning drives 2023-07-24 11:58:55 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" 2023-07-24 11:58:55 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 70464) 2023-07-24 11:59:04 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" 2023-07-24 11:59:04 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-004 --management-ips=10.108.119.146 --join-ips=10.108.61.86,10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 71798) 2023-07-24 11:59:30 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:59:35 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 11:59:36 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 11:59:40 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 11:59:40 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 11:59:41 mbc divider - INFO: Adding S3 protocol to frontend container 2023-07-24 11:59:41 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 5,6,7,27" 2023-07-24 11:59:41 mbc divider - INFO: Waiting for s3 container to be Ready 2023-07-24 11:59:46 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:59:51 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:59:56 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 11:59:56 mbc divider - INFO: Checking etcd Health 2023-07-24 11:59:56 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 11:59:56 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 11:59:56 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 11:59:57 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 4" 2023-07-24 11:59:57 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka cluster host 4 -J" 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 4 --no-unimprint" 2023-07-24 11:59:57 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 11:59:57 mbc divider - INFO: Finished cleanup 2023-07-24 11:59:57 mbc divider - INFO: running: /bin/sh -c "weka local status -v" Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 s3, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 70464 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Removed from cluster: Not reachable by the cluster (49 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 69091 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 71798 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (47 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND s3 (s3): Status: RUNNING PID: 73578 Version: 4.1.0.77 Last Message: API Port: 14000 2023-07-24 12:00:02 mbc divider - INFO: Finished moving server to MBC architecture HOST 10.108.119.146 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.119.146 to rejoin cluster All Nodes belonging to 10.108.119.146 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.61.86 scp -p /tmp/mbc_divider_script.py 10.108.61.86:/tmp > /dev/null Warning: Permanently added '10.108.61.86' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.61.86 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:00:03 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:00:03 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:00:03 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:00:03 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:00:03 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 5 -J" 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:06:2e:fe/10.108.48.187/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:02:89:09/10.108.131.175/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:02:57:a1/10.108.171.134/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:b2:b8/10.108.190.177/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:06:1a:7b/10.108.24.153/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:03:53:22/10.108.235.154/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:02:eb:41/10.108.112.67/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:02:d0:bd/10.108.86.237/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:6e:43/10.108.18.197/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:02:27:ec/10.108.134.169/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:b9:f5/10.108.96.125/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:09:38:fc/10.108.220.52/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:03:60:39/10.108.64.250/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - DEBUG: Adding net devices: 02:00:17:03:e5:d0/10.108.157.73/16/10.108.0.1 2023-07-24 12:00:04 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:00:04 mbc divider - INFO: Checking etcd Health 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:00:04 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:00:04 mbc divider - WARNING: Draining S3 container 2023-07-24 12:00:04 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster drain 5" 2023-07-24 12:01:24 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:24 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:25 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:25 mbc divider - INFO: Got 1 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:26 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:26 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:26 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:26 mbc divider - INFO: Got 2 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:27 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:27 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:27 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:27 mbc divider - INFO: Got 3 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:28 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:29 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:29 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:29 mbc divider - INFO: Got 4 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:30 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:30 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:30 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:30 mbc divider - INFO: Got 5 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:31 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:31 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:31 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:32 mbc divider - INFO: Got 6 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:33 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:33 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:33 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:33 mbc divider - INFO: Got 7 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:34 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:34 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:34 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:34 mbc divider - INFO: Got 8 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:35 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:35 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:35 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:36 mbc divider - INFO: Got 9 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:37 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:01:37 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:01:37 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=5" 2023-07-24 12:01:37 mbc divider - INFO: Got 10 successful drain checks in a row on host ms-sbc2-5; We need 10 2023-07-24 12:01:37 mbc divider - WARNING: Removing container from S3 cluster 2023-07-24 12:01:37 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,6,7" 2023-07-24 12:01:44 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:01:44 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:01:44 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:01:44 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:01:44 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:01:49 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:01:49 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:01:49 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 2023-07-24 12:01:49 mbc divider - INFO: Management IPs selected: --management-ips=10.108.61.86 2023-07-24 12:01:49 mbc divider - INFO: Stopping old container 2023-07-24 12:01:49 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:01:58 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:2e:fe/10.108.48.187/16/10.108.0.1 02:00:17:02:89:09/10.108.131.175/16/10.108.0.1 02:00:17:02:57:a1/10.108.171.134/16/10.108.0.1 02:00:17:0a:b2:b8/10.108.190.177/16/10.108.0.1 02:00:17:06:1a:7b/10.108.24.153/16/10.108.0.1 02:00:17:03:53:22/10.108.235.154/16/10.108.0.1 02:00:17:02:eb:41/10.108.112.67/16/10.108.0.1 02:00:17:02:d0:bd/10.108.86.237/16/10.108.0.1 02:00:17:0b:6e:43/10.108.18.197/16/10.108.0.1 02:00:17:02:27:ec/10.108.134.169/16/10.108.0.1 02:00:17:0b:b9:f5/10.108.96.125/16/10.108.0.1 02:00:17:09:38:fc/10.108.220.52/16/10.108.0.1 02:00:17:03:60:39/10.108.64.250/16/10.108.0.1 02:00:17:03:e5:d0/10.108.157.73/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:01:58 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:06:2e:fe/10.108.48.187/16/10.108.0.1 02:00:17:02:89:09/10.108.131.175/16/10.108.0.1 02:00:17:02:57:a1/10.108.171.134/16/10.108.0.1 02:00:17:0a:b2:b8/10.108.190.177/16/10.108.0.1 02:00:17:06:1a:7b/10.108.24.153/16/10.108.0.1 02:00:17:03:53:22/10.108.235.154/16/10.108.0.1 02:00:17:02:eb:41/10.108.112.67/16/10.108.0.1 02:00:17:02:d0:bd/10.108.86.237/16/10.108.0.1 02:00:17:0b:6e:43/10.108.18.197/16/10.108.0.1 02:00:17:02:27:ec/10.108.134.169/16/10.108.0.1 02:00:17:0b:b9:f5/10.108.96.125/16/10.108.0.1 02:00:17:09:38:fc/10.108.220.52/16/10.108.0.1 02:00:17:03:60:39/10.108.64.250/16/10.108.0.1 02:00:17:03:e5:d0/10.108.157.73/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [10, 16, 20, 2, 26, 8, 14, 6, 18, 22, 24, 12, 4, 28], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:01:58 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:01:58 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:01:58 mbc divider - INFO: Starting new containers 2023-07-24 12:01:58 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" 2023-07-24 12:01:58 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 71141) 2023-07-24 12:02:20 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:02:25 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:02:26 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:02:31 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:02:35 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:02:35 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 5 -J" 2023-07-24 12:02:36 mbc divider - DEBUG: Waiting for nodes ['NodeId<100>', 'NodeId<101>', 'NodeId<102>', 'NodeId<103>', 'NodeId<104>', 'NodeId<105>', 'NodeId<106>', 'NodeId<107>', 'NodeId<108>', 'NodeId<109>', 'NodeId<110>', 'NodeId<111>', 'NodeId<112>', 'NodeId<113>', 'NodeId<114>'] to reach DOWN state 2023-07-24 12:02:36 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 5 -J" 2023-07-24 12:02:36 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:02:36 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 5 -J" 2023-07-24 12:02:36 mbc divider - INFO: The following drives will be moved to the new container: ['55f6c87e-8a48-43ab-ae50-bf8ca1b37430', 'f4bdaea8-fe43-46cf-8572-36f7882cba5f', 'b93ffd5c-95f5-4b19-9012-9e3674b3e5f1', '46a57713-c9a3-4304-b457-c532b204111e', 'e8dae016-f802-4cc7-8700-98994c77c174', '481a8d19-74c5-45f6-884a-cd431a684cc8', '673e5314-8191-4f4b-9257-400967778116'] 2023-07-24 12:02:36 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 28" 2023-07-24 12:02:38 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 5 -J" 2023-07-24 12:02:38 mbc divider - INFO: Done scanning drives 2023-07-24 12:02:38 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" 2023-07-24 12:02:38 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 72573) 2023-07-24 12:02:49 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" 2023-07-24 12:02:49 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-005 --management-ips=10.108.61.86 --join-ips=10.108.131.98,10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 74008) 2023-07-24 12:03:13 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:03:18 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 12:03:19 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:03:24 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 12:03:24 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 12:03:24 mbc divider - INFO: Adding S3 protocol to frontend container 2023-07-24 12:03:24 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,6,7,30" 2023-07-24 12:03:24 mbc divider - INFO: Waiting for s3 container to be Ready 2023-07-24 12:03:29 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:03:34 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:03:39 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:03:40 mbc divider - INFO: Checking etcd Health 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:03:40 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:03:40 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 5" 2023-07-24 12:03:40 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka cluster host 5 -J" 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 5 --no-unimprint" 2023-07-24 12:03:40 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:03:40 mbc divider - INFO: Finished cleanup 2023-07-24 12:03:40 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 12:03:45 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 s3, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 72573 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (55 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 71141 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 74008 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (43 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND s3 (s3): Status: RUNNING PID: 75716 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.61.86 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.61.86 to rejoin cluster All Nodes belonging to 10.108.61.86 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.131.98 scp -p /tmp/mbc_divider_script.py 10.108.131.98:/tmp > /dev/null Warning: Permanently added '10.108.131.98' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.131.98 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:03:46 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:03:46 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:03:46 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:03:46 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:03:46 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 6 -J" 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 00:00:17:02:8d:cd/10.108.168.54/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 00:00:17:02:d6:69/10.108.222.94/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:03:bb:30/10.108.213.27/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:09:32:7c/10.108.126.188/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:09:4f:05/10.108.164.81/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:09:11:a7/10.108.245.153/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:03:4b:e9/10.108.252.107/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:06:8e:e7/10.108.219.94/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:28:92/10.108.176.100/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:06:75:b9/10.108.204.123/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:06:a2:f7/10.108.194.118/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:03:77:79/10.108.66.137/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:09:11:3a/10.108.152.252/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - DEBUG: Adding net devices: 02:00:17:09:d2:8a/10.108.2.1/16/10.108.0.1 2023-07-24 12:03:47 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:03:47 mbc divider - INFO: Checking etcd Health 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:03:47 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:03:47 mbc divider - WARNING: Draining S3 container 2023-07-24 12:03:47 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster drain 6" 2023-07-24 12:05:07 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:07 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:08 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:08 mbc divider - INFO: Got 1 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:09 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:09 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:09 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:09 mbc divider - INFO: Got 2 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:10 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:10 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:10 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:10 mbc divider - INFO: Got 3 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:11 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:12 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:12 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:12 mbc divider - INFO: Got 4 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:13 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:13 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:13 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:13 mbc divider - INFO: Got 5 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:14 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:14 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:14 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:15 mbc divider - INFO: Got 6 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:16 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:16 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:16 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:16 mbc divider - INFO: Got 7 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:17 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:17 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:17 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:17 mbc divider - INFO: Got 8 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:18 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:18 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:18 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:19 mbc divider - INFO: Got 9 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:20 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:05:20 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:05:20 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=6" 2023-07-24 12:05:20 mbc divider - INFO: Got 10 successful drain checks in a row on host ms-sbc2-6; We need 10 2023-07-24 12:05:20 mbc divider - WARNING: Removing container from S3 cluster 2023-07-24 12:05:20 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,30,7" 2023-07-24 12:05:27 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:05:27 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:05:27 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:05:27 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:05:27 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:05:32 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:05:32 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:05:32 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 2023-07-24 12:05:32 mbc divider - INFO: Management IPs selected: --management-ips=10.108.131.98 2023-07-24 12:05:32 mbc divider - INFO: Stopping old container 2023-07-24 12:05:32 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:05:41 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 00:00:17:02:8d:cd/10.108.168.54/16/10.108.0.1 00:00:17:02:d6:69/10.108.222.94/16/10.108.0.1 02:00:17:03:bb:30/10.108.213.27/16/10.108.0.1 02:00:17:09:32:7c/10.108.126.188/16/10.108.0.1 02:00:17:09:4f:05/10.108.164.81/16/10.108.0.1 02:00:17:09:11:a7/10.108.245.153/16/10.108.0.1 02:00:17:03:4b:e9/10.108.252.107/16/10.108.0.1 02:00:17:06:8e:e7/10.108.219.94/16/10.108.0.1 02:00:17:0a:28:92/10.108.176.100/16/10.108.0.1 02:00:17:06:75:b9/10.108.204.123/16/10.108.0.1 02:00:17:06:a2:f7/10.108.194.118/16/10.108.0.1 02:00:17:03:77:79/10.108.66.137/16/10.108.0.1 02:00:17:09:11:3a/10.108.152.252/16/10.108.0.1 02:00:17:09:d2:8a/10.108.2.1/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:05:41 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 00:00:17:02:8d:cd/10.108.168.54/16/10.108.0.1 00:00:17:02:d6:69/10.108.222.94/16/10.108.0.1 02:00:17:03:bb:30/10.108.213.27/16/10.108.0.1 02:00:17:09:32:7c/10.108.126.188/16/10.108.0.1 02:00:17:09:4f:05/10.108.164.81/16/10.108.0.1 02:00:17:09:11:a7/10.108.245.153/16/10.108.0.1 02:00:17:03:4b:e9/10.108.252.107/16/10.108.0.1 02:00:17:06:8e:e7/10.108.219.94/16/10.108.0.1 02:00:17:0a:28:92/10.108.176.100/16/10.108.0.1 02:00:17:06:75:b9/10.108.204.123/16/10.108.0.1 02:00:17:06:a2:f7/10.108.194.118/16/10.108.0.1 02:00:17:03:77:79/10.108.66.137/16/10.108.0.1 02:00:17:09:11:3a/10.108.152.252/16/10.108.0.1 02:00:17:09:d2:8a/10.108.2.1/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [26, 8, 10, 6, 14, 24, 12, 22, 18, 16, 20, 4, 28, 2], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:05:41 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:05:41 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:05:41 mbc divider - INFO: Starting new containers 2023-07-24 12:05:41 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" 2023-07-24 12:05:41 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 72433) 2023-07-24 12:06:03 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:06:08 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:06:09 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:06:14 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:06:18 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:06:18 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 6 -J" 2023-07-24 12:06:18 mbc divider - DEBUG: Waiting for nodes ['NodeId<120>', 'NodeId<121>', 'NodeId<122>', 'NodeId<123>', 'NodeId<124>', 'NodeId<125>', 'NodeId<126>', 'NodeId<127>', 'NodeId<128>', 'NodeId<129>', 'NodeId<130>', 'NodeId<131>', 'NodeId<132>', 'NodeId<133>', 'NodeId<134>'] to reach DOWN state 2023-07-24 12:06:18 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 6 -J" 2023-07-24 12:06:19 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:06:19 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 6 -J" 2023-07-24 12:06:19 mbc divider - INFO: The following drives will be moved to the new container: ['bbb000f4-6790-44f0-b889-ac017c4ba40f', '10468999-a936-4d22-8623-b25345ae3c86', 'be5fb97e-e50f-4002-843a-c1958eac0f40', '83cbd387-ef07-4bd4-a680-f3a54fd9a241', 'da713982-d77b-4c6c-95c9-5abb938337a1', '803dd45f-50a5-48b6-97bd-ca82fa76c735', 'c26efcf8-107b-4d0d-a78d-8fc3a08eddc9'] 2023-07-24 12:06:19 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 31" 2023-07-24 12:06:21 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 6 -J" 2023-07-24 12:06:21 mbc divider - INFO: Done scanning drives 2023-07-24 12:06:21 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" 2023-07-24 12:06:21 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 73860) 2023-07-24 12:06:31 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" 2023-07-24 12:06:31 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-006 --management-ips=10.108.131.98 --join-ips=10.108.60.82,10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 75179) 2023-07-24 12:07:01 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:07:07 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 12:07:08 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:07:13 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 12:07:13 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 12:07:13 mbc divider - INFO: Adding S3 protocol to frontend container 2023-07-24 12:07:13 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,30,7,33" 2023-07-24 12:07:14 mbc divider - INFO: Waiting for s3 container to be Ready 2023-07-24 12:07:19 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:07:25 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:07:30 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:07:31 mbc divider - INFO: Checking etcd Health 2023-07-24 12:07:31 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:07:31 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:07:31 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:07:31 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:07:31 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:07:31 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:07:32 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 6" 2023-07-24 12:07:32 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:07:32 mbc divider - INFO: running: /bin/sh -c "weka cluster host 6 -J" 2023-07-24 12:07:32 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 6 --no-unimprint" 2023-07-24 12:07:32 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:07:32 mbc divider - INFO: Finished cleanup 2023-07-24 12:07:32 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 12:07:38 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 s3, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 73860 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (2 minutes ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 72433 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 75179 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (51 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND s3 (s3): Status: RUNNING PID: 76945 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.131.98 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.131.98 to rejoin cluster All Nodes belonging to 10.108.131.98 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.60.82 scp -p /tmp/mbc_divider_script.py 10.108.60.82:/tmp > /dev/null Warning: Permanently added '10.108.60.82' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.60.82 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:07:39 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:07:39 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:07:39 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:07:39 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:07:39 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:07:39 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 7 -J" 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 00:00:17:02:3c:c9/10.108.68.182/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 00:00:17:02:7c:ad/10.108.67.38/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:02:39:91/10.108.166.132/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:09:32:e2/10.108.187.102/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:cd:ca/10.108.163.90/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:06:9f:a2/10.108.118.234/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:87:ae/10.108.240.123/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:58:2c/10.108.163.194/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:02:db:5c/10.108.171.30/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:48:a8/10.108.176.113/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:12:40/10.108.44.159/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:6a:ec/10.108.146.93/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 00:00:17:02:84:0b/10.108.6.152/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:48:1f/10.108.107.63/16/10.108.0.1 2023-07-24 12:07:40 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:07:40 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:07:40 mbc divider - INFO: Checking etcd Health 2023-07-24 12:07:40 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:07:40 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:07:40 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:07:40 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:07:40 mbc divider - WARNING: Draining S3 container 2023-07-24 12:07:40 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster drain 7" 2023-07-24 12:09:00 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:00 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:01 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:01 mbc divider - INFO: Got 1 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:02 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:02 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:02 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:02 mbc divider - INFO: Got 2 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:03 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:03 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:03 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:03 mbc divider - INFO: Got 3 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:04 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:05 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:05 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:05 mbc divider - INFO: Got 4 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:06 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:06 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:06 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:06 mbc divider - INFO: Got 5 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:07 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:07 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:07 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:07 mbc divider - INFO: Got 6 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:08 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:09 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:09 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:09 mbc divider - INFO: Got 7 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:10 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:10 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:10 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:10 mbc divider - INFO: Got 8 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:11 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:11 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:11 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:12 mbc divider - INFO: Got 9 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:13 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster -J" 2023-07-24 12:09:13 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 curl -sk http://127.0.0.1:9000/minio/drain/mode" 2023-07-24 12:09:13 mbc divider - INFO: running: /bin/sh -c "weka debug jrpc container_get_drain_status hostId=7" 2023-07-24 12:09:13 mbc divider - INFO: Got 10 successful drain checks in a row on host ms-sbc2-7; We need 10 2023-07-24 12:09:13 mbc divider - WARNING: Removing container from S3 cluster 2023-07-24 12:09:13 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,30,33" 2023-07-24 12:09:20 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:09:20 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:09:20 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:09:20 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:09:20 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:09:25 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:09:25 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:09:25 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 2023-07-24 12:09:25 mbc divider - INFO: Management IPs selected: --management-ips=10.108.60.82 2023-07-24 12:09:25 mbc divider - INFO: Stopping old container 2023-07-24 12:09:25 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:09:34 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 00:00:17:02:3c:c9/10.108.68.182/16/10.108.0.1 00:00:17:02:7c:ad/10.108.67.38/16/10.108.0.1 02:00:17:02:39:91/10.108.166.132/16/10.108.0.1 02:00:17:09:32:e2/10.108.187.102/16/10.108.0.1 02:00:17:0a:cd:ca/10.108.163.90/16/10.108.0.1 02:00:17:06:9f:a2/10.108.118.234/16/10.108.0.1 02:00:17:0b:87:ae/10.108.240.123/16/10.108.0.1 02:00:17:0a:58:2c/10.108.163.194/16/10.108.0.1 02:00:17:02:db:5c/10.108.171.30/16/10.108.0.1 02:00:17:0a:48:a8/10.108.176.113/16/10.108.0.1 02:00:17:0a:12:40/10.108.44.159/16/10.108.0.1 02:00:17:0a:6a:ec/10.108.146.93/16/10.108.0.1 00:00:17:02:84:0b/10.108.6.152/16/10.108.0.1 02:00:17:0a:48:1f/10.108.107.63/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:09:34 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 00:00:17:02:3c:c9/10.108.68.182/16/10.108.0.1 00:00:17:02:7c:ad/10.108.67.38/16/10.108.0.1 02:00:17:02:39:91/10.108.166.132/16/10.108.0.1 02:00:17:09:32:e2/10.108.187.102/16/10.108.0.1 02:00:17:0a:cd:ca/10.108.163.90/16/10.108.0.1 02:00:17:06:9f:a2/10.108.118.234/16/10.108.0.1 02:00:17:0b:87:ae/10.108.240.123/16/10.108.0.1 02:00:17:0a:58:2c/10.108.163.194/16/10.108.0.1 02:00:17:02:db:5c/10.108.171.30/16/10.108.0.1 02:00:17:0a:48:a8/10.108.176.113/16/10.108.0.1 02:00:17:0a:12:40/10.108.44.159/16/10.108.0.1 02:00:17:0a:6a:ec/10.108.146.93/16/10.108.0.1 00:00:17:02:84:0b/10.108.6.152/16/10.108.0.1 02:00:17:0a:48:1f/10.108.107.63/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [6, 2, 26, 20, 10, 28, 18, 4, 12, 16, 14, 22, 8, 24], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:09:34 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:09:34 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:09:34 mbc divider - INFO: Starting new containers 2023-07-24 12:09:34 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:09:34 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 75877) 2023-07-24 12:09:56 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:10:00 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:10:01 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:10:07 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:10:11 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:10:11 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 7 -J" 2023-07-24 12:10:11 mbc divider - DEBUG: Waiting for nodes ['NodeId<140>', 'NodeId<141>', 'NodeId<142>', 'NodeId<143>', 'NodeId<144>', 'NodeId<145>', 'NodeId<146>', 'NodeId<147>', 'NodeId<148>', 'NodeId<149>', 'NodeId<150>', 'NodeId<151>', 'NodeId<152>', 'NodeId<153>', 'NodeId<154>'] to reach DOWN state 2023-07-24 12:10:11 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 7 -J" 2023-07-24 12:10:11 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:10:11 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 7 -J" 2023-07-24 12:10:12 mbc divider - INFO: The following drives will be moved to the new container: ['c4659bad-b18d-45af-bcb4-f1da72707889', '67f6891c-85f1-41da-b9c8-1da928223d40', '1f166352-811e-4cc6-9ec3-e6631d7d7a3b', '96892e9d-d630-46b3-98e5-4722652d7461', '07babbff-39a5-4fe6-8e4f-31b69d87027c', '3e91766f-dc8f-43f1-9aca-6f29f7e1a06e', 'c37d2cae-30e3-4f1a-b7ef-774b455173b0'] 2023-07-24 12:10:12 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 34" 2023-07-24 12:10:14 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 7 -J" 2023-07-24 12:10:14 mbc divider - INFO: Done scanning drives 2023-07-24 12:10:14 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:10:14 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 77410) 2023-07-24 12:10:26 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:10:26 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-007 --management-ips=10.108.60.82 --join-ips=10.108.12.47,10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 78759) 2023-07-24 12:10:48 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:10:53 mbc divider - INFO: Waiting for ContainerType.FRONTEND containers to reach READY state, currently:READY 2023-07-24 12:10:54 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:10:58 mbc divider - INFO: running: /bin/sh -c "weka local resources -C frontend0 -J" 2023-07-24 12:10:59 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0 -P 14200" 2023-07-24 12:10:59 mbc divider - INFO: Adding S3 protocol to frontend container 2023-07-24 12:10:59 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster update --host 27,30,33,36" 2023-07-24 12:10:59 mbc divider - INFO: Waiting for s3 container to be Ready 2023-07-24 12:11:04 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:11:09 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:11:14 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:11:14 mbc divider - INFO: Checking etcd Health 2023-07-24 12:11:14 mbc divider - INFO: running: /bin/sh -c " weka local exec -C s3 etcdctl endpoint health --cluster -w json" 2023-07-24 12:11:15 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:11:15 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 7" 2023-07-24 12:11:15 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka cluster host 7 -J" 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 7 --no-unimprint" 2023-07-24 12:11:15 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:11:15 mbc divider - INFO: Finished cleanup 2023-07-24 12:11:15 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 12:11:20 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 4/5 running (1 s3, 4 weka) Nodes: 19/32 running (13 UNKNOWN, 19 READY) Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 77410 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (53 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 2: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 3: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 4: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 75877 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 78759 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (43 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND s3 (s3): Status: RUNNING PID: 80393 Version: 4.1.0.77 Last Message: API Port: 14000 HOST 10.108.60.82 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.60.82 to rejoin cluster All Nodes belonging to 10.108.60.82 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.12.47 scp -p /tmp/mbc_divider_script.py 10.108.12.47:/tmp > /dev/null Warning: Permanently added '10.108.12.47' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.12.47 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:11:21 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 8 -J" 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:09:cd:c4/10.108.203.103/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:05:aa/10.108.125.77/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:d2:ac/10.108.60.16/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:02:5e:1a/10.108.43.1/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:af:60/10.108.148.5/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:e6:b1/10.108.117.226/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:03:0c:b8/10.108.193.117/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:06:25:94/10.108.201.126/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:b3:77/10.108.141.53/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:06:24:97/10.108.177.55/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 00:00:17:02:b6:39/10.108.34.29/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:e0:11/10.108.100.72/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:06:cc:ed/10.108.244.56/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:d6:c1/10.108.202.190/16/10.108.0.1 2023-07-24 12:11:21 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:11:21 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:11:22 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:11:22 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:11:22 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:11:22 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:11:22 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:11:22 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:11:27 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:11:27 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:11:27 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 2023-07-24 12:11:27 mbc divider - INFO: Management IPs selected: --management-ips=10.108.12.47 2023-07-24 12:11:27 mbc divider - INFO: Stopping old container 2023-07-24 12:11:27 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:11:36 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:cd:c4/10.108.203.103/16/10.108.0.1 02:00:17:0a:05:aa/10.108.125.77/16/10.108.0.1 02:00:17:0b:d2:ac/10.108.60.16/16/10.108.0.1 02:00:17:02:5e:1a/10.108.43.1/16/10.108.0.1 02:00:17:0b:af:60/10.108.148.5/16/10.108.0.1 02:00:17:0b:e6:b1/10.108.117.226/16/10.108.0.1 02:00:17:03:0c:b8/10.108.193.117/16/10.108.0.1 02:00:17:06:25:94/10.108.201.126/16/10.108.0.1 02:00:17:0b:b3:77/10.108.141.53/16/10.108.0.1 02:00:17:06:24:97/10.108.177.55/16/10.108.0.1 00:00:17:02:b6:39/10.108.34.29/16/10.108.0.1 02:00:17:0a:e0:11/10.108.100.72/16/10.108.0.1 02:00:17:06:cc:ed/10.108.244.56/16/10.108.0.1 02:00:17:0a:d6:c1/10.108.202.190/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:11:36 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:09:cd:c4/10.108.203.103/16/10.108.0.1 02:00:17:0a:05:aa/10.108.125.77/16/10.108.0.1 02:00:17:0b:d2:ac/10.108.60.16/16/10.108.0.1 02:00:17:02:5e:1a/10.108.43.1/16/10.108.0.1 02:00:17:0b:af:60/10.108.148.5/16/10.108.0.1 02:00:17:0b:e6:b1/10.108.117.226/16/10.108.0.1 02:00:17:03:0c:b8/10.108.193.117/16/10.108.0.1 02:00:17:06:25:94/10.108.201.126/16/10.108.0.1 02:00:17:0b:b3:77/10.108.141.53/16/10.108.0.1 02:00:17:06:24:97/10.108.177.55/16/10.108.0.1 00:00:17:02:b6:39/10.108.34.29/16/10.108.0.1 02:00:17:0a:e0:11/10.108.100.72/16/10.108.0.1 02:00:17:06:cc:ed/10.108.244.56/16/10.108.0.1 02:00:17:0a:d6:c1/10.108.202.190/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [20, 6, 24, 4, 14, 2, 8, 10, 28, 18, 22, 26, 16, 12], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:11:36 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:11:36 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:11:36 mbc divider - INFO: Starting new containers 2023-07-24 12:11:36 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:11:36 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up drives0: Allocated network device "0000:00:08.0" (with identifier "0000:00:08.0") to slots [1] on "ms-sbc2-8":"drives0" (1/1) drives0: Allocated core 16 to slot 1 on "ms-sbc2-8":"drives0" (1/1) drives0: Starting hugepages allocation for "ms-sbc2-8":"drives0" drives0: Container "ms-sbc2-8":"drives0" allocated 712 out of 712 required hugepages after 1 retries drives0: Allocated 1424MB hugepages memory from 1 NUMA nodes for "ms-sbc2-8":"drives0" drives0: Bandwidth of "ms-sbc2-8":"drives0" set to unlimited drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-8":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-8":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-8":"drives0" Container "drives0" is ready (pid = 65387) 2023-07-24 12:12:04 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:12:08 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:12:09 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:12:15 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:12:19 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:12:19 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 8 -J" 2023-07-24 12:12:19 mbc divider - DEBUG: Waiting for nodes ['NodeId<160>', 'NodeId<161>', 'NodeId<162>', 'NodeId<163>', 'NodeId<164>', 'NodeId<165>', 'NodeId<166>', 'NodeId<167>', 'NodeId<168>', 'NodeId<169>', 'NodeId<170>', 'NodeId<171>', 'NodeId<172>', 'NodeId<173>', 'NodeId<174>'] to reach DOWN state 2023-07-24 12:12:19 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 8 -J" 2023-07-24 12:12:19 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:12:19 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 8 -J" 2023-07-24 12:12:20 mbc divider - INFO: The following drives will be moved to the new container: ['18edf355-8558-41e9-8bcb-878c52283029', '68af4d65-57b7-4b3b-bf38-92106a835c3a', '03c2eb68-fda7-443a-897f-a389e4e7244a', 'ed4351cd-b34a-49b5-bae3-bc5a0974a469', '7ffd706c-146e-48c2-9067-a6ec7789cf3d', '3fe58865-2e74-44ce-a260-3de15d93f26a', 'b2f92fc7-4f5c-46aa-ac9b-473e36f1d07c'] 2023-07-24 12:12:20 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 37" 2023-07-24 12:12:22 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 8 -J" 2023-07-24 12:12:22 mbc divider - INFO: Done scanning drives 2023-07-24 12:12:22 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:12:22 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 66859) 2023-07-24 12:12:32 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" 2023-07-24 12:12:32 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-008 --management-ips=10.108.12.47 --join-ips=10.108.208.219,10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 68299) 2023-07-24 12:12:42 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:12:42 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 8" 2023-07-24 12:12:42 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka cluster host 8 -J" 2023-07-24 12:12:42 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 8 --no-unimprint" 2023-07-24 12:12:43 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:12:43 mbc divider - INFO: Finished cleanup 2023-07-24 12:12:43 mbc divider - INFO: running: /bin/sh -c "weka local status -v" Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 3/4 running (4 weka) Nodes: 6/32 running (6 READY, 1 CLUSTERIZED, 24 UNKNOWN, 1 INIT) NOTICE: Some nodes services are not ready yet. Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 66859 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: CLUSTERIZED Status Message: Successfully joined cluster Last Message: Removed from cluster: Not reachable by the cluster (15 seconds ago) Roles: MANAGEMENT Slot 1: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 2: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 65387 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 68299 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (12 seconds ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND 2023-07-24 12:12:55 mbc divider - INFO: Finished moving server to MBC architecture HOST 10.108.12.47 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.12.47 to rejoin cluster All Nodes belonging to 10.108.12.47 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.208.219 scp -p /tmp/mbc_divider_script.py 10.108.208.219:/tmp > /dev/null Warning: Permanently added '10.108.208.219' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.208.219 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:12:56 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:12:56 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:12:56 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:12:56 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:12:56 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:12:56 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 9 -J" 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:03:cd:5a/10.108.113.110/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 00:00:17:02:c2:ad/10.108.222.101/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:4e:2f/10.108.206.113/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:b4:2c/10.108.56.193/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:06:ae:28/10.108.99.123/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:09:1b:8d/10.108.196.44/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:bf:52/10.108.186.142/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:03:fc:56/10.108.253.58/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:06:80:65/10.108.210.85/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 00:00:17:02:c4:72/10.108.79.2/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:06:24:32/10.108.122.68/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:09:ac:88/10.108.216.221/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:06:e0:6f/10.108.63.231/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:4d:4d/10.108.188.183/16/10.108.0.1 2023-07-24 12:12:57 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:12:57 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:12:57 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:12:57 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:12:57 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:12:57 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:12:57 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:12:57 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:13:02 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:13:02 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:13:02 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 2023-07-24 12:13:02 mbc divider - INFO: Management IPs selected: --management-ips=10.108.208.219 2023-07-24 12:13:02 mbc divider - INFO: Stopping old container 2023-07-24 12:13:02 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:13:11 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:03:cd:5a/10.108.113.110/16/10.108.0.1 00:00:17:02:c2:ad/10.108.222.101/16/10.108.0.1 02:00:17:0a:4e:2f/10.108.206.113/16/10.108.0.1 02:00:17:0b:b4:2c/10.108.56.193/16/10.108.0.1 02:00:17:06:ae:28/10.108.99.123/16/10.108.0.1 02:00:17:09:1b:8d/10.108.196.44/16/10.108.0.1 02:00:17:0b:bf:52/10.108.186.142/16/10.108.0.1 02:00:17:03:fc:56/10.108.253.58/16/10.108.0.1 02:00:17:06:80:65/10.108.210.85/16/10.108.0.1 00:00:17:02:c4:72/10.108.79.2/16/10.108.0.1 02:00:17:06:24:32/10.108.122.68/16/10.108.0.1 02:00:17:09:ac:88/10.108.216.221/16/10.108.0.1 02:00:17:06:e0:6f/10.108.63.231/16/10.108.0.1 02:00:17:0a:4d:4d/10.108.188.183/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:13:11 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:03:cd:5a/10.108.113.110/16/10.108.0.1 00:00:17:02:c2:ad/10.108.222.101/16/10.108.0.1 02:00:17:0a:4e:2f/10.108.206.113/16/10.108.0.1 02:00:17:0b:b4:2c/10.108.56.193/16/10.108.0.1 02:00:17:06:ae:28/10.108.99.123/16/10.108.0.1 02:00:17:09:1b:8d/10.108.196.44/16/10.108.0.1 02:00:17:0b:bf:52/10.108.186.142/16/10.108.0.1 02:00:17:03:fc:56/10.108.253.58/16/10.108.0.1 02:00:17:06:80:65/10.108.210.85/16/10.108.0.1 00:00:17:02:c4:72/10.108.79.2/16/10.108.0.1 02:00:17:06:24:32/10.108.122.68/16/10.108.0.1 02:00:17:09:ac:88/10.108.216.221/16/10.108.0.1 02:00:17:06:e0:6f/10.108.63.231/16/10.108.0.1 02:00:17:0a:4d:4d/10.108.188.183/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [6, 2, 12, 8, 16, 18, 24, 26, 28, 4, 20, 10, 14, 22], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:13:11 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:13:11 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:13:11 mbc divider - INFO: Starting new containers 2023-07-24 12:13:11 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" 2023-07-24 12:13:11 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "drives0" is ready (pid = 65506) 2023-07-24 12:13:33 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:13:37 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:13:38 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:13:44 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:13:48 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:13:48 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 9 -J" 2023-07-24 12:13:48 mbc divider - DEBUG: Waiting for nodes ['NodeId<180>', 'NodeId<181>', 'NodeId<182>', 'NodeId<183>', 'NodeId<184>', 'NodeId<185>', 'NodeId<186>', 'NodeId<187>', 'NodeId<188>', 'NodeId<189>', 'NodeId<190>', 'NodeId<191>', 'NodeId<192>', 'NodeId<193>', 'NodeId<194>'] to reach DOWN state 2023-07-24 12:13:48 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 9 -J" 2023-07-24 12:13:48 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:13:48 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 9 -J" 2023-07-24 12:13:49 mbc divider - INFO: The following drives will be moved to the new container: ['cb5f6f57-95b8-4ecd-a868-685aed95db22', '372455f4-4546-44e8-9694-52a3ca93e9f5', 'c7d178b0-7c82-4572-8130-0ccdf44d3208', '57a076f7-6a1d-4d03-94d3-407f07d50b30', '995b39a4-a858-4306-a3e4-37278f85ed12', '75621a28-b219-4006-81a3-765618430e08', '304c63df-2206-481d-b900-ef0660dd9aba'] 2023-07-24 12:13:49 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 40" 2023-07-24 12:13:51 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 9 -J" 2023-07-24 12:13:51 mbc divider - INFO: Done scanning drives 2023-07-24 12:13:51 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" 2023-07-24 12:13:51 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 66919) 2023-07-24 12:14:03 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" 2023-07-24 12:14:03 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-009 --management-ips=10.108.208.219 --join-ips=10.108.219.170,10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 68268) 2023-07-24 12:14:11 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:14:11 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:14:12 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:14:12 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:14:12 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:14:12 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 9" 2023-07-24 12:14:12 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:14:12 mbc divider - INFO: running: /bin/sh -c "weka cluster host 9 -J" 2023-07-24 12:14:13 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 9 --no-unimprint" 2023-07-24 12:14:13 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:14:13 mbc divider - INFO: Finished cleanup 2023-07-24 12:14:13 mbc divider - INFO: running: /bin/sh -c "weka local status -v" Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 3/4 running (4 weka) Nodes: 6/32 running (11 INIT, 1 CLUSTERIZED, 14 UNKNOWN, 6 READY) NOTICE: Some nodes services are not ready yet. Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 66919 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (17 seconds ago) Roles: MANAGEMENT Slot 1: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 2: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 3: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 4: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 5: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 6: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 7: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 8: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 9: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 10: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 11: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 12: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 65506 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 68268 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: CLUSTERIZED Status Message: Successfully joined cluster Last Message: Configuration snapshot pulled (5 seconds ago) Roles: MANAGEMENT Slot 1: Node State: UNKNOWN Last Message: Roles: FRONTEND 2023-07-24 12:14:21 mbc divider - INFO: Finished moving server to MBC architecture HOST 10.108.208.219 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.208.219 to rejoin cluster All Nodes belonging to 10.108.208.219 in UP status. VERIFYING /tmp/mbc_divider_script.py PERMISSIONS File Permission are set correctly VERIFYING /tmp/resources_generator.py PERMISSIONS File Permission are set correctly DISTRIBUTING FILE TO HOST 10.108.219.170 scp -p /tmp/mbc_divider_script.py 10.108.219.170:/tmp > /dev/null Warning: Permanently added '10.108.219.170' (ECDSA) to the list of known hosts. /tmp/mbc_divider_script.py transferred successfully /tmp/resources_generator.py transferred successfully ====================================== EXECUTING CONVERSION TO MBC ON HOST 10.108.219.170 ====================================== RUNNING COMMAND: /tmp/mbc_divider_script.py --force 2023-07-24 12:14:29 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:14:29 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:14:29 mbc divider - INFO: Backing up resources to /root/resources.json.backup 2023-07-24 12:14:29 mbc divider - INFO: running: /bin/sh -c "weka local resources export /root/resources.json.backup" Exported resources to /root/resources.json.backup 2023-07-24 12:14:29 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:14:29 mbc divider - INFO: running: /bin/sh -c "weka cluster host net 10 -J" 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:cd:97/10.108.101.170/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:03:9d:e0/10.108.36.128/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:09:41:1a/10.108.10.232/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:43:ec/10.108.174.233/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:41:21/10.108.25.24/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:02:57:c5/10.108.115.193/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:06:9e:92/10.108.39.113/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0a:22:e8/10.108.145.94/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 00:00:17:02:46:e5/10.108.169.220/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:02:0a:41/10.108.118.68/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:22:53/10.108.87.251/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:09:e8:0a/10.108.219.160/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 00:00:17:02:c6:e2/10.108.229.36/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - DEBUG: Adding net devices: 02:00:17:0b:21:b8/10.108.206.88/16/10.108.0.1 2023-07-24 12:14:30 mbc divider - INFO: Checking for active protocols on the host 2023-07-24 12:14:30 mbc divider - INFO: running: /bin/sh -c "weka s3 cluster status -J" 2023-07-24 12:14:30 mbc divider - INFO: running: /bin/sh -c "weka smb cluster status -J" 2023-07-24 12:14:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:14:30 mbc divider - INFO: There is NFS configured in this cluster 2023-07-24 12:14:30 mbc divider - INFO: running: /bin/sh -c "weka nfs interface-group -J" 2023-07-24 12:14:30 mbc divider - WARNING: There is nfs setup in this cluster 2023-07-24 12:14:30 mbc divider - INFO: Validating no protocols containers are running 2023-07-24 12:14:35 mbc divider - INFO: running: /bin/sh -c "weka local ps -J" 2023-07-24 12:14:35 mbc divider - INFO: running: /bin/sh -c "weka cluster host -b -J" 2023-07-24 12:14:35 mbc divider - INFO: IPs for Joining the cluster: --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 2023-07-24 12:14:35 mbc divider - INFO: Management IPs selected: --management-ips=10.108.219.170 2023-07-24 12:14:35 mbc divider - INFO: Stopping old container 2023-07-24 12:14:35 mbc divider - INFO: running: /bin/sh -c "weka local stop" Stopping Weka on this host Stopping containers 2023-07-24 12:14:44 mbc divider - INFO: Running resources-generator with cmd: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:0b:cd:97/10.108.101.170/16/10.108.0.1 02:00:17:03:9d:e0/10.108.36.128/16/10.108.0.1 02:00:17:09:41:1a/10.108.10.232/16/10.108.0.1 02:00:17:0b:43:ec/10.108.174.233/16/10.108.0.1 02:00:17:0b:41:21/10.108.25.24/16/10.108.0.1 02:00:17:02:57:c5/10.108.115.193/16/10.108.0.1 02:00:17:06:9e:92/10.108.39.113/16/10.108.0.1 02:00:17:0a:22:e8/10.108.145.94/16/10.108.0.1 00:00:17:02:46:e5/10.108.169.220/16/10.108.0.1 02:00:17:02:0a:41/10.108.118.68/16/10.108.0.1 02:00:17:0b:22:53/10.108.87.251/16/10.108.0.1 02:00:17:09:e8:0a/10.108.219.160/16/10.108.0.1 00:00:17:02:c6:e2/10.108.229.36/16/10.108.0.1 02:00:17:0b:21:b8/10.108.206.88/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" 2023-07-24 12:14:44 mbc divider - INFO: running: /bin/sh -c "/tmp/resources_generator.py --net 02:00:17:0b:cd:97/10.108.101.170/16/10.108.0.1 02:00:17:03:9d:e0/10.108.36.128/16/10.108.0.1 02:00:17:09:41:1a/10.108.10.232/16/10.108.0.1 02:00:17:0b:43:ec/10.108.174.233/16/10.108.0.1 02:00:17:0b:41:21/10.108.25.24/16/10.108.0.1 02:00:17:02:57:c5/10.108.115.193/16/10.108.0.1 02:00:17:06:9e:92/10.108.39.113/16/10.108.0.1 02:00:17:0a:22:e8/10.108.145.94/16/10.108.0.1 00:00:17:02:46:e5/10.108.169.220/16/10.108.0.1 02:00:17:02:0a:41/10.108.118.68/16/10.108.0.1 02:00:17:0b:22:53/10.108.87.251/16/10.108.0.1 02:00:17:09:e8:0a/10.108.219.160/16/10.108.0.1 00:00:17:02:c6:e2/10.108.229.36/16/10.108.0.1 02:00:17:0b:21:b8/10.108.206.88/16/10.108.0.1 --compute-dedicated-cores 12 --drive-dedicated-cores 1 --frontend-dedicated-cores 1 -f" INFO:resources generator:Drives to be allocated: [] INFO:resources generator:NUMA 0 cores: [4, 14, 22, 26, 8, 18, 20, 2, 24, 28, 16, 6, 12, 10], total memory: 90.197509765625 GiB INFO:resources generator:num_containers_by_role[FRONTEND]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[DRIVES]: 1, nodes count: 1 INFO:resources generator:num_containers_by_role[COMPUTE]: 1, nodes count: 12 INFO:resources generator:Added FRONTEND container resources INFO:resources generator:Added DRIVES container resources INFO:resources generator:Added COMPUTE container resources INFO:resources generator:14 io nodes on NUMA 0 INFO:resources generator:allocating 24.466776218265295 GiB for COMPUTE container, (12 nodes) 2023-07-24 12:14:44 mbc divider - INFO: Releasing old hugepages allocation 2023-07-24 12:14:44 mbc divider - INFO: running: /bin/sh -c "find /opt/weka/data/agent/containers/state/default/huge* -name weka_\* -delete" 2023-07-24 12:14:44 mbc divider - INFO: Starting new containers 2023-07-24 12:14:44 mbc divider - INFO: Starting container of type DRIVE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" 2023-07-24 12:14:44 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=drives0 --resources-path=drives0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" Successfully imported resources from drives0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named drives0 Preparing version 4.1.0.77 of container drives0 NICS were imported from resources file Successfully set up container drives0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up drives0: Allocated network device "0000:00:10.0" (with identifier "0000:00:10.0") to slots [1] on "ms-sbc2-10":"drives0" (1/1) drives0: Allocated core 12 to slot 1 on "ms-sbc2-10":"drives0" (1/1) drives0: Starting hugepages allocation for "ms-sbc2-10":"drives0" drives0: Container "ms-sbc2-10":"drives0" allocated 712 out of 712 required hugepages after 1 retries drives0: Allocated 1424MB hugepages memory from 1 NUMA nodes for "ms-sbc2-10":"drives0" drives0: Bandwidth of "ms-sbc2-10":"drives0" set to unlimited drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-10":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-10":"drives0" drives0: Disabled NUMA Balancing on dedicated host "ms-sbc2-10":"drives0" Container "drives0" is ready (pid = 66121) 2023-07-24 12:15:12 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:15:17 mbc divider - INFO: Waiting for ContainerType.DRIVE containers to reach READY state, currently:READY 2023-07-24 12:15:18 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:15:23 mbc divider - INFO: running: /bin/sh -c "weka local status -J" 2023-07-24 12:15:28 mbc divider - INFO: running: /bin/sh -c "weka debug manhole getServerInfo --slot=0" 2023-07-24 12:15:28 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 10 -J" 2023-07-24 12:15:28 mbc divider - DEBUG: Waiting for nodes ['NodeId<200>', 'NodeId<201>', 'NodeId<202>', 'NodeId<203>', 'NodeId<204>', 'NodeId<205>', 'NodeId<206>', 'NodeId<207>', 'NodeId<208>', 'NodeId<209>', 'NodeId<210>', 'NodeId<211>', 'NodeId<212>', 'NodeId<213>', 'NodeId<214>'] to reach DOWN state 2023-07-24 12:15:28 mbc divider - INFO: running: /bin/sh -c "weka cluster nodes --host 10 -J" 2023-07-24 12:15:28 mbc divider - DEBUG: all nodes are DOWN 2023-07-24 12:15:28 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 10 -J" 2023-07-24 12:15:28 mbc divider - INFO: The following drives will be moved to the new container: ['39341d97-dbc7-4764-a73b-a3aab7b1a663', '51b7e1e1-6bfd-49a2-8783-1ec7af5ef922', 'd79afa00-3126-4a8d-83fd-edba82b02582', '1c0632c5-bb27-4007-8fe9-ee7187ac5ac9', '8905ff45-144c-4915-bafc-6cf0c60fe52f', '179f1c0b-eb8b-41bd-96a5-dcd29318094d', 'd11b386d-c1bd-4a20-bb39-0dc80b2160dd'] 2023-07-24 12:15:28 mbc divider - INFO: running: /bin/sh -c "weka cluster drive scan 43" 2023-07-24 12:15:30 mbc divider - INFO: running: /bin/sh -c "weka cluster drive --host 10 -J" 2023-07-24 12:15:30 mbc divider - INFO: Done scanning drives 2023-07-24 12:15:30 mbc divider - INFO: Starting container of type COMPUTE using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" 2023-07-24 12:15:30 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=compute0 --resources-path=compute0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" Successfully imported resources from compute0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named compute0 Preparing version 4.1.0.77 of container compute0 NICS were imported from resources file Successfully set up container compute0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "compute0" is ready (pid = 67502) 2023-07-24 12:15:43 mbc divider - INFO: Starting container of type FRONTEND using the following command: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" 2023-07-24 12:15:43 mbc divider - INFO: running: /bin/sh -c "weka local setup host --timeout 10m --name=frontend0 --resources-path=frontend0.json --failure-domain DOM-010 --management-ips=10.108.219.170 --join-ips=10.108.123.20,10.108.123.20,10.108.123.20,10.108.63.220,10.108.63.220,10.108.63.220,10.108.72.27,10.108.72.27,10.108.72.27,10.108.216.52,10.108.216.52 --disable" Successfully imported resources from frontend0.json Downloading version 4.1.0.77 Version 4.1.0.77 already downloaded Created Weka container named frontend0 Preparing version 4.1.0.77 of container frontend0 NICS were imported from resources file Successfully set up container frontend0 Starting container The NetworkManager service is running, we aren't allowed to stop it, but we were told to run anyway, so trying Waiting for container to start up Container "frontend0" is ready (pid = 68954) 2023-07-24 12:15:52 mbc divider - INFO: Waiting for buckets redistribution 2023-07-24 12:15:52 mbc divider - INFO: running: /bin/sh -c "weka status -J" 2023-07-24 12:15:52 mbc divider - INFO: running: /bin/sh -c "weka local enable" Enabling all containers available on the system Containers succesfully enabled. Run "weka local ps" to see info about all of them 2023-07-24 12:15:52 mbc divider - INFO: Starting cleanup: The old container will be removed locally and removed from the cluster 2023-07-24 12:15:52 mbc divider - INFO: running: /bin/sh -c "weka local disable default " 2023-07-24 12:15:52 mbc divider - INFO: running: /bin/sh -c "weka cluster host deactivate 10" 2023-07-24 12:15:53 mbc divider - INFO: Waiting for host to deactivate 2023-07-24 12:15:53 mbc divider - INFO: running: /bin/sh -c "weka cluster host 10 -J" 2023-07-24 12:15:53 mbc divider - INFO: running: /bin/sh -c "weka cluster host remove 10 --no-unimprint" 2023-07-24 12:15:53 mbc divider - INFO: the old container default is disabled and stopped, please remove it after the conversion is done 2023-07-24 12:15:53 mbc divider - INFO: Finished cleanup 2023-07-24 12:15:53 mbc divider - INFO: running: /bin/sh -c "weka local status -v" 2023-07-24 12:16:03 mbc divider - INFO: Finished moving server to MBC architecture Weka v4.1.0.77 (CLI build 4.1.0.77) Containers: 3/4 running (4 weka) Nodes: 9/32 running (8 INIT, 1 ALLOCATING_RESOURCES, 14 UNKNOWN, 9 READY) NOTICE: Some nodes services are not ready yet. Mounts: No active WekaFS mounts Container Information: compute0 (weka): Status: RUNNING PID: 67502 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14300 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (18 seconds ago) Roles: MANAGEMENT Slot 1: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 2: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 3: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 4: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 5: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 6: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 7: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 8: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 9: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 10: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE Slot 11: Node State: READY Status Message: Ready Last Message: Roles: COMPUTE Slot 12: Node State: INIT Status Message: Initializing Last Message: Roles: COMPUTE default (weka): Status: STOPPED Version: 4.1.0.77 Last Message: Container stopped due to user request (1 minute ago) Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: FRONTEND Slot 2: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 3: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 4: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 5: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 6: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 7: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 8: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 9: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 10: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 11: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 12: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 13: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: COMPUTE Slot 14: Node State: UNKNOWN Last Message: SIGTERM signal received (1 minute ago) Roles: DRIVES drives0 (weka): Status: RUNNING PID: 66121 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14000 Nodes: Slot 0: Node State: READY Status Message: Ready Last Message: Configuration snapshot pulled (1 minute ago) Roles: MANAGEMENT Slot 1: Node State: READY Status Message: Ready Last Message: Roles: DRIVES frontend0 (weka): Status: RUNNING PID: 68954 Version: 4.1.0.77 Last Message: Cluster Membership: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) API Port: 14200 Nodes: Slot 0: Node State: ALLOCATING_RESOURCES Status Message: Disabled NUMA balancing Last Message: Configuration snapshot pulled (6 seconds ago) Roles: MANAGEMENT Slot 1: Node State: UNKNOWN Last Message: Roles: FRONTEND HOST 10.108.219.170 WAS SUCCESSFULLY CONVERTED TO MBC VERIFYING REBUILD STATUS COMPLETE Waiting for rebuild to complete please standby... Rebuild complete. VERIFYING NODE STATUS Waiting for nodes belonging to 10.108.219.170 to rejoin cluster All Nodes belonging to 10.108.219.170 in UP status. Done converting cluster to MBC, please clean up the unconnected default container WekaIO v4.1.0.77 (CLI build 4.1.0.77) cluster: ms-sbc2 (2c2fc1ea-1994-4e52-8454-d6448e252330) status: OK (33 backend containers UP, 77 drives UP) protection: 9+2 hot spare: 1 failure domains (1.50 TiB) drive storage: 15.06 TiB total cloud: connected license: OK, valid thru 2023-08-23T07:00:23Z io status: STARTED 2 hours ago (153/154 io-nodes UP, 396 Buckets UP) link layer: Ethernet clients: 2 connected reads: 0 B/s (0 IO/s) writes: 3.72 MiB/s (955 IO/s) operations: 997 ops/s alerts: 3 active alerts, use `weka alerts` to list them [root@ms-sbc2-0 ~] 2023-07-24 12:16:09 $