Runs Management - fedml run
FedML Run CLI Overview
Manage runs on the TensorOpera AI Platform
Usage: fedml run [OPTIONS] COMMAND [ARGS]...
Manage runs on the TensorOpera AI Platform.
Options:
-h, --help Show this message and exit.
-k, --api_key TEXT The user API key.
-v, --version TEXT Version of TensorOpera AI Platform. It should be dev,
test or release.
-pf, --platform TEXT The platform name at the TensorOpera AI Platform
(options: octopus, parrot, spider, beehive, falcon,
launch, default is falcon).
Commands:
list List runs from the TensorOpera AI Platform.
logs Get logs of run from the TensorOpera AI Platform.
status Get status of run from the TensorOpera AI Platform.
stop Stop a run from the TensorOpera AI Platform.
fedml run list [OPTIONS]
List runs from the TensorOpera AI Platform.
Options
| Option | Description |
|---|---|
--help or -h | Show this message and exit. |
--run_name or -r | Run name at the TensorOpera AI Platform. |
--run_id or -rid | Run id at the TensorOpera AI Platform. |
--api_key or -k | The user API key. |
--version or -v | Version of TensorOpera AI Platform. It should be dev, test or release. |
--platform or -pf | The platform name at the TensorOpera AI Platform (options: octopus, parrot, spider, beehive, falcon, launch, default is falcon). |
Example
List all runs on the TensorOpera AI Platform.
feml run list
Found the following matched runs.
+----------------------+---------------------+----------+---------------------+------------------+------+
| Run Name | Run ID | Status | Created | Spend Time(hour) | Cost |
+----------------------+---------------------+----------+---------------------+------------------+------+
| tight_ready | 1684458113152978944 | FINISHED | 2023-07-27 06:58:04 | 0.0333 | 2.0 |
| shorter_tax | 1684458685260238848 | FINISHED | 2023-07-27 07:00:20 | 0.0333 | 2.0 |
| swam_fellow | 1684500824392339456 | FINISHED | 2023-07-27 09:47:47 | 0.0333 | 2.0 |
| national_your | 1684753343311908864 | FINISHED | 2023-07-28 02:31:13 | 0.0333 | 2.0 |
+----------------------+---------------------+----------+---------------------+------------------+------+
List selected runs on the TensorOpera AI Platform.
fedml run list -r tight_ready
Found the following matched runs.
+-------------+---------------------+----------+---------------------+------------------+------+
| Run Name | Run ID | Status | Created | Spend Time(hour) | Cost |
+-------------+---------------------+----------+---------------------+------------------+------+
| tight_ready | 1684458113152978944 | FINISHED | 2023-07-27 06:58:04 | 0.0333 | 2.0 |
+-------------+---------------------+----------+---------------------+------------------+------+
fedml run logs [OPTIONS]
Get logs of run from the TensorOpera AI Platform.
Options
| Option | Description |
|---|---|
--help or -h | Show this message and exit. |
--run_id or -rid | Run id at the TensorOpera AI Platform. |
--need_all_logs or -a | Boolean value representing if all logs are needed. Default to True. |
--page_num or -pn | request page num for logs. --need_all_logs should be set to False if you want to use this option. |
--page_size or -ps | request page size for logs, --need_all_logs should be set to False if you want to use this option. |
--api_key or -k | The user API key. |
--version or -v | Version of TensorOpera AI Platform. It should be dev, test or release. |
--platform or -pf | The platform name at the TensorOpera AI Platform (options: octopus, parrot, spider, beehive, falcon, launch, default is falcon). |
Example
fedml run logs -rid 1716563514434392064
Logs summary info is as follows.
+---------------------+-----------------+---------------------------------------------------------------------------------------+
| Run ID | Total Log Lines | Log URL |
+---------------------+-----------------+---------------------------------------------------------------------------------------+
| 1716563514434392064 | 11 | https://s3.us-west-1.amazonaws.com/fedml/fedml-logs/fedml-run-1716563514434392064.log |
+---------------------+-----------------+---------------------------------------------------------------------------------------+
Logs URL for each device is as follows.
+---------------------+-------------------------+---------------------------------------------------------------------------------------------------------------------------+
| Device ID | Device Name | Device Log URL |
+---------------------+-------------------------+---------------------------------------------------------------------------------------------------------------------------+
| 1684824138201567232 | NVIDIA A100-SXM4-80GB:8 | https://s3.us-west-1.amazonaws.com/fedml/fedml-logs/fedml-run-1714535384211394560-edge-1684824138201567232%40user-214.log |
+---------------------+-------------------------+---------------------------------------------------------------------------------------------------------------------------+
All logs is as follows.
[FedML-Client @device-id-1684824138201567232] [Mon, 23 Oct 2023 14:13:30 -0700] [INFO]-----GPU Machine scheduling successful-----
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:13:54 -0700] [INFO] [mlops_metrics.py:287:report_server_id_status] report_server_id_status. message_json = {"run_id": 1716563514434392064, "edge_id": 201649, "status": "STARTING"}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:13:54 -0700] [INFO] [mlops_metrics.py:229:report_server_device_status_to_web_ui] report_server_device_status. msg = {'run_id': 1716563514434392064, 'edge_id': 201649, 'status': 'STARTING', 'role': 'normal', 'version': 'v1.0'}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:13:54 -0700] [INFO] [mlops_metrics.py:229:report_server_device_status_to_web_ui] report_server_device_status. msg = {'run_id': 1716563514434392064, 'edge_id': 201649, 'status': 'STARTING', 'role': 'normal', 'version': 'v1.0'}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:13:54 -0700] [INFO] [server_runner.py:502:run_impl] Detect all status of Edge ids: [1684824138201567232]
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [ERROR] [server_runner.py:934:detect_edges_status] There are inactive edge devices. Inactivate edge id list is as follows. [1684824138201567232]
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [INFO] [mlops_metrics.py:287:report_server_id_status] report_server_id_status. message_json = {"run_id": 1716563514434392064, "edge_id": 201649, "status": "FAILED", "server_id": 201649}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [INFO] [mlops_metrics.py:229:report_server_device_status_to_web_ui] report_server_device_status. msg = {'run_id': 1716563514434392064, 'edge_id': 201649, 'status': 'FAILED', 'role': 'normal', 'version': 'v1.0'}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [ERROR] [server_runner.py:1441:send_exit_train_with_exception_request_to_edges] exit_train_with_exception: send topic flserver_agent/1684824138201567232/exit_train_with_exception
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [INFO] [mlops_metrics.py:158:common_broadcast_client_training_status] report_client_training_status. message_json = {"edge_id": 1684824138201567232, "run_id": 1716563514434392064, "status": "FAILED"}
[FedML-Server @device-id-201649] [Mon, 23 Oct 2023 14:38:59 -0700] [INFO] [server_runner.py:438:run] Release resources.
fedml run status [OPTIONS]
Get status of run from the TensorOpera AI Platform.
Options
| Option | Description |
|---|---|
--help or -h | Show this message and exit |
--run_name or -r | Run name at the TensorOpera AI Platform. |
--run_id or -rid | Run id at the TensorOpera AI Platform. |
--api_key or -k | The user API key. |
--version or -v | Version of TensorOpera AI Platform. It should be dev, test or release. |
--platform or -pf | The platform name at the TensorOpera AI Platform (options: octopus, parrot, spider, beehive, falcon, launch, default is falcon). |
Example
❯ fedml run status -r particular_determine
Found the following matched runs.
+----------------------+---------------------+----------+---------------------+------------------+------+
| Run Name | Run ID | Status | Created | Spend Time(hour) | Cost |
+----------------------+---------------------+----------+---------------------+------------------+------+
| particular_determine | 1684754107195330560 | FINISHED | 2023-07-28 02:34:15 | 0.0333 | 2.0 |
+----------------------+---------------------+----------+---------------------+------------------+------+
fedml run stop [OPTIONS]
Stop a run from the TensorOpera AI Platform.
Options
| Option | Description |
|---|---|
--help or -h | Show this message and exit. |
--run_id or -rid | Id of the run. |
--api_key or -k | The user API key. |
--version or -v | Version of TensorOpera AI Platform. It should be dev, test or release. |
--platform or -pf | The platform name at the TensorOpera AI Platform (options: octopus, parrot, spider, beehive, falcon, launch, default is falcon). |
Example
fedml run stop -rid 1716563514434392064
Run 1716563514434392064 is stopped successfully.