MediaPipe の Hand Tracking データを外部に保存する必要があった。

「【MediaPipe】Multi Hand Trackingから検出データを抽出した」を基にさせてもらい、Windows でデータを取得できるようにした。

上記Webが扱っているMediaPipe のバージョンとこれを執筆時点のMediaPipe のバージョンが異なるため、執筆時点のMediaPipe のバージョンに合わせる修正をした。下記のビルドに利用したファイルを https://github.com/kunsen-an/mediapipe に置いた。

環境

試したソフトウェアなどのバージョンは以下の通りである。

• Windows 11 Pro 22000.556
• MediaPipe v0.8.9 (14 Dec 2021)

ビルドのための、bazel や Visual C++、Pythonは、Windows 10にMediaPipe をインストール (WSL不使用) で設定したものを利用している。これらは最新バージョンとは異なる。

Hand Tracking のグラフの変更

データのファイルへの保存は、PassThroughCalculatorを用いた「【MediaPipe】Multi Hand Trackingから検出データを抽出した」を基にしている。

ただし、「【MediaPipe】Multi Hand Trackingから検出データを抽出した」は、MediaPipe v0.7.4 を基にしており、v0.8.9では Hand Tracking のグラフが変更されているのでそのまま動作させることができなかった。

MediaPipe v0.8.9 の Hand Tracking のグラフ　mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt を Mediapipe Visualizer で可視化すると次の通りであった。

このHandLandMarkTrackingCpu と HandRenderer の間に、PathThroughCalculator を追加した(mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt)。Mediapipe Visualizer で可視化したものは次の通りである。

mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt　は以下の通り。

# MediaPipe graph that performs hands tracking on desktop with TensorFlow
# Lite on CPU.
# Used in the example in
# mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out.
# This file is based on mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

# CPU image. (ImageFrame)
input_stream: "input_video"

# CPU image. (ImageFrame)
output_stream: "output_video"


output_stream: "output_palm_detections"
output_stream: "output_landmarks"
output_stream: "output_palm_rects"
output_stream: "output_hand_rects"


# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingCpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:multi_palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:multi_hand_rects"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:multi_palm_rects"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_cpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:output_palm_detections"
  input_stream: "LANDMARKS:output_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:output_palm_rects"
  input_stream: "NORM_RECTS:1:output_hand_rects"
  output_stream: "IMAGE:output_video"
}

# Streams to get results of detection and landmark
node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_palm_detections"
  output_stream: "output_palm_detections"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "landmarks"
  output_stream: "output_landmarks"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_palm_rects"
  output_stream: "output_palm_rects"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_hand_rects"
  output_stream: "output_hand_rects"
}

demo_run_graph_main.cc の変更

グラフを実行して、PassThroughCalculator のデータを保存するために、mediapipe/examples/desktop/demo_run_graph_main.cc を変更し、mediapipe/examples/desktop/demo_run_graph_main_out.cc とした。

以前のMediaPipe v0.7.4　ではフレームごとにランドマークや手の矩形、手の平、手の平の矩形などが出力されていたようであるが、MediaPipe v0.8.9では、実際に検出されたときにしかデータ出力されない。このため、すべてのデータが揃うのを待っていては適切なタイミングでデータを保存できない。

やむを得ないので、PassThroughCalculator にPoller を追加し、そのキューが空でなければ(poller_?????.QueueSize() > 0)、データがあるとし、すべてのデータが揃った際にファイルに(save_dataがtrueの場合に)データを出力することにした。すべてのデータが揃わなくてもデータを保存した方がよい場合には、ファイルへ出力する(save_dataの)条件を変更する必要がある。

データが揃わない場合(save_dataが falseの場合)には、processTypeList でキューの処理は行うが、ファイルへのデータ出力は行わない。この処理のタイミングがフローによって異なるため、キューの処理よりも入力ビデオの処理が速い場合には、データ出力のタイミングがずれる可能性がある。現時点では、処理をするCPUの性能が十分に高ければ、そのようなことはないと考えている。念のために、キューの長さが1より大きくなったら処理が間に合っていないとして、警告(LOG(WARNING))を出力するようにしている。

mediapipe/examples/desktop/demo_run_graph_main_out.cc　は以下の通りである。

// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// An example of sending OpenCV webcam frames into a MediaPipe graph.

// This file is based on mediapipe/examples/desktop/demo_run_graph_main.cc and
// https://dev.classmethod.jp/articles/mediapipe-extract-data-from-multi-hand-tracking/ .

#include <cstdlib>

#include <iostream>
#include <filesystem>
#include <fstream>
#include <sys/stat.h>
#include <cstdio>
#include <regex>


#include "absl/flags/flag.h"
#include "absl/flags/parse.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/image_frame.h"
#include "mediapipe/framework/formats/image_frame_opencv.h"
#include "mediapipe/framework/port/file_helpers.h"
#include "mediapipe/framework/port/opencv_highgui_inc.h"
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
#include "mediapipe/framework/port/opencv_video_inc.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

constexpr char kInputStream[] = "input_video";
constexpr char kOutputStream[] = "output_video";
constexpr char kWindowName[] = "MediaPipe";


#include "mediapipe/framework/formats/detection.pb.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/formats/rect.pb.h"
#include "mediapipe/framework/timestamp.h"

#include <direct.h>

constexpr char kOutputPalmDetections[] = "output_palm_detections";
constexpr char kOutputLandmarks[] = "output_landmarks";
constexpr char kOutputPalmRects[] = "output_palm_rects";
constexpr char kOutputHandRects[] = "output_hand_rects";


ABSL_FLAG(std::string, calculator_graph_config_file, "",
          "Name of file containing text format CalculatorGraphConfig proto.");
ABSL_FLAG(std::string, input_video_path, "",
          "Full path of video to load. "
          "If not provided, attempt to use a webcam.");
ABSL_FLAG(std::string, output_video_path, "",
          "Full path of where to save result (.mp4 only). "
          "If not provided, show result in a window.");

std::string output_dirpath = std::string("./result");
std::string output_data_extension = std::string(".bin");


std::string outputFilePath(mediapipe::Timestamp timestamp, std::string type, int index, std::string postfix) {
	std::ostringstream os;

	os << output_dirpath + "/"
		<< timestamp << "_"
		<< type << "_"
		<< std::to_string(index)
		<< postfix;
	return os.str();
}

std::string outputFilePath(mediapipe::Timestamp timestamp, std::string postfix) {
	std::ostringstream os;

	os << output_dirpath + "/"
		<< timestamp << "_"
		<< postfix;
	return os.str();
}

template<typename TVector>
bool
processTypeList(mediapipe::OutputStreamPoller& poller, std::string type, bool save) {
	bool ret = false;
	mediapipe::Packet	packet;
	// process landmarks
	if ( poller.QueueSize() > 0 && poller.Next(&packet) ) {
		if ( poller.QueueSize() > 1 ) LOG(WARNING) << "QueueSize: " << poller.QueueSize();
		auto &output = packet.Get<TVector>();
		if ( save ) {	// save data to output file if save is true
			for (int j = 0; j < output.size(); j++)
			{
				std::string filePath = outputFilePath(packet.Timestamp(), type, j,output_data_extension);
				std::ofstream outputfile(filePath);

				std::string serializedStr;
				output[j].SerializeToString(&serializedStr);
				outputfile << serializedStr << std::flush;
			}
			ret = save;
		}
	}
	return ret;
}


absl::Status RunMPPGraph() {
  std::string calculator_graph_config_contents;
  MP_RETURN_IF_ERROR(mediapipe::file::GetContents(
      absl::GetFlag(FLAGS_calculator_graph_config_file),
      &calculator_graph_config_contents));
  LOG(INFO) << "Get calculator graph config contents: "
            << calculator_graph_config_contents;
  mediapipe::CalculatorGraphConfig config =
      mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(
          calculator_graph_config_contents);

  LOG(INFO) << "Initialize the calculator graph.";
  mediapipe::CalculatorGraph graph;
  MP_RETURN_IF_ERROR(graph.Initialize(config));

  LOG(INFO) << "Initialize the camera or load the video.";
  cv::VideoCapture capture;
  const bool load_video = !absl::GetFlag(FLAGS_input_video_path).empty();
  if (load_video) {
    capture.open(absl::GetFlag(FLAGS_input_video_path));
  } else {
    capture.open(0);
  }
  RET_CHECK(capture.isOpened());

  cv::VideoWriter writer;
  const bool save_video = !absl::GetFlag(FLAGS_output_video_path).empty();
  if (!save_video) {
    cv::namedWindow(kWindowName, /*flags=WINDOW_AUTOSIZE*/ 1);
#if (CV_MAJOR_VERSION >= 3) && (CV_MINOR_VERSION >= 2)
    capture.set(cv::CAP_PROP_FRAME_WIDTH, 640);
    capture.set(cv::CAP_PROP_FRAME_HEIGHT, 480);
    capture.set(cv::CAP_PROP_FPS, 30);
#endif
  }

  LOG(INFO) << "Start running the calculator graph.";


  // Connect pollers to graph
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_video,
                   graph.AddOutputStreamPoller(kOutputStream));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_detections,
                   graph.AddOutputStreamPoller(kOutputPalmDetections));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_landmarks,
                   graph.AddOutputStreamPoller(kOutputLandmarks));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_rects,
                   graph.AddOutputStreamPoller(kOutputPalmRects));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_hand_rects,
                   graph.AddOutputStreamPoller(kOutputHandRects));

  
  MP_RETURN_IF_ERROR(graph.StartRun({}));

  LOG(INFO) << "Start grabbing and processing frames.";
  bool grab_frames = true;

  // define output folder path
  std::string flag_output_video_path = absl::GetFlag(FLAGS_output_video_path);
  std::filesystem::path output_video_path = flag_output_video_path;
  std::string output_video_dir = output_video_path.parent_path().string();
  LOG(INFO) << "output_video_dir:" << output_video_dir;

  if ( ! output_video_dir.empty() ) {
	  output_dirpath =  output_video_dir;
  }
  _mkdir(output_dirpath.c_str()); // for windows
  
  while (grab_frames)
  {
    // Capture opencv camera or video frame.
    cv::Mat camera_frame_raw;
    capture >> camera_frame_raw;

    if (camera_frame_raw.empty()) {
      if (!load_video) {
        LOG(INFO) << "Ignore empty frames from camera.";
        continue;
      }
      LOG(INFO) << "Empty frame, end of video reached.";
      break;
    }
    cv::Mat camera_frame, flipped_frame;
    cv::cvtColor(camera_frame_raw, camera_frame, cv::COLOR_BGR2RGB);

    // Wrap Mat into an ImageFrame.
    auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
        mediapipe::ImageFormat::SRGB, camera_frame.cols, camera_frame.rows,
        mediapipe::ImageFrame::kDefaultAlignmentBoundary);
    cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
    camera_frame.copyTo(input_frame_mat);

    // Send image packet into the graph.
    size_t frame_timestamp_us =
        (double)cv::getTickCount() / (double)cv::getTickFrequency() * 1e6;
    MP_RETURN_IF_ERROR(graph.AddPacketToInputStream(
        kInputStream, mediapipe::Adopt(input_frame.release())
                          .At(mediapipe::Timestamp(frame_timestamp_us))));



    // Get the graph result packet, or stop if that fails.
    mediapipe::Packet packet_video;

    // check if data exist from graph
    if (!poller_video.Next(&packet_video) ) {
		break;
	}

    // get output from graph
    auto &output_video = packet_video.Get<mediapipe::ImageFrame>();
	auto packet_video_timestamp =  packet_video.Timestamp();
	LOG(INFO) << "packet_video.Timestamp:" << packet_video_timestamp;
	
	// save_data is true if all data is available (If partial data should be saved, change the following conditions)
	bool save_data = (poller_palm_detections.QueueSize() > 0)
					&& (poller_palm_rects.QueueSize() > 0)
					&& (poller_hand_rects.QueueSize() > 0)
					&& (poller_landmarks.QueueSize() > 0);

	// process palm detections
	processTypeList<std::vector<mediapipe::Detection>>(poller_palm_detections, std::string("palm_detections"), save_data);

	// process palm rects
	processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_palm_rects, std::string("palm_rects"), save_data);

	// process hand rects
	processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_hand_rects, std::string("hand_rects"), save_data);

	// process landmarks
	processTypeList<std::vector<mediapipe::NormalizedLandmarkList>>(poller_landmarks, std::string("landmarks"), save_data);


	
    // Convert back to opencv for display or saving.
    cv::Mat output_frame_mat = mediapipe::formats::MatView(&output_video);
    cv::cvtColor(output_frame_mat, output_frame_mat, cv::COLOR_RGB2BGR);
    if (save_video)
    {
      if (!writer.isOpened())
      {
        LOG(INFO) << "Prepare video writer:" << flag_output_video_path;
        writer.open(flag_output_video_path,
                    mediapipe::fourcc('a', 'v', 'c', '1'), // .mp4
                    capture.get(cv::CAP_PROP_FPS), output_frame_mat.size());
        RET_CHECK(writer.isOpened());
      }
      writer.write(output_frame_mat);
    }


	// show output image
	cv::imshow(kWindowName, output_frame_mat);
	// Press any key to exit.
	const int pressed_key = cv::waitKey(5);
	if (pressed_key >= 0 && pressed_key != 255)
		grab_frames = false;


	if ( !save_data ) continue;

    // save input frame to file
	std::string inputFramePath = outputFilePath(mediapipe::Timestamp(frame_timestamp_us), std::string("inputFrame.jpg"));
    cv::imwrite(inputFramePath, input_frame_mat);
	
    // save output frame to file
	std::string outputFramePath = outputFilePath(packet_video_timestamp, "outputFrame.jpg");
    cv::imwrite(outputFramePath, output_frame_mat);
  }

  LOG(INFO) << "Shutting down.";
  if (writer.isOpened()) writer.release();
  MP_RETURN_IF_ERROR(graph.CloseInputStream(kInputStream));
  return graph.WaitUntilDone();
}

int main(int argc, char** argv) {
  google::InitGoogleLogging(argv[0]);
  absl::ParseCommandLine(argc, argv);
  absl::Status run_status = RunMPPGraph();
  if (!run_status.ok()) {
    LOG(ERROR) << "Failed to run the graph: " << run_status.message();
    return EXIT_FAILURE;
  } else {
    LOG(INFO) << "Success!";
  }
  return EXIT_SUCCESS;
}

ビルド

ビルドするターゲットを mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out にし、基になる グラフをmediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt に、C++プログラムを mediapipe/examples/desktop/demo_run_graph_main_out.cc　にし、PathThroughCalculator を利用するため、bazel のBUILDファイルを変更した。

mediapipe/examples/desktop/BUILD

name =  demo_run_graph_main_out の cc_library を追加し、依存しているものに以下を追加した。

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

licenses(["notice"])

package(default_visibility = [
    "//visibility:public",
])

cc_library(
    name = "simple_run_graph_main",
    srcs = ["simple_run_graph_main.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:map_util",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:ret_check",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework/port:statusor",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
        "@com_google_absl//absl/strings",
    ],
)

cc_library(
    name = "demo_run_graph_main",
    srcs = ["demo_run_graph_main.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
    ],
)

cc_library(
    name = "demo_run_graph_main_out",
    srcs = ["demo_run_graph_main_out.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
		"//mediapipe/framework/formats:detection_cc_proto",
        "//mediapipe/framework/formats:landmark_cc_proto",
        "//mediapipe/framework/formats:rect_cc_proto",
    ],
)

# Linux only.
# Must have a GPU with EGL support:
# ex: sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
# (or similar nvidia/amd equivalent)
cc_library(
    name = "demo_run_graph_main_gpu",
    srcs = ["demo_run_graph_main_gpu.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "//mediapipe/gpu:gl_calculator_helper",
        "//mediapipe/gpu:gpu_buffer",
        "//mediapipe/gpu:gpu_shared_data_internal",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
    ],
)

mediapipe/examples/desktop/hand_tracking/BUILD

name =  hand_tracking_cpu_out の cc_library を追加し、依存しているものはhand_tracking_cpuを基にして以下とした。

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

licenses(["notice"])

package(default_visibility = ["//mediapipe/examples:__subpackages__"])

cc_binary(
    name = "hand_tracking_tflite",
    deps = [
        "//mediapipe/examples/desktop:simple_run_graph_main",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)

cc_binary(
    name = "hand_tracking_cpu",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)


cc_binary(
    name = "hand_tracking_cpu_out",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main_out",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)

# Linux only
cc_binary(
    name = "hand_tracking_gpu",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main_gpu",
        "//mediapipe/graphs/hand_tracking:mobile_calculators",
    ],
)

mediapipe/graphs/hand_tracking/BUILD

name = "desktop_tflite_calculators" の cc_library の依存しているものに "//mediapipe/calculators/core:pass_through_calculator" を追加した。

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

load(
    "//mediapipe/framework/tool:mediapipe_graph.bzl",
    "mediapipe_binary_graph",
)

licenses(["notice"])

package(default_visibility = ["//visibility:public"])

exports_files(glob([
    "*.pbtxt",
]))

cc_library(
    name = "desktop_offline_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/core:gate_calculator",
        "//mediapipe/calculators/core:immediate_mux_calculator",
        "//mediapipe/calculators/core:packet_inner_join_calculator",
        "//mediapipe/calculators/core:previous_loopback_calculator",
        "//mediapipe/calculators/video:opencv_video_decoder_calculator",
        "//mediapipe/calculators/video:opencv_video_encoder_calculator",
    ],
)

cc_library(
    name = "desktop_tflite_calculators",
    deps = [
        ":desktop_offline_calculators",
        "//mediapipe/calculators/core:constant_side_packet_calculator",
        "//mediapipe/calculators/core:merge_calculator",
        "//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_cpu",
        "//mediapipe/modules/hand_landmark:hand_landmark_tracking_cpu",
		"//mediapipe/calculators/core:pass_through_calculator",
    ],
)

mediapipe_binary_graph(
    name = "hand_tracking_desktop_live_binary_graph",
    graph = "hand_tracking_desktop_live.pbtxt",
    output_name = "hand_tracking_desktop_live.binarypb",
    deps = [":desktop_tflite_calculators"],
)

cc_library(
    name = "mobile_calculators",
    deps = [
        "//mediapipe/calculators/core:constant_side_packet_calculator",
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_gpu",
        "//mediapipe/modules/hand_landmark:hand_landmark_tracking_gpu",
    ],
)

mediapipe_binary_graph(
    name = "hand_tracking_mobile_gpu_binary_graph",
    graph = "hand_tracking_mobile.pbtxt",
    output_name = "hand_tracking_mobile_gpu.binarypb",
    deps = [":mobile_calculators"],
)

cc_library(
    name = "detection_mobile_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/util:annotation_overlay_calculator",
        "//mediapipe/calculators/util:detections_to_render_data_calculator",
        "//mediapipe/modules/palm_detection:palm_detection_gpu",
    ],
)

mediapipe_binary_graph(
    name = "hand_detection_mobile_gpu_binary_graph",
    graph = "hand_detection_mobile.pbtxt",
    output_name = "hand_detection_mobile_gpu.binarypb",
    deps = [":detection_mobile_calculators"],
)

環境設定

利用したツールなどは以下の設定を行ってコマンドプロンプト内でビルドした。

set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC
set BAZEL_VC_FULL_VERSION=14.28.29333
set BAZEL_WINSDK_FULL_VERSION=10.0.19041.0
set PYTHON_BIN_PATH="C://python39//python.exe"
cd D:\mediapipe_repo\mediapipe

以下では、この設定の下での例を示す。

bazel build の実行

D:\mediapipe_repo\mediapipe で以下のbazel build を実行し、hand_tracking_cpu_out.exe　をビルドした。

bazel-bin\mediapipe\examples\desktop\hand_tracking\hand_tracking_cpu_out.exe　に実行ファイルが構築された。

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH=%PYTHON_BIN_PATH% mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out

hand_tracking_cpu_out.exeの実行

ビルドした hand_tracking_cpu_out.exe を以下のコマンドで実行した。

bazel-bin\mediapipe\examples\desktop\hand_tracking\hand_tracking_cpu_out.exe --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt

以下の例では、set GLOG_logtostderr=1　を実行しておき、ログを標準エラー出力に表示している。この環境変数の設定を行わないと、画面には出力されない。

実行時のプレビューウィンドウ

実行時のログの情報出力

ログの出力レベルを変更するには環境変数を設定する。set GLOG_minloglevel=1　を実行するすることで、INFOレベルのメッセージは出力せず、WARNING以上を出力する。値に2、 3 を指定することで、それぞれERROR以上、FATAL になる。

データ出力

hand_tracking_cpu_out.cc の中の std::string output_dirpath = std::string("./result"); で出力を指定している。この例では、カレントディレクトリをD:\mediapipe_repo\mediapipe として実行しているので、　D:\mediapipe_repo\mediapipe\result　にデータが出力される。

出力された .bin ファイルは人が可読形式になっていないので、必要に応じて変換を行う必要がある。