MediaPipe の Hand Tracking データを外部に保存する必要があった。

「【MediaPipe】Multi Hand Trackingから検出データを抽出した」を基にさせてもらい、Windows でデータを取得できるようにした。

上記Webが扱っているMediaPipe のバージョンとこれを執筆時点のMediaPipe のバージョンが異なるため、執筆時点のMediaPipe のバージョンに合わせる修正をした。下記のビルドに利用したファイルを https://github.com/kunsen-an/mediapipe に置いた。

1 環境
2 Hand Tracking のグラフの変更
3 demo_run_graph_main.cc の変更
4 ビルド
5 hand_tracking_cpu_out.exeの実行

環境

試したソフトウェアなどのバージョンは以下の通りである。

Windows 11 Pro 22000.556
MediaPipe v0.8.9 (14 Dec 2021)

ビルドのための、bazel や Visual C++、Pythonは、Windows 10にMediaPipe をインストール (WSL不使用) で設定したものを利用している。これらは最新バージョンとは異なる。

Hand Tracking のグラフの変更

データのファイルへの保存は、PassThroughCalculatorを用いた「【MediaPipe】Multi Hand Trackingから検出データを抽出した」を基にしている。

ただし、「【MediaPipe】Multi Hand Trackingから検出データを抽出した」は、MediaPipe v0.7.4 を基にしており、v0.8.9では Hand Tracking のグラフが変更されているのでそのまま動作させることができなかった。

MediaPipe v0.8.9 の Hand Tracking のグラフ　mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt を Mediapipe Visualizer で可視化すると次の通りであった。

このHandLandMarkTrackingCpu と HandRenderer の間に、PathThroughCalculator を追加した(mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt)。Mediapipe Visualizer で可視化したものは次の通りである。

mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt　は以下の通り。

# MediaPipe graph that performs hands tracking on desktop with TensorFlow
# Lite on CPU.
# Used in the example in
# mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out.
# This file is based on mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

# CPU image. (ImageFrame)
input_stream: "input_video"

# CPU image. (ImageFrame)
output_stream: "output_video"


output_stream: "output_palm_detections"
output_stream: "output_landmarks"
output_stream: "output_palm_rects"
output_stream: "output_hand_rects"


# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingCpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:multi_palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:multi_hand_rects"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:multi_palm_rects"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_cpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:output_palm_detections"
  input_stream: "LANDMARKS:output_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:output_palm_rects"
  input_stream: "NORM_RECTS:1:output_hand_rects"
  output_stream: "IMAGE:output_video"
}

# Streams to get results of detection and landmark
node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_palm_detections"
  output_stream: "output_palm_detections"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "landmarks"
  output_stream: "output_landmarks"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_palm_rects"
  output_stream: "output_palm_rects"
}

node {
  calculator: "PassThroughCalculator"
  input_stream: "multi_hand_rects"
  output_stream: "output_hand_rects"
}

# MediaPipe graph that performs hands tracking on desktop with TensorFlow

# Lite on CPU.

# Used in the example in

# mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out.

# This file is based on mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

# CPU image. (ImageFrame)

input_stream: "input_video"

# CPU image. (ImageFrame)

output_stream: "output_video"

output_stream: "output_palm_detections"

output_stream: "output_landmarks"

output_stream: "output_palm_rects"

output_stream: "output_hand_rects"

# Generates side packet cotaining max number of hands to detect/track.

node {

calculator: "ConstantSidePacketCalculator"

output_side_packet: "PACKET:num_hands"

node_options: {

[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {

packet { int_value: 2 }

}

# Detects/tracks hand landmarks.

node {

calculator: "HandLandmarkTrackingCpu"

input_stream: "IMAGE:input_video"

input_side_packet: "NUM_HANDS:num_hands"

output_stream: "LANDMARKS:landmarks"

output_stream: "HANDEDNESS:handedness"

output_stream: "PALM_DETECTIONS:multi_palm_detections"

output_stream: "HAND_ROIS_FROM_LANDMARKS:multi_hand_rects"

output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:multi_palm_rects"

}

# Subgraph that renders annotations and overlays them on top of the input

# images (see hand_renderer_cpu.pbtxt).

node {

calculator: "HandRendererSubgraph"

input_stream: "IMAGE:input_video"

input_stream: "DETECTIONS:output_palm_detections"

input_stream: "LANDMARKS:output_landmarks"

input_stream: "HANDEDNESS:handedness"

input_stream: "NORM_RECTS:0:output_palm_rects"

input_stream: "NORM_RECTS:1:output_hand_rects"

output_stream: "IMAGE:output_video"

}

# Streams to get results of detection and landmark

node {

calculator: "PassThroughCalculator"

input_stream: "multi_palm_detections"

output_stream: "output_palm_detections"

}

node {

calculator: "PassThroughCalculator"

input_stream: "landmarks"

output_stream: "output_landmarks"

}

node {

calculator: "PassThroughCalculator"

input_stream: "multi_palm_rects"

output_stream: "output_palm_rects"

}

node {

calculator: "PassThroughCalculator"

input_stream: "multi_hand_rects"

output_stream: "output_hand_rects"

}

demo_run_graph_main.cc の変更

グラフを実行して、PassThroughCalculator のデータを保存するために、mediapipe/examples/desktop/demo_run_graph_main.cc を変更し、mediapipe/examples/desktop/demo_run_graph_main_out.cc とした。

以前のMediaPipe v0.7.4　ではフレームごとにランドマークや手の矩形、手の平、手の平の矩形などが出力されていたようであるが、MediaPipe v0.8.9では、実際に検出されたときにしかデータ出力されない。このため、すべてのデータが揃うのを待っていては適切なタイミングでデータを保存できない。

やむを得ないので、PassThroughCalculator にPoller を追加し、そのキューが空でなければ(poller_?????.QueueSize() > 0)、データがあるとし、すべてのデータが揃った際にファイルに(save_dataがtrueの場合に)データを出力することにした。すべてのデータが揃わなくてもデータを保存した方がよい場合には、ファイルへ出力する(save_dataの)条件を変更する必要がある。

データが揃わない場合(save_dataが falseの場合)には、processTypeList でキューの処理は行うが、ファイルへのデータ出力は行わない。この処理のタイミングがフローによって異なるため、キューの処理よりも入力ビデオの処理が速い場合には、データ出力のタイミングがずれる可能性がある。現時点では、処理をするCPUの性能が十分に高ければ、そのようなことはないと考えている。念のために、キューの長さが1より大きくなったら処理が間に合っていないとして、警告(LOG(WARNING))を出力するようにしている。

mediapipe/examples/desktop/demo_run_graph_main_out.cc　は以下の通りである。

// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// An example of sending OpenCV webcam frames into a MediaPipe graph.

// This file is based on mediapipe/examples/desktop/demo_run_graph_main.cc and
// https://dev.classmethod.jp/articles/mediapipe-extract-data-from-multi-hand-tracking/ .

#include <cstdlib>

#include <iostream>
#include <filesystem>
#include <fstream>
#include <sys/stat.h>
#include <cstdio>
#include <regex>


#include "absl/flags/flag.h"
#include "absl/flags/parse.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/image_frame.h"
#include "mediapipe/framework/formats/image_frame_opencv.h"
#include "mediapipe/framework/port/file_helpers.h"
#include "mediapipe/framework/port/opencv_highgui_inc.h"
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
#include "mediapipe/framework/port/opencv_video_inc.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

constexpr char kInputStream[] = "input_video";
constexpr char kOutputStream[] = "output_video";
constexpr char kWindowName[] = "MediaPipe";


#include "mediapipe/framework/formats/detection.pb.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/formats/rect.pb.h"
#include "mediapipe/framework/timestamp.h"

#include <direct.h>

constexpr char kOutputPalmDetections[] = "output_palm_detections";
constexpr char kOutputLandmarks[] = "output_landmarks";
constexpr char kOutputPalmRects[] = "output_palm_rects";
constexpr char kOutputHandRects[] = "output_hand_rects";


ABSL_FLAG(std::string, calculator_graph_config_file, "",
          "Name of file containing text format CalculatorGraphConfig proto.");
ABSL_FLAG(std::string, input_video_path, "",
          "Full path of video to load. "
          "If not provided, attempt to use a webcam.");
ABSL_FLAG(std::string, output_video_path, "",
          "Full path of where to save result (.mp4 only). "
          "If not provided, show result in a window.");

std::string output_dirpath = std::string("./result");
std::string output_data_extension = std::string(".bin");


std::string outputFilePath(mediapipe::Timestamp timestamp, std::string type, int index, std::string postfix) {
	std::ostringstream os;

	os << output_dirpath + "/"
		<< timestamp << "_"
		<< type << "_"
		<< std::to_string(index)
		<< postfix;
	return os.str();
}

std::string outputFilePath(mediapipe::Timestamp timestamp, std::string postfix) {
	std::ostringstream os;

	os << output_dirpath + "/"
		<< timestamp << "_"
		<< postfix;
	return os.str();
}

template<typename TVector>
bool
processTypeList(mediapipe::OutputStreamPoller& poller, std::string type, bool save) {
	bool ret = false;
	mediapipe::Packet	packet;
	// process landmarks
	if ( poller.QueueSize() > 0 && poller.Next(&packet) ) {
		if ( poller.QueueSize() > 1 ) LOG(WARNING) << "QueueSize: " << poller.QueueSize();
		auto &output = packet.Get<TVector>();
		if ( save ) {	// save data to output file if save is true
			for (int j = 0; j < output.size(); j++)
			{
				std::string filePath = outputFilePath(packet.Timestamp(), type, j,output_data_extension);
				std::ofstream outputfile(filePath);

				std::string serializedStr;
				output[j].SerializeToString(&serializedStr);
				outputfile << serializedStr << std::flush;
			}
			ret = save;
		}
	}
	return ret;
}


absl::Status RunMPPGraph() {
  std::string calculator_graph_config_contents;
  MP_RETURN_IF_ERROR(mediapipe::file::GetContents(
      absl::GetFlag(FLAGS_calculator_graph_config_file),
      &calculator_graph_config_contents));
  LOG(INFO) << "Get calculator graph config contents: "
            << calculator_graph_config_contents;
  mediapipe::CalculatorGraphConfig config =
      mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(
          calculator_graph_config_contents);

  LOG(INFO) << "Initialize the calculator graph.";
  mediapipe::CalculatorGraph graph;
  MP_RETURN_IF_ERROR(graph.Initialize(config));

  LOG(INFO) << "Initialize the camera or load the video.";
  cv::VideoCapture capture;
  const bool load_video = !absl::GetFlag(FLAGS_input_video_path).empty();
  if (load_video) {
    capture.open(absl::GetFlag(FLAGS_input_video_path));
  } else {
    capture.open(0);
  }
  RET_CHECK(capture.isOpened());

  cv::VideoWriter writer;
  const bool save_video = !absl::GetFlag(FLAGS_output_video_path).empty();
  if (!save_video) {
    cv::namedWindow(kWindowName, /*flags=WINDOW_AUTOSIZE*/ 1);
#if (CV_MAJOR_VERSION >= 3) && (CV_MINOR_VERSION >= 2)
    capture.set(cv::CAP_PROP_FRAME_WIDTH, 640);
    capture.set(cv::CAP_PROP_FRAME_HEIGHT, 480);
    capture.set(cv::CAP_PROP_FPS, 30);
#endif
  }

  LOG(INFO) << "Start running the calculator graph.";


  // Connect pollers to graph
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_video,
                   graph.AddOutputStreamPoller(kOutputStream));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_detections,
                   graph.AddOutputStreamPoller(kOutputPalmDetections));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_landmarks,
                   graph.AddOutputStreamPoller(kOutputLandmarks));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_rects,
                   graph.AddOutputStreamPoller(kOutputPalmRects));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_hand_rects,
                   graph.AddOutputStreamPoller(kOutputHandRects));

  
  MP_RETURN_IF_ERROR(graph.StartRun({}));

  LOG(INFO) << "Start grabbing and processing frames.";
  bool grab_frames = true;

  // define output folder path
  std::string flag_output_video_path = absl::GetFlag(FLAGS_output_video_path);
  std::filesystem::path output_video_path = flag_output_video_path;
  std::string output_video_dir = output_video_path.parent_path().string();
  LOG(INFO) << "output_video_dir:" << output_video_dir;

  if ( ! output_video_dir.empty() ) {
	  output_dirpath =  output_video_dir;
  }
  _mkdir(output_dirpath.c_str()); // for windows
  
  while (grab_frames)
  {
    // Capture opencv camera or video frame.
    cv::Mat camera_frame_raw;
    capture >> camera_frame_raw;

    if (camera_frame_raw.empty()) {
      if (!load_video) {
        LOG(INFO) << "Ignore empty frames from camera.";
        continue;
      }
      LOG(INFO) << "Empty frame, end of video reached.";
      break;
    }
    cv::Mat camera_frame, flipped_frame;
    cv::cvtColor(camera_frame_raw, camera_frame, cv::COLOR_BGR2RGB);

    // Wrap Mat into an ImageFrame.
    auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
        mediapipe::ImageFormat::SRGB, camera_frame.cols, camera_frame.rows,
        mediapipe::ImageFrame::kDefaultAlignmentBoundary);
    cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
    camera_frame.copyTo(input_frame_mat);

    // Send image packet into the graph.
    size_t frame_timestamp_us =
        (double)cv::getTickCount() / (double)cv::getTickFrequency() * 1e6;
    MP_RETURN_IF_ERROR(graph.AddPacketToInputStream(
        kInputStream, mediapipe::Adopt(input_frame.release())
                          .At(mediapipe::Timestamp(frame_timestamp_us))));



    // Get the graph result packet, or stop if that fails.
    mediapipe::Packet packet_video;

    // check if data exist from graph
    if (!poller_video.Next(&packet_video) ) {
		break;
	}

    // get output from graph
    auto &output_video = packet_video.Get<mediapipe::ImageFrame>();
	auto packet_video_timestamp =  packet_video.Timestamp();
	LOG(INFO) << "packet_video.Timestamp:" << packet_video_timestamp;
	
	// save_data is true if all data is available (If partial data should be saved, change the following conditions)
	bool save_data = (poller_palm_detections.QueueSize() > 0)
					&& (poller_palm_rects.QueueSize() > 0)
					&& (poller_hand_rects.QueueSize() > 0)
					&& (poller_landmarks.QueueSize() > 0);

	// process palm detections
	processTypeList<std::vector<mediapipe::Detection>>(poller_palm_detections, std::string("palm_detections"), save_data);

	// process palm rects
	processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_palm_rects, std::string("palm_rects"), save_data);

	// process hand rects
	processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_hand_rects, std::string("hand_rects"), save_data);

	// process landmarks
	processTypeList<std::vector<mediapipe::NormalizedLandmarkList>>(poller_landmarks, std::string("landmarks"), save_data);


	
    // Convert back to opencv for display or saving.
    cv::Mat output_frame_mat = mediapipe::formats::MatView(&output_video);
    cv::cvtColor(output_frame_mat, output_frame_mat, cv::COLOR_RGB2BGR);
    if (save_video)
    {
      if (!writer.isOpened())
      {
        LOG(INFO) << "Prepare video writer:" << flag_output_video_path;
        writer.open(flag_output_video_path,
                    mediapipe::fourcc('a', 'v', 'c', '1'), // .mp4
                    capture.get(cv::CAP_PROP_FPS), output_frame_mat.size());
        RET_CHECK(writer.isOpened());
      }
      writer.write(output_frame_mat);
    }


	// show output image
	cv::imshow(kWindowName, output_frame_mat);
	// Press any key to exit.
	const int pressed_key = cv::waitKey(5);
	if (pressed_key >= 0 && pressed_key != 255)
		grab_frames = false;


	if ( !save_data ) continue;

    // save input frame to file
	std::string inputFramePath = outputFilePath(mediapipe::Timestamp(frame_timestamp_us), std::string("inputFrame.jpg"));
    cv::imwrite(inputFramePath, input_frame_mat);
	
    // save output frame to file
	std::string outputFramePath = outputFilePath(packet_video_timestamp, "outputFrame.jpg");
    cv::imwrite(outputFramePath, output_frame_mat);
  }

  LOG(INFO) << "Shutting down.";
  if (writer.isOpened()) writer.release();
  MP_RETURN_IF_ERROR(graph.CloseInputStream(kInputStream));
  return graph.WaitUntilDone();
}

int main(int argc, char** argv) {
  google::InitGoogleLogging(argv[0]);
  absl::ParseCommandLine(argc, argv);
  absl::Status run_status = RunMPPGraph();
  if (!run_status.ok()) {
    LOG(ERROR) << "Failed to run the graph: " << run_status.message();
    return EXIT_FAILURE;
  } else {
    LOG(INFO) << "Success!";
  }
  return EXIT_SUCCESS;
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

// Licensed under the Apache License, Version 2.0 (the "License");

// you may not use this file except in compliance with the License.

// You may obtain a copy of the License at

// http://www.apache.org/licenses/LICENSE-2.0

// Unless required by applicable law or agreed to in writing, software

// distributed under the License is distributed on an "AS IS" BASIS,

// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

// See the License for the specific language governing permissions and

// limitations under the License.

// An example of sending OpenCV webcam frames into a MediaPipe graph.

// This file is based on mediapipe/examples/desktop/demo_run_graph_main.cc and

// https://dev.classmethod.jp/articles/mediapipe-extract-data-from-multi-hand-tracking/ .

#include <cstdlib>

#include <iostream>

#include <filesystem>

#include <fstream>

#include <sys/stat.h>

#include <cstdio>

#include <regex>

#include "absl/flags/flag.h"

#include "absl/flags/parse.h"

#include "mediapipe/framework/calculator_framework.h"

#include "mediapipe/framework/formats/image_frame.h"

#include "mediapipe/framework/formats/image_frame_opencv.h"

#include "mediapipe/framework/port/file_helpers.h"

#include "mediapipe/framework/port/opencv_highgui_inc.h"

#include "mediapipe/framework/port/opencv_imgproc_inc.h"

#include "mediapipe/framework/port/opencv_video_inc.h"

#include "mediapipe/framework/port/parse_text_proto.h"

#include "mediapipe/framework/port/status.h"

constexpr char kInputStream[] = "input_video";

constexpr char kOutputStream[] = "output_video";

constexpr char kWindowName[] = "MediaPipe";

#include "mediapipe/framework/formats/detection.pb.h"

#include "mediapipe/framework/formats/landmark.pb.h"

#include "mediapipe/framework/formats/rect.pb.h"

#include "mediapipe/framework/timestamp.h"

#include <direct.h>

constexpr char kOutputPalmDetections[] = "output_palm_detections";

constexpr char kOutputLandmarks[] = "output_landmarks";

constexpr char kOutputPalmRects[] = "output_palm_rects";

constexpr char kOutputHandRects[] = "output_hand_rects";

ABSL_FLAG(std::string, calculator_graph_config_file, "",

"Name of file containing text format CalculatorGraphConfig proto.");

ABSL_FLAG(std::string, input_video_path, "",

"Full path of video to load. "

"If not provided, attempt to use a webcam.");

ABSL_FLAG(std::string, output_video_path, "",

"Full path of where to save result (.mp4 only). "

"If not provided, show result in a window.");

std::string output_dirpath = std::string("./result");

std::string output_data_extension = std::string(".bin");

std::string outputFilePath(mediapipe::Timestamp timestamp, std::string type, int index, std::string postfix) {

std::ostringstream os;

os << output_dirpath + "/"

<< timestamp << "_"

<< type << "_"

<< std::to_string(index)

<< postfix;

return os.str();

}

std::string outputFilePath(mediapipe::Timestamp timestamp, std::string postfix) {

std::ostringstream os;

os << output_dirpath + "/"

<< timestamp << "_"

<< postfix;

return os.str();

}

template<typename TVector>

bool

processTypeList(mediapipe::OutputStreamPoller& poller, std::string type, bool save) {

bool ret = false;

mediapipe::Packet packet;

// process landmarks

if ( poller.QueueSize() > 0 && poller.Next(&packet) ) {

if ( poller.QueueSize() > 1 ) LOG(WARNING) << "QueueSize: " << poller.QueueSize();

auto &output = packet.Get<TVector>();

if ( save ) { // save data to output file if save is true

for (int j = 0; j < output.size(); j++)

{

std::string filePath = outputFilePath(packet.Timestamp(), type, j,output_data_extension);

std::ofstream outputfile(filePath);

std::string serializedStr;

output[j].SerializeToString(&serializedStr);

outputfile << serializedStr << std::flush;

}

ret = save;

}

return ret;

}

absl::Status RunMPPGraph() {

std::string calculator_graph_config_contents;

MP_RETURN_IF_ERROR(mediapipe::file::GetContents(

absl::GetFlag(FLAGS_calculator_graph_config_file),

&calculator_graph_config_contents));

LOG(INFO) << "Get calculator graph config contents: "

<< calculator_graph_config_contents;

mediapipe::CalculatorGraphConfig config =

mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(

calculator_graph_config_contents);

LOG(INFO) << "Initialize the calculator graph.";

mediapipe::CalculatorGraph graph;

MP_RETURN_IF_ERROR(graph.Initialize(config));

LOG(INFO) << "Initialize the camera or load the video.";

cv::VideoCapture capture;

const bool load_video = !absl::GetFlag(FLAGS_input_video_path).empty();

if (load_video) {

capture.open(absl::GetFlag(FLAGS_input_video_path));

} else {

capture.open(0);

}

RET_CHECK(capture.isOpened());

cv::VideoWriter writer;

const bool save_video = !absl::GetFlag(FLAGS_output_video_path).empty();

if (!save_video) {

cv::namedWindow(kWindowName, /*flags=WINDOW_AUTOSIZE*/ 1);

#if (CV_MAJOR_VERSION >= 3) && (CV_MINOR_VERSION >= 2)

capture.set(cv::CAP_PROP_FRAME_WIDTH, 640);

capture.set(cv::CAP_PROP_FRAME_HEIGHT, 480);

capture.set(cv::CAP_PROP_FPS, 30);

#endif

}

LOG(INFO) << "Start running the calculator graph.";

// Connect pollers to graph

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_video,

graph.AddOutputStreamPoller(kOutputStream));

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_detections,

graph.AddOutputStreamPoller(kOutputPalmDetections));

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_landmarks,

graph.AddOutputStreamPoller(kOutputLandmarks));

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_palm_rects,

graph.AddOutputStreamPoller(kOutputPalmRects));

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_hand_rects,

graph.AddOutputStreamPoller(kOutputHandRects));

MP_RETURN_IF_ERROR(graph.StartRun({}));

LOG(INFO) << "Start grabbing and processing frames.";

bool grab_frames = true;

// define output folder path

std::string flag_output_video_path = absl::GetFlag(FLAGS_output_video_path);

std::filesystem::path output_video_path = flag_output_video_path;

std::string output_video_dir = output_video_path.parent_path().string();

LOG(INFO) << "output_video_dir:" << output_video_dir;

if ( ! output_video_dir.empty() ) {

output_dirpath = output_video_dir;

}

_mkdir(output_dirpath.c_str()); // for windows

while (grab_frames)

{

// Capture opencv camera or video frame.

cv::Mat camera_frame_raw;

capture >> camera_frame_raw;

if (camera_frame_raw.empty()) {

if (!load_video) {

LOG(INFO) << "Ignore empty frames from camera.";

continue;

}

LOG(INFO) << "Empty frame, end of video reached.";

break;

}

cv::Mat camera_frame, flipped_frame;

cv::cvtColor(camera_frame_raw, camera_frame, cv::COLOR_BGR2RGB);

// Wrap Mat into an ImageFrame.

auto input_frame = absl::make_unique<mediapipe::ImageFrame>(

mediapipe::ImageFormat::SRGB, camera_frame.cols, camera_frame.rows,

mediapipe::ImageFrame::kDefaultAlignmentBoundary);

cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());

camera_frame.copyTo(input_frame_mat);

// Send image packet into the graph.

size_t frame_timestamp_us =

(double)cv::getTickCount() / (double)cv::getTickFrequency() * 1e6;

MP_RETURN_IF_ERROR(graph.AddPacketToInputStream(

kInputStream, mediapipe::Adopt(input_frame.release())

.At(mediapipe::Timestamp(frame_timestamp_us))));

// Get the graph result packet, or stop if that fails.

mediapipe::Packet packet_video;

// check if data exist from graph

if (!poller_video.Next(&packet_video) ) {

break;

}

// get output from graph

auto &output_video = packet_video.Get<mediapipe::ImageFrame>();

auto packet_video_timestamp = packet_video.Timestamp();

LOG(INFO) << "packet_video.Timestamp:" << packet_video_timestamp;

// save_data is true if all data is available (If partial data should be saved, change the following conditions)

bool save_data = (poller_palm_detections.QueueSize() > 0)

&& (poller_palm_rects.QueueSize() > 0)

&& (poller_hand_rects.QueueSize() > 0)

&& (poller_landmarks.QueueSize() > 0);

// process palm detections

processTypeList<std::vector<mediapipe::Detection>>(poller_palm_detections, std::string("palm_detections"), save_data);

// process palm rects

processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_palm_rects, std::string("palm_rects"), save_data);

// process hand rects

processTypeList<std::vector<mediapipe::NormalizedRect>>(poller_hand_rects, std::string("hand_rects"), save_data);

// process landmarks

processTypeList<std::vector<mediapipe::NormalizedLandmarkList>>(poller_landmarks, std::string("landmarks"), save_data);

// Convert back to opencv for display or saving.

cv::Mat output_frame_mat = mediapipe::formats::MatView(&output_video);

cv::cvtColor(output_frame_mat, output_frame_mat, cv::COLOR_RGB2BGR);

if (save_video)

{

if (!writer.isOpened())

{

LOG(INFO) << "Prepare video writer:" << flag_output_video_path;

writer.open(flag_output_video_path,

mediapipe::fourcc('a', 'v', 'c', '1'), // .mp4

capture.get(cv::CAP_PROP_FPS), output_frame_mat.size());

RET_CHECK(writer.isOpened());

}

writer.write(output_frame_mat);

}

// show output image

cv::imshow(kWindowName, output_frame_mat);

// Press any key to exit.

const int pressed_key = cv::waitKey(5);

if (pressed_key >= 0 && pressed_key != 255)

grab_frames = false;

if ( !save_data ) continue;

// save input frame to file

std::string inputFramePath = outputFilePath(mediapipe::Timestamp(frame_timestamp_us), std::string("inputFrame.jpg"));

cv::imwrite(inputFramePath, input_frame_mat);

// save output frame to file

std::string outputFramePath = outputFilePath(packet_video_timestamp, "outputFrame.jpg");

cv::imwrite(outputFramePath, output_frame_mat);

}

LOG(INFO) << "Shutting down.";

if (writer.isOpened()) writer.release();

MP_RETURN_IF_ERROR(graph.CloseInputStream(kInputStream));

return graph.WaitUntilDone();

}

int main(int argc, char** argv) {

google::InitGoogleLogging(argv[0]);

absl::ParseCommandLine(argc, argv);

absl::Status run_status = RunMPPGraph();

if (!run_status.ok()) {

LOG(ERROR) << "Failed to run the graph: " << run_status.message();

return EXIT_FAILURE;

} else {

LOG(INFO) << "Success!";

}

return EXIT_SUCCESS;

}

ビルド

ビルドするターゲットを mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out にし、基になるグラフをmediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt に、C++プログラムを mediapipe/examples/desktop/demo_run_graph_main_out.cc　にし、PathThroughCalculator を利用するため、bazel のBUILDファイルを変更した。

mediapipe/examples/desktop/BUILD

name = demo_run_graph_main_out の cc_library を追加し、依存しているものに以下を追加した。

"//mediapipe/framework/formats:detection_cc_proto",

"//mediapipe/framework/formats:landmark_cc_proto",

"//mediapipe/framework/formats:rect_cc_proto"

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

licenses(["notice"])

package(default_visibility = [
    "//visibility:public",
])

cc_library(
    name = "simple_run_graph_main",
    srcs = ["simple_run_graph_main.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:map_util",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:ret_check",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework/port:statusor",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
        "@com_google_absl//absl/strings",
    ],
)

cc_library(
    name = "demo_run_graph_main",
    srcs = ["demo_run_graph_main.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
    ],
)

cc_library(
    name = "demo_run_graph_main_out",
    srcs = ["demo_run_graph_main_out.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
		"//mediapipe/framework/formats:detection_cc_proto",
        "//mediapipe/framework/formats:landmark_cc_proto",
        "//mediapipe/framework/formats:rect_cc_proto",
    ],
)

# Linux only.
# Must have a GPU with EGL support:
# ex: sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
# (or similar nvidia/amd equivalent)
cc_library(
    name = "demo_run_graph_main_gpu",
    srcs = ["demo_run_graph_main_gpu.cc"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/port:file_helpers",
        "//mediapipe/framework/port:opencv_highgui",
        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:opencv_video",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "//mediapipe/gpu:gl_calculator_helper",
        "//mediapipe/gpu:gpu_buffer",
        "//mediapipe/gpu:gpu_shared_data_internal",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
    ],
)

100

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

licenses(["notice"])

package(default_visibility = [

"//visibility:public",

])

cc_library(

name = "simple_run_graph_main",

srcs = ["simple_run_graph_main.cc"],

deps = [

"//mediapipe/framework:calculator_framework",

"//mediapipe/framework/port:file_helpers",

"//mediapipe/framework/port:map_util",

"//mediapipe/framework/port:parse_text_proto",

"//mediapipe/framework/port:ret_check",

"//mediapipe/framework/port:status",

"//mediapipe/framework/port:statusor",

"@com_google_absl//absl/flags:flag",

"@com_google_absl//absl/flags:parse",

"@com_google_absl//absl/strings",

)

cc_library(

name = "demo_run_graph_main",

srcs = ["demo_run_graph_main.cc"],

deps = [

"//mediapipe/framework:calculator_framework",

"//mediapipe/framework/formats:image_frame",

"//mediapipe/framework/formats:image_frame_opencv",

"//mediapipe/framework/port:file_helpers",

"//mediapipe/framework/port:opencv_highgui",

"//mediapipe/framework/port:opencv_imgproc",

"//mediapipe/framework/port:opencv_video",

"//mediapipe/framework/port:parse_text_proto",

"//mediapipe/framework/port:status",

"@com_google_absl//absl/flags:flag",

"@com_google_absl//absl/flags:parse",

)

cc_library(

name = "demo_run_graph_main_out",

srcs = ["demo_run_graph_main_out.cc"],

deps = [

"//mediapipe/framework:calculator_framework",

"//mediapipe/framework/formats:image_frame",

"//mediapipe/framework/formats:image_frame_opencv",

"//mediapipe/framework/port:file_helpers",

"//mediapipe/framework/port:opencv_highgui",

"//mediapipe/framework/port:opencv_imgproc",

"//mediapipe/framework/port:opencv_video",

"//mediapipe/framework/port:parse_text_proto",

"//mediapipe/framework/port:status",

"@com_google_absl//absl/flags:flag",

"@com_google_absl//absl/flags:parse",

"//mediapipe/framework/formats:detection_cc_proto",

"//mediapipe/framework/formats:landmark_cc_proto",

"//mediapipe/framework/formats:rect_cc_proto",

)

# Linux only.

# Must have a GPU with EGL support:

# ex: sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev

# (or similar nvidia/amd equivalent)

cc_library(

name = "demo_run_graph_main_gpu",

srcs = ["demo_run_graph_main_gpu.cc"],

deps = [

"//mediapipe/framework:calculator_framework",

"//mediapipe/framework/formats:image_frame",

"//mediapipe/framework/formats:image_frame_opencv",

"//mediapipe/framework/port:file_helpers",

"//mediapipe/framework/port:opencv_highgui",

"//mediapipe/framework/port:opencv_imgproc",

"//mediapipe/framework/port:opencv_video",

"//mediapipe/framework/port:parse_text_proto",

"//mediapipe/framework/port:status",

"//mediapipe/gpu:gl_calculator_helper",

"//mediapipe/gpu:gpu_buffer",

"//mediapipe/gpu:gpu_shared_data_internal",

"@com_google_absl//absl/flags:flag",

"@com_google_absl//absl/flags:parse",

)

mediapipe/examples/desktop/hand_tracking/BUILD

name = hand_tracking_cpu_out の cc_library を追加し、依存しているものはhand_tracking_cpuを基にして以下とした。

"//mediapipe/examples/desktop:demo_run_graph_main_out", "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators"

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

licenses(["notice"])

package(default_visibility = ["//mediapipe/examples:__subpackages__"])

cc_binary(
    name = "hand_tracking_tflite",
    deps = [
        "//mediapipe/examples/desktop:simple_run_graph_main",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)

cc_binary(
    name = "hand_tracking_cpu",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)


cc_binary(
    name = "hand_tracking_cpu_out",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main_out",
        "//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",
    ],
)

# Linux only
cc_binary(
    name = "hand_tracking_gpu",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main_gpu",
        "//mediapipe/graphs/hand_tracking:mobile_calculators",
    ],
)

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

licenses(["notice"])

package(default_visibility = ["//mediapipe/examples:__subpackages__"])

cc_binary(

name = "hand_tracking_tflite",

deps = [

"//mediapipe/examples/desktop:simple_run_graph_main",

"//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",

)

cc_binary(

name = "hand_tracking_cpu",

deps = [

"//mediapipe/examples/desktop:demo_run_graph_main",

"//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",

)

cc_binary(

name = "hand_tracking_cpu_out",

deps = [

"//mediapipe/examples/desktop:demo_run_graph_main_out",

"//mediapipe/graphs/hand_tracking:desktop_tflite_calculators",

)

# Linux only

cc_binary(

name = "hand_tracking_gpu",

deps = [

"//mediapipe/examples/desktop:demo_run_graph_main_gpu",

"//mediapipe/graphs/hand_tracking:mobile_calculators",

)

mediapipe/graphs/hand_tracking/BUILD

name = "desktop_tflite_calculators" の cc_library の依存しているものに "//mediapipe/calculators/core:pass_through_calculator" を追加した。

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

load(
    "//mediapipe/framework/tool:mediapipe_graph.bzl",
    "mediapipe_binary_graph",
)

licenses(["notice"])

package(default_visibility = ["//visibility:public"])

exports_files(glob([
    "*.pbtxt",
]))

cc_library(
    name = "desktop_offline_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/core:gate_calculator",
        "//mediapipe/calculators/core:immediate_mux_calculator",
        "//mediapipe/calculators/core:packet_inner_join_calculator",
        "//mediapipe/calculators/core:previous_loopback_calculator",
        "//mediapipe/calculators/video:opencv_video_decoder_calculator",
        "//mediapipe/calculators/video:opencv_video_encoder_calculator",
    ],
)

cc_library(
    name = "desktop_tflite_calculators",
    deps = [
        ":desktop_offline_calculators",
        "//mediapipe/calculators/core:constant_side_packet_calculator",
        "//mediapipe/calculators/core:merge_calculator",
        "//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_cpu",
        "//mediapipe/modules/hand_landmark:hand_landmark_tracking_cpu",
		"//mediapipe/calculators/core:pass_through_calculator",
    ],
)

mediapipe_binary_graph(
    name = "hand_tracking_desktop_live_binary_graph",
    graph = "hand_tracking_desktop_live.pbtxt",
    output_name = "hand_tracking_desktop_live.binarypb",
    deps = [":desktop_tflite_calculators"],
)

cc_library(
    name = "mobile_calculators",
    deps = [
        "//mediapipe/calculators/core:constant_side_packet_calculator",
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_gpu",
        "//mediapipe/modules/hand_landmark:hand_landmark_tracking_gpu",
    ],
)

mediapipe_binary_graph(
    name = "hand_tracking_mobile_gpu_binary_graph",
    graph = "hand_tracking_mobile.pbtxt",
    output_name = "hand_tracking_mobile_gpu.binarypb",
    deps = [":mobile_calculators"],
)

cc_library(
    name = "detection_mobile_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/util:annotation_overlay_calculator",
        "//mediapipe/calculators/util:detections_to_render_data_calculator",
        "//mediapipe/modules/palm_detection:palm_detection_gpu",
    ],
)

mediapipe_binary_graph(
    name = "hand_detection_mobile_gpu_binary_graph",
    graph = "hand_detection_mobile.pbtxt",
    output_name = "hand_detection_mobile_gpu.binarypb",
    deps = [":detection_mobile_calculators"],
)

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

load(

"//mediapipe/framework/tool:mediapipe_graph.bzl",

"mediapipe_binary_graph",

)

licenses(["notice"])

package(default_visibility = ["//visibility:public"])

exports_files(glob([

"*.pbtxt",

]))

cc_library(

name = "desktop_offline_calculators",

deps = [

"//mediapipe/calculators/core:flow_limiter_calculator",

"//mediapipe/calculators/core:gate_calculator",

"//mediapipe/calculators/core:immediate_mux_calculator",

"//mediapipe/calculators/core:packet_inner_join_calculator",

"//mediapipe/calculators/core:previous_loopback_calculator",

"//mediapipe/calculators/video:opencv_video_decoder_calculator",

"//mediapipe/calculators/video:opencv_video_encoder_calculator",

)

cc_library(

name = "desktop_tflite_calculators",

deps = [

":desktop_offline_calculators",

"//mediapipe/calculators/core:constant_side_packet_calculator",

"//mediapipe/calculators/core:merge_calculator",

"//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_cpu",

"//mediapipe/modules/hand_landmark:hand_landmark_tracking_cpu",

"//mediapipe/calculators/core:pass_through_calculator",

)

mediapipe_binary_graph(

name = "hand_tracking_desktop_live_binary_graph",

graph = "hand_tracking_desktop_live.pbtxt",

output_name = "hand_tracking_desktop_live.binarypb",

deps = [":desktop_tflite_calculators"],

)

cc_library(

name = "mobile_calculators",

deps = [

"//mediapipe/calculators/core:constant_side_packet_calculator",

"//mediapipe/calculators/core:flow_limiter_calculator",

"//mediapipe/graphs/hand_tracking/subgraphs:hand_renderer_gpu",

"//mediapipe/modules/hand_landmark:hand_landmark_tracking_gpu",

)

mediapipe_binary_graph(

name = "hand_tracking_mobile_gpu_binary_graph",

graph = "hand_tracking_mobile.pbtxt",

output_name = "hand_tracking_mobile_gpu.binarypb",

deps = [":mobile_calculators"],

)

cc_library(

name = "detection_mobile_calculators",

deps = [

"//mediapipe/calculators/core:flow_limiter_calculator",

"//mediapipe/calculators/util:annotation_overlay_calculator",

"//mediapipe/calculators/util:detections_to_render_data_calculator",

"//mediapipe/modules/palm_detection:palm_detection_gpu",

)

mediapipe_binary_graph(

name = "hand_detection_mobile_gpu_binary_graph",

graph = "hand_detection_mobile.pbtxt",

output_name = "hand_detection_mobile_gpu.binarypb",

deps = [":detection_mobile_calculators"],

)

環境設定

利用したツールなどは以下の設定を行ってコマンドプロンプト内でビルドした。

set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC
set BAZEL_VC_FULL_VERSION=14.28.29333
set BAZEL_WINSDK_FULL_VERSION=10.0.19041.0
set PYTHON_BIN_PATH="C://python39//python.exe"
cd D:\mediapipe_repo\mediapipe

set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools

set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC

set BAZEL_VC_FULL_VERSION=14.28.29333

set BAZEL_WINSDK_FULL_VERSION=10.0.19041.0

set PYTHON_BIN_PATH="C://python39//python.exe"

cd D:\mediapipe_repo\mediapipe

以下では、この設定の下での例を示す。

bazel build の実行

D:\mediapipe_repo\mediapipe で以下のbazel build を実行し、hand_tracking_cpu_out.exe　をビルドした。

bazel-bin\mediapipe\examples\desktop\hand_tracking\hand_tracking_cpu_out.exe　に実行ファイルが構築された。

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH=%PYTHON_BIN_PATH% mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out

1	bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH=%PYTHON_BIN_PATH% mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu_out

hand_tracking_cpu_out.exeの実行

ビルドした hand_tracking_cpu_out.exe を以下のコマンドで実行した。

bazel-bin\mediapipe\examples\desktop\hand_tracking\hand_tracking_cpu_out.exe --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt

1	bazel-bin\mediapipe\examples\desktop\hand_tracking\hand_tracking_cpu_out.exe --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_out.pbtxt

以下の例では、set GLOG_logtostderr=1　を実行しておき、ログを標準エラー出力に表示している。この環境変数の設定を行わないと、画面には出力されない。

実行時のプレビューウィンドウ

実行時のログの情報出力

ログの出力レベルを変更するには環境変数を設定する。set GLOG_minloglevel=1　を実行するすることで、INFOレベルのメッセージは出力せず、WARNING以上を出力する。値に2、 3 を指定することで、それぞれERROR以上、FATAL になる。

データ出力

hand_tracking_cpu_out.cc の中の std::string output_dirpath = std::string("./result"); で出力を指定している。この例では、カレントディレクトリをD:\mediapipe_repo\mediapipe として実行しているので、　D:\mediapipe_repo\mediapipe\result　にデータが出力される。

出力された .bin ファイルは人が可読形式になっていないので、必要に応じて変換を行う必要がある。