How to detect hands, eyes, or other things by the live camera in the flutter

We were developing one app that we need to implement hand detection and eye detection features from a live camera feed. So we have done a lot of R&D on object (hand and eye) detection.

We found some solutions for it but all are different for hand and eye detection. we need only one solution which can acquire both hand and eye detection in a flutter. So after some more R&D, we came up with one solution which can fulfil our requirements.

We have used the image labelling functionality of thegoole_ml_vision pub package to detect hands and eyes.

Let me tell you how we used it for Hand and Eyes detection.

Add pub dependency

google_ml_vision: ^0.0.7
camera: 0.9.5

Init Image labeller and camera

final ImageLabeler _imageLabeler = GoogleVision.instance.imageLabeler();
CameraController? camera;
bool palmFound = false;
int palmFoundCount = 0;

Detect Hand or Eye from a live camera feed

Future<void> _initializeCamera() async {
  final CameraDescription description = await getCamera(_direction);

  camera = CameraController(
    description,
    ResolutionPreset.high,
    enableAudio: false,
  );
  await camera!.initialize();
  update();
  await camera!.startImageStream((CameraImage image) {
    if (!palmFound.value) {
      detect(
        image: image,
        detectInImage: _getDetectionMethod(),
        imageRotation: description.sensorOrientation,
      ).then(
        (dynamic results) {
          for (var element in results) {
            //// you can use 'eyelash' for eye detection.
            if (element.text!.toLowerCase() == 'hand' &&
                element.confidence! > 0.75) {
              palmFoundCount = palmFoundCount + 1;
              if (palmFoundCount == 50) {
                palmFound = true;
                if (palmFound) {
                  // do your code if hand detect.
               }
              }
            }
          }
        },
      );
    }
  });
}

static Future<dynamic> detect({
  required CameraImage image,
  required Future<dynamic> Function(GoogleVisionImage image) detectInImage,
  required int imageRotation,
}) async {
  return detectInImage(
    GoogleVisionImage.fromBytes(
      _concatenatePlanes(image.planes),
      _buildMetaData(image, _rotationIntToImageRotation(imageRotation)),
    ),
  );
}

static Uint8List _concatenatePlanes(List<Plane> planes) {
  final WriteBuffer allBytes = WriteBuffer();
  for (var plane in planes) {
    allBytes.putUint8List(plane.bytes);
  }
  return allBytes.done().buffer.asUint8List();
}

static GoogleVisionImageMetadata _buildMetaData(
  CameraImage image,
  ImageRotation rotation,
) {
  return GoogleVisionImageMetadata(
    rawFormat: image.format.raw,
    size: Size(image.width.toDouble(), image.height.toDouble()),
    rotation: rotation,
    planeData: image.planes.map(
      (Plane plane) {
        return GoogleVisionImagePlaneMetadata(
          bytesPerRow: plane.bytesPerRow,
          height: plane.height,
          width: plane.width,
        );
      },
    ).toList(),
  );
}

static ImageRotation _rotationIntToImageRotation(int rotation) {
  switch (rotation) {
    case 0:
      return ImageRotation.rotation0;
    case 90:
      return ImageRotation.rotation90;
    case 180:
      return ImageRotation.rotation180;
    default:
      assert(rotation == 270);
      return ImageRotation.rotation270;
  }
}

static Future<CameraDescription> getCamera(CameraLensDirection dir) async {
  return availableCameras().then(
    (List<CameraDescription> cameras) => cameras.firstWhere(
      (CameraDescription camera) => camera.lensDirection == dir,
    ),
  );
}

Future<dynamic> Function(GoogleVisionImage image) _getDetectionMethod() {
  return _imageLabeler.processImage;
}

First, init camera and image labeller from goole_ml_vision package.

then start startImageStream on the camera it will give you continuously live images from the camera feed. Then process that detect method It will provide you with different types of labels with the confidence of things which are available on that image.

In this way, we can simply detect the hand and eye from the live camera feed.

You can also find or detect labels from picked or capture images as explained below.

Detect Hand or Eye from the image file

File image;
checkEyelashFoundOrNotInImage() async {
  final GoogleVisionImage visionImage = GoogleVisionImage.fromFile(image!);
  List<ImageLabel> results;
  results = await _imageLabeler.processImage(visionImage);
  for (var element in results) {
    if (kDebugMode) {
      print('${element.text} - ${element.confidence}');
    }
    if (element.text!.toLowerCase() == 'hand' &&
      element.confidence! > 0.75) {
      palmFound = true;
      break;
    }
  }
}

You also can detect a face, barcode and text recognised by this package.

final BarcodeDetector barcodeDetector = GoogleVision.instance.barcodeDetector();
final FaceDetector faceDetector = GoogleVision.instance.faceDetector();
final ImageLabeler labeler = GoogleVision.instance.imageLabeler();
final TextRecognizer textRecognizer = GoogleVision.instance.textRecognizer();

To detect face with landmarks, the hand with landmarks check out this Github repo which uses tensor flow models.

A Flutter plugin to use the capabilities of on-device Google ML Kit Vision APIs. To use this plugin, add google_ml_vision as a dependency in your pubspec.yaml file.

pub.dev

GitHub - Inficial-Infotech/object-detection-demo: A object detection demo

A new Flutter project. This project is a starting point for a Flutter application. A few resources to get you started if this is your first Flutter project.

github.com

GitHub - JaeHeee/FlutterWithMediaPipe: Flutter with MediaPipe ML models

Flutter with MediaPipe Project. This is Flutter project using MediaPipe ML models.

github.com

• Detect face on real-time
• Detect hand on real-time
• Detect face landmark on real-time
• Estimate body pose on real-time
• Visualize inference results

Happy Hunting!