@capacitor-mlkit/face-mesh-detection¶

Unofficial Capacitor plugin for ML Kit Face Mesh Detection.¹

Installation¶

npm install @capacitor-mlkit/face-mesh-detection@next
npx cap sync

Attention: Please use the next tag to install the latest version of the plugin due to issue #179.

Android¶

Variables¶

This plugin will use the following project variables (defined in your app’s variables.gradle file):

$mlkitFaceMeshDetectionVersion version of com.google.mlkit:face-mesh-detection (default: 16.0.0-beta1)

Configuration¶

No configuration required for this plugin.

Demo¶

A working example can be found here: robingenz/capacitor-mlkit-plugin-demo

Usage¶

import { FaceMeshDetection, UseCase } from '@capacitor-mlkit/face-mesh-detection';

const processImage = async () => {
  const { faceMeshs } = await FaceMeshDetection.processImage({
    path: 'path/to/image.jpg',
    useCase: UseCase.FaceMesh,
  });
  return faceMeshs;
};

processImage(...)¶

processImage(options: ProcessImageOptions) => Promise<ProcessImageResult>

Detects face mesh from the supplied image.

Only available on Android.

Param	Type
`options`	`ProcessImageOptions`

Returns: Promise<ProcessImageResult>

Since: 5.3.0

Interfaces¶

ProcessImageResult¶

Prop	Type	Description	Since
`faceMeshs`	`FaceMesh[]`	The detected face meshs.	5.3.0

FaceMesh¶

Represents a face mesh detected by FaceMeshDetector.

When BoundingBoxOnly is selected, FaceMesh only contains valid bounding box.

When FaceMesh is selected, FaceMesh also contains a group of 468 3D face mesh points and related triangle information. Each point is represented by FaceMeshPoint describing a specific position in detected face. The triangle information is a group of 3 FaceMeshPoints representing a valid surface on Face (e.g. a valid small surface on nose tip).

Prop	Type	Description	Since
`bounds`	`Rect`	Returns the axis-aligned bounding rectangle of the detected face mesh.	5.3.0
`contours`	`Contours`	Returns contours with a list of `FaceMeshPoint` representing the detected face.	5.3.0
`faceMeshPoints`	`FaceMeshPoint[]`	Returns a list of `FaceMeshPoint` representing the whole detected face.	5.3.0
`triangles`	`Triangle[]`	Returns a list of `Triangle` representing logical triangle surfaces of detected face. Each `Triangle` contains 3 `FaceMeshPoint`, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh.	5.3.0

Rect¶

Rect holds four integer coordinates for a rectangle.

Prop	Type	Description	Since
`left`	`number`	The X coordinate of the left side of the rectangle.	5.3.0
`top`	`number`	The Y coordinate of the top of the rectangle.	5.3.0
`right`	`number`	The X coordinate of the right side of the rectangle.	5.3.0
`bottom`	`number`	The Y coordinate of the bottom of the rectangle.	5.3.0

Contours¶

Represents contours with their face mesh points.

Prop	Type	Description	Since
`faceOval`	`FaceMeshPoint[]`	Returns all points for the `FaceOval` contour.	5.3.0
`leftEyebrowTop`	`FaceMeshPoint[]`	Returns all points for the `LeftEyebrowTop` contour.	5.3.0
`leftEyebrowBottom`	`FaceMeshPoint[]`	Returns all points for the `LeftEyebrowBottom` contour.	5.3.0
`rightEyebrowTop`	`FaceMeshPoint[]`	Returns all points for the `RightEyebrowTop` contour.	5.3.0
`rightEyebrowBottom`	`FaceMeshPoint[]`	Returns all points for the `RightEyebrowBottom` contour.	5.3.0
`leftEye`	`FaceMeshPoint[]`	Returns all points for the `LeftEye` contour.	5.3.0
`rightEye`	`FaceMeshPoint[]`	Returns all points for the `RightEye` contour.	5.3.0
`upperLipTop`	`FaceMeshPoint[]`	Returns all points for the `UpperLipTop` contour.	5.3.0
`upperLipBottom`	`FaceMeshPoint[]`	Returns all points for the `UpperLipBottom` contour.	5.3.0
`lowerLipTop`	`FaceMeshPoint[]`	Returns all points for the `LowerLipTop` contour.	5.3.0
`lowerLipBottom`	`FaceMeshPoint[]`	Returns all points for the `LowerLipBottom` contour.	5.3.0
`noseBridge`	`FaceMeshPoint[]`	Returns all points for the `NoseBridge` contour.	5.3.0

FaceMeshPoint¶

Represents a 3D point in face mesh.

The index is an unique ID meaning a fixed position on face, ranging from 0 to 467.

In Point3D, x and y are pixel location of detected face in InputImage. z is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points.

Prop	Type	Description	Since
`index`	`number`	Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value.	5.3.0
`point`	`Point3D`	Gets a 3D point in face mesh. Inside `Point3D`, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera.	5.3.0

Point3D¶

Represents a 3D point.

Prop	Type	Description	Since
`x`	`number`	Returns the X value of the point.	5.3.0
`y`	`number`	Returns the Y value of the point.	5.3.0
`z`	`number`	Returns the Z value of the point.	5.3.0

Triangle¶

Represents a triangle with 3 generic points.

Prop	Type	Description	Since
`points`	`FaceMeshPoint[]`	Returns all points inside the `Triangle`.	5.3.0

ProcessImageOptions¶

Prop	Type	Description	Default	Since
`path`	`string`	The local path to the image file.		5.3.0
`useCase`	`UseCase`	Sets the use case. When `BoundingBoxOnly` is selected, the returned `FaceMesh` only contains bounding box. When `FaceMesh` is selected, the returned `FaceMesh` contains bounding box as well as 468 `FaceMeshPoint` and triangle information. It detects at most 2 faces in this case and it is slower than `BoundingBoxOnly`.	`UseCase.FaceMesh`	5.3.0

Enums¶

UseCase¶

Members	Value	Description	Since
`BoundingBoxOnly`	`0`	Return bounding box for detected face.	5.3.0
`FaceMesh`	`1`	Return face mesh info for detected face. It detects at most 2 faces in this use case.	5.3.0

Terms & Privacy¶

This plugin uses the Google ML Kit:

Changelog¶

See CHANGELOG.md.

License¶

See LICENSE.

This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries. ↩