--- title: Capacitor ML Kit Face Mesh Detection Plugin for Android description: Unofficial Capacitor plugin for Google ML Kit Face Mesh Detection SDK to detect faces and their mesh in images. tags: - Android search: boost: 2 faq: true github_repo: capawesome-team/capacitor-mlkit npm_package: "@capacitor-mlkit/face-mesh-detection" --- # Capacitor ML Kit Face Mesh Detection Plugin Unofficial Capacitor plugin for [ML Kit Face Mesh Detection](https://developers.google.com/ml-kit/vision/face-mesh-detection).[^1]
Deliver Live Updates to your Capacitor app with Capawesome Cloud
## Use Cases The Face Mesh Detection plugin is typically used whenever an app needs detailed facial geometry, for example: - **Face filters and effects**: Use the 468 3D face mesh points and triangle surfaces to align virtual makeup, masks, or other effects with a face. - **Facial feature analysis**: Work with the contours of the face oval, eyebrows, eyes, lips, and nose bridge. - **Fast face localization**: Use the `BoundingBoxOnly` use case to quickly get the bounding boxes of detected faces. - **Selfie tools**: Analyze up to 2 faces in detail using the full face mesh information. ## Compatibility | Plugin Version | Capacitor Version | Status | | -------------- | ----------------- | -------------- | | 8.x.x | >=8.x.x | Active support | | 6.x.x | 6.x.x | Deprecated | ## Installation You can use our **AI-Assisted Setup** to install the plugin. Add the [Capawesome Skills](https://github.com/capawesome-team/skills) to your AI tool using the following command: ```bash npx skills add capawesome-team/skills --skill capacitor-plugins ``` Then use the following prompt: ``` Use the `capacitor-plugins` skill from `capawesome-team/skills` to install the `@capacitor-mlkit/face-mesh-detection` plugin in my project. ``` If you prefer **Manual Setup**, install the plugin by running the following commands and follow the platform-specific instructions below: ```bash npm install @capacitor-mlkit/face-mesh-detection npx cap sync ``` ### Android #### Variables If needed, you can define the following project variable in your app’s `variables.gradle` file to change the default version of the dependency: - `$mlkitFaceMeshDetectionVersion` version of `com.google.mlkit:face-mesh-detection` (default: `16.0.0-beta1`) This can be useful if you encounter dependency conflicts with other plugins in your project. ## Configuration No configuration required for this plugin. ## Demo A working example can be found here: [robingenz/capacitor-mlkit-plugin-demo](https://github.com/robingenz/capacitor-mlkit-plugin-demo) ## Usage The following example shows how to detect face meshes in an image. ### Detect face meshes in an image Detect face meshes in an image at a local path. Choose the `BoundingBoxOnly` use case for fast bounding box detection or the `FaceMesh` use case to also receive the 468 3D face mesh points and triangle information. Only available on Android: ```typescript import { FaceMeshDetection, UseCase } from '@capacitor-mlkit/face-mesh-detection'; const processImage = async () => { const { faceMeshs } = await FaceMeshDetection.processImage({ path: 'path/to/image.jpg', useCase: UseCase.FaceMesh, }); return faceMeshs; }; ``` ## API * [`processImage(...)`](#processimage) * [Interfaces](#interfaces) * [Enums](#enums) ### processImage(...) ```typescript processImage(options: ProcessImageOptions) => Promise ``` Detects face mesh from the supplied image. Only available on Android. | Param | Type | | ------------- | ------------------------------------------------------------------- | | **`options`** | ProcessImageOptions | **Returns:** Promise<ProcessImageResult> **Since:** 5.3.0 -------------------- ### Interfaces #### ProcessImageResult | Prop | Type | Description | Since | | --------------- | ----------------------- | ------------------------ | ----- | | **`faceMeshs`** | FaceMesh[] | The detected face meshs. | 5.3.0 | #### FaceMesh Represents a face mesh detected by `FaceMeshDetector`. When `BoundingBoxOnly` is selected, `FaceMesh` only contains valid bounding box. When `FaceMesh` is selected, `FaceMesh` also contains a group of 468 3D face mesh points and related triangle information. Each point is represented by `FaceMeshPoint` describing a specific position in detected face. The triangle information is a group of 3 `FaceMeshPoint`s representing a valid surface on Face (e.g. a valid small surface on nose tip). | Prop | Type | Description | Since | | -------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | | **`bounds`** | Rect | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 | | **`contours`** | Contours | Returns contours with a list of `FaceMeshPoint` representing the detected face. | 5.3.0 | | **`faceMeshPoints`** | FaceMeshPoint[] | Returns a list of `FaceMeshPoint` representing the whole detected face. | 5.3.0 | | **`triangles`** | Triangle[] | Returns a list of `Triangle` representing logical triangle surfaces of detected face. Each `Triangle` contains 3 `FaceMeshPoint`, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 | #### Rect Rect holds four integer coordinates for a rectangle. | Prop | Type | Description | Since | | ------------ | ------------------- | ---------------------------------------------------- | ----- | | **`left`** | number | The X coordinate of the left side of the rectangle. | 5.3.0 | | **`top`** | number | The Y coordinate of the top of the rectangle. | 5.3.0 | | **`right`** | number | The X coordinate of the right side of the rectangle. | 5.3.0 | | **`bottom`** | number | The Y coordinate of the bottom of the rectangle. | 5.3.0 | #### Contours Represents contours with their face mesh points. | Prop | Type | Description | Since | | ------------------------ | ---------------------------- | -------------------------------------------------------- | ----- | | **`faceOval`** | FaceMeshPoint[] | Returns all points for the `FaceOval` contour. | 5.3.0 | | **`leftEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowTop` contour. | 5.3.0 | | **`leftEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowBottom` contour. | 5.3.0 | | **`rightEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowTop` contour. | 5.3.0 | | **`rightEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowBottom` contour. | 5.3.0 | | **`leftEye`** | FaceMeshPoint[] | Returns all points for the `LeftEye` contour. | 5.3.0 | | **`rightEye`** | FaceMeshPoint[] | Returns all points for the `RightEye` contour. | 5.3.0 | | **`upperLipTop`** | FaceMeshPoint[] | Returns all points for the `UpperLipTop` contour. | 5.3.0 | | **`upperLipBottom`** | FaceMeshPoint[] | Returns all points for the `UpperLipBottom` contour. | 5.3.0 | | **`lowerLipTop`** | FaceMeshPoint[] | Returns all points for the `LowerLipTop` contour. | 5.3.0 | | **`lowerLipBottom`** | FaceMeshPoint[] | Returns all points for the `LowerLipBottom` contour. | 5.3.0 | | **`noseBridge`** | FaceMeshPoint[] | Returns all points for the `NoseBridge` contour. | 5.3.0 | #### FaceMeshPoint Represents a 3D point in face mesh. The index is an unique ID meaning a fixed position on face, ranging from 0 to 467. In `Point3D`, `x` and `y` are pixel location of detected face in `InputImage`. `z` is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points. | Prop | Type | Description | Since | | ----------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | | **`index`** | number | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 | | **`point`** | Point3D | Gets a 3D point in face mesh. Inside `Point3D`, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 | #### Point3D Represents a 3D point. | Prop | Type | Description | Since | | ------- | ------------------- | --------------------------------- | ----- | | **`x`** | number | Returns the X value of the point. | 5.3.0 | | **`y`** | number | Returns the Y value of the point. | 5.3.0 | | **`z`** | number | Returns the Z value of the point. | 5.3.0 | #### Triangle Represents a triangle with 3 generic points. | Prop | Type | Description | Since | | ------------ | ---------------------------- | ----------------------------------------------------------------- | ----- | | **`points`** | FaceMeshPoint[] | Returns all points inside the `Triangle`. | 5.3.0 | #### ProcessImageOptions | Prop | Type | Description | Default | Since | | ------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | ----- | | **`path`** | string | The local path to the image file. | | 5.3.0 | | **`useCase`** | UseCase | Sets the use case. When `BoundingBoxOnly` is selected, the returned `FaceMesh` only contains bounding box. When `FaceMesh` is selected, the returned `FaceMesh` contains bounding box as well as 468 `FaceMeshPoint` and triangle information. It detects at most 2 faces in this case and it is slower than `BoundingBoxOnly`. | UseCase.FaceMesh | 5.3.0 | ### Enums #### UseCase | Members | Value | Description | Since | | --------------------- | -------------- | ------------------------------------------------------------------------------------- | ----- | | **`BoundingBoxOnly`** | 0 | Return bounding box for detected face. | 5.3.0 | | **`FaceMesh`** | 1 | Return face mesh info for detected face. It detects at most 2 faces in this use case. | 5.3.0 | ## FAQ ### Which platforms are supported by this plugin? The `processImage(...)` method is only available on Android. iOS and Web are not supported by the underlying ML Kit Face Mesh Detection SDK. ### What is the difference between the `BoundingBoxOnly` and `FaceMesh` use cases? With `BoundingBoxOnly`, the result only contains a valid bounding box for each detected face, which is faster. With `FaceMesh` (the default), the result additionally contains a group of 468 3D face mesh points, contours, and triangle information, but it detects at most 2 faces and is slower. ### What exactly is a face mesh point? A face mesh point is a 3D point on a detected face with an index between 0 and 467, where each index always refers to the same fixed position on the face. The `x` and `y` values are pixel locations in the input image, while the `z` value describes the distance to the camera: the smaller the value, the closer that point is to the camera. ### What is the difference between this plugin and the Face Detection plugin? The [ML Kit Face Detection](https://capawesome.io/docs/sdks/capacitor/mlkit/face-detection/) plugin detects faces with bounding boxes, landmarks, contours, and classifications such as smiling probability, and works on Android and iOS. This plugin returns a much denser mesh of 468 3D points and triangle information for detailed facial geometry, but is only available on Android. ### Can I use this plugin with Ionic, React, Vue or Angular? Yes, the plugin is framework-agnostic. It works in any Capacitor app regardless of the web framework, including Ionic with Angular, React, or Vue, as well as plain JavaScript projects. ## Related Plugins - [ML Kit Face Detection](https://capawesome.io/docs/sdks/capacitor/mlkit/face-detection/): Detect faces with landmarks, contours, and classifications in images. - [ML Kit Selfie Segmentation](https://capawesome.io/docs/sdks/capacitor/mlkit/selfie-segmentation/): Segment selfies from the background with ML Kit Selfie Segmentation. - [ML Kit Subject Segmentation](https://capawesome.io/docs/sdks/capacitor/mlkit/subject-segmentation/): Separate subjects from the background with ML Kit Subject Segmentation. ## Terms & Privacy This plugin uses the [Google ML Kit](https://developers.google.com/ml-kit): - [Terms & Privacy](https://developers.google.com/ml-kit/terms) - [Android Data Disclosure](https://developers.google.com/ml-kit/android-data-disclosure) - [iOS Data Disclosure](https://developers.google.com/ml-kit/ios-data-disclosure) ## Newsletter Stay up to date with the latest news and updates about the Capawesome, Capacitor, and Ionic ecosystem by subscribing to our [Capawesome Newsletter](https://cloud.capawesome.io/newsletter/). ## Changelog See [CHANGELOG.md](https://github.com/capawesome-team/capacitor-mlkit/blob/main/packages/face-mesh-detection/CHANGELOG.md). ## License See [LICENSE](https://github.com/capawesome-team/capacitor-mlkit/blob/main/packages/face-mesh-detection/LICENSE). [^1]: This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries.