Collecting data for mouth opening research project

[yqzhishen]

Hello all. The dev team has decided to start a research project aiming to bring mouth opening parameter to DiffSinger. The project roughly take 3 stages:

1. Collect mouth-related data and recording pairs with specific devices (this is why we need your help!)
2. Train an NN-based estimator to extract mouth opening curve from recordings (will be released for everyone to use without requiring specific devices)
3. Integrate the estimator into DiffSinger, bringing ability for variance models to predict mouth opening, and for acoustic models to accept such control.

How to participate

For stage 1, we have planned to use TrueDepth cameras on Apple devices. Via this technology, we can use many attributes provided by Apple ARKit (such as jawOpen and mouthClose) to define our own mouth opening parameter. Here are the steps to collect and contribute your data:

1. Device requirements: iPhone or iPad with Face ID (iPhone X or above, or iPad Pro), and corresponding holders.
2. Search "LipsSync" in App Store and install the tool. Don't worry - this tool is completely offline and only collects your mouth data (several simple attributes) and your voice. No videos or face models are stored or shared.
3. Use a holder to hold your device in place, with your face facing the front camera, and adjust the distance from the carema to your face at about 25~50cm to capture your entire face while achieving best precision.
4. Verify your device: start LipsSync, click on the record button, keep your head still, produce common vowel sounds in a natural pattern, and stabilize the readings displayed on the screen to ensure that the relevant indications are as expected. After this step, you can switch to "File" tab and delete the clip you just recorded, as it's useless for formal steps.
5. Try to find a quiet place. Go back to "Recorder" tab. Click on the record button, try to keep your head still and sing (or speak) a number of phrases naturally, paying attention to keep your mouth closed as much as possible when you don't make a sound.
6. Go to "Files" tab, select your recordings and tap "Zip and Share". Send an email to [email protected], in which leave your nickname/signature, Discord username (optional), homepage url (optional), device name, singing language and attach the exported zip file.
7. Synchronized recording: you can use other microphones to record synchronized recordings (WAV or FLAC format, mono, 24kHz+ sampling rate, 16bit or higher). You don't need to align them perfectly - just ensure they contain the same content and we have internal tools to do so. Please name the synchronized recordings in order so that they correspond to the exported recordings. Attach another zip file in the email and leave a description on how the synchronized recordings correspond to exported ones.

---

If you have any questions or troubles, please first leave a comment here and don't send data that you think may be abnormal. All data will be used for mouth opening estimator training only, and will not be redistributed or published. We cannot guarantee the success of the program at this time, but we will continuously share our progress here. All those who have contributed valid data will be included in the acknowledgement list if the project succeeds.