Simple Real-Time Voice Conversion with Artificial Intelligence & Machine Learning | by Ankit Arora | Mar, 2024

Voice conversion is a fascinating area of research and application within the domain of artificial intelligence and machine learning.

It involves the transformation of a speaker’s voice from one characteristic to another, often referred to as mapping the source voice to a target voice.

In this article, we will delve into the process of building a real-time voice conversion system using Node.js for the backend, TensorFlow.js for machine learning capabilities in the browser, and an API for seamless integration.

We will cover data collection, preprocessing, model development, and API deployment, providing code examples and explanations at each step.

Data collection is a crucial initial step in building a voice conversion system.

You need a diverse dataset containing recordings of both source and target voices.

These recordings should cover a range of speech patterns, accents, and emotions to ensure the model’s robustness.

Code Example (Node.js):

// Code for recording audio samples using Web Audio API
navigator.mediaDevices.getUserMedia({ audio: true })
.then(function(stream) {
// Record audio using the MediaRecorder API
const mediaRecorder = new MediaRecorder(stream);
const chunks = [];
mediaRecorder.ondataavailable = function(event) {
chunks.push(event.data);
}
mediaRecorder.onstop = function() {
// Convert audio chunks to a single Blob
const audioBlob = new Blob(chunks, { type: 'audio/wav' });
// Upload audioBlob to the server or save locally
}
// Start recording
mediaRecorder.start();
// Stop recording after a specified duration
setTimeout(function() {
mediaRecorder.stop();
}, 5000); // Recording duration (e.g., 5 seconds)
})
.catch(function(err) {
console.error('Error accessing microphone:', err);
});

Preprocessing involves transforming raw audio data into a format suitable for input to the machine learning model.

Common preprocessing techniques include extracting features like Mel-frequency cepstral coefficients (MFCCs) or spectrograms from the audio signals.

Code Example (TensorFlow.js):

// Code for preprocessing audio data using TensorFlow.js
const preprocessAudio = (audioData) => {
// Convert audio data to spectrogram
const spectrogram = tf.browser.audio.spectrogram(
tf.browser.audio.decodeWav(audioData),
{ fftSize: 512, hopLength: 256 }
);
// Normalize spectrogram
const normalizedSpectrogram = tf.scalar(255).div(spectrogram.max()).toFloat();
// Expand dimensions to match model input shape
const processedData = normalizedSpectrogram.expandDims(0);
return processedData;
};

For this article, let’s demonstrate how to train a simple model for voice conversion using TensorFlow.js. We’ll create a basic model using the Layers API and train it using sample data.

// Define and compile the model
const model = tf.sequential();
model.add(tf.layers.dense({ units: 100, inputShape: [10] }));
model.add(tf.layers.dense({ units: 1 }));
// Train the model
const xTrain = tf.randomNormal([100, 10]);
const yTrain = tf.randomNormal([100, 1]);
model.fit(xTrain, yTrain, {
epochs: 100,
callbacks: {
onEpochEnd: (epoch, logs) => console.log(`Epoch ${epoch}: loss = ${logs.loss}`)
}
});

Once we have our model ready, we can deploy it as an API using Node.js and Express. The API will receive audio data, preprocess it, perform voice conversion using the model, and return the converted audio data.

Code Example (Node.js — API Endpoint):

const express = require('express');
const multer = require('multer');
const tf = require('@tensorflow/tfjs-node');
const app = express();
const PORT = process.env.PORT || 3000;
const upload = multer({ dest: 'uploads/' });
// Define and compile the model
const model = tf.sequential();
model.add(tf.layers.dense({ units: 100, inputShape: [10] }));
model.add(tf.layers.dense({ units: 1 }));
model.compile({ loss: 'meanSquaredError', optimizer: 'sgd' });
// Train the model
const xTrain = tf.randomNormal([100, 10]);
const yTrain = tf.randomNormal([100, 1]);
model.fit(xTrain, yTrain, {
epochs: 100,
callbacks: {
onEpochEnd: (epoch, logs) => console.log(`Epoch ${epoch}: loss = ${logs.loss}`)
}
});
app.post('/convert-trainable', upload.single('audio'), async (req, res) => {
try {
const audioData = req.file.buffer;
const processedData = preprocessAudio(audioData);
const convertedAudioData = await model.predict(processedData);
// Further post-processing if required
res.send(convertedAudioData);
} catch (error) {
console.error('Error:', error);
res.status(500).send('An error occurred during voice conversion.');
}
});
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});

Explanation

• The sample input data represents the original audio file containing the sentence “Hello, how are you today?” spoken in the source voice.
• After processing through the voice conversion model, the output data represents the same sentence but is transformed to sound as if it was spoken in the target voice.
• In this simplified example, we assume that the voice conversion model successfully maps the characteristics of the source voice to those of the target voice, resulting in an output audio file that sounds as if it was spoken by the target speaker.

In this article, we have explored the process of building a real-time voice conversion system using Node.js and TensorFlow.js.

We covered data collection, preprocessing, model development, and API deployment, providing code examples and explanations at each step.

Voice conversion has numerous applications in entertainment, accessibility, and human-computer interaction, and with the power of modern machine learning frameworks, building such systems has become more accessible than ever before.