Lab A - Tonal Analysis

This lab allows you to practice analysing an audio signal input in the frequency domain, and to extract some features. Skills learnt during this lab would be useful for an analysis system which is based on the extraction of tonal features for example for F0/pitch detection, tuning, intonation, vibrato analysis etc.

Overall aims:

1) To be able to process audio signals to extract tonal parameters

2) To be able to use tonal parameters to calculate tonal attributes

Stages:

1) Creating test files

2) Extracting the fundamental frequency

3) Using spectrograms

4) Calculating fundamental frequency over time

5) Creating more complex test files

6) Analysing more complex tones with spectrogram

7) Creating a complex tone with vibrato

8) Analysing vibrato

9) Further work

1. Creating test files

The first section of this lab involves making and using test audio. Having ‘ground truth’ files enables us to know that our system is performing as expected - it is worth also doing this for your assignment in some way!

Make 4 sine waves of different fundamental frequencies (make sure you know what they are!) each with a length of 2 seconds. Check that they are correct using plot and soundsc. Use descriptive variable names.

Make a melody by combining the four sine waves in sequence.

2. Extracting the fundamental frequency

Write a function which uses fft to plot the frequency spectrum of the single tones you have created.

fft returns the complex and imaginary parts of the frequency domain version of the time domain signal. To get the frequency magnitude response, you will also need to use abs.

Reminder: the spectrum returned by fft contains N samples (where N is the number of points specified, or if no value is specified, the length of the input signal) from 0Hz to the sampling frequency Fs. Notice that the lower half of the spectrum is reflected in the upper half of the plot. This is a consequence of the waveform being sampled. Only the N/2 frequencies up to just below Fs/2 carry useful information, the higher frequencies are redundant and can be ignored (usually they are not plotted).

Now, change the scale of the horizontal axis of the spectrum to kHz.

Also change the vertical axis to plot the magnitude spectra in decibels (dBs).

Calculate the fundamental frequency from the data returned by fft.

3. Using spectrograms

Although using fft to perform a frequency analysis on a whole signal gives us some very general information about what is present, it does not tell us about where in time components at particular frequencies occur – we need to analyse this with a time-frequency analysis tool.

A very common example of a time-frequency analysis tool is the spectrogram, which describes how the spectrum of a signal changes over time. It does this by dividing the signal up into small (possibly overlapping and tapered) segments and performing a Fourier analysis (as is used for fft) for each one, which shows how energy is distributed in frequency for each of these segments.

Handily, there is a function called spectrogram within Matlab which carries out all of these steps for us and plots the distribution of a signal’s energy over time and frequency in a two-dimensional plot.

spectrogram(X,WINDOW,NOVERLAP,NFFT,Fs, freqloc);

The first input to this function is a vector containing the single-channel (i.e. mono) signal that we wish to analyse.

The second input is a scalar (i.e. a single numeric value) that specifies how long each segment (or ‘window’) should be.

The third input is the number of samples each segment of the vector overlaps.

The fourth input is the number of frequency points used to calculate the discrete Fourier transforms.

The fifth input is the sampling frequency.

The sixth input is the location of the frequency axis: ‘xaxis’ or ‘yaxis’.

Plot the spectrogram of your melodic sequence. If you do not assign any outputs, spectrogram creates a plot of the data.

Experiment with changing the window length and overlap size. Increasing the window length gives us better frequency resolution in our spectrogram. This comes at a cost of poorer time-resolution - this is not a problem for a simple sinusoidal input such as this, but may be something you need to think about later.

Assigning outputs to spectrogram means we can access the data, rather than just producing a nice plot.

[s,f,t] = spectrogram(...) returns:

s = complex frequency data of dimensions f by t

f = specified frequencies

t = specified time intervals

See if you can recreate the output from spectrogram using the output data - don’t worry about the colorbar values for now. Remember that Matlab figures start incrementing from the top left but spectrograms start at the bottom left. A helpful plot function might be imagesc, as might be the call axis xy.

It is also possible to build your own spectrogram by using fft multiple times whilst stepping through the signal.

4. Calculating fundamental frequency over time using spectrogram

In a similar way to that used in Section 2, calculate the fundamental frequencies of the melodic sequence and plot against time. Hint: the output from spectrogram will help here.

Expand this to determine the onset time of each note.

Why might it not always be accurate to do this in the frequency domain?

5. Creating more complex test files

Now create some sawtooth tones with the same fundamental frequencies and lengths as the sine waves you made in Section 1. Build the sawtooth tones from sinusoidal components rather than using the Matlab function - include 10-20 harmonics. Create a melodic sequence in the same way as in Section 1. You should have the same sequence twice, using sine and sawtooth tones.

6. Analysing more complex tones with spectrogram

Use the spectrogram function to analyse the sawtooth tone sequence you have created – are the results as you expect? What happens if you calculate the fundamental frequency of a more complex tone?

7. Creating a complex tone with vibrato

Vibrato is a frequency modulation of the fundamental and harmonics of a signal, usually around 4-8Hz. We are going to synthesise a complex tone with vibrato to act as an approximated singing signal.

Use the following code as a starting point to alter your sawtooth tone generation code to frequency-modulate each harmonic. We are modulating the frequency by changing the value passed to sine for each sample.

lfo_f = 5; %modulating frequency (rate)

lfo_depth = 5; %modulation depth (extent)

phase_adjustment = lfo_depth.*sin(2*pi*lfo_f .*t);

Harmonic = sin(2*pi*fc*t + phase_adjustment);

Then create a sequence using the same fundamental frequencies as your sine sequence and complex tone sequence. You should now have 3 sequences: a sine sequence, a sawtooth sequence, and a vibrato sawtooth sequence. These need to match in both frequencies used and length of notes, for reasons that you will see later. Plot these sequences and make sure you can see the harmonics and vibrato.

8. Analysing vibrato

Calculate the fundamental frequency over time of your complex vibrato sequence. Do you capture the vibrato information? What might you need to adjust in your spectrogram calculation?

To analyse the vibrato for separate notes, we need to know where notes start and end. This is why it is important that your sequences all match in both fundamental frequencies used and length of notes in the melody. We are fabricating a fairly complex step in vibrato analysis to enable us to look at the final output - determining where the notes are in the signal.

We are going to use the note onset time information from the sine sequence (calculated in Section 4) to analyse the periodicity of the vibrato on each note in turn to determine the vibrato rate.

Use the following code as a start. Findpeaks finds the local maxima in a signal - we can use this to calculate the period and therefore the frequency of the vibrato.

[peaks, locations] = findpeaks(fundamental(noteOnset1:noteOnset2-1); %we are interested in a specific section of the signal

vibratoMaxLocations = t(locations); %where t is the output from spectrogram which gives the time increments

vibratoPeriod = mean(diff(vibratoMaxLocations)); %calculates the mean difference between each time value

vibratoFrequency = 1/vibratoPeriod;

Make sure you understand this code - ask if you are not sure. The frequency you calculate should match the LFO value that you specified when creating the complex tone with vibrato.

Download the example file (lc_vss_212_cv7.wav). This is a real-world vibrato singing example, for which we do not have ground truth fundamental frequency values. How does this differ to the synthesised example when you plot the spectrogram? How might you go about analysis of the vibrato in this example?

9. Further work

There are other methods of F0 estimation that are more sophisticated than just taking values from the FFT, and which incorporate the note onset requirement. Some of these can be found in toolboxes that other people have written - see if you can find any online or from literature and explore their inputs, usage, and outputs.

It is also possible to implement tonal analysis in Pure Data. If you wish, there are various objects similar to the functions explored in this lab, in particular:

? rfft~

? ifft~

? fiddle~

Explore [rfft], [ifft] and [fiddle] by working through the examples in this tutorial (http://pd-tutorial.com/english/ch03s08.html), particularly 3.8.1.1 and 3.8.3.1

What other features could you extract from a spectral analysis of the audio input file?

This lab allows you to practice analysing an audio signal input in the frequency domain, and to extract some features. Skills learnt during this lab would be useful for an analysis system which is based on the extraction of tonal features for example for F0/pitch detection, tuning, intonation, vibrato analysis etc.

Overall aims:

1) To be able to process audio signals to extract tonal parameters

2) To be able to use tonal parameters to calculate tonal attributes

Stages:

1) Creating test files

2) Extracting the fundamental frequency

3) Using spectrograms

4) Calculating fundamental frequency over time

5) Creating more complex test files

6) Analysing more complex tones with spectrogram

7) Creating a complex tone with vibrato

8) Analysing vibrato

9) Further work

1. Creating test files

The first section of this lab involves making and using test audio. Having ‘ground truth’ files enables us to know that our system is performing as expected - it is worth also doing this for your assignment in some way!

Make 4 sine waves of different fundamental frequencies (make sure you know what they are!) each with a length of 2 seconds. Check that they are correct using plot and soundsc. Use descriptive variable names.

Make a melody by combining the four sine waves in sequence.

2. Extracting the fundamental frequency

Write a function which uses fft to plot the frequency spectrum of the single tones you have created.

fft returns the complex and imaginary parts of the frequency domain version of the time domain signal. To get the frequency magnitude response, you will also need to use abs.

Reminder: the spectrum returned by fft contains N samples (where N is the number of points specified, or if no value is specified, the length of the input signal) from 0Hz to the sampling frequency Fs. Notice that the lower half of the spectrum is reflected in the upper half of the plot. This is a consequence of the waveform being sampled. Only the N/2 frequencies up to just below Fs/2 carry useful information, the higher frequencies are redundant and can be ignored (usually they are not plotted).

Now, change the scale of the horizontal axis of the spectrum to kHz.

Also change the vertical axis to plot the magnitude spectra in decibels (dBs).

Calculate the fundamental frequency from the data returned by fft.

3. Using spectrograms

Although using fft to perform a frequency analysis on a whole signal gives us some very general information about what is present, it does not tell us about where in time components at particular frequencies occur – we need to analyse this with a time-frequency analysis tool.

A very common example of a time-frequency analysis tool is the spectrogram, which describes how the spectrum of a signal changes over time. It does this by dividing the signal up into small (possibly overlapping and tapered) segments and performing a Fourier analysis (as is used for fft) for each one, which shows how energy is distributed in frequency for each of these segments.

Handily, there is a function called spectrogram within Matlab which carries out all of these steps for us and plots the distribution of a signal’s energy over time and frequency in a two-dimensional plot.

spectrogram(X,WINDOW,NOVERLAP,NFFT,Fs, freqloc);

The first input to this function is a vector containing the single-channel (i.e. mono) signal that we wish to analyse.

The second input is a scalar (i.e. a single numeric value) that specifies how long each segment (or ‘window’) should be.

The third input is the number of samples each segment of the vector overlaps.

The fourth input is the number of frequency points used to calculate the discrete Fourier transforms.

The fifth input is the sampling frequency.

The sixth input is the location of the frequency axis: ‘xaxis’ or ‘yaxis’.

Plot the spectrogram of your melodic sequence. If you do not assign any outputs, spectrogram creates a plot of the data.

Experiment with changing the window length and overlap size. Increasing the window length gives us better frequency resolution in our spectrogram. This comes at a cost of poorer time-resolution - this is not a problem for a simple sinusoidal input such as this, but may be something you need to think about later.

Assigning outputs to spectrogram means we can access the data, rather than just producing a nice plot.

[s,f,t] = spectrogram(...) returns:

s = complex frequency data of dimensions f by t

f = specified frequencies

t = specified time intervals

See if you can recreate the output from spectrogram using the output data - don’t worry about the colorbar values for now. Remember that Matlab figures start incrementing from the top left but spectrograms start at the bottom left. A helpful plot function might be imagesc, as might be the call axis xy.

It is also possible to build your own spectrogram by using fft multiple times whilst stepping through the signal.

4. Calculating fundamental frequency over time using spectrogram

In a similar way to that used in Section 2, calculate the fundamental frequencies of the melodic sequence and plot against time. Hint: the output from spectrogram will help here.

Expand this to determine the onset time of each note.

Why might it not always be accurate to do this in the frequency domain?

5. Creating more complex test files

Now create some sawtooth tones with the same fundamental frequencies and lengths as the sine waves you made in Section 1. Build the sawtooth tones from sinusoidal components rather than using the Matlab function - include 10-20 harmonics. Create a melodic sequence in the same way as in Section 1. You should have the same sequence twice, using sine and sawtooth tones.

6. Analysing more complex tones with spectrogram

Use the spectrogram function to analyse the sawtooth tone sequence you have created – are the results as you expect? What happens if you calculate the fundamental frequency of a more complex tone?

7. Creating a complex tone with vibrato

Vibrato is a frequency modulation of the fundamental and harmonics of a signal, usually around 4-8Hz. We are going to synthesise a complex tone with vibrato to act as an approximated singing signal.

Use the following code as a starting point to alter your sawtooth tone generation code to frequency-modulate each harmonic. We are modulating the frequency by changing the value passed to sine for each sample.

lfo_f = 5; %modulating frequency (rate)

lfo_depth = 5; %modulation depth (extent)

phase_adjustment = lfo_depth.*sin(2*pi*lfo_f .*t);

Harmonic = sin(2*pi*fc*t + phase_adjustment);

Then create a sequence using the same fundamental frequencies as your sine sequence and complex tone sequence. You should now have 3 sequences: a sine sequence, a sawtooth sequence, and a vibrato sawtooth sequence. These need to match in both frequencies used and length of notes, for reasons that you will see later. Plot these sequences and make sure you can see the harmonics and vibrato.

8. Analysing vibrato

Calculate the fundamental frequency over time of your complex vibrato sequence. Do you capture the vibrato information? What might you need to adjust in your spectrogram calculation?

To analyse the vibrato for separate notes, we need to know where notes start and end. This is why it is important that your sequences all match in both fundamental frequencies used and length of notes in the melody. We are fabricating a fairly complex step in vibrato analysis to enable us to look at the final output - determining where the notes are in the signal.

We are going to use the note onset time information from the sine sequence (calculated in Section 4) to analyse the periodicity of the vibrato on each note in turn to determine the vibrato rate.

Use the following code as a start. Findpeaks finds the local maxima in a signal - we can use this to calculate the period and therefore the frequency of the vibrato.

[peaks, locations] = findpeaks(fundamental(noteOnset1:noteOnset2-1); %we are interested in a specific section of the signal

vibratoMaxLocations = t(locations); %where t is the output from spectrogram which gives the time increments

vibratoPeriod = mean(diff(vibratoMaxLocations)); %calculates the mean difference between each time value

vibratoFrequency = 1/vibratoPeriod;

Make sure you understand this code - ask if you are not sure. The frequency you calculate should match the LFO value that you specified when creating the complex tone with vibrato.

Download the example file (lc_vss_212_cv7.wav). This is a real-world vibrato singing example, for which we do not have ground truth fundamental frequency values. How does this differ to the synthesised example when you plot the spectrogram? How might you go about analysis of the vibrato in this example?

9. Further work

There are other methods of F0 estimation that are more sophisticated than just taking values from the FFT, and which incorporate the note onset requirement. Some of these can be found in toolboxes that other people have written - see if you can find any online or from literature and explore their inputs, usage, and outputs.

It is also possible to implement tonal analysis in Pure Data. If you wish, there are various objects similar to the functions explored in this lab, in particular:

? rfft~

? ifft~

? fiddle~

Explore [rfft], [ifft] and [fiddle] by working through the examples in this tutorial (http://pd-tutorial.com/english/ch03s08.html), particularly 3.8.1.1 and 3.8.3.1

What other features could you extract from a spectral analysis of the audio input file?

SUPPLEMENTARY ASSESSMENTSubject Code and Name HFR603 Hotel Finance & RevenueAssessment Revenue Management Scenario EvaluationIndividual/Group IndividualLength 1,000 wordsLearning Outcomes This assessment...Unit CybersecurityUnit code ISYS1002Assignment 2 Security consultation report and guideline.Due Date Learning 24 September 2021 11:00 PMOutcomesGraduate 3, 6Attributes 3, 4 & 5Weight 30% of overall...Assignment: The Impact of Nursing Informatics on Patient Outcomes and Patient Care EfficienciesIn the Discussion for this module, you considered the interaction of nurse informaticists with other specialists...Assessment 3 InformationSubject Code: DATA4700Subject Name: Digital Marketing and Competitive AdvantageAssessment Title: Competitive Advantage Analytics ReportAssessment Type: Report (1500) + Statement...Assessment 4 InformationSubject Code: DATA4900Subject Name: Innovation and Creativity in Business AnalyticsAssessment Title: UX, CX and Ethical innovation case studyAssessment Type: Individual reportWord...Tutorial Activity 2 Submission LinkTutorial Activity 2 - HC3152. Due date 21/09/2021Students need to answer the two following tutorial questions.Answers need to present in the point form, include 100 to...Assessment DetailsQualification Code/Title BSB50120 - Diploma of BusinessAssessment Type Assessment -03 (Project) Time allowed 5 weeksDue Date 15th of August 2021 Location AHIC Term / Year 3 of 2021Unit...**Show All Questions**