Tutorial: Validating Videos of Facial Affect

Today, I made a little experiment to validate videos of facial effect. Your institute probably has the Ekman faces CDROM, right? If you do, you can verify it has over a hundred pictures of very retro looking faces of people including Paul Ekman himself (if younger), enacting six different emotions – happy, angry, sad, disgusted, surprised, fearful – and a neutral baseline (sometimes). You could argue that enough is enough, these pictures are too old, or perhaps that emotions in the wild are rarely statically displayed. My reason was less obvious: for a project on the affective and cognitive effects of touch, somehow we ended up with this giant construct of a study which includes a bar in virtual reality, where one meets a 3D model displaying some emotion, then touches the participant, whose EEG we measure. I promise you it honestly makes sense and is great fun to do, but in the mean time, it’s clear that we need to validate animated (as opposed to static) faces to know in advance that the emotions are recognized consistently.

Main structure

Enter today’s tutorial, which presents an easy, yet quite powerful and reasonably intuitive way to validate videos of facial affect. My goal was to be able to advertise the experiment in such a way that even my colleagues wouldn’t be too annoyed: it should require little if any instruction and take very little time. For the tutorial, the working assumption is, here and in the future, is that the reader has read the E-Primer, and can easily follow what I’m doing.

The experiment has the structure shown right.

Slide by slide

Let's see what happens in each of these slides:

• First, Instructions are shown (Instruction), consisting of a short reminder of what the subject is supposed to do.
• Every Trial (TrialList*), there’s a countdown (each 333 ms, just here to give a cinematic quality to the experiment):

The 3 above is a "hard" counter, as opposed to some attribute referencing "soft" counter, so in slide Countdown2, for example, it shows "[DoneN] 2". It also shows how many trials are done [DoneN] of how many in total. There are 38 trials, so it should say 5/38 in trial 5, for instance. This is handled in the preceding inline, CountTrial, which simply states:

ctrial = ctrial + 1

c.SetAttrib "DoneN", CStr(ctrial) & "/38"

Ctrial is declared in the user script (press alt+5, go to user tab), which says "Dim ctrial as integer" - assigning the value 0 to the integer variable ctrial (for "current" trial). Thus, the inline above adds 1 to the ctrial, then converts the number 1 into “1” (usually unnecessary in Basic, but irresistible for anyone who uses other languages), adding “/38” to it and putting it in the list, under the attribute DoneN.

Generate PreRun/ PostRun

Intermezzo: PreRun
In the latest E-Prime 2, one is required to make sure at what point in time the attributes are fed into the attributes, which is probably good for timing or debugging, but if you come from E-Prime 1, it can be confusing, because the default is exactly the other way around. To check, go to the slide’s properties and look under Script Generation (right).

Notice Generate PreRun: if set to the default (inherit), it will be the same as the procedure above it – TrialProc – which will say “TopOfProcedure”. In other words, at the beginning of the procedure, E-Prime will “pre-run” the Slide, which calls attribute [DoneN], but fails, since at that point there is no DoneN yet. Of course, the attribute DoneN will only exist after the CountTrial inline, so here we need to set, for each slide, Generate PreRun to “BeforeObjectRun” (as shown).

Back to Slide by slide
To continue:

• ShowFilm, then, shows the same as the CountDown slides, except with, instead of a TextDisplay, a MovieDisplay subobject.

As in the CountDown, I added a little white border of 3 width and gave a black background colour; it ensures the nice cinematic effect. Some people scuff at such “frivolities”: proper science is not about pretty displays and unnecessary aspects but instead focusses on the essence! It’s a similar attitudes to those academics whose Powerpoints are never anything but black (default font) against a white background, because otherwise it’s just not serious enough. I’m of the opposed opinion and think that people are more easily underwhelmed by lack of interest than find the presenter or programmer too grandiose. In most software one can use to present (experiments as well as powerpoints), it takes mere minutes to not make something look incredibly amateurish, which I think is a reasonable precaution anyone should take.

Secondly, notice the double attribute reference – a trick I learnt from David McFarlane of the E-Prime googlegroup. It says [VideoDir][VideoFile], and works exactly like two concatenated references, so if [videodir] happens to be "Videos\" and the current file is "Happy01.wmv", then the end result is Videos\Happy01.wmv. So, this is useful if you ever happen to move the files around.

This is shown in E-Studio

• The Answer-slide shows the final display in the trial procedure, shown after the film stopped running:

This is shown in runtime, after subject pressed 3.

The first time a subject sees the screen, they see the list of emotions, which is randomised, no current response, and no suggestion to press enter. However, after pressing the number 3, it shows this:

To implement all this functionality (random emotions, enter continue, left to go back, response to emotion mapping), I used only 13 lines of code.

Randomise emotions within trial
There are only two lists in this experiment:

Left: TrialList. Right: Nested (ordering) list.

Left, there’s the obvious ones: VideoDir (mentioned), VideoFile (the .wmv files, which worked, unlike the .avi files, well in E-Prime!), XEmotionShown (I almost always assign a leading X to independent variables and Y to dependent ones, because they show up easily in the data files – here, AngerMO and AngerMC refer to Anger Mouth Open and Anger Mouth Closed versions of the same emotion), YRESP (empty), DoPressEnter (empty), ExtraAllowable(empty). The list goes on to have 31 more trials. To the right is the nested list (referenced in TrialList’s Nested), showing the 6 emotions + neutral. In the Answer slide, these answers are displayed using colon syntax, which calls the row of the Emotion to draw from (but starting at 0). So, if the nested list wouldn’t be randomised, it’d show up as:

1: [Emotion:0] => 1: Happy
2: [Emotion:1] => 2: Sad

Now, because the order of the nested list is randomised, the response-choices are effectively balanced. 

Replay video

If this is clear, then my tiny bit of code, brilliantly named RecodeAnswerGoBackMaybe, shouldn’t be too hard. First off, if the subject presses the left button, there’s a bit of magic to rewind the movie:

If Answer.RESP = "{LEFTARROW}" Then 'to go back

      Dim theMovie As SlideMovie

      Set theMovie =_

CSlideMovie(ShowFilm.States(ShowFilm.ActiveState).Objects("Movie1"))

      theMovie.Stop

      theMovie.Load

      GoTo PreCountDown

End If

Response to emotion mapping

Which accounts for 7 out of the 13 lines of code and accomplishes merely to check for LEFT, to rewind the movie subobject, and to go back to the label. Incidentally, I had to guess the bit about the movie from the E-Basic help’s information on SlideSoundOut, because there’s as of yet no mention of movies in the current version of the E-Basic help.

More interestingly, the other 6 lines of code:

If IsNumeric(Answer.RESP) Then 'handing all number responses

      'getting the answered emotion from the nested list

      c.SetAttrib "YRESP",c.GetAttrib("Emotion:" & CStr(CInt(Answer.RESP)-1))

      'making it possible to press enter

      c.SetAttrib "DoPressEnter", "Press ENTER To continue"

      c.SetAttrib "ExtraAllowable", "{ENTER}"

      GoTo PreAnswer   

End If

If any number is pressed (isNumeric), we change the YRESP attribute by calling from nested list. Thus, given the above example, if a person types 2, YRESP is set not to 2 (which would be difficult to analyse later on), but to Emotion:(2-1) ergo Emotion:1 ergo Sad. This is also shown back to the subject immediately, so they may change their mind. 

Usability: Show and allow finalised response

Furthermore, DoPressEnter is changed so it shows a prompt (Press ENTER to continue). Enter doesn’t normally do anything, since the Answer slide has, as allowable: 1234567{LEFTARROW}[ExtraAllowable]

ExtraAllowable is normally nothing (see list above), so the subject can’t accidentally press Enter without making a choice. 

In Conclusion

In the video at the head of this post, I tried to show all the functionality in the experiment. It shows two trials, and a repetition of the second trial (because I pressed enter). You are also free to download it from here, though I can’t make the video files themselves available unless we have something published. Have fun E-Priming!

* Note that I did not use a BlockList. Given the size of the study - more a validation than a real experiment - and lack of training or other type of phase in the experiment, I did not find it necessary. However, we do generally recommend its use to keep an amount of consistency. Personally, I also tend to add attributes to an otherwise unused list and use them like "global variables" - say, the general background colour or the experiment, or the duration of each stimulus in an RSVP paradigm. It is generally much easier to keep track of such parameters in the list than it is in code.