Minutes from RT Teleconference Oct. 30, 2008

• The RT mailing list will change to rt09@nist.gov. rt07 will be discontinued.
• Question: Will RT be combined with RT?
• Ans: No. Lack of funding/data. RT will focus on meeting data.
• Question: Will there be any lecture data?
• Ans: No. None available.
• Question: The Call for Participation omitted the IHM condition, was this on purpose?
• Ans: No. It should have been included.

• Dev/Training resources

• JF: To get access to new data, need to have license agreements. Previously signed agreements are sufficient for previous participants until the RT-09 agreements are ready. New participants must sign the RT-07 agreements too get training resources and then they must sign the RT-09 agreements when there available.
• JF: NIST will not be supplying data already in the LDC catalog. Participants must contact them directly.
• Video data for the RT-06 data is available. Send NIST a disk in order to get it.  The RT-07 video data will be ready in a month.

• Evaluation Data Contributions
• NIST will prepare a Conference data test set similar to last year.
• Data will be slightly different than last year. The AMIDA data includes teleconference phone:
• Mike L: There is a telecon phone, but the remote person has a head mic and there is a mic array in the room as well.
• Andreas S.: The issue from the last evaluation with telecon phones involved the IHM condition where the speaker had no corresponding IHM channel. This raises the issue of how IHM is treated: as a contrast condition, or as a technology application of interest. 
• JF: One way to handle it would be to exclude remote participant speech from IHM scoring.
• JF: Since Thomas H. wasn't on the phone, JF thought the issue of how to handle the IHM condition should be tabled.
  

• Action Item: Discuss the issue next telecon.

• DvL.: TNO has some Dutch meeting data that might be useful for diarization tests. Would the community be interested in working with this data?
• New Training resources: NIST can do modest amount of Quick Transcriptions for new NIST training resources. This is to give people experience with our new data which includes 4 NIST Mark III arrays and higher resolution cameras.
  

• New idea: Online Diarization
• DVL: In the AMIDA project, they are interested in online treatment of meeting data. For instance a remote connection join in and already see results. It's a different problem  from offline speaker diarization. He proposes doing an online diarization task, without using info from the future. Perhaps allow a 1-second lag prior to output for example.  This would be closer to real-time applications.
  
• AS: How would we specify realtime arrival? It's hard to simulate the condition?
  
• DVL: Make it a 'realistic' simulation based on look ahead.
  
• JF: We've administered similar evaluations that have been successful.
  

• Action Item: Propose to specify this condition in the eval plan for all tasks. Only allow information you have processed up to N seconds in the future.
  

• New Idea: Explicit context switching (Dr. Kapuska, please check/correct this)
• Dr. Kapuska: In meetings, it would be useful to know when a person is addressing a person, addressing a computing resource, or talking about someone. Does the data  support this?
• JF: No. That would require a Wizard of Oz setup to test and the meetings we use are targeted at human-human communication.
  

• Beam Forming Assessment task:
• JF: Two ways to evaluate beamforming: (1) evaluate the benefit to an application (STT) (2) measure the signal directly. This task focuses on method (2).
• JF: The blind source separation community uses a number of metrics to measure signal quality.  Would like to employ them on this dataset. Has anyone tried to assess quality? (No one has).

• Action Item: JF will send out a list of known metrics.
  

• Leveraging multimodal/mic array output: What can be done to share processed data?
• Xavi's BeamformIt tool: http://www.icsi.berkeley.edu/~xanguera/beamformit/index.html
  

• Eval data released: Mar 2
• Submissions due: Mar 27
• Results Released: April 10
• Workshop: Late May (We need a venue.  Any ideas/voluteers?)
  

• Next telecon: 11/20/2008 (NIST has a confilict the week of Nov. 11).