Open positions
Learning procedural knowledge through interaction for assisted living
In many domestic and industrial settings, robots need to learn new skills from the human partner. Learning procedural knowledge (also called task learning, e.g., cooking, assembling furniture, etc.) is a hallmark ability of robots. It requires a suite of capabilities such as visual perception, language understanding, reasoning, manipulation, and human-robot interaction. To achieve robust and trustworthy robot learning, a key element is the intelligence related to the establishment and communication of agent mental model of the task.

This project aims to develop technologies that endow robots with the ability to learn procedural knowledge through human-in-the-loop (interactive) learning. The learning process can be designed as an iterative cycle of robot learning (establishment of mental model) and communicating the learning outcome to a human teacher via novel information display techniques. We want to train the robot not only to capture the task knowledge, but more importantly, to let the human know how much has been learned (or not learned). The later requires a quantifiable measurement of the uncertainty level at which the robot infers about a source state and a target state (e.g., “put the cup on the shelf” may be ambiguous if two cups are visible in the scene). In this case, the robot may resolve the ambiguity by asking the human teacher questions (e.g., “Do you mean the red one?”). This leads to many interesting and intriguing research questions, such as (1) How to represent task knowledge in a format that can be communicated easily between human and robot? (2) How to capture simple commonsense knowledge, such as object attributes, affordance, which can be used to restrict the task representation? (3) How to establish an uncertainty measure of robot mental model of the task? (4) How to design an interaction protocol to allow robot to ask relevant questions to communicate its learning outcome? (5) How to capture human feedback and transfer it into new knowledge?

Learning outcome: the candidate will learn to develop novel techniques in computer vision, machine learning and human-robot interaction for the robot task learning. Relevant domains of research include visual recognition, visual question answering, human-in-the-loop reinforcement learning, augmented reality, and human-robot interaction. He/she will explore novel application cases such as assisted living in the domestic setting or collaborative robots in industries.

Participants: Qianli Xu (A*STAR) Joo Hwee Lim (A*STAR), Christophe Jouffrais (CNRS)
Contact: Christophe.Jouffrais_AT_cnrs.fr (replace _AT_ by @)

Augmenting human memory using egocentric view via a wearable device

Human working memory is limited and can only store a few chunks (< 10) of audio-textual and visual-spatial information over a short period of time. In complex tasks where there are dynamic changes in the environment and steps, working memory can be overwhelmed and this can lead to deterioration in performance. In this project, the objective is to investigate the use of egocentric view from a camera embedded in a wearable device to augment working memory and develop a framework and models to (i) understand the dynamics in the environment from the user’s point of view, (ii) capture chunks of information that will aid the user with the task by synergizing with long term memory and (iii) provide information to the user when detecting a ‘forget’ event.

Publicly available egocentric datasets such as EPIC-KITCHENS, Charades, EGO-4D etc, will be used in this project to develop the models and the framework will be implemented in a wearable device to demonstrate its utility in an identified use case. An example will be a cooking task whereby the recipe and ingredients form the long term memory and the working memory consists of the individual steps and instructions. The developed system will have to understand and record the steps that the user has performed and remind the user if critical steps/information are forgotten.

Participants: Lai Xing NG, A*STAR, Joo Hwee LIM, A*STAR, C. Jouffrais, CNRS

Contact: christophe.jouffrais_AT_cnrs.fr (replace _AT_ by @)

Obstacle (anomaly) detection using spiking neural network and event based camera during drone flight

When a drone is flying along its planned flight path, it is possible that there may be unforeseen obstacles which may lead to an accident. As the space of possible obstacles is ill-defined, this can be framed as an anomaly detection problem. Traditional RGB cameras and neural networks may be able to detect the obstacle given time but during drone flight, the detection may not be fast enough to avoid collision. In this project, we will explore the use of spiking neural networks and event based camera for obstacle detection during drone flights. Event based cameras only send a signal (a ‘spike’) when a change is detected in the visual inputs, which drastically reduces the amount of information that needs to be analyzed. The data can be processed with spiking neural networks to detect obstacles.

Publically available event based camera datasets (MVSEC and DSEC) will be used for the project. As obstacles are generally not labelled or available in the data, there may be some effort required to label or synthetically create obstacles in the datasets. Candidate will have to propose and implement a spiking neural network for obstacle detection and conduct experiments and benchmark against other relevant networks. The proposed network will also be implemented in a drone for evaluation.

Participants: Lai Xing NG, A*STAR, Joo Hwee LIM, A*STAR, C. Jouffrais, CNRS

Contact: christophe.jouffrais_AT_cnrs.fr (replace _AT_ with @)

PhD offer in NLP for crisis management

**Context **
Recently, social media has been widely used to generate valuable information in crisis situations (ecological crises, sanitary crises, etc.), showing that traditional means of communication between population and rescue departments (e.g., phone calls) are clearly suboptimal (Olteanu et al.,2015). One of the main characteristics of messages posted during crisis events is that both urgent and non urgent messages are drowned in a deluge of off-topic or personal messages that may contain crisis-related keywords. New tools are thus needed to early access useful information that will allow the emergency units to anticipate actions, providing therefore actionable information that will help to set priorities for the human teams and decide appropriate rescue action (Spiliopoulou et al. 2020, McCreadie et al, 2019;Imran et al., 2015).

**Job description**
The thesis takes place within the Descartes project (https:// www.cnrsatcreate.cnrs.fr/descartes/), a large France-Singapore collaboration project on applying AI to urban systems. The project will make use of Natural language processing (NLP) models to analyse and extract users’ intentions, making detection of user future actions and the AI conversation systems more intelligence and practical. We target intentions for urban crisis management and study in particular: (1) text classification in extremely imbalanced datasets, (2) transfer learning strategies to handle different types of crisis, and finally (3) Analysis of data in real-time.

**Supervision**:

The thesis will happen within the France-Singapore collaboration, with advisors from both sides. The student will be registered at the University of Toulouse, and part of the IRIT lab (www.irit.fr), but is expected to spend a good part of the thesis in Singapore at the A*star Institute for Infocomm Research (I2R, https://www.a-star.edu.sg/i2r), with funding provided by the Descartes program. The thesis will be supervised on the French side by Farah Benamara (IRIT) and co-advised by Jian Su from the Astar lab. The French advisor will also spend time at Astar during the thesis.

**Additional information**

–>Duration of the position: 36 months –> Starting date: October 2022

**Ideal Candidate** –>Master degree in Computer science with solid background in NLP and/or machine learning –> A good experience in deep learning approaches for NLP –> Good programming skills in Python

**Application** Send your CV together with a list of publications to Farah Benamara (farah.benamara@irit.fr) and Jian Su (sujian@i2r.a-star.edu.sg) before 20th February 2022

**References**
Olteanu, A., Vieweg, S., and Castillo, C. (2015). What to Expect When the
Unexpected Happens: Social Media Communications Across Crises. In
Proceedings of the 18th ACM Conference on Computer Supported Cooperative
Work & Social Computing, CSCW ’15, pages 994–1009
McCreadie, R., Buntain, C., and Soboroff, I. (2019). TREC incident streams: Finding actionable information on social media. In Zeno Franco,
et al., editors, Proceedings of the 16th International Conference on
Information Systems for Crisis Response and Management, Valencia, Spain,
May 19-22, 2019. ISCRAM Association.
M. Imran, C. Castillo, F. Diaz, S. Vieweg, Processing Social Media
Messages in Mass Emergency: A Survey, ACM Computing Surveys 47 (2015)
67:1–67:38.
Spiliopoulou E, Maza S.M, Hovy E, Hauptmann A. Event-Related Bias Removal
for Real-time Disaster Events. Findings at EMNLP 2020

PhD offer in NLP for Text to SQL Transformation

**Context **
Large amounts of information are stored in structured and semi-structured
knowledge bases(e.g. Database). In order to effectively retrieve and
analyze the data, it is necessary to interact with the database through
programming languages such as SQL. The difficulty of proficient use of SQL
hinders non-technical users and the data utilization rate. People urgently
need technology and tools to break down the barriers between user and
structured data, that’s what the TEXT-to-SQL(T2S) task aims to solve.
Thus, it will have great potentials in applications. Since 2017, many
Business Intelligence tools have provided functions that can support
natural language queries, but they are focusing on simple queries. In this
study, we are aiming at the solution for complex queries as well as quick
adaptation to new applications.
**Job description**
The thesis takes place within the Descartes project ( https://
www.cnrsatcreate.cnrs.fr/descartes/), a large France-Singapore
collaboration project on applying AI to urban systems. The project will
make use of Natural language processing (NLP) models to analyse and
extract users’ intentions, making detection of user future actions and the
AI conversation systems more intelligence and practical. We target
intentions for urban crisis management and study in particular: (1) Few-
shot learning and data augmentations, (2) Weakly-supervised learning and
(3) Multi-turn dialogue TEXT-2-SQL
**Supervision**:
The thesis will happen within the France-Singapore collaboration, with
advisors from both sides. The student will be registered at the University
of Toulouse and part of the IRIT lab (www.irit.fr), but is expected to
spend a good part of the thesis in Singapore at the A*star Institute for
Infocomm Research (I2R, https://www.a-star.edu.sg/i2r), with funding
provided by the Descartes program.
The thesis will be supervised on the French side by Farah Benamara (IRIT)
and co-advised by Bin Chen and Jian Su from the A*star I2R. The French
advisor will also spend time at A*star during the thesis.
**Additional information**
Duration of the position: 36 months
Starting date: October 2022
**Ideal Candidate**
Master degree in Computer science with solid background in NLP and/or
machine learning
A good experience in deep learning approaches for NLP
Good programming skills in Python
**Application**
Send your CV together with a list of publications to Farah Benamara
(farah.benamara@irit.fr) and Bin Chen (bchen@i2r.a-star.edu.sg) before
February 20th
**References**
Tao Y, Rui Zhang, Kai Y, Michihiro Y, 2018b. Spider: A largescale human-
labeled dataset for complex and crossdomain semantic parsing and text-to-
SQL task. In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 3911–3921

 

DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020.
RYANSQL: Recursively applying sketch-based slot fillings for complex text-
to-SQL in cross-domain databases. arXiv preprint arXiv:2004.03125.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and
Dongmei Zhang. 2019. Towards complex text-to-SQL in crossdomain database
with intermediate representation. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 4524–4535.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew
Richardson. 2020b. RATSQL: Relation-aware schema encoding and linking for
text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pages 7567–7578, Online.
Association for Computational Linguistics

AI-based adaptive human-computer interfaces

We consider in this project the design of adaptive user interfaces, following an AI-based recommendation approach integrating sequential and active learning techniques.

Description

In general, AI-based recommender systems can benefit significantly from sequential (such as multi-armed bandits) or reinforcement learning [2], and active learning techniques [5], when user interests / profiles are unknown, hardly generalizable, or highly dynamic. Such conditions arise in particular in cold-start scenarios, i.e., when addressing a diverse or highly dynamic user base. Online and adaptive recommendation algorithms must therefore (re)learn preferences over time and continuously, striking a balance between exploiting popular recommendation options and exploring new ones, that may improve overall user satisfaction.

When the task at hand is the design of an adapted user interfaces (AUIs) [1], for atypical interface users (such as users with certain disabilities), such techniques can enable the gathering of necessary feedback, in order to understand the key profile features of the user and to adapt to them.  In this sense, AUI design using AI techniques morphs into an approach of incremental recommendation and data completion, at the intersection of recommender systems and active learning.

Data selection methods, such as active learning and core-set selection, are well approaches for improving the data efficiency of predictive models [5]. In short, they assist the learning procedure by prioritizing the selection of relevant unlabeled samples for human labeling, with the goal of maximizing the model performance with minimal labeling cost.

More precisely, on one hand, the learning / recommendation process must request user feedback on interface design alternatives in a principled and optimal way, reminiscent of conversational systems [3] and active-learning in recommender systems [4]. On the other hand, since the context of feedback gathering can be seen as a highly constrained one, interactive mechanisms that can collect the most relevant and complementary data at the right moment require approaches that are online by design. This means in short that they are able to integrate the optimization objective and the task of feedback gathering.

Participants: Christophe Jouffrais (DR CNRS, director of IPAL), Bogdan Cautis (IPAL,  Professeur des Universités – U. Paris Saclay).

Contact: Christophe.Jouffrais_AT_cnrs.fr (replace _AT_ by @)

References

[1] E. Brulé, G. Bailly, A. M. Brock, F. Valentin, G. Denis, C. Jouffrais. MapSense: Multi-Sensory Interactive Maps for Children Living with Visual Impairments. CHI 2016.

[2] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book; MIT Press, 1998.

[3] Sun et al. Conversational Recommender System. In ACM SIGIR 2018.

[4] Rubens et al. Active Learning in Recommender Systems. In Recommender Systems Handbook, 2015.

[5] G. Luo. A Review of Automatic Selection Methods for Machine Learning Algorithms and Hyper-Parameter Values. NetMAHIB, 5(1):18, 2016

PhD position - Can GPS guidance lead to cognitive mapping?
Eye-gaze patterns during teaching/learning

Study of the human factors (eye gaze) on task learning. The aim consists in finding out the eye-gaze patterns of the student during task learning (e.g., when mistakes are likely to happen, when is assistance needed, etc.), which may help the design of future Augmented Human tech.

Contact: C. Jouffrais christophe.jouffrais_AT_ cnrs.fr (replace _AT_ by @)

Augmenting human memory using egocentric view via a wearable device

Human working memory is limited and can only store a few chunks (< 10) of audio-textual and visual-spatial information over a short period of time. In complex tasks where there are dynamic changes in the environment and steps, working memory can be overwhelmed and this can lead to deterioration in performance. In this project, the objective is to investigate the use of egocentric view from a camera embedded in a wearable device to augment working memory and develop a framework and models to (i) understand the dynamics in the environment from the user’s point of view, (ii) capture chunks of information that will aid the user with the task by synergizing with long term memory and (iii) provide information to the user when detecting a ‘forget’ event.

Publicly available egocentric datasets such as EPIC-KITCHENS, Charades, EGO-4D etc, will be used in this project to develop the models and the framework will be implemented in a wearable device to demonstrate its utility in an identified use case. An example will be a cooking task whereby the recipe and ingredients form the long term memory and the working memory consists of the individual steps and instructions. The developed system will have to understand and record the steps that the user has performed and remind the user if critical steps/information are forgotten.

Participants: Lai Xing NG, A*STAR, Joo Hwee LIM, A*STAR, C. Jouffrais, CNRS

Contact: christophe.jouffrais_AT_cnrs.fr (replace _AT_ by @)

Obstacle (anomaly) detection using spiking neural network and event based camera during drone flight

When a drone is flying along its planned flight path, it is possible that there may be unforeseen obstacles which may lead to an accident. As the space of possible obstacles is ill-defined, this can be framed as an anomaly detection problem. Traditional RGB cameras and neural networks may be able to detect the obstacle given time but during drone flight, the detection may not be fast enough to avoid collision. In this project, we will explore the use of spiking neural networks and event based camera for obstacle detection during drone flights. Event based cameras only send a signal (a ‘spike’) when a change is detected in the visual inputs, which drastically reduces the amount of information that needs to be analyzed. The data can be processed with spiking neural networks to detect obstacles.

Publically available event based camera datasets (MVSEC and DSEC) will be used for the project. As obstacles are generally not labelled or available in the data, there may be some effort required to label or synthetically create obstacles in the datasets. Candidate will have to propose and implement a spiking neural network for obstacle detection and conduct experiments and benchmark against other relevant networks. The proposed network will also be implemented in a drone for evaluation.

Participants: Lai Xing NG, A*STAR, Joo Hwee LIM, A*STAR, C. Jouffrais, CNRS

Contact: christophe.jouffrais_AT_cnrs.fr (replace _AT_ with @)

PhD offer in NLP for crisis management

**Context **
Recently, social media has been widely used to generate valuable information in crisis situations (ecological crises, sanitary crises, etc.), showing that traditional means of communication between population and rescue departments (e.g., phone calls) are clearly suboptimal (Olteanu et al.,2015). One of the main characteristics of messages posted during crisis events is that both urgent and non urgent messages are drowned in a deluge of off-topic or personal messages that may contain crisis-related keywords. New tools are thus needed to early access useful information that will allow the emergency units to anticipate actions, providing therefore actionable information that will help to set priorities for the human teams and decide appropriate rescue action (Spiliopoulou et al. 2020, McCreadie et al, 2019;Imran et al., 2015).

**Job description**
The thesis takes place within the Descartes project (https:// www.cnrsatcreate.cnrs.fr/descartes/), a large France-Singapore collaboration project on applying AI to urban systems. The project will make use of Natural language processing (NLP) models to analyse and extract users’ intentions, making detection of user future actions and the AI conversation systems more intelligence and practical. We target intentions for urban crisis management and study in particular: (1) text classification in extremely imbalanced datasets, (2) transfer learning strategies to handle different types of crisis, and finally (3) Analysis of data in real-time.

**Supervision**:

The thesis will happen within the France-Singapore collaboration, with advisors from both sides. The student will be registered at the University of Toulouse, and part of the IRIT lab (www.irit.fr), but is expected to spend a good part of the thesis in Singapore at the A*star Institute for Infocomm Research (I2R, https://www.a-star.edu.sg/i2r), with funding provided by the Descartes program. The thesis will be supervised on the French side by Farah Benamara (IRIT) and co-advised by Jian Su from the Astar lab. The French advisor will also spend time at Astar during the thesis.

**Additional information**

–>Duration of the position: 36 months –> Starting date: October 2022

**Ideal Candidate** –>Master degree in Computer science with solid background in NLP and/or machine learning –> A good experience in deep learning approaches for NLP –> Good programming skills in Python

**Application** Send your CV together with a list of publications to Farah Benamara (farah.benamara@irit.fr) and Jian Su (sujian@i2r.a-star.edu.sg) before 20th February 2022

**References**
Olteanu, A., Vieweg, S., and Castillo, C. (2015). What to Expect When the
Unexpected Happens: Social Media Communications Across Crises. In
Proceedings of the 18th ACM Conference on Computer Supported Cooperative
Work & Social Computing, CSCW ’15, pages 994–1009
McCreadie, R., Buntain, C., and Soboroff, I. (2019). TREC incident streams: Finding actionable information on social media. In Zeno Franco,
et al., editors, Proceedings of the 16th International Conference on
Information Systems for Crisis Response and Management, Valencia, Spain,
May 19-22, 2019. ISCRAM Association.
M. Imran, C. Castillo, F. Diaz, S. Vieweg, Processing Social Media
Messages in Mass Emergency: A Survey, ACM Computing Surveys 47 (2015)
67:1–67:38.
Spiliopoulou E, Maza S.M, Hovy E, Hauptmann A. Event-Related Bias Removal
for Real-time Disaster Events. Findings at EMNLP 2020

PhD offer in NLP for Text to SQL Transformation

**Context **
Large amounts of information are stored in structured and semi-structured
knowledge bases(e.g. Database). In order to effectively retrieve and
analyze the data, it is necessary to interact with the database through
programming languages such as SQL. The difficulty of proficient use of SQL
hinders non-technical users and the data utilization rate. People urgently
need technology and tools to break down the barriers between user and
structured data, that’s what the TEXT-to-SQL(T2S) task aims to solve.
Thus, it will have great potentials in applications. Since 2017, many
Business Intelligence tools have provided functions that can support
natural language queries, but they are focusing on simple queries. In this
study, we are aiming at the solution for complex queries as well as quick
adaptation to new applications.
**Job description**
The thesis takes place within the Descartes project ( https://
www.cnrsatcreate.cnrs.fr/descartes/), a large France-Singapore
collaboration project on applying AI to urban systems. The project will
make use of Natural language processing (NLP) models to analyse and
extract users’ intentions, making detection of user future actions and the
AI conversation systems more intelligence and practical. We target
intentions for urban crisis management and study in particular: (1) Few-
shot learning and data augmentations, (2) Weakly-supervised learning and
(3) Multi-turn dialogue TEXT-2-SQL
**Supervision**:
The thesis will happen within the France-Singapore collaboration, with
advisors from both sides. The student will be registered at the University
of Toulouse and part of the IRIT lab (www.irit.fr), but is expected to
spend a good part of the thesis in Singapore at the A*star Institute for
Infocomm Research (I2R, https://www.a-star.edu.sg/i2r), with funding
provided by the Descartes program.
The thesis will be supervised on the French side by Farah Benamara (IRIT)
and co-advised by Bin Chen and Jian Su from the A*star I2R. The French
advisor will also spend time at A*star during the thesis.
**Additional information**
Duration of the position: 36 months
Starting date: October 2022
**Ideal Candidate**
Master degree in Computer science with solid background in NLP and/or
machine learning
A good experience in deep learning approaches for NLP
Good programming skills in Python
**Application**
Send your CV together with a list of publications to Farah Benamara
(farah.benamara@irit.fr) and Bin Chen (bchen@i2r.a-star.edu.sg) before
February 20th
**References**
Tao Y, Rui Zhang, Kai Y, Michihiro Y, 2018b. Spider: A largescale human-
labeled dataset for complex and crossdomain semantic parsing and text-to-
SQL task. In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 3911–3921

 

DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020.
RYANSQL: Recursively applying sketch-based slot fillings for complex text-
to-SQL in cross-domain databases. arXiv preprint arXiv:2004.03125.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and
Dongmei Zhang. 2019. Towards complex text-to-SQL in crossdomain database
with intermediate representation. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 4524–4535.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew
Richardson. 2020b. RATSQL: Relation-aware schema encoding and linking for
text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pages 7567–7578, Online.
Association for Computational Linguistics

AI-based adaptive human-computer interfaces

We consider in this project the design of adaptive user interfaces, following an AI-based recommendation approach integrating sequential and active learning techniques.

Description

In general, AI-based recommender systems can benefit significantly from sequential (such as multi-armed bandits) or reinforcement learning [2], and active learning techniques [5], when user interests / profiles are unknown, hardly generalizable, or highly dynamic. Such conditions arise in particular in cold-start scenarios, i.e., when addressing a diverse or highly dynamic user base. Online and adaptive recommendation algorithms must therefore (re)learn preferences over time and continuously, striking a balance between exploiting popular recommendation options and exploring new ones, that may improve overall user satisfaction.

When the task at hand is the design of an adapted user interfaces (AUIs) [1], for atypical interface users (such as users with certain disabilities), such techniques can enable the gathering of necessary feedback, in order to understand the key profile features of the user and to adapt to them.  In this sense, AUI design using AI techniques morphs into an approach of incremental recommendation and data completion, at the intersection of recommender systems and active learning.

Data selection methods, such as active learning and core-set selection, are well approaches for improving the data efficiency of predictive models [5]. In short, they assist the learning procedure by prioritizing the selection of relevant unlabeled samples for human labeling, with the goal of maximizing the model performance with minimal labeling cost.

More precisely, on one hand, the learning / recommendation process must request user feedback on interface design alternatives in a principled and optimal way, reminiscent of conversational systems [3] and active-learning in recommender systems [4]. On the other hand, since the context of feedback gathering can be seen as a highly constrained one, interactive mechanisms that can collect the most relevant and complementary data at the right moment require approaches that are online by design. This means in short that they are able to integrate the optimization objective and the task of feedback gathering.

Participants: Christophe Jouffrais (DR CNRS, director of IPAL), Bogdan Cautis (IPAL,  Professeur des Universités – U. Paris Saclay).

Contact: Christophe.Jouffrais_AT_cnrs.fr (replace _AT_ by @)

References

[1] E. Brulé, G. Bailly, A. M. Brock, F. Valentin, G. Denis, C. Jouffrais. MapSense: Multi-Sensory Interactive Maps for Children Living with Visual Impairments. CHI 2016.

[2] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book; MIT Press, 1998.

[3] Sun et al. Conversational Recommender System. In ACM SIGIR 2018.

[4] Rubens et al. Active Learning in Recommender Systems. In Recommender Systems Handbook, 2015.

[5] G. Luo. A Review of Automatic Selection Methods for Machine Learning Algorithms and Hyper-Parameter Values. NetMAHIB, 5(1):18, 2016

PhD position - Can GPS guidance lead to cognitive mapping?
Eye-gaze patterns during teaching/learning

Study of the human factors (eye gaze) on task learning. The aim consists in finding out the eye-gaze patterns of the student during task learning (e.g., when mistakes are likely to happen, when is assistance needed, etc.), which may help the design of future Augmented Human tech.

Contact: C. Jouffrais christophe.jouffrais_AT_ cnrs.fr (replace _AT_ by @)

Internships and PhD positions in the framework of the Descartes Program

All the following positions are open for Interns, and can be continued by a PhD. All these positions will take place in the exciting scientific environment of the Descartes collaborative program. See here for more in Descartes: Descartes presentation

Starting date

The starting date will be in early 2022. All the master internships can lead to a PhD in France or Singapore. Interns who aim to do a PhD will be preferred.

Application

Send an email to the corresponding supervisors with the following documents: 

  • Complete CV (with possible publications)
  • Letter of motivation
  • transcripts of records since L1 or Prepa
  • Report of a previous internship

You may apply to more than one proposal. In this case, please send the documents to all supervisors and mention it in your message.

Proposal #1 - Understanding the environment from drones with multiples sensors

Supervisors: Lai-Xing Ng (Contact: ng_lai_xing@i2r.a-star.edu.sg) and Benoit Cottereau (Contact: benoit.cottereau@cnrs.fr)

Abstract: Drones, or machines in general, have a multitude of sensors that provide information about the surroundings. Existing works on drone perception often use image-based sensors, such as RGB cameras and depth cameras. Image-based sensors are susceptible to motion blur as well as variation in illumination and thus do not work well when the drone is fast-moving. For teleoperated drones, the human operator can only rely on the live video feed of a single camera and the restricted field-of-view affects the human understanding of the drone’s environment. In this project, the aim is to utilize the available sensors and provide a human operator a perspective of being at the drone’s location. Selected candidate will work on developing novel approaches on how distributed sensors can communicate, collaborate (including changing what they sense) and process the signals in an energy-efficient way to extract meaningful information from the scene, in response to existing knowledge models (long term memory) and real-time interaction and decisions from humans, and send back the information to humans for visualization. Research tasks include:

  • Process different types of sensory signals (e.g. data collected from event-based cameras, synchronous cameras, and other sensors) for scene understanding (e.g. object detection and localization) using neuromorphic systems based on artificial neural networks and embedded on a single or multiple drones.
  • Extract meaningful information (3D layout of the scene, objects of interest, threats, etc.) and combine with existing knowledge models.
  • Provide meaningful multimodal feedback to the user based on a wearable device (e.g. smart glasses) that should provide remote (augmented) perception.

Expected skills: The candidate should be willing to work in an international environment which involves Singapore and France, have a good level in English and very good programming skills (in Matlab, Python or C++).

Proposal #2 - Drone Piloting from Different Perspectives

Supervisors: Shen ZHAO (zhaosd@comp.nus.edu.sg) and Christophe JOUFFRAIS (Christophe.Jouffrais@cnrs.fr)

Abstract: New technologies such as mixed reality, natural or wearable interfaces, as well as Artificial Intelligence are beginning to take hold in production facilities. They promise performance gains but can also improve safety and comfort in interactions between human operators and semi-autonomous systems. For these technologies to be accepted and deployed, human factors must be considered.

In this internship, we will design and evaluate a multisensory interface for drone piloting from different perspectives. The aim of the project will be to define the characteristics of a multimodal interface for the control of semi-autonomous drones. Behavioral experiments and the analysis of the collected data will allow the selection of the most suitable parameters for both the design of the interfaces and the evaluation of the human-system interaction. 

Expected skills: The candidate must have skills in human-computer interaction, cognitive science and/or human factors. He-She should be willing to work in an international environment which involves Singapore and France, have a good level in English and good programming skills.

Proposal #3 - Future Video Prediction using Generative Models

Supervisors: Ying SUN (suny@i2r.a-star.edu.sg) and Christophe JOUFFRAIS (Christophe.Jouffrais@cnrs.fr)

AbstractLearning to predict the future is an important research problem in machine learning and artificial intelligence. In this project, we focus on the task of predicting future frames in videos, i.e., video prediction, given a sequence of previous frames. Recently, deep-learning-based methods have emerged as a promising approach for video prediction, especially generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). VAEs can generate various plausible outcomes, however, the predicted frames are blurry and of low quality. While GAN-based models tend to produce higher quality future frames, adversarial training is unstable and may lead to model collapse. Therefore, we will explore state-of-the-art generative models for video prediction and develop new strategies to address the limitations of existing methods.

Expected skills: The candidate should be willing to work in an international environment which involves Singapore and France, have a good level in English and very good programming skills (in Matlab, Python or C++).

Proposal #4 - human-in-the-loop learning

Supervisors: Lai-Xing Ng (Contact: ng_lai_xing@i2r.a-star.edu.sg), Wei-Tsang Ooi (Contact: ooiwt@comp.nus.edu.sg)  and Axel Carlier (Contact: Axel.Carlier@toulouse-inp.fr)

AbstractWhile deep learning has brought important advances in many domains, large labeled datasets are required to ensure good model performances. Several models for collaborations between human and machine learning have been proposed to overcome this limitation and try to decrease the need for labeled data. In active learning, the model explicitly chooses data samples for humans to label, which is then fed into the training process in an online fashion. 

Unlike learning from a large number of pre-labeled data samples, human inputs in human-in-the-loop learning have a larger impact or even overriding effects on machine decisions. Such human-AI collaboration models make it possible for malicious humans to impact the outcome of machine learning models.

In this internship, we plan to build upon existing criteria, such as for example expected model output change (EMOC), to study possible trade-off between impact of (possible malicious) input vs. how fast a model can learn from humans.

Proposal #5 - Interactive Explainable AI

Supervisors: Christophe Hurter (christophe.hurter@enac.fr), Brian Lim (brianlim@comp.nus.edu.sg), and Jamie Ng (jamie@i2r.a-star.edu.sg)

AbstractWith AI capabilities, drones can be used to automatically inspect airplanes and buildings to improve the safety of these structures. However, it is ultimately dependent on the human operator to verify the severity of defects. In this project, we will develop interactive methods to make AI explainable for drone operators and inspectors to interpret and verify the image predictions. This research will investigate how to support user understanding of AI decisions using interactive visualization, and explainable AI. 

We are looking for talented candidates to join our multidisciplinary team.  The project looks into how robotics, computer vision, artificial intelligence, virtual/augmented reality and human-computer interaction can lead to effective human-AI collaboration. We are looking for candidates with passion in research and development, as well as in translating R&D technology into industry applications. 

Expected Skills:

  • Qualification/Field of Study: Bachelors or Masters
  • Technical skills: Strong programming skills (e.g., Python, Javascript, C++).
  • Experience and Knowledge: Computer Vision and Machine Learning, Virtual Reality, Human-Computer Interaction, etc.
  • Aptitude: Critical thinker, self-motivated, can work both independently and in teams, with good analytical and communication skills.

 

Previous Internship Positions

2020  Master Internship positions

Multimodal feedback in a virtual scene

Designing non-visual multimodal feedback to help with navigation in a virtual scene. The student will be helping to develop audio and tactile feedback to guide a user in navigation through a virtual scene. These two modalities will be integrated in a system that attempts to understand user preferences, obtain their feedback for human-in-the-loop reinforcement learning and evaluate our approach. The job task involves help to prepare a software library that, given a 3D virtual scene and a route, renders the orientations and directions to the users using audio (via text-to-speech) and tactile feedback. Both signals will be sent to an earphone and one or two wrist-based tactile bands with motors to provide spatial cues.

Date: project start between May and Dec 2020

Contact: C. Jouffrais christophe.jouffrais  (@) ipal.cnrs.fr  and Shen Zhao dcszs  (@)nus.edu.sg

Easier Scene Understanding with Deep Learning using Context

In this project, the intern will study the problem of scene understanding from a given image using a deep neural network.  Current state-of-the-art methods require a complex and deep network and a large amount of training data.  We will explore how having prior context information about the scene can simplify the problem, and thus the complexity of the network as well as the amount of training data required.

The intern will assist the researcher in experimenting with different neural network models and how the context information can be integrated into the training and inferencing phase of the problem.

Date: project start between May and Dec 2020

Contact: Axel Carlier Axel.Carlier (@) enseeiht.fr and Wei Tsang Ooi ooiwt (@) comp.nus.edu.sg

How To Fool a Deep Neural Net with another Deep Neural Net

Deep neural network has been proved successful in computer vision and natural language. Nevertheless, the research literature has shown that they can be vulnerable if we change several pixels of an image of a dog so that the model may make a wrong prediction. Such a mutated image is called an adversarial sample for the neural network. Such perturbation-based approach looks for adversarial samples from a low and detailed perspective. In this research, we investigate a new adversarial sample generation technique by exploring GAN (Generative Adversarial Network). We are exploring how to use GAN to generate adversarial samples from a higher perspective. More specifically, we are exploring to generate a face of Bob which has never appeared in the training set but can be mistakenly classified as Alice to fool some face recognition system.

Date: project start between May and Dec 2020

Contact: Blaise Genest blaise.genest (@) irisa.fr and Jin Song Dong dcsdjs (@) nus.edu.sg

Neural Network for Differential Equations

Differential equations are one of the main tools for the modelling, simulation and analysis of complex systems in most domains of science and engineering. Neural networks have recently been shown to be able to effectively and efficiently solve differential equations. In fact, several possible approaches are still under investigation. In this project, the researcher will implement and evaluate several existing and new approaches to represent and solve systems of differential equations with neural networks. The student researcher may also be involved in the development of applications of the work to hydrology, meteorology and climate change.

Date: project start between May and Dec 2020

Contact: Talel Abdessalem Talel.Abdessalem (@) telecom-paristech.fr and Stephane Bressan steph (@) nus.edu.sg

 

Previous positions:

      2019 Internships:

      • Towards ageing-well through trusted intelligent systems based on AI, IoT and Formal Analysis
      • Android development of urban mobility app using Fitbit and environment APIs (app implementation, analysis, reasoning)
      • Front and back-end dev and data analysis Node.js (Machine learning, IoT for health, on site validation)
      • Software IoT architecture (refactoring, optimization of platform to enhance large-scale deployments)
      • Web-based visualisation of GeoJSON (interactions in WebGIS environment)
      • Continuous and nonintrusive vital sign monitoring using optical fibre sleep mat (machine learning, sleep cycles data analysis) in collaboration with Khoo Teck Puat Hospital (KTPH) and Singapore University of Technology and Design (SUTD)

      Get to know more:
      Scientists worldwide are welcome to join our challenges! IPAL provides great opportunities to researchers and students from all nationalities who desire to blossom in an excellent international research laboratory. We are committed to provide a unique platform for candidates to begin research and develop their skills in a top-ranked university fully supported by distinguished and world-renowned researchers from Singapore and France.
      CNRS and Universities mobility: If you are already a researcher working for the CNRS, we will be very honored to welcome you in our laboratory. Please have a look at the CNRS website for the procedure, do come in touch with us to prepare a joint ambitious projects, able to boost your carreer, and do not hesitate to contact us for further assistance: CNRS Mobility website
      Singapore, a high-tech and world-class scientific environmentIn a very competitive scientific environment, surrounded by dynamic and talented scientists and supported by one of the best basic and translational research infrastructures worldwide, working in Singapore is a valuable experience. In partnership with the National University of Singapore and the Agency for Science, Technology and Research institutes, world-class scientists from all major scientific centres in the world, are exchanging and sharing with us all year long, generating a prolific scientific osmosis.
      Open PhD Positions and Regular PhD applications: In order to work with IPAL, you need to come in touch with one of our staff during your first year. Please look at our research goalsaxesprojects and publications and you will quickly understand what competencies we will always welcome. Don’t hesitate to contact us if needed. Beside the open position(s) above, a regular submission can be done via the graduates portal NUS, School of Computing, Computer Science Dpt. or NUS, Faculty of Engineering depending on your profile. See also the PhD Programme at NUS School of Computing. Another possibility to get a NUS degree at IPAL is to go for the SINGA – Singapore International Graduate Award programme or the ARAP – A*STAR Research Attachment Programme both funded by A*STAR, with a graduation through NUS in the case of IPAL. Last but not least, regular applications can be done via the EDITE doctoral scool (Informatics, Telecommunications and Electronics) of the University Pierre and Marie Curie, Paris, France or the Doctoral School for Computer Sciences, Applied and Pure Mathematics (MSTII) of the University Joseph Fourier, Grenoble 1, France, for a French PhD while working at IPAL in Singapore or in a collaborative way with highly reputable CNRS labs in France. Please get in touch with us to define your project before application in this case.

      Previous positions:

      2017 Support Team

      2017 Master Internship Proposals

      Internships hosted by our partners on joint projects:

      2017 Post-doc fellowship @BII:

      2016 Master Internship Proposals

      Internships hosted by our partners on joint projects:

      2016 PhD positions

      2014 PhD positions

      2015 Master internship positions

      Internships hosted by our partners on joint projects:

      2014 Master internship positions