Gautham Vasan

The main outcome of my research stay was the development of a novel learning method named Composite Soft Actor-Critic, which can be particularly effective in tasks where an agent needs to learn a control policy using trial-and-error interactions with its environment.

The overarching goal of our project is to integrate very noisy signals decoded from a patient's brain, which include preference and failure information, into a Reinforcement Learning (RL) framework [1], and with the evaluation of this setup for skill learning and adaptation of assistive robots. In the long term, we hope to develop this into a sample-efficient and intuitive method for controlling various technological systems with the help of brain signals and adapting them to the users' preferences and needs. As a first step towards this ambitious goal, we worked on developing a robust, sample-efficient learning algorithm that can handle noisy reward signals (i.e., preference information from a user).

Soft Actor-Critic is a state-of-the-art algorithm in the field of RL specifically designed for continuous action spaces. While this method achieves the best performance in most robotics benchmark tasks, it suffers in the case of noisy reward signals and long task horizons. Composite Q-learning is an off-policy reinforcement learning method that divides the long-term value into smaller time scales.

Github Gautham Vasan

Mirrored Bilateral Training of a Myoelectric Prosthesis with a non-amputated arm via Actor-Critic Reinforcement Learning

This algorithm has been shown to be robust to noisy rewards and is well-suited for long-horizon tasks. However, it is sensitive to the choice of hyper-parameters and is not as robust as Soft Actor-Critic.

We blend the ideas of two existing RL algorithms, namely Composite Q-learning [2] and Soft Actor-Critic [3] and introduce a novel learning method called “Composite Soft Actor-Critic”. This line of research entailed experimenting with advanced optimization techniques, reward shaping and mathematical derivations to extend the ideas of Composite Q-learning to Soft Actor-Critic. Our approach can potentially side-step the issues that assail the original methods.

Guided by mentors and fueled by discussions with peers, my project is evolving into a potential submission at IEEE Robotics and Automation Letters. With these findings in hand, I'm eager to integrate them into real-world applications, especially with user studies involving EEG signal feedback from human users.

Github Gautham Vasan

Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning

The academic vibe at the Universität Freiburg was invigorating. The university's AI labs, interdisciplinary interactions, and collaborative ethos provided a fertile ground for exploration. Engaging in workshops and seminars opened doors to fresh perspectives, fostering innovation and broadening my horizons. I’m particularly thankful to Joschka Boedecker, Erfan Azad, Jan Ole von Hartz, Jasper Hoffmann, Lisa Graf and Gabriel Kalweit for several insightful discussions and feedback on my work.

Being an Indian in Germany was a delightful experience. Everyday experiences like exploring the Black Forest and joining local festivities added vibrancy to my journey. Simple cultural differences, like it is customary for university students to rap their knuckles on the desks after each lecture, was quite fascinating, to say the least. While my command of German is weak, the locals were always pleasant and helpful with directions and recommendations in and around Freiburg.

My time at Universität Freiburg has been transformational. Exposure to top-notch research, mingling with experts, and adapting to a new academic scene have honed my skills, setting the stage for my future research and career trajectory. Beyond the academic realm, I formed bonds that transcend borders. The blend of cultures and intellectual exchanges nurtured friendships that I'm sure will endure. Among the cherished memories, hiking in the Black Forest region and having beers at Kastaniengarten stand out as a unique cultural immersion.

In sum, my research stint at Universität Freiburg has been a journey of innovation, collaboration, and personal evolution. The synergy of focused research, cultural immersion, and academic camaraderie has propelled me towards an exciting path in reinforcement learning.