The goal of this 3-month Forschungspraxis or 6-month thesis is to develop a symbolic task planner based on pretrained multi-modality LLMs and integrate it into existing task and motion planning (TAMP) frameworks.
TAMP has proven effective in addressing a wide range of long-horizon tasks within a single framework[1]. However, the integrated symbolic planner typically depends on manually crafted domain descriptions to operate correctly, which constrains the framework's adaptability. Conversely, LLM excels at reasoning with "common sense" and is resilient to flawed prompt inputs[2]. How could it be used to address the problems in TAMP?
Your tasks:
- Understand the principle of classic TAMP (eTAMP[1])
- Explore the roles of pretrained LLMs in eTAMP: planner or planner assistant?
- Evaluate the proposed methods in benchmark tasks and compare them with baselines
- Document everything
Requirements:
- Experiences with physical simulators (e.g., PyBullet)
- Knowledge of robot motion planning
- Knowledge of PDDL language
- Experience with Git
Contact:
Dr. Tianyu Ren (tianyu.ren@tum.de)
Georg-Brauchle-Ring 60-62, 80992 München
Refenrences:
[1] Ren T, Chalvatzaki G, Peters J. Extended tree search for robot task and motion planning. arXiv preprint, 2021.
[2] Liang J, Huang W, Xia F, et al. Code as policies: Language model programs for embodied control ICRA. IEEE, 2023
To apply:
Send your personal info as attachment to the contact person with the following naming
FirstnameLastname_ Forschungspraxis/Thesis_Staringdate.pdf
e.g., TianyuRen_Thesis_26062024.pdf
Robot Control
The actuator presented in [1,2], based on the Bi-Stiffness Actuation (BSA) concept, consists of five modules, i.e., motor, spring, brake, clutch, and link. With these modules, the actuator can be configured in various operating modes. It also consists of various sensors which enable multiple sensor feedback. The low-level controller currently implemented on this actuator is a cascaded current/velocity/position controller where each loop is PI-based. This cascaded controller is implemented on only the motor module, leaving other modules without low-level controllers.
The objective of this 6-month master’s thesis is to develop a novel Modular Control Framework for Modular Actuator with Multiple Sensor Feedback. By exploiting the modular structure characteristic of the BSA, the effects of each element such as link decoupling, the performance of the elastic element, stiffness brake, and motor control can be tested independently. Moreover, other factors such as friction, hysteresis, effects of coupling and mechanical play between parts, and the effect of switching which creates instantaneous change in velocity can also be analyzed and their effect reduced through control at low-level. To this end, the objective is to consider each module separately and develop a control system that enhances its performance. The end goal is to integrate the controllers into a generalized framework for the combined system.
Requirements for this position
- Dynamic systems modeling techniques
- Background in hardware development and system integration
- Good programming skills with Matlab, Python, and C++
- Experience with system modeling techniques
- Background in robotics with basic understanding of manipulator kinematics and dynamics.
- Working skills in the Ubuntu operating system
- Knowledge of parameter estimation algorithms and sensor fusion frameworks such as Kalman filter is a plus
- Experience in design of experiments is a plus
- E. P. Fortunić, M. C. Yildirim, D. Ossadnik, A. Swikir, S. Abdolshah and S. Haddadin, “Optimally Controlling the Timing of Energy Transfer in Elastic Joints: Experimental Validation of the Bi-Stiffness Actuation Concept,” in IEEE Robotics and Automation Letters, vol. 8, no. 12, pp. 8106-8113, Dec. 2023, doi: 10.1109/LRA.2023.3325782.
- D. Ossadnik et al., “BSA - Bi-Stiffness Actuation for optimally exploiting intrinsic compliance and inertial coupling effects in elastic joint robots,” 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 3536-3543, doi: 10.1109/IROS47612.2022.9981928.
- K. Samuel, R. Oboe and S. Oh, “A Reduced-Order Multisensor-Based Force Observer,” in IEEE Transactions on Industrial Electronics, vol. 69, no. 5, pp. 4946-4956, May 2022, doi: 10.1109/TIE.2021.3086719.
To apply, send your CV, short motivation, and transcripts to Samuel Kangwagye (s.kangwagye(at)tum.de).
The actuator presented in [1,2], based on the Bi-Stiffness Actuation (BSA) concept, consists of five modules, i.e., motor, spring, brake, clutch, and link. With these modules, the actuator can be configured in various operating modes. It also consists of various sensors which enable multiple sensor feedback. The low-level controller currently implemented on this actuator is a cascaded current/velocity/position controller where each loop is PI-based. This cascaded controller is implemented on only the motor module, leaving other modules without low-level controllers.
The objective of this 6-month master’s thesis is to develop a novel optimized framework for integration of direct (sensor) and indirect measurements within the modular actuator to improve measurement accuracy while accounting for sensor redundancy. The cases include:
- Parameters that can be estimated indirectly – develop the estimation algorithm(s)
- Parameters that can be measured with sensors – develop the sensor fusion algorithm(s)
- Parameters that can be estimated by combining direct and indirect methods – develop the integration algorithm(s)
Data integration/fusion frameworks such as the Kalman filter, etc., will be utilized. See example in [3].
Requirements for this position
- Dynamic systems modeling techniques
- Background in hardware development and system integration
- Good programming skills with Matlab, Python, and C++
- Experience with system modeling techniques
- Background in robotics with basic understanding of manipulator kinematics and dynamics.
- Working skills in the Ubuntu operating system
- Knowledge of parameter estimation algorithms and sensor fusion frameworks such as Kalman filter is a plus
- Experience in design of experiments is a plus
- E. P. Fortunić, M. C. Yildirim, D. Ossadnik, A. Swikir, S. Abdolshah and S. Haddadin, “Optimally Controlling the Timing of Energy Transfer in Elastic Joints: Experimental Validation of the Bi-Stiffness Actuation Concept,” in IEEE Robotics and Automation Letters, vol. 8, no. 12, pp. 8106-8113, Dec. 2023, doi: 10.1109/LRA.2023.3325782.
- D. Ossadnik et al., “BSA - Bi-Stiffness Actuation for optimally exploiting intrinsic compliance and inertial coupling effects in elastic joint robots,” 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 3536-3543, doi: 10.1109/IROS47612.2022.9981928.
- K. Samuel, R. Oboe and S. Oh, “A Reduced-Order Multisensor-Based Force Observer,” in IEEE Transactions on Industrial Electronics, vol. 69, no. 5, pp. 4946-4956, May 2022, doi: 10.1109/TIE.2021.3086719.
To apply, send your CV, short motivation, and transcripts to Samuel Kangwagye (s.kangwagye(at)tum.de).
Project-Typ: Forschungspraxis/Ingenieurspraxis (possible thesis extension) Application deadline: September 15, 2024
Efficient and fast manipulation is still a big challenge in the robotics community. Traditionally, generating fast motion requires scaling of actuator power. Recently, however, more attention has been paid to “additional ways” for mechanical energy storage and release in order to keep the actuator power requirements baseline lower (just enough to satisfy general manipulation requirements). To introduce “fast mode” manipulation energy could “be injected” from mechanical elements present in the system. It can be useful for tasks such as throwing or other explosive maneuvers.
Bi-Stiffness Actuation (BSA) concept [1] is the physical realization of the previously mentioned idea. There, a switch-and-hold mechanism is used for full link decoupling while simultaneously breaking the spring element (allowing controlled storage and energy release). Changing modes within the actuator (clutch engagement and disengagement) is followed by the impulsive switch of dynamics.
Students are expected to study and understand the physical and mathematical representations of developed concepts. Apply and gain an understanding of multi-DoF manipulator systems, their control, and classifications. Further, using the state-of-the-art simulation frameworks develop a codebase for its control. The work will be foundational for further research, thus the student is expected to follow best coding practices and document his work.
Requirements from candidates:
- Knowledge of Matlab, C++, Python
- Working skills in Ubuntu operating system
- Familiarity with ROS
- Robotics (Forward, backward dynamics and kinematics)
- Proficiency in English C1, reading academic papers
- Plus are:
- Knowledge of working with Gazebo/MuJoCo
- Familiarity with GIT
- DesignPatterns for coding
- Familiarity with Docker
- Googletest (or other testing framework)
[1] Ossadnik, Dennis, et al. "BSA-Bi-Stiffness Actuation for optimally exploiting intrinsic compliance and inertial coupling effects in elastic joint robots." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.
To apply, you can send your CV, and short motivation to:
Supervisor M.Sc. Vasilije Rakcevic
Pos-type: Forschungspraxis/Internship, possible thesis extention. Application deadline: September 15, 2024
Brushless motors are growing in popularity for Robotics applications. In particular, due to their high power density, these motors can be used with smaller gear ratios to deliver the torque and speed requirements. For example, the key to MIT mini cheetah's success was BLDC adaptation within the Proprioceptive Actuator concept [1].
This task is continuation of the previous work. For this semester, the initial goal is to familiarize by contributing software feature developments, to support various test scenarios of BLDC actuator on already developed testing station. Idea is to in this manner gain insights about physical properties of BLDC actuators, and to mathematically describe them. One should design simple test scenarios and program the control for them (that includes programming absorber side - how the loading will look like, alongside the motor being tested). Collect, visualise and analyse data from experiments (plotting power, efficiency, etc.). Make conclusions about the relation between physical properties of different actuators (high level design choices like inrunner or outrunner, number of poles, control algorithm, etc.) and collected results.
Second part will be joining us to further work on Electronics, Software or Mechanics, for realization of the novel actuator concepts. It's worth mentioning that you will be supported by inhouse developed solutions for BLDC control, available testbed, etc.
Please check attached PDF for a bit more detailed description (To open it, click on the Picture above).
What you will gain:
- Hands-on experience and in-depth understanding of Brushless Motors, and their control
- Best practices for Embedded software development
- Experience building, prototyping, 3d Printing
- Working with DataSheets and Documentations of various Devices
- Hacking electronic signals (via oscilloscope, etc.)
- Insights in our System Development and access to our community
Requirements from candidates:
- Knowledge of C, Matlab
- Working skills in Ubuntu operating system
- Understanding how Motors work
- Basics in Electronics and Mechanics
- Proficiency in English C1, reading academic papers
- Plus are:
- Familiarity with GIT
- Embedded software development
- Robotics
To apply, you can send your CV, and short motivation to:
Supervisor
M.Sc. Vasilije Rakcevic
[1] P. M. Wensing, A. Wang, S. Seok, D. Otten, J. Lang and S. Kim, "Proprioceptive Actuator Design in the MIT Cheetah: Impact Mitigation and High-Bandwidth Physical Interaction for Dynamic Legged Robots," in IEEE Transactions on Robotics, vol. 33, no. 3, pp. 509-522, June 2017, doi: 10.1109/TRO.2016.2640183.
Brain Computer Interfaces
The potential for invasive Brain Computer Interface (iBCI) systems to dramatically improve the livelihoods of those who have lost motor functionality has increased in recent years. This improvement has emerged due in part to advances in the field of deep learning, but the performance of these models varies significantly depending on the quality of the data collection and the design of the deep learning model architectures. This project focuses on designing and refining deep learning architectures tailored for decoding spiking neural data, with an emphasis on handling sparse and binary input data, continuous output kinematic features, and maintaining a small footprint to facilitate real-time inference. Research with this project will provide experience in pushing the boundaries of what's possible with iBCI systems and will involve working in a collaborate environment for one of the most difficult decoding tasks in the bio-engineering field. Researchers will have access to state-of-the-art facilities and computational resources and will benefit from mentorship opportunities and professional development.
Tasks may include a few of the below (to be discussed depending on your interest and background)
- Further develop and optimize an existing decoding pipeline in python
- Develop state-of-the-art deep learning models based on convolutional and/or recurrent neural networks
- Investigate intermediate latent spaces that may exist between the neural data and the associated kinematic features
- Create a reporting mechanism to make assessment of different architectures efficient, which will enhance our ability to rapidly iterate and improve decoding models
Prerequisites
- Proven experience with building deep learning architectures in Tensorflow
- Proficiency in python
Helpful but not required
- Experience with real-time inference with trained models
- Theoretical understanding of manifolds and dimensionality reduction methods
Position
Current project is targeted for a research internship or semester thesis. It’s not a paid position, nor a master thesis ad.
Related Literature
- Decoding arm kinematics: Paper 1 (https://journals.sagepub.com/doi/full/10.1177/0278364914561535) Paper 2 (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(12)61816-9/)
- Backend Communication: (https://www.biorxiv.org/content/10.1101/2023.08.08.552473v1.full.pdf)
For more information, please contact:
Dr. Alexander Craik (alex.craik(at)tum.de)
Ioannis Xygonakis (ioannis.xygonakis(at)tum.de)
Dexterous manipulation
The goal of this 3-month Forschungspraxis/6-month thesis is to finetune OpenVLA[4] for multifingered grasping or manipulation tasks. The platform to be used for testing is a 7 DoF arm equipped with a multifingered hand.
Background:
The emergence of language models, such as GPT and Llama, has significantly enhanced the capabilities of robots by endowing them with common knowledge, thereby enabling them to address complex tasks in a more generalized manner. Vision language action model integrates language with perception and action. Existing works show that pretrained vision language action model has the ability to control multiple robots out of the box in an end-to-end way. OCTO[1] integrates state-of-the-art vision, language, and control models to enable robotic control via natural language commands. RT2[2] expresses actions as tokens and trains on a large-scale dataset to enhance robotic performance. VoxPoser[3] employs a modular algorithm that constructs a 3D value map by considering affordances and constraints, facilitating more precise control. OpenVLA[4] utilizes a fine-tuning approach, resulting in higher success rates across multiple tasks. In dataset of these models, the common end effector is two-finger gripper. This limitation restricts the models' ability to perform tasks of dexterous hand operations. Yuyang et al. [5] show dexterous has the ability to adapt to more complex robotic tasks.
Your tasks:
- Train and test OpenVLA
- Collect demonstration from a 7DoF arm and a multifingered hand
- Finetune OpenVLA model to fulfill multifingered grasping or manipulation tasks
- Implement the experiments on the real robots
Requirements:
- Knowledge of large language models (LLM) or foundation models
- Experiences or knowledge from related Robotic courses
- Knowledge of Pytorch or similar machine-learning library
- knowledge and experience in Git, Linux is a plus
Helpful but not required:
Experience with diffusion policy, reinforcement learning, MoveIt, ROS
Contacts:
Dr. Shuang Li (li.shuang(at)tum.de)
Contacting best before 30.09.2024
Location/s: Georg-Brauchle-Ring 60-62, 80992 München
References:
[1] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, & Sergey Levine. (2024). Octo: An Open-Source Generalist Robot Policy.
[2] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, & Brianna Zitkovich. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.
[3] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, & Li Fei-Fei. (2023). VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models.
[4] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, & Chelsea Finn. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
[5] Li, Y., Liu, B., Geng, Y., Li, P., Yang, Y., Zhu, Y., Liu, T., and Huang, S. 2024. Grasp Multiple Objects With One Hand. IEEE Robotics and Automation Letters, 9(5), p.4027–4034
Robotic hand modeling
Type: Forschungspraxis
Diverse robotic hands have been designed and developed to achieve the human hand's performance in daily tasks. The tendon-driven robotic hand offers the advantage that the motors are set remotely and actuate the hand via tendons that mimic the human musculotendon system. A dynamic model is necessary to control such a complex multi DoFs system. Additionally, having a model enables us to simulate the hand functions with the tendon forces as control input and validate the controller hypothesis.
As the hand group in MIRMI, we are interested in tendon-driven robotic hand design, modeling, control and planning in order to explore and improve the hand manipulations.
Tasks:
- Implementing a dynamic model of the hand prototype using a developing MATLAB modelling toolbox
- Further development and summarizing of the modelling toolbox
- Model parameter identification with experiments
- Dynamic simulation using the generated model
Prerequisites:
- Knowledge of robot kinematic and dynamic modelling
- Basic knowledge of impedance control
- Experience with MATLAB/Simulink programming
- Basic understanding of hand manipulation
- Willing to explore new knowledge domain
If you are interested in this topic and gain more hands-on experience with modelling, please do not hesitate to contact us.
Contact:
Supervisor: M.Sc. Junnan Li (junnan.li(at)tum.de)
Contacting best before 30.09.2024
You can find more research opportunities at the Chairs of our Principal Investigators!
Data Protection Information
When you apply for a position with the Technical University of Munich (TUM), you are submitting personal information. With regard to personal information, please take note of the Datenschutzhinweise gemäß Art. 13 Datenschutz-Grundverordnung (DSGVO) zur Erhebung und Verarbeitung von personenbezogenen Daten im Rahmen Ihrer Bewerbung. (data protection information on collecting and processing personal data contained in your application in accordance with Art. 13 of the General Data Protection Regulation (GDPR)). By submitting your application, you confirm that you have acknowledged the above data protection information of TUM.