GRASP
Open-world tabletop manipulation system using neuro-symbolic reasoning to execute natural language sorting instructions for warehouse environments.
GRASP is an internship project at USG/UMD under Dr. Romel Gomez, built on top of JarvisSort, a gantry-based automated sorting system originally developed by a CPSE Summer team. I redesigned it into a fully open-world tabletop manipulation system that lets a robot understand and execute natural language instructions such as “move bottles to the bottom and all green things to the top,” with potential applications in assistive technology, warehouse automation, and lab environments.
The core pipeline runs perception -> reasoning -> action. GroundingDINO handles open-vocabulary object detection, feeding detections into an LLM that generates a goal state from the natural language instruction. The system then compares the current shelf state against the goal state to determine which objects need to move and in what order, triggering and confirming sorting tasks autonomously. A client-server architecture with Raspberry Pi handles vision execution, camera streaming, annotated outputs, and automated label and environment processes.
End-to-end pick-and-place control maps detections directly to real motor behavior–yaw, pitch, roll, claw open/close, object grasp and release–integrating both an Arducam and a Logitech camera feed. The hardware runs on a CoreXY gantry with a PCA9685 I2C multi-servo controller and a differential-drive claw mechanism. I rebuilt and optimized the hardware integration loops repeatedly throughout the project, including gantry axle and idler fixes, rewiring the multi-servo controller, setting new angle and limit states, and validating the claw and differential drive for reliable autonomous placement.