GOAT Robotics received the Make in India Excellence Award 2025, presented by DPIIT, Government of India.

How AI Simplifies Tote Perception Development

How GOAT Robotics combines synthetic training data, RGB-D vision and geometric processing to build a more repeatable tote-detection workflow.

By Naveen Sakthivel — — 4 min read

Workflow diagram: simulate known tote geometry, learn with an RGB-D detector, check with geometric refinement, then review scenes and model versions

A person can glance at a tote stand and recognise the next container to handle. For a robot, that glance becomes a series of measurements: where does the tote begin, which rim is exposed, how is it oriented, and where is its centre in three-dimensional space?

At GOAT Robotics, we are developing an AI-based perception workflow for this task. The aim is practical: give the automation system a useful description of the tote, while making the process of developing and testing that perception system easier to repeat.

Actual training-generator screenshot showing the camera view and an external view of the same simulated tote scene

Captured from the running training generator: the camera view on the left and the same simulated scene on the right.

Start with a defined task

Our detector has a specific job: identify the tote boundary and estimate its rim centre and orientation from colour and depth data. The rectangular rim also provides corner coordinates that can help define an approach direction.

The detector does not drive the robot arm or decide how a gripper should close. Those responsibilities belong to motion planning and control. Keeping perception separate lets the team work on tote recognition independently of gripping and robot trajectories.

Use simulation to create examples and labels together

Collecting and labelling every arrangement by hand would be slow. A tote may be empty or filled, slightly shifted, leaning against its neighbour or partially nested. The camera can observe the same arrangement from a different angle, and the depth sensor can return incomplete measurements around edges.

We start with the tote’s known geometry and generate synthetic scenes. Because the simulator knows the position of each tote, it can produce a training observation and its reference pose together. This reduces the manual annotation needed for synthetic data and makes it possible to repeat a difficult scene.

The variations must still resemble the intended operation. Our training scope includes contained objects, modest placement changes and realistic camera variation. Generating more unusual scenes is useful only when they represent conditions the system may actually encounter.

Actual optimizer minibatch showing processed scene inputs, target heatmaps and the model’s current forward-pass output

A real optimizer minibatch from the training pipeline, showing inputs alongside target heatmaps and model output. These are simulation-based training observations, not customer-site photographs.

Combine learned recognition with measured geometry

Colour helps describe appearance; depth adds distance and surface structure. Together, they give the detector more information than a flat rectangle in a photograph.

Our workflow pairs a learned pose proposal with geometric refinement. The model proposes where the rim is. The refinement stage compares that proposal with the measured depth structure and the expected tote geometry.

This combination gives the workflow two complementary tools: learned recognition for locating the tote and geometry for interpreting its shape in three dimensions.

Make iteration easier to inspect

AI also helped us build the development tools around the detector: scene-generation code, training utilities, visual previews and calibration interfaces. AI assistance made it easier to turn an idea into something we could inspect and test; engineers still had to check the geometry, assumptions and results.

That tooling matters. We can generate varied scenes, train from saved checkpoints and review the process through a shared set of visual tools. Versioned releases keep model weights and their development records organised.

Four-step loop: simulate known tote geometry, learn from RGB-D, apply geometric checks, and review scenes and model versions

The development loop connects generated data, learned perception and development review. Each iteration builds on a saved development record.

Make the workflow easier to follow

Visual tools bring the development process together. Scene previews show how totes and their contents are arranged. Camera views make it easier to understand what the perception system sees. Calibration interfaces reuse known tote dimensions and illustrate the installation measurements that still need to be collected.

Saved checkpoints let development continue from previous work. Keeping model versions, scene settings and engineering notes together gives the team a clear record of each iteration.

A simpler way to develop tote perception

The value of AI here is a more manageable engineering process: less manual labelling of synthetic scenes, repeatable tests, inspectable failures and a clearer separation between perception and robot control.

That gives our team a better way to develop tote-handling perception — and a practical foundation for the next stage of integration.

Exploring tote-handling automation? Talk to GOAT Robotics about your material flow and tote-handling workflow.

See a GOAT AMR working on your floor

We bring a robot and an engineer to your site. Tell us what you move, how far and how often, and we’ll recommend the right robot and fleet size.

Chat with sales on WhatsApp