diff --git a/documents/rulebook.tex b/documents/rulebook.tex index d4496ffc..1417d5cf 100644 --- a/documents/rulebook.tex +++ b/documents/rulebook.tex @@ -117,7 +117,7 @@ \chapter{Tests in Stage II} %%% FINALS %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\input{tests/Finals} +\input{tasks/Finals} \printabx diff --git a/tests/Finals.tex b/tasks/Finals.tex similarity index 100% rename from tests/Finals.tex rename to tasks/Finals.tex diff --git a/tasks/CleanUp.tex b/tasks/old/CleanUp.tex similarity index 100% rename from tasks/CleanUp.tex rename to tasks/old/CleanUp.tex diff --git a/tests/CocktailParty.tex b/tasks/old/CocktailParty.tex similarity index 100% rename from tests/CocktailParty.tex rename to tasks/old/CocktailParty.tex diff --git a/tests/EEGPSR.tex b/tasks/old/EEGPSR.tex similarity index 100% rename from tests/EEGPSR.tex rename to tasks/old/EEGPSR.tex diff --git a/tasks/Farewell.tex b/tasks/old/Farewell.tex similarity index 100% rename from tasks/Farewell.tex rename to tasks/old/Farewell.tex diff --git a/tasks/FindMyDisk.tex b/tasks/old/FindMyDisk.tex similarity index 100% rename from tasks/FindMyDisk.tex rename to tasks/old/FindMyDisk.tex diff --git a/tasks/FindMyMates.tex b/tasks/old/FindMyMates.tex similarity index 100% rename from tasks/FindMyMates.tex rename to tasks/old/FindMyMates.tex diff --git a/tasks/HandMeThat.tex b/tasks/old/HandMeThat.tex similarity index 100% rename from tasks/HandMeThat.tex rename to tasks/old/HandMeThat.tex diff --git a/tests/HelpMeCarry.tex b/tasks/old/HelpMeCarry.tex similarity index 100% rename from tests/HelpMeCarry.tex rename to tasks/old/HelpMeCarry.tex diff --git a/tests/OpenChallenge.tex b/tasks/old/OpenChallenge.tex similarity index 100% rename from tests/OpenChallenge.tex rename to tasks/old/OpenChallenge.tex diff --git a/tests/PnG.tex b/tasks/old/PnG.tex similarity index 100% rename from tests/PnG.tex rename to tasks/old/PnG.tex diff --git a/tests/SPR.tex b/tasks/old/SPR.tex similarity index 100% rename from tests/SPR.tex rename to tasks/old/SPR.tex diff --git a/tasks/ServingDrinks.tex b/tasks/old/ServingDrinks.tex similarity index 100% rename from tasks/ServingDrinks.tex rename to tasks/old/ServingDrinks.tex diff --git a/tasks/SmoothieChef.tex b/tasks/old/SmoothieChef.tex similarity index 100% rename from tasks/SmoothieChef.tex rename to tasks/old/SmoothieChef.tex diff --git a/tasks/TakeOutGarbage.tex b/tasks/old/TakeOutGarbage.tex similarity index 100% rename from tasks/TakeOutGarbage.tex rename to tasks/old/TakeOutGarbage.tex diff --git a/tasks/TidyUp.tex b/tasks/old/TidyUp.tex similarity index 100% rename from tasks/TidyUp.tex rename to tasks/old/TidyUp.tex diff --git a/tests/TourGuide.tex b/tasks/old/TourGuide.tex similarity index 100% rename from tests/TourGuide.tex rename to tasks/old/TourGuide.tex diff --git a/tasks/WhatisThat.tex b/tasks/old/WhatisThat.tex similarity index 100% rename from tasks/WhatisThat.tex rename to tasks/old/WhatisThat.tex diff --git a/tasks/WhereIsThis.tex b/tasks/old/WhereIsThis.tex similarity index 100% rename from tasks/WhereIsThis.tex rename to tasks/old/WhereIsThis.tex diff --git a/tests/GPSR.tex b/tests/GPSR.tex deleted file mode 100644 index 92751c97..00000000 --- a/tests/GPSR.tex +++ /dev/null @@ -1,89 +0,0 @@ -\section{General Purpose Service Robot} - -% -% MAURICIO @2017 -% Short instructions as in 2012 rulebook -% -This test evaluates the abilities of the robot that are required throughout the set of tests in stage I of this and previous years' RuleBooks. In this test the robot has to solve multiple tasks upon request. That is, the test is not incorporated into a (predefined) story and there is neither a predefined order of tasks nor a predefined set of actions. The actions that are to be carried out by the robot are chosen randomly by the referees from a larger set of actions. These actions are organized in three categories with different complexity. Scoring thereby depends on the complexity class. - -% -% MAURICIO @2017 -% Same focus as in 2012 rulebook. There should be no problem since this is SOLVED -% -\subsection{Focus} -This test particularly focuses on the following aspects: -\begin{itemize} - \item No predefined order of actions to carry out (to get away from state machine-like behavior programming). - \item Increased complexity in speech recognition. - \item Environmental (high-level) reasoning. - \item Efficient and fast task execution (speed). -\end{itemize} - - -% -% MAURICIO @2017 -% Test has been shorten based in 2012 one, however, categories have changed -% to bypass NLP and avoid problems agreed by TC. -% -% All extra explanations have been sent to corresponding appendix. -% -\subsection{Task} - -\begin{enumerate} - \item \textbf{Entering and command retrieval:} The robot enters the arena and drives to a designated position where it has to wait for further commands. - - \item \textbf{Command generation:} A command is generated randomly, depending on the command category chosen by the team (see below). Commands are generated by the generator which is made publicly available at https://github.com/kyordhel/GPSRCmdGen. \\ - - \item \textbf{Command categories:} The team may choose from the following three categories: - \begin{enumerate} - \item \textbf{Category I:} Tasks with a low difficulty degree. - \item \textbf{Category II:} Tasks with a moderate difficulty degree. - \item \textbf{Category III:} Tasks with a high difficulty degree or with incomplete/erroneous information. - \end{enumerate} - - \item \textbf{Task assignment:} The robot is given the command by the operator and may directly start to work on the task assignment. - - \item \textbf{Returning to the operator:} After accomplishing the assigned task, the robot has to move back to the operator to retrieve the next command (i.e., go back to 1. without the need of re-entering the arena). The robot can work on at most three commands. After the third command, it has to leave the arena. - - \item \textbf{Exiting the arena:} After accomplishing the assigned task, the robot has to leave the arena. - - The robot must prove it has understood the given command by repeating it (Please see the remarks about this in section~\ref{sec:gpsr_remarks}). -\end{enumerate} - -\subsection{Additional rules and remarks} -\label{sec:gpsr_remarks} -\begin{enumerate} - \item \textbf{Referees:} Since the score system in this test involves a subjective evaluation of the robot's behavior, the referees are EC/TC members. - - \item \textbf{Category selection:} For every of the three commands given to the robot, the team chooses the desired command category. - - \item \textbf{Operator:} - \begin{itemize} - \item The person operating the robot is one of the referees (default operator). - \item If the robot appears to consistently not be able to understand the operator, the referees ask the team to use a custom operator or bypassing speech recognition (\refsec{rule:asrcontinue}). - \end{itemize} - - \item \textbf{Retrieving the command:} The robot must show it has understood the given command by repeating the command (i.e.~stating all the required information to accomplish the task). - \\ - \textit{Note:} Referees must have sufficient evidence proving the robot is actively trying to execute the commanded tasks to score. Robots skipping command execution will not receive points for understanding the command. - - \item \textbf{Incremental scoring:} Scoring depends on the category chosen by the team leader and the previous successfully accomplished command. Thereby, scoring for a second and third command depends on how well the robot solved (not understood) a first and second command respectively. Referees determine how well the command was accomplished and its impact on the incremental scoring of subsequent commands. -\end{enumerate} - -\subsection{Referee and OC instructions} -\textbf{2h before test:} -\begin{itemize} - \item Specify and announce the entrance and exit door - \item Specify and announce the command retrieval point -\end{itemize} - -\subsection {Audio Data Recollection} -Teams are encouraged to submit to the TC the audio data recorded during the test, specially that which was captured during speech recoginition. If so, teams are urged to provide it with annotation of what the user said and what was recognized. Audio files are expected to be mono, one per microphone (in the case array recordings), of a sample rate equal to or higher than 16 kHz, and with a sample size of at least 16 bits. Depending on the quality of the recordings and their annothations, an official certificate that formalizes these efforts may be provided to submitting teams. - -\newpage -\subsection{Score sheet} -\input{scoresheets/GPSR.tex} - -% Local Variables: -% TeX-master: "Rulebook" -% End: diff --git a/tests/Restaurant.tex b/tests/Restaurant.tex deleted file mode 100644 index ad3c6116..00000000 --- a/tests/Restaurant.tex +++ /dev/null @@ -1,140 +0,0 @@ -\section{Restaurant} -The robots are tested in a real environment such as a real restaurant or a shopping mall. -There are \emph{two} robots helping clients in the restaurant at the same time. - -\subsection{Focus} -This test focuses on online mapping, safe navigation in previously unknown environments, gesture detection, human-robot interaction, and manipulation in a real environment. - -The robot will need to create its own map from the environment and then move within it to handle human requests, such as delivering drinks or snacks, while people are walking around. -As this test is performed with 2 robots (2 teams, each with their own 1 robot) in parallel, the robots will also have to avoid each other. - -\subsection{Setup} -\begin{enumerate} - \item \textbf{Location:} A real restaurant fully equipped with a \enquote{Professional Barman} i.e.~the operator and at least three tables with \enquote{Professional Clients}. -\end{enumerate} - -\subsection{Task} -\begin{enumerate} - \item \textbf{Start:} The robot starts at a designated starting position. After the start signal is given, the robot may look around to keep an \textit{eye} on the tables. - The location of the tables is not taught to the robot via some training phase. - - \item \textbf{Calling:} A guest will ask for the robot's attention by waving \emph{and} calling it out using voice. - The robot must state out loud that it has detected the call. - % A robot must also indicate roughly where it has detected the call (e.g.~\enquote{to my left}) - In case both robots notice the same call, the \textit{Professional Barman} will tell one of the robots to take the order. - The barman will say the robot's name followed by \enquote{Take the order} e.g.~\enquote{R2D2, take the order}. - The other robot will simply have to wait for another call. - If the robot \textit{not} commanded to take the order still goes, it will be commanded to wait (e.g.~\enquote{C3PO, Wait}). - In case the robot keeps going after that, the emergency button will be used to stop the robot. - -% \item \textbf{Parallel orders:} It may occur that two guests want to order something at the same time and thus wave at the same time. -% The referees will arange these two guests such that each is clearly visible to (in front of) one of the robots. - - \item \textbf{Ordering}: The robot must ask the person what he or she wants to order. See Orders below for details about ordering. - \item \textbf{Avoiding random person:} At any time while going to any of the tables or to the \textit{Kitchen}, a person may step on the robot's path. - It is expected of the robot to avoid that person or stop and wait for it to move away. - - \item \textbf{Delivering phase:} - \begin{enumerate} - \item \textbf{Repeating the order:} Once again in the kitchen, the robot recites the orders for each table (e.g.~\textit{\enquote{Hamburger with fries for table A and Orange juice for table B}}, to the \textit{Professional Barman}. - The \textit{Professional Barman} will serve the order and place it into a tray on the Kitchen-bar. - If the barman cannot understand the order that the robot repeats, he cannot hand out the order and no points can be awarded for reciting the order. - - \item \textbf{Grabbing a beverage:} The robot must grab a can of the appropriate drink from a set of cans on the Kitchen-bar. - - \item \textbf{Grabbing a combo:} The robot must carry a tray with the ordering from the kitchen-bar. - Teams must indicate beforehand whether the robot is able to grasp the plate itself, whether it needs a tray or whether the plate needs to be handed to the robot. - - \item \textbf{Delivery:} The robot must place the order on the table. - If the robot is not able to do this, the robot is allowed to hand over the order, but the client is not allowed to shift his/her chair or stand up. - The robot must help the client, not the other way around. - \end{enumerate} - - \item \textbf{Next customer, please:} When the robot is in the kitchen, the \textit{Professional Barman} will ask the robot to either find a new client to serve or to stop the test. - The barman will either tell the robot \enquote{R2D2, Wait} to make it wait for another client or \enquote{R2D2, Stop the test} to end the test for that robot. -\end{enumerate} - -\textbf{Orders:} The menu offers Beverages and Combos. An order may be a Beverage or Combo. Some guest(s) will order a Combo while another will order a Beverage. - A Combo is a combination of two of the food items from the set of objects~\ref{rule:scenario_objects}, e.g.~\enquote{noodles with peanuts} or \enquote{noodles and peanuts}. - Guests also prefer to state their order in a natural way, as they would in a restaurant operated by humans. - -\begin{figure}[tbp] - \centering - \includegraphics[width=0.5\columnwidth]{images/restaurant.png} - \caption{Restaurant test: example setup.} - \label{fig:restaurant} -\end{figure} - -\subsection{Additional rules and remarks} - -\begin{itemize} - \item \textbf{Safety!} This test takes place in a public area. That is, there may be people standing, sitting or walking around the area throughout the test. The robot is expected to not even slightly touch anything and is immediately stopped in case of danger. - - \item \textbf{Referees and guidance:} For safety reasons, the referees in this test are TC members. One of the referees follows the robot and is always in reach of the emergency button. - - \item \textbf{Start:} There is no fixed start signal in this test, it starts when both robots are ready (though within a reasonable time). - - \item \textbf{Order:} The way the user provides information to the robot is up to the robot's team. A natural interaction is preferred. - - \item \textbf{Location:} This test can be arranged in any real restaurant or shopping mall. If this is not possible, the test can be conducted in an arbitrary room containing the appropriate locations. - The only requirement is that this room is not part of the arena and that the teams do not know the room beforehand. - The exact location, including the object and delivery locations, will be defined by the technical committee on site (and in corporation with the local organization). - In addition, to avoid unnecessary time investment for navigation, the distances between tables and the \enquote{Kitchen Bar} will be minimal. - - \item \textbf{Disturbances from outside:} If a person from the audience (severely) interferes with the robot in a way that makes it impossible to solve the task, the teams may repeat the test immediately. - - \item \textbf{Learning tables:} Of course, it can only be sure that a robot correctly remembered where an order is suppposed to be delivered when it is able to go there after grabbing the order. - - \item \textbf{Instruction:} The robot interacts with the operators, not the team. That is, the team is only allowed to (very!) briefly instruct the \textit{Professional Barman} - \begin{itemize} - \item How to the tell the robot the order has been served - \end{itemize} - It is not allowed to the team to instruct the clients on how to get robot's attention. It shall be done in a natural way like when interacting with a human waiter. - - \item \textbf{Kitchen-bar:} The \textit{Kitchen-bar} will be a table located at the restaurant's kitchen, next to the place where the robot started. - The robot may ask on which side of the robot the Kitchen-bar is, e.g.~on its left or right side. It may ask this at any time, but it is better if the robot infers this itself. - It has the following setup. - \begin{itemize} - \item \textbf{Barman:} A \textit{Professional Barman} (member of the TC) will be at the other side of the Kitchen-bar to take the order provided by the robot and serve it in the official tray. - \item \textbf{Beverages:} Beverages will be located on the Kitchen-bar next to the \textit{Professional Barman}. - \end{itemize} - -\end{itemize} - -% \subsection{Data recording} -% Please record the following data (See~\refsec{rule:datarecording}): -% \begin{itemize} -% \item Audio -% \item Commands -% \item Mapping data -% \item Images -% \item Plans -% \end{itemize} - -\subsubsection{Referee instructions} - -The referee needs to -\begin{itemize} - \item Prepare orders for each client in advance, so that there can be no confusion. These orders must also be available at the kitchen. -\end{itemize} - -% \subsubsection{OC instructions} - -% \textbf{2 hours before the test} -% \begin{itemize} -% \item -% \item -% \end{itemize} -% \textbf{During the test} -% \begin{itemize} -% \item -% \item -% \end{itemize} - -\newpage -\subsection{Score sheet} -\input{scoresheets/Restaurant.tex} - -% Local Variables: -% TeX-master: "Rulebook" -% End: diff --git a/tests/StoringGroceries.tex b/tests/StoringGroceries.tex deleted file mode 100644 index 4fd693f4..00000000 --- a/tests/StoringGroceries.tex +++ /dev/null @@ -1,125 +0,0 @@ -\section{Storing Groceries [DSPL \& OPL]} -The robot helps by storing newly bought groceries in the cupboard next to the objects of the same kind that are already there; for instance by placing fresh apples near other apples. - -\subsection{Goal} -The robot has to correctly identify and manipulate objects at different heights, grouping them by category and likelihood. - -\subsection{Focus} -This test focuses on the detection and recognition of objects and their features, as well as object manipulation. - -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% -% Setup -% -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\begin{minipage}{0.70\textwidth} - \subsection{Setup} - \begin{enumerate} - \item \textbf{Location:} This test can take place either inside or outside the arena. The testing area must have a bookcase or cupboard, and a nearby table. The maximum distance between the Table and the Cupboard is 2 meters. - \item \textbf{Start position:} The robot starts between the cupboard and the table in a random orientation, but facing towards the Cupboard. - \item \textbf{Cupboard:} The cupboard has 5 shelves between 0.0m and 1.80m from the ground and contains several objects grouped by category or likeliness (See~\ref{rule:scenario_objects}). The cupboard has at least one free space for starting a new set. - \begin{itemize} - \item \textbf{Door:} The cupboard has a single door, which is closed initially. - This door encloses some of the objects, covering up to one half of the cupboard (e.g.~the left or bottom half), as indicated by the hatched area in Figure~\ref{fig:storing_groceries_shelf}. - \end{itemize} - \item \textbf{Table:} The table has at least 5 objects (but no more than 10). If not all objects fit on the table, they will be added as the robot frees up space. - \end{enumerate} -\end{minipage}\hfill -\begin{minipage}{0.25\textwidth} - \begin{figure}[H] - \centering - \includegraphics[width=\textwidth]{images/storing_groceries.png}% - \vspace{-10pt} - %\caption{Example shelf where objects will be placed.} - \caption{Shelf} - \label{fig:storing_groceries_shelf} - \end{figure} -\end{minipage} - -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% -% Task -% -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{Task} -\begin{enumerate} - \item \textbf{Evaluating the situation:} The robot inspects its surrounding and analyzing the best course of action. In any order, the robot has to: - \begin{itemize} - \item \textit{Inspect the cupboard} (locating and categorizing existing groceries). - \item \textit{Open the cupboard's door.} If the robot can't open the door, it may ask the Referee to do it. - \item \textit{Inspect the table} (analyze the newly bought groceries, i.e.~objects). - \end{itemize} - - \item \textbf{Moving objects:} The robot moves as many objects as possible in the given time - (only the first five score) - from the Table to the Cupboard, allocating similar objects all together. - Stacking is allowed. - \begin{itemize} - \item Objects of the same type (i.e.~identical known objects or akin alike objects) must be placed one next to the other. - \item If the Cupboard has no object of the same type, then objects must be grouped by category (e.g.~drinks with drinks, snacks with snacks, etc) - \item If the Cupboard has no similar object, the robot must clearly state its decision on how to solve the problem. For instance, the robot can start a new set in a free space for either all unknown objects or all objects sharing a particular feature (color, shape, function, etc.). - \item Moving two objects at a time (2-handed manipulation) is allowed. - \end{itemize} - - \textbf{Note:} Either before or after grasping an object the robot may announce the name of the object found. - \item \textbf{Repeat:} This repeats until the time is up or all groceries are stored. -\end{enumerate} - - - -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% -% Additional Rules -% -% %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{Additional rules and remarks} -\begin{enumerate} - \item \textbf{Bypassing Manipulation:} Bypassing object manipulation via the CONTINUE rule (see~\refsec{rule:asrcontinue}) is not allowed during this test. - \item \textbf{No setup:} There is no setup time. - \item \textbf{Startup:} The robot can be started with a simple voice command or via a start button (see~\refsec{rule:start_signal}). - \item \textbf{Single try:} The robot must be able to start from the first attempt. There is no restart for this test. If the robot is unable to start it must be removed immediately. - \item \textbf{Collisions:} Slightly touching the cupboard is tolerated (but not advised). Crushing objects or any other form of a major collision terminates the test immediately (see~\refsec{rule:safetyfirst}). - \item \textbf{Clear area:} The robot may assume that the direct vicinity of the cupboard and table are clear, and that the robot can move slightly backwards for its task. - \item \textbf{Objects:} The 10 objects are evenly distributed in random fashion including - 3 known objects, - 3 alike objects, - 2 unknown objects, and - 2 special objects (bowl, cloth, dish, etc.). - \item \textbf{Table} The table's rough location will be announced beforehand, having it's position either left, right, or behind the robot. - % \item \textbf{Timing:} The robot has to successfully place the first object within the first two minutes, otherwise the test is ended. If the robot opens the cupboard door by on its own, one additional minute is added to the 2-minutes limit. The maximum time for this test is 5 minutes. -\end{enumerate} - -% \subsection{Data recording} -% Please record the following data (See~\refsec{rule:datarecording}): -% \begin{itemize} -% \item Images -% \item Plans -% \end{itemize} - -\subsection{OC instructions} - -\textbf{2 hours before the test} -\begin{itemize} - \item Announce the startup location for robots. - % This test is about manipulation and object recognition, NOT about finding furniture - % Finding a table is done in the Restaurant task - \item Announce which table will be used in the test. - \item Announce a rough location for the table. -\end{itemize} - -\subsection{Referee instructions} -The referee needs to -\begin{itemize} - \item Place the objects in the cupboard and a few of the same class on the table. New items can be placed when there is room or the robot asks for more objects. - \item Close the door of the cupboard. - \item Put objects on the table and the corresponding objects in the cupboard: 3 known objects, 2 alike and 5 unknown objects. -\end{itemize} - - -\newpage -\subsection{Score sheet} -\input{scoresheets/StoringGroceries.tex} - -% Local Variables: -% TeX-master: "Rulebook" -% End: