AprilTags for FTC Blocks
This tutorial describes an alternate way for FTC Blocks teams to detect the POWERPLAY Signal Sleeve using AprilTag.
Caveat: this is intended as a learn-by-doing exercise, not a place to "grab some code". There is no complete OpMode at the end of this tutorial, but a patient student will soon be detecting AprilTags successfully.
The officially suggested technology is FTC-ML, using Machine Learning to train a custom TensorFlow Lite model. The resulting model of a team's Sleeve can be used in Blocks or Java Sample OpModes for TensorFlow Object Detection (TFOD).
FTC teams are free to use other methods to detect the Sleeve. Blocks or Java teams can use a color sensor, which works at close range. Detecting color or other simple features via webcam (without TFOD) is also possible.
A popular camera-based technology is AprilTag, a scanned image similar to a QR Code. Its effectiveness and quicker set-up have drawn many teams who program in Java, especially teams using Android Studio.
Blocks teams will use three major resources here:
-
AprilTag: an open-source library for scanning formatted images
-
EasyOpenCV: an FTC-optimized version of OpenCV, an image processing library
-
myBlocks: custom Blocks created in OnBot Java (OBJ)
Much credit to EasyOpenCV developer @Windwoes, myBlocks developer @lizlooney, and the smart people at UMich/AprilTag.
This tutorial supports webcam and the internal camera of an Android RC phone.
This capability exists for an FTC Robot Controller (RC) running Android 7 and higher. Moto G 2nd Gen and Moto G 3rd Gen RC phones cannot use external libraries.
Developed at the University of Michigan, AprilTag is like a 2D barcode or a simplified QR Code. It contains a numeric ID code and can be used for location and orientation.
AprilTag is a type of visual fiducial, or fiducial marker, containing information and designed for easy recognition.
These samples represent different formats, or families. Here you will choose a single AprilTag family, then choose 3 tags from that family. Each tag will represent the POWERPLAY parking objective of Zone 1, Zone 2 or Zone 3.
We suggest a common family called 36h11. They're very nice people.
Download this PDF file AprilTag_0-20_family36h11.pdf showing the numbers 0 through 20 from the 36h11 family (capacity of 587). Each number is the ID code of that tag.
Click the Download button (green arrow above). Save it on your computer in a new folder called "AprilTag for Blocks". Other files will be stored here.
Which images to put on the Sleeve? Here we'll make the obvious and boring choice of the numbers 1, 2 and 3. You are free to choose other standard (numeric) ID codes, from any family, or generate your own custom AprilTags. You might prefer having unique tags.
Later you can print these images on a Signal Sleeve in accordance with FTC Game Manual 1, Section 7.5, as amended.
You are welcome to study AprilTag's advantages and how it works. But you can use it without learning those details. This is the benefit of libraries in programming -- other people have figured out those details, allowing you to focus on your own project goals.
Many people drive cars, without knowing how an internal combustion engine works. Just turn the key, and step on the gas pedal.
You need only one file, containing the AprilTag library. This is called an Android Archive or .aar file. Download the following file, saving it in the same folder called "AprilTag for Blocks":
https://repo1.maven.org/maven2/org/openftc/apriltag/1.1.1/apriltag-1.1.1.aar
Where do .aar files come from? Android Archive is a compiled/packaged ready-to-use version of a library, so the user doesn't need to build from source code. Sometimes the developer of a library or repository makes an .aar or .jar (Java Archive) available for public download, to encourage its use.
In this case, the open-source AprilTag library is written in C, not Java. EasyOpenCV developer @Windwoes used JNI code to create this .aar file that's compatible with Java. Hooray!
How new is this resource? The current version 1.1.1 was released on Dec. 22, 2022.
EasyOpenCV is an FTC-optimized version of OpenCV, an open-source computer vision library providing many ways to capture and process images from cameras.
The EasyOpenCV repository provides an Android Archive that supports FTC myBlocks. At the following page, download only the .aar file, saving it in the same folder called "AprilTag for Blocks":
https://github.com/OpenFTC/EasyOpenCV/releases/tag/v1.5.2
Again, you are welcome to learn all about EasyOpenCV's many useful features and how they work. This tutorial focuses only on detecting an AprilTag's ID code.
Turn on your Robot Controller (Control Hub or RC phone), and connect your computer to the RC's WiFi network as usual. In the Chrome browser, open the FTC programming interface and click OnBotJava.
At the top left, click the large black "Upload" arrow (see green circle below). Navigate to your "AprilTag for Blocks" folder, and upload both .aar files:
-
apriltag-1.1.1.aar -
EasyOpenCV_v1.5.2_OBJ_bundle.aar
As shown above in the yellow rectangle, this action creates a folder called ExternalLibraries containing the two uploaded library Archive files.
At this point you must restart the Robot Controller app, to properly load these libraries. For Control Hub, turn its power switch off, then turn it back on. For RC phone, simply Exit the RC app and re-open it.
Now these two libraries can provide their Java classes related to AprilTag and computer vision. Below we will create Java methods that use these classes.
To make these Java methods available to Blocks (as myBlocks), you must read and generally follow some Java code. Not to worry! The following two tutorials help you learn what the Java code is doing.
Read at least the first 4 sections of this tutorial on myBlocks:
http://ftc-docs.firstinspires.org/programming_resources/shared/myblocks/index.html
And read about External Libraries here:
Pay attention to Step 3B, where a library method is provided to Blocks users with a myBlock.
After studying that background, your next (coding) steps are:
-
use library classes to process camera images and detected AprilTags
-
annotate these methods to appear as myBlocks
Annotation is simply one @ExportToBlocks command placed immediately before each method to be available in Blocks. From the tutorials, here's an example annotation:
@ExportToBlocks (
comment = "This myBlock returns the hypotenuse (longest side) of the right triangle " +
"with legs whose lengths are specified by the two given numbers.",
tooltip = "calculate hypotenuse of 2 sides",
parameterLabels = {"side a", "side b"},
parameterDefaultValues = {"3.0", "4.0"}
)Optional annotation labels are heading and color. Fun to try!
Click the following link, to see the Java program that contains myBlocks methods:
This page has no "Download" button. Click the Raw button (green arrow above) to view only the Java code.
Right-click on the text, select "Save as...".
Navigate to your "AprilTag for Blocks" folder. Before clicking "Save", remove ".txt" if it was added to the filename (yellow arrow above).
Now the file AprilTagIdCode.java is downloaded to your computer.
In OnBot Java, click the Upload Files icon. This places the file in the teamcode folder.
Note: Windows might add the ".txt" extension without showing you. This file won't upload to the teamcode folder as required. In Windows Explorer, click the View tab at the top, then check the box to show File name extensions. Now you can see and remove the ".txt" extension using Rename or the F2 key. Then try the upload again.
Still in OnBot Java (OBJ), click the filename to open it. Read through the Java code, looking at the 8 methods and their myBlock annotations. Each one will become a myBlock that appears in the Blocks menu, under Java Classes.
Each annotation includes a short comment describing the method. In general you'll use these methods in the order they are defined:
-
createAprilTagDetector()(for webcam) -
createAprilTagDetector()(for RC phone camera) -
startAprilTagDetector() -
getAllDetections() -
getHowManyDetections() -
getOneDetection() -
getID() -
closeAprilTagDetector()
This Java file is a simplified version of a more capable sample provided by @Windwoes. It can provide location and orientation information -- something to consider!
Finally you must set up the connection to the camera, and the tools to process its images.
EasyOpenCV does not use Vuforia to access the camera, as for TensorFlow Object Detection (TFOD). Instead it uses OpenCV software.
Here you'll create a "pipeline", the steps to collect and process images from the camera. You can learn more about pipelines here.
Click the following link, to see the Java program that operates the pipeline:
Follow the same steps as above, to download the Java code to your computer:
- click Raw (green arrow above)
- right-click on text, select "Save as..."
- navigate to "AprilTag for Blocks" folder
- remove .txt if added, then click Save
In OnBot Java, click the Upload Files icon to place this file in the teamcode folder.
Again, Windows might add the ".txt" extension without showing you. In Windows Explorer, click the View tab at the top, then check the box to show File name extensions. Remove the ".txt" extension using Rename or the F2 key, then upload.
There's no need here to study this Java code; it's highly technical and won't be edited for this exercise. The program collects a camera image using OpenCV, searches it for one or more AprilTags, and draws a colorful box projecting from each AprilTag. This pipeline can also provide location and orientation data, but we focus here on simply getting an AprilTag ID code.
Still at the OnBot Java screen, click the round wrench icon at lower right. This compiles all the code appearing at the left-side menu, preparing it to actually run.
Any build errors must be resolved before proceeding. Carefully read the messages for the filename, line number and type of error.
If you had other OnBot Java programs with serious problems, consider Download and Delete, or comment out all their code with keyboard CTRL-A and CTRL-/.
Continue this, until clicking the Build Everything icon gives "Build SUCCESSFUL!".
At last! Let's make a Blocks OpMode to detect an AprilTag. Click Blocks at the top red interface bar.
Click Create New Op Mode, name it Sample_AprilTag_ID_Code, then click OK. You don't need an existing Sample OpMode -- you will use your own custom myBlocks!
First change the OpMode type from TeleOp to Autonomous; this sample will not use the gamepads.
At the left side menu (palette or toolbox), click the new entry called Java Classes to see all classes that contain myBlocks.
Your class is called AprilTagIdCode, the name of the Java file containing the methods. You reviewed the 8 methods above, and now they appear as purple and green myBlocks.
Numeric and text parameters appear as pre-filled sockets. These default values can be edited directly in Blocks, if needed.
At the Variables menu, click Create variable... to make a new Variable called "myDetector". This Java object is the name of your pipeline to collect and process camera images.
In the open white workspace, assign myDetector with the createAprilTagDetector myBlock from the AprilTag toolbox. Choose the version for WEBCAM or RC PHONE (internal camera), to be used on the RC device now connected to your programming laptop.
Again, choose the myBlock for this robot's camera type. Here is the version for RC Phone internal camera:
Still in the open white workspace, pull out another myBlock called startAprilTagDetector. Plug in the Variable, or Java object, myDetector which was created in the previous step.
This myBlock uses a default camera resolution of 640 x 480 pixels. The values here must be supported by your camera and adequate for detecting your AprilTag on the Signal Sleeve at the intended distance.
If the values aren't supported, your OpMode will give an error message that should list the actual resolutions supported by your webcam or phone camera:
Use the default values for now, but return here to enter a legal, adequate resolution as needed.
Now place these first two myBlock expressions in the INIT section of your OpMode, before waitForStart.
To review: you assigned a newly created pipeline to a Variable, or Java object, named myDetector. Then you started running that pipeline.
In this example, you will use the INIT period only for starting the pipeline. Actual evaluation of the images will begin after touching the START button on the Driver Station.
Later you could begin evaluating the Sleeve images before START, but don't let your code "decide" too quickly on the ID code! FTC field randomization might take a while, or it might be re-done, or your camera might pick up confusing background motion and images before the actual start of Autonomous.
We can already test this INIT portion of the OpMode, to verify the pipeline is working.
Click Save OpMode, then connect/pair your Driver Station device (Driver Hub or DS phone) to the Robot Controller (Control Hub or RC phone).
If using a webcam, it should be UVC-compatible and plugged into the Robot Controller. Verify that your createAprilTagDetector myBlock shows the webcam's name from the active configuration.
Select your OpMode from the Autonomous menu, turn off the 30-second timer, then touch INIT. Don't touch START!
If using an RC phone's internal camera, a video preview will appear on the RC screen.
On the Driver Station, touch the 3-dots menu icon at top right, and select the new item Camera Stream.
This should be familiar from your work with TFOD. The display shows the camera's view; touch the image to update it. (It's not live video, to conserve bandwidth.)
Now point the camera at these AprilTag ID codes of 1, 2 and 3. Refresh the Camera Stream.
You could also open the PDF file containing the numbers 0 through 20, in the family 36h11. Or search Google for AprilTag images.
WOW! You should see colorful boxes projecting from the detected AprilTags. The boxes are unrelated to your goal here, but they do verify that your OpMode's AprilTag detection is working. Hooray!
If desired, you can specify camera orientation in the OnBot Java file AprilTagIdCode.java, in the method startAprilTagDetector(). This would allow, for example, a webcam to be mounted sideways, providing an upright preview in Camera Stream. This is optional, since AprilTags can be detected in any orientation.
Look closely to see the X, Y and Z axis at the center of each AprilTag. You can also see the larger blue and green box protruding from each AprilTag. These indicators relate to pose (position and orientation), beyond the scope of this tutorial. Yes, it's possible to navigate based on the Signal Sleeve!
Select Camera Stream again from the 3-dots menu to close it. Stop the OpMode, there's nothing to run yet.
Now you know the camera and pipeline are working properly. Turn off the Driver Station, to preserve battery life.
Under the Variables menu, choose Create New Variable, call it allDetections.
Assign this new variable with getAllDetections, specifying your pipeline name myDetector.
Now this new Variable, or Java object, contains the data from any and all detected AprilTags.
Place this expression inside the green repeat while loop, just after the blue comment Put loop blocks here.
allDetections is a Java list of all detections (if any) and their data. Each detection in the list has an index number, starting with zero.
Create another new Variable, called numberOfDetections.
Assign this with getHowManyDetections, specifying the collected set of allDetections.
Place this in the OpMode, just after the getAllDetections myBlock.
Now let's find out if an AprilTag was detected in the current loop cycle.
This tutorial uses a logic approach intended to be very easy to follow, not the most robust or efficient. Here the goal is detecting one and only one AprilTag -- probably the one on your Signal Sleeve.
From the Logic menu, pull out the Block with [ IF DO - ELSE IF DO - ELSE ]. For now, keep it floating in the open white workspace -- easier to see what you're building.
This example logic assumes you care about 3 scenarios: no detections, 2 or more detections, or (best of all) exactly one detection. The last scenario is the 'leftover'. Set these up in your Logic structure.
The IF-DO structure (yellow rectangle) and Compare Block (orange rectangle) are found in the Logic toolbox. Raw numbers like 0 and 2 are found in the Math toolbox (green oval).
Add messages for the two scenarios that don't call for action: zero detections, and two or more detections.
For the 'exactly-one-detection' scenario, make a new Variable to hold its data, called singleDetection. and assign it with getOneDetection, index 0 (the first and only one).
Place this in the else branch of the IF-DO structure.
Finally we can extract what we want, the ID code. Make another new Variable called detectedID, and assign it with getID.
Place this below the getOneDetection myBlock.
Display this result on the DS screen:
Add this as the third and final item in the else branch of the IF-DO structure. Review this structure to make sure the logic is clear to you.
Now place the entire blue IF-DO structure into the OpMode, after the green getHowManyDetections myBlock, and before Telemetry.update which sends any addData messages to the Driver Station screen.
Done! Click Save Op Mode, turn on the Driver Station, and run this OpMode.
Touch INIT, and preview the Camera Stream -- always a good idea to check camera angle and distance, and verify the pipeline is working.
Select Camera Stream again to close it, then touch the START arrow.
Congratulations!! Observe your Telemetry, and have fun!
This tutorial has now brought you to the same place as the official Blocks TFOD Sample OpMode for recognizing the standard POWERPLAY Signal. Namely, the main repeat while loop can report a correct recognition -- forever.
You must decide how that loop should stop repeating. Your OpMode now loops until the Stop button is touched (while opModeIsActive).
Some choices for ending the loop:
- (a) immediately upon any detection (even if multiple)
- (b) immediately upon a single detection (and no others)
- (c) save 10 (or 100) single detections, use the most frequent
- (d) 10 (or 100) identical detections in a row
- (e) loop for time, use only the last single detection
- (f) loop for time, use the most frequent detection
- (g) ...any other process or logic for choosing
To help decide, carefully observe the Telemetry report on the Driver Station screen. Or consider saving the detection data to a spreadsheet, for later evaluation (see separate article on FTC Datalogging).
To exit based on time, see the separate article on FTC Timers. Scroll down to its first example, Loop for Time.
This tutorial's logic made it easier to use the very simple choice (b): exit the loop when one (and only one) AprilTag has been detected. At that moment, the Variable detectedID should contain the number 1, 2 or 3 needed for Autonomous parking.
You could set a Variable, or Boolean "flag", when that event has happened -- only in the ELSE branch of your IF-DO structure.
Or, here's an alternate way to exit the loop:
Both of these have a problem: what if no AprilTag is ever detected? Your robot will sit there for the entire Autonomous period.
Consider a time limit, established in the INIT section (e.g. 5 seconds):
This example also covers the consequence of reaching the time limit: detectedID is never assigned a detected value of 1, 2 or 3. If its value is still 999 as intialized, there was no detection and you can decide what the robot should do.
Or, just pre-assign a default code (1, 2 or 3 instead of 999) to detectedID, to be treated afterwards as a real detection. What's the expected value of a 1/3 chance of 20 points?
You now have no further use for the pipeline of camera images being collected and processed. To save CPU cycles, use this myBlock:
Place this Block immediately after exiting from the green repeat while loop.
Print your AprilTag images on a POWERPLAY Signal Sleeve in accordance with FTC Game Manual 1, Section 7.5, as amended.
Experiment with tag sizes and positions, camera placement and angle, lighting, backgrounds, etc. You may notice detection differences between a flat vs. curved image. Determine your best conditions, and document this in your Engineering Notebook or Portfolio.
Adjust your Blocks code as needed, based on real-world conditions of the FTC field and your robot. This might include camera resolution, described above for startAprilTagDetector, or detection logic.
Be careful when increasing camera resolution above the default 640 x 480 pixels. Higher resolution can cause the frame rate to decrease, which may increase the time needed for detection.
You may also specify camera orientation in the OnBot Java file AprilTagIdCode.java, in the method startAprilTagDetector().
For advanced adjustments, see a separate tutorial on webcam settings.
After exiting the detection loop and deactivating the pipeline, you can now add a new 3-way IF-DO structure to drive the robot to the correct POWERPLAY parking zone.
Use Blocks Functions for each sequence of driving actions. This may include first scoring the preloaded Cone, and perhaps more, before parking.
Hint: Functions may contain Functions! Consider using a Function for any common action/sequence that appears in multiple code sections.
Did we really need all those intermediate Variables? For the learning opportunity, yes.
In theory, the following expression works, if an AprilTag ID code is always detected immediately:
In reality, the myBlock getOneDetection crashes if there are no detections; the array list from getAllDetections has zero length.
But with your new OnBot Java knowledge, you could just make... One myBlock to rule them all.
This tutorial described how Blocks teams can detect the Signal Sleeve in POWERPLAY using a fiducial image similar to a QR Code.
It required learning about three major resources:
- AprilTag: an open-source library for scanning formatted images
- EasyOpenCV: an FTC-optimized version of OpenCV, an image processing library
- myBlocks: custom Blocks created in OnBot Java (OBJ)
Webcam and RC phone camera are both supported.
This method can be compared to other ways of earning the Sleeve bonus using Blocks, including FTC Machine Learning to develop a custom TensorFlow Lite model.
This tutorial brought you to the same place as the official Blocks TFOD Sample OpMode for recognizing the standard POWERPLAY Signal. Namely, the main repeat while loop can report a correct recognition -- forever. A few exit options are provided here, but still you must decide how to proceed from detection.
Best of luck in POWERPLAY!
============
Much credit to EasyOpenCV developer @Windwoes, myBlocks developer @lizlooney, and the smart people at UMich/AprilTag.
Questions, comments and corrections to [email protected]
.
-
TensorFlow 2023-2024