Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Visiond should detect/test for valid modes/resolutions #640

Closed
cglusky opened this issue Oct 30, 2017 · 12 comments
Closed

Visiond should detect/test for valid modes/resolutions #640

cglusky opened this issue Oct 30, 2017 · 12 comments
Assignees
@fnoop
Labels
enhancement Vision
Milestone
1.2

Comments

@cglusky
Copy link
Member

cglusky commented Oct 30, 2017

More details on bootstraping MinnowBoard here:
#495

Two issues of note for @fnoop:

  1. gst vaapi did not appear to install during bootstrap
  2. Need option for verbose output to log from gstreamer

Running a MinnowBoard Turbot Dual with Ubuntu and Maverick 1.1.3 (env=flight)

Elp 2mp USB 2.0 camera plugged into USB 3 port. After several hours of messing the key was to get just what the camera would support for pixel format, size and FPS.

To help find what camera supports....
v4l2-ctl --list-formats-ext

Using the output from above was key to my success with this camera and making visiond/gstreamer happy. I chose 800x600@20fps as a starting point....

[visiond]

# On/off switch for maverick-visiond, set to False to disable
enable = True

# Camera device, almost always /dev/video0
camera_device = /dev/video0

# Camera format, if not specified here will autodetect yuv->mjpeg->h264
format = yuv

# Pixel Format, could be (fourcc)YUV2, I420 etc
pixelformat = YUY2

# Encoder, if not specified will default to h264.  Values are h264, mjpeg or none
encoder = h264

# Resolution width of video stream, must be valid for camera
width = 800

# Resolution height of video stream, must be valid for camera
height = 600

# Framerate of video stream, must be valid for camera
framerate = 20

# Brightness - 0 is automatic
brightness = 0

# Stream input type: v4l2 (fpv), appsrc (cv)
input = v4l2

# Stream output type: file (save video), udp (stream video), wcast (wifibroadcast)
output = udp

# Output destination: filename (file output), IP address (udp output), Interface (wcast output)
output_dest = 192.168.2.184
# output_dest = wcast0

# Output port: Port number (eg. 5000) for network destination, Channel for wifibroadcast output (eg. 1)
output_port = 5000
# output_port = 1

Output to visiond log

2017-10-30 01:01:29,885 - INFO - Starting maverick-visiond
2017-10-30 01:01:29,888 - INFO - Config read from /srv/maverick/data/config/vision/maverick-visiond.conf
2017-10-30 01:01:29,890 - DEBUG - V4l2 device input: Camera 1:2
2017-10-30 01:01:29,891 - DEBUG - driver: uvcvideo
2017-10-30 01:01:29,891 - DEBUG - card: USB 2.0 Camera
2017-10-30 01:01:29,891 - DEBUG - Camera control: Brightness
2017-10-30 01:01:29,892 - DEBUG - Camera control: Contrast
2017-10-30 01:01:29,892 - DEBUG - Camera control: Saturation
2017-10-30 01:01:29,892 - DEBUG - Camera control: Hue
2017-10-30 01:01:29,893 - DEBUG - Camera control: White Balance Temperature, Auto
2017-10-30 01:01:29,894 - DEBUG - Camera control: Gamma
2017-10-30 01:01:29,894 - DEBUG - Camera control: Gain
2017-10-30 01:01:29,894 - DEBUG - Camera control: Power Line Frequency
2017-10-30 01:01:29,895 - DEBUG - Camera control: White Balance Temperature
2017-10-30 01:01:29,895 - DEBUG - Camera control: Sharpness
2017-10-30 01:01:29,895 - DEBUG - Camera control: Backlight Compensation
2017-10-30 01:01:29,896 - DEBUG - Camera format: Motion-JPEG
2017-10-30 01:01:29,896 - DEBUG - Camera format: YUYV 4:2:2
2017-10-30 01:01:29,897 - INFO - Creating stream object - camera:/dev/video0, stream:yuv, pixelformat:YUY2, encoder:h264, size:(800 x 600 / 20), output:udp, brightness:0
2017-10-30 01:01:29,897 - INFO - Attaching input 'v4l2': /dev/video0
2017-10-30 01:01:29,914 - INFO - Attaching stream 'yuv'
2017-10-30 01:01:29,922 - INFO - Attaching encoding 'h264'
2017-10-30 01:01:30,110 - INFO - No hardware encoder detected, using software x264 encoder
2017-10-30 01:01:30,124 - INFO - Attaching payload 'h264'
2017-10-30 01:01:30,127 - DEBUG - h264parse element created
2017-10-30 01:01:30,132 - DEBUG - Attaching h264pay to h264parse
2017-10-30 01:01:30,133 - INFO - Attaching output 'udp', sending to 192.168.2.184:5000
2017-10-30 01:01:30,142 - INFO - Pipeline: "v4l2-source /dev/video0 ! capsfilter 'video/x-raw, format=(string)YUY2, width=(int)800, height=(int)600, framerate=(fraction)20/1' ! queue ! autovideoconvert ! x264-encode ! h264parse ! pay0 ! udpsink"
2017-10-30 01:01:30,149 - INFO - Starting camera stream

Receiving with the following on Windows10 using gst

gst-launch-1.0 -v udpsrc port=9000 ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=f

@fnoop
Copy link
Member

fnoop commented Oct 30, 2017

@cglusky Thanks for the report! Could you possibly paste the visiond output with a default config file? I'd like to see what it autodetects, and why it doesn't work.

@cglusky
Copy link
Member Author

cglusky commented Oct 30, 2017

Defaults seems to get close but the camera I have does not support the resultion and frame rate you have set so I get a "not negotiated". I have to run it through gst launch with the -v flag to see what is happening since visiond log stops recording once you get to "Starting camera stream"

BTW, here's the gstreamer pipeline I cobbled together to approximate what I think visiond is putting out. This results in the not negotiated error:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)30/1' ! queue ! autovideoconvert ! x264enc ! h264parse ! rtph264pay ! udpsink host=192.168.2.184 port=5000

If I change the above to 800x600@20fps it works fine

Here's what is recorded in visiond.log when i run with the defaults....

2017-10-30 21:52:36,945 - INFO - Starting maverick-visiond
2017-10-30 21:52:36,948 - INFO - Config read from /srv/maverick/data/config/vision/maverick-visiond.conf
2017-10-30 21:52:36,950 - DEBUG - V4l2 device input: Camera 1:2
2017-10-30 21:52:36,950 - INFO - v4l2 device /dev/video0 is a camera, autoselecting
2017-10-30 21:52:36,951 - DEBUG - V4l2 device input: Camera 1:2
2017-10-30 21:52:36,951 - DEBUG - driver: uvcvideo
2017-10-30 21:52:36,952 - DEBUG - card: USB 2.0 Camera
2017-10-30 21:52:36,952 - DEBUG - Camera control: Brightness
2017-10-30 21:52:36,952 - DEBUG - Camera control: Contrast
2017-10-30 21:52:36,953 - DEBUG - Camera control: Saturation
2017-10-30 21:52:36,953 - DEBUG - Camera control: Hue
2017-10-30 21:52:36,953 - DEBUG - Camera control: White Balance Temperature, Auto
2017-10-30 21:52:36,954 - DEBUG - Camera control: Gamma
2017-10-30 21:52:36,954 - DEBUG - Camera control: Gain
2017-10-30 21:52:36,955 - DEBUG - Camera control: Power Line Frequency
2017-10-30 21:52:36,955 - DEBUG - Camera control: White Balance Temperature
2017-10-30 21:52:36,955 - DEBUG - Camera control: Sharpness
2017-10-30 21:52:36,956 - DEBUG - Camera control: Backlight Compensation
2017-10-30 21:52:36,956 - DEBUG - Camera format: Motion-JPEG
2017-10-30 21:52:36,957 - DEBUG - Camera format: YUYV 4:2:2
2017-10-30 21:52:36,958 - INFO - Camera YUV stream available, using yuv stream
2017-10-30 21:52:36,958 - INFO - Creating stream object - camera:/dev/video0, stream:yuv, pixelformat:YUY2, encoder:h264, size:(1280 x 720 / 30), output:udp, brightness:0
2017-10-30 21:52:36,958 - INFO - Attaching input 'v4l2': /dev/video0
2017-10-30 21:52:36,976 - INFO - Attaching stream 'yuv'
2017-10-30 21:52:36,984 - INFO - Attaching encoding 'h264'
2017-10-30 21:52:37,162 - INFO - No hardware encoder detected, using software x264 encoder
2017-10-30 21:52:37,175 - INFO - Attaching payload 'h264'
2017-10-30 21:52:37,179 - DEBUG - h264parse element created
2017-10-30 21:52:37,185 - DEBUG - Attaching h264pay to h264parse
2017-10-30 21:52:37,185 - INFO - Attaching output 'udp', sending to 192.168.2.184:5000
2017-10-30 21:52:37,194 - INFO - Pipeline: "v4l2-source /dev/video0 ! capsfilter 'video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)30/1' ! queue ! autovideoconvert ! x264-encode ! h264parse ! pay0 ! udpsink"
2017-10-30 21:52:37,203 - INFO - Starting camera stream

@cglusky
Copy link
Member Author

cglusky commented Oct 30, 2017

Here's what my camera supports. So I can change just the frame rate to 9 and get it to work. Actually not bad for a USB web cam over IP!

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'MJPG' (compressed)
        Name        : Motion-JPEG
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.008s (120.101 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.008s (120.101 fps)

        Index       : 1
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 1920x1080
                        Interval: Discrete 0.167s (6.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.111s (9.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.167s (6.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.050s (20.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.167s (6.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.033s (30.000 fps)

@fnoop
Copy link
Member

fnoop commented Oct 30, 2017 •
edited
Loading

Thanks, useful info. I'll try and enhance the autodetection to take the actual modes available into account and choose a sensible default.
According to your formats, you should be able to use 'mjpeg' as the format and get much higher framerates. This will depend on vaapi being available as it will need hardware jpg decompresser available to get the better framerates. You only really need raw video for CV work, although visiond does choose it by default.

@fnoop
Copy link
Member

fnoop commented Oct 30, 2017

Also if you use mjpeg as the input format and the output format visiond knows to just repackage the frames - this is the minimum work involved so much less load on the system and power demands, and also minimal latency as mjpeg is frame by frame not multi-frame analysis like h264 etc. It is (much) higher bandwidth, but has it's advantages.

@cglusky
Copy link
Member Author

cglusky commented Oct 30, 2017

So switched to mjpeg on the format and left the h264 encode. Video shows up now in QGC. Not sure why QGC would not show the YUV video since they are both sent via h264enc. I get the first frame in Mission Planner HUD and it freezes. Seems to be a known problem with MP. Camera will not stream at all unless I perfectly match resolution and frame rate for given format. Since this camera does not appear to be uvc compliant don't think I can tweak output. All good to know.

@cglusky
Copy link
Member Author

cglusky commented Oct 31, 2017 •
edited
Loading

Just a few more notes that may save someone else some time later. Ditched the Elp USB cam and plugged in RealSense R200. Shows up as 3 video devices. The first two appear to perform magic CV/3D stuff and the last looks like our RGB target web cam.

ls-l /dev/video* shows the 3 devices enumerated...

Adding the -d switch to v4l2-ctl utility lets us see what each device supports, but from doing a bit of digging there may be some questions as to whether they are all really supported.

IntelRealSense/librealsense#169

I did test 848x480@15 and it worked fine.

v4l2-ctl --list-formats-ext -d 2
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 960x540
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 848x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 640x360
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 424x240
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 320x180
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)

        Index       : 1
        Type        : Video Capture
        Pixel Format: 'pRAA'
        Name        : 10-bit Bayer RGRG/GBGB Packed
                Size: Discrete 2400x1081
                        Interval: Discrete 0.033s (30.000 fps)

@fnoop fnoop added this to the 1.1.5 milestone Nov 12, 2017
@fnoop fnoop self-assigned this Nov 12, 2017
@fnoop
Copy link
Member

fnoop commented Nov 29, 2017

A new 'debug' option has been added to maverick-visiond.conf that sets the gstdebug level.

@fnoop fnoop mentioned this issue Nov 29, 2017
Closed
@fnoop
Copy link
Member

fnoop commented Nov 29, 2017

  1. gst vaapi did not appear to install during bootstrap
    Separate issue Gstreamer vaapi not installed on intel platforms #696 raised

@fnoop fnoop changed the title Visiond and Gstreamer on MinnowBoard Visiond should detect/test for valid modes/resolutions Nov 29, 2017
@fnoop fnoop modified the milestones: 1.1.5, 1.1.6 Nov 29, 2017
@cglusky
Copy link
Member Author

cglusky commented Nov 30, 2017

OK. I will test this with #696 as well. the Elp cam I have should be a good test for this fix. Thanks!

@fnoop fnoop modified the milestones: 1.1.6, Sometime in the distant future Dec 1, 2017
@cglusky
Copy link
Member Author

cglusky commented Dec 2, 2017

Tested with Elp cam. Debug info helps. Agree with the Sometime in future tag. Not critical. Feature not a bug at this point.

@fnoop fnoop added the Vision label May 8, 2019
@fnoop fnoop modified the milestones: 2.0, 1.2 Feb 27, 2020
@fnoop fnoop mentioned this issue Mar 13, 2020
Open
@fnoop
Copy link
Member

fnoop commented Mar 13, 2020

I've hit similar problems with other usb cameras. Moved to goodrobots/visiond#19

@fnoop fnoop closed this as completed Mar 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fnoop fnoop

Labels
enhancement Vision
Projects
None yet
Milestone
1.2
Development

No branches or pull requests
2 participants
@cglusky @fnoop