Auball
Back to AU Robot Servers
Contents |
Ball finder plugin
A plugin intended for finding coloured balls in a camera image. This example is an image (ball08.png) from a Kinect camera with 2 balls at daylight.
Figure 78 (and 50). There are two balls, a red and a blue. These are marked in the second image (50) by the plugin, the white circle is a failed circle candidate. The red line is the top limit, where to look for balls.
The code in the function is explained below.
Sourse image
The code in the file ufuncball.cpp to get the sourse image is
The UImage type is a class that holds a version of the image that can be transferred to the client and a cv::Mat version for openCV manipulation.
Line 87 get the image from the image pool with the number specified in the (global) variable "poolImg=18" with the call 'varSourceImg->getInt()'.
convert to HSV
The same image in HSV format
Figure 45. A high hue value shown as blue, a high saturation in green (e.g. the dark red in the front and the blue lit in the back), the intensity (value) is shown in red (ex. the cup in the front). White objects have a hue of 0, the hue of dark objects is unreliable.
Code
101 // split into planes 102 splitIntoHSVChannels(img, imgHSV, imgHue, imgSat, imgInt, imgRGB);
This is a call to a function that converts to HSV and splits the HSV image into 3 grayscale images
This function takes 6 images as parameters. The source updates the cv::Mat version of the image in line 536, and then makes a color conversion from BGR to HSV using the 'cv::COLOR_BGR2HSV' parameter in line 537. The image is then loaded back to the image pool and given a new name in lines 539 and 540.
The splitting of the color planes are in line 543 into an array with 3 cv::Mat objects. The remaining lines are to get images back to the image pool.
Crop
The image is cropped to the interesting Range Of Interest, here the top 200 rows are removed (parameter 'topLine').
NB! the parameters used in this example is listed at the bottom of this page.
Split into planes
The individual planes shown as grayscale images.
Figure 48. Intensity, this is the grayscale version of the original image and has 3 lines marked, these are used in the analysis to the left. The balls has a rather low intensity, but never gets below a value of 50.
Figure 46. Hue, 0 is yellow (dark like the table, green is about 50 (rather dark), the blue ball is about 120, the red ball is about 165. Maximum is 179, then it is folds back to 0. The plot shows 3 lines and it is clear that the red ball does not differ too much from the surrounding floor colour. It is further visible that there some colour variation across the ball.
Figure 47. Saturation, the brighter the more saturated the colour. It is visible that the balls have a better saturation than most of the rest of the image.
Enhanced image
The enhanced image is an attempt to promote the characteristic hue, saturation and intensity of the balls, in such a way that a following edge filter has an easy task in finding the ball edges and suppress all other edges.
The hue is used as the main enhancement, here emphasizing the two selected hues 'redHue=165' and 'blueHue=112'. Low saturation and low intensity are further removed.
Figure 53. A grayscale image, where all pixels with hue values of the two colours ('redHue' and 'blueHue') in the range specified by 'colLim' are set to white (255), if the hue is further away, then the highest distance from the desired hue is used. If the saturation is below 'limitSat=70' then the value is set to 0 (dark), the same if the intensity is below 'limitVal=70'.
Code
The filter is implemented in a sub-function:
121 // enhance contrast based on known hue for red and blue ball 122 use = enhanceDesiredHue(use, imgSat->mat, varLimitSat->getInt(), imgInt->mat, varLimitValue->getInt());
The enhanced is constructed from HSV pixel values so that each enhanced pixel e = f(h,s,v)
Some of these values Ls = redHue, Lb = blueHue and 13 (redLim and blueLim) are setable parameters (see list at bottom of this page). The hue value is from 0 to 179. The red range if folded if reached outside these limits.
These parameters are fetched at the start of the enhancement function
The resulting image is set to be available in the image pool as image 45+8 = 53.
Each pixel in the resulting image is based on the same pixel position in the HSV planes as described above, the actual code is:
Opening
The filtered image is likely to have smaller areas enhanced, these are removed/reduced by an opening filter.
Figure 54. Result of an opening operation with a 3x3 (all ones) erosion (twice, if 'opening=2') followed by dilation the same number of times.
Code
The opening code is straight forward - a number of erodes followed by the same number of dilate.
124 cv::Mat buffer; 125 cv::Mat mask = (cv::Mat_<char>(3,3) << 126 1, 1, 1, 127 1, 1, 1, 128 1, 1, 1); 129 cv::erode(use, buffer, mask, cv::Point(-1,-1), varOpening->getInt(0)); 130 cv::dilate(buffer, use, mask, cv::Point(-1,-1), varOpening->getInt(0));
Smoothing
The resulting image is then smoothed to get softer edges better suitable for a canny edge detector.
Figure 55. The smoothing is a Gauss blur with a mask size of 5 ('filter="1 5"').
Code
Blur filtering is optional but recommended.
137 if (varFilter->getBool()) 138 { 139 int n = varFilter->getInt(1); 140 if (true) 141 { 142 printf("Using median filter (%dx%d)\n", n,n); 143 cv::medianBlur(use, buffer, n); 144 } 145 else 146 { 147 printf("Using gauss filter (%dx%d)\n", n,n); 148 cv::GaussianBlur(use, buffer, cv::Size(n, n), 0 , 0); 149 } 150 use = buffer.clone(); 151 }
A median blur is used here, but a Gaussian blur was probably better.
Hough transform
The Hough transform is performed on the filtered image with a number of parameters ('hough="700 70 5").
The used canny filter has a high limit of 700 (and a low limit of 350 (half)). The second parameter 70 represents the voting of there is a circle with a centre at this position. The last parameter 5 is the resolution of the centre position, in this case all centre votes within 5 pixels are counted.
The Hough transform found these circles.
Hough circles found 3 circles AUBall:: 0 (361, 93, 27) Hue=122, sat=114 - blue(122)= 0 off, red(166)=-44 off (OK: blue) AUBall:: 1 (469, 91, 30) Hue=163, sat=187 - blue(122)= 41 off, red(166)= -3 off (OK: red) AUBall:: 2 ( 61, 45, 24) Hue=141, sat=127 - blue(122)= 19 off, red(166)=-25 off (NOT: low intensity)
The numbers in brackets are pixel position and circle radius, then the HSV values and how far the hue is from the two colors. In the last bracket is the colour detection (1 for red and 2 for blue).
Figure 50. The found circles (balls?) are shown in the original image with blue and red circles. The removed top part is shown as a red line. The white ring is a rejected circle.
Code
The result of the Hough transform is delivered in a vector with 3 floats (line 165).
164 // prepare result circles 165 std::vector<cv::Vec3f> circles; 166 std::vector<int> usable; // 0 = not, 1 = red, 2= blue 167 // find circles 168 cv::HoughCircles(use, circles, cv::HOUGH_GRADIENT, varHough->getInt(2), 169 use.rows/16, // change this value to detect circles with different distances to each other 170 varHough->getInt(0), // Canny top parameter [0..1000] 171 varHough->getInt(1), // circle quality limit 172 varSize->getInt(0), // minimum radius in pixels 173 varSize->getInt(1) // maximum radius in pixels 174 ); 175 isOK = circles.size() > 0; 176 printf("Hough circles found %d circles\n", (int)circles.size());
Canny
The Hough includes a canny edge detector, to make the image binary.
Figure 51. A replica of the Canny edge filtered image that the Hough transform uses as the basis for estimation.
Ball color
The code
179 findUsableCircles(circles, usable, imgHue, varTopLine->getInt(), imgRGB);
checkes the colour in the centre of the found circles, and annotates with 1 or 2 if the colour is within the red or blue range, and further paints circles in the RGB image. Other circles are deleted.
Reporting
The rest of the code (line 205 to line 245) is for reporting the result to the client and to the MRC.
Parameters
The code uses a number of parameters, these are
>> var ball <help subject="var list" name="ball"> Description: camera based ball detect (compiled Feb 16 2020 09:23:37) poolImg=77 (r/w) image pool number to use as source poolDebugImg=45 (r/w) first image pool number to use for interim images redHue=166 (r/w) hue value (in HSV formet) for red ball range [0-180] (~120=red) redCnt=1 (r) Number of red balls found in last image blueHue=122 (r/w) hue value (in HSV formet) for blue ball range [0-180] (~0=blue) blueCnt=1 (r) Number of blue balls found in last image BallSize=0.12 (r/w) Size of the ball (diameter [m]) topLine=200 (r/w) is the topmost line that could be a ball on the floor. size=15 60 (r/w) size limits of ball in pixels [min max] hough=400 40 2 (r/w) params for Hough (canny high [0-1000], hough vote [0..255], Hough resolution 1(fine)..8(rough)) colLim=13 13 (rw) Color limit (+/-) for circle center hue match [red blue] [0..180] mrc=1 (rw) Should result be send to MRC (smrcl) debug=1 (rw) make more debug images and printout filter=1 5 (rw) smooth image before detect [filter 0-1, size NxN] opening=2 (rw) Opening before Hough circles limitSat=80 (rw) do not use pixels with saturation lower than this (0..255) limitVal=50 (rw) do not use pixels with a lower V-value (intensity) lower than this (0..255) (H: has time series, L: is logged, R: replay) </help> <var info="done"/>
They can be modified with eg:
>> var ball.topLine=175 <var name="ball.topLine" typ="d" size="1" value="175"/> >> var ball.hough="300 40 2" <var name="ball.hough" typ="d" size="3" value="300 40 2" oldValue="400 40 2"/>
To change the top line cut to 150 pixels rather than 200, and change one of the Hough parameters.