Patrick McGuire: Ball Tracking with OpenCV

Monday, September 14, 2015

Ball Tracking with OpenCV

Today marks the 100th blog post on PyImageSearch.

100 posts. It’s hard to believe it, but it’s true.

When I started PyImageSearch back in January of 2014, I had no idea what the blog would turn into. I didn’t know how it would evolve and mature. And I most certainly did not know how popular it would become. After 100 blog posts, I think the answer is obvious now, although I struggled to put it into words (ironic, since I’m a writer) until I saw this tweet from @si2w:

I couldn’t agree more. And I hope the rest of the PyImageSearch readers do as well.

It’s been an incredible ride and I really have you, the PyImageSearch readers to thank. Without you, this blog really wouldn’t have been possible.

That said, to make the 100th blog post special, I thought I would do something a fun — ball tracking with OpenCV:

The goal here is fair self-explanatory:

Step #1: Detect the presence of a colored ball using computer vision techniques.
Step #2: Track the ball as it moves around in the video frames, drawing its previous positions as it moves.

The end product should look similar to the GIF and video above.

After reading this blog post, you’ll have a good idea on how to track balls (and other objects) in video streams using Python and OpenCV.

JUMP TO CODE

Ball tracking with OpenCV

Let’s get this example started. Open up a new file, name it

ball_tracking.py

, and we’ll get coding:

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
        help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=64,
        help="max buffer size")
args = vars(ap.parse_args())

Lines 2-6 handle importing our necessary packages. We’ll be using

deque

, a list-like data structure with super fast appends and pops to maintain a list of the past N (x, y)-locations of the ball in our video stream. Maintaining such a queue allows us to draw the “contrail” of the ball as its being tracked.

We’ll also be using

imutils

, my collection of OpenCV convenience functions to make a few basic tasks (like resizing) much easier. If you don’t already have

imutils

installed on your system, you can grab the source from GitHub or just use

pip

to install it:

$ pip install imutils

From there, Lines 9-14 handle parsing our command line arguments. The first switch,

--video

is the (optional) path to our example video file. If this switch is supplied, then OpenCV will grab a pointer to the video file and read frames from it. Otherwise, if this switch is not supplied, then OpenCV will try to access our webcam.

If this your first time running this script, I suggest using the

--video

switch to start: this will demonstrate the functionality of the Python script to you, then you can modify the script, video file, and webcam access to your liking.

A second optional argument,

--buffer

is the maximum size of our

deque

, which maintains a list of the previous (x, y)-coordinates of the ball we are tracking. This

deque

allows us to draw the “contrail” of the ball, detailing its past locations. A smaller queue will lead to a shorter tail whereas a larger queue will create a longer tail (since more points are being tracked):

Figure 1: An example of a short contrail (buffer=32) on the left, and a longer contrail (buffer=128) on the right. Notice that as the size of the buffer increases, so does the length of the contrail.

Now that our command line arguments are parsed, let’s look at some more code:

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
        help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=64,
        help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "green"
# ball in the HSV color space, then initialize the
# list of tracked points
greenLower = (29, 86, 6)
greenUpper = (64, 255, 255)
pts = deque(maxlen=args["buffer"])

# if a video path was not supplied, grab the reference
# to the webcam
if not args.get("video", False):
        camera = cv2.VideoCapture(0)

# otherwise, grab a reference to the video file
else:
        camera = cv2.VideoCapture(args["video"])

Lines 19 and 20 define the lower and upper boundaries of the color green in the HSV color space (which I determined using the range-detector script in the

imutils

library). These color boundaries will allow us to detect the green ball in our video file. Line 21 then initializes our

deque

pts

using the supplied maximum buffer size (which defaults to

From there, we need to grab access to our

camera

pointer. If a

--video

switch was not supplied, then we grab reference to our webcam (Lines 25 and 26). Otherwise, if a video file path was supplied, then we open it for reading and grab a reference pointer on Lines 29 and 30.

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
        help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=64,
        help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "green"
# ball in the HSV color space, then initialize the
# list of tracked points
greenLower = (29, 86, 6)
greenUpper = (64, 255, 255)
pts = deque(maxlen=args["buffer"])

# if a video path was not supplied, grab the reference
# to the webcam
if not args.get("video", False):
        camera = cv2.VideoCapture(0)

# otherwise, grab a reference to the video file
else:
        camera = cv2.VideoCapture(args["video"])

# keep looping
while True:
        # grab the current frame
        (grabbed, frame) = camera.read()

        # if we are viewing a video and we did not grab a frame,
        # then we have reached the end of the video
        if args.get("video") and not grabbed:
                break

        # resize the frame, blur it, and convert it to the HSV
        # color space
        frame = imutils.resize(frame, width=600)
        blurred = cv2.GaussianBlur(frame, (11, 11), 0)
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

        # construct a mask for the color "green", then perform
        # a series of dilations and erosions to remove any small
        # blobs left in the mask
        mask = cv2.inRange(hsv, greenLower, greenUpper)
        mask = cv2.erode(mask, None, iterations=2)
        mask = cv2.dilate(mask, None, iterations=2)

Line 33 starts a loop that will continue until (1) we press the

key, indicating that we want to terminate the script or (2) our video file reaches its end and runs out of frames.

Line 35 makes a call to the

read

method of our

camera

pointer which returns a 2-tuple. The first entry in the tuple,

grabbed

is a boolean indicating whether the

frame

was successfully read or not. The

frame

is the video frame itself.

In the case we are reading from a video file and the frame is not successfully read, then we know we are at the end of the video and can break from the

while

loop (Lines 39 and 40).

Lines 44-46 preprocess our

frame

a bit. First, we resize the frame to have a width of 600px. Downsizing the

frame

allows us to process the frame faster, leading to an increase in FPS (since we have less image data to process). We’ll then blur the

frame

to reduce high frequency noise and allow us to focus on the structural objects inside the

frame

, such as the ball. Finally, we’ll convert the

frame

to the HSV color space.

Lines 51 handles the actual localization of the green ball in the frame by making a call to

cv2.inRange

. We first supply the lower HSV color boundaries for the color green, followed by the upper HSV boundaries. The output of

cv2.inRange

is a binary

mask

, like this one:

Figure 2: Generating a mask for the green ball using the cv2.inRange function.

As we can see, we have successfully detected the green ball in the image. A series of erosions and dilations (Lines 52 and 53) remove any small blobs that my be left on the mask.

Alright, time to perform compute the contour (i.e. outline) of the green ball and draw it on our

frame

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
        help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=64,
        help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "green"
# ball in the HSV color space, then initialize the
# list of tracked points
greenLower = (29, 86, 6)
greenUpper = (64, 255, 255)
pts = deque(maxlen=args["buffer"])

# if a video path was not supplied, grab the reference
# to the webcam
if not args.get("video", False):
        camera = cv2.VideoCapture(0)

# otherwise, grab a reference to the video file
else:
        camera = cv2.VideoCapture(args["video"])

# keep looping
while True:
        # grab the current frame
        (grabbed, frame) = camera.read()

        # if we are viewing a video and we did not grab a frame,
        # then we have reached the end of the video
        if args.get("video") and not grabbed:
                break

        # resize the frame, blur it, and convert it to the HSV
        # color space
        frame = imutils.resize(frame, width=600)
        blurred = cv2.GaussianBlur(frame, (11, 11), 0)
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

        # construct a mask for the color "green", then perform
        # a series of dilations and erosions to remove any small
        # blobs left in the mask
        mask = cv2.inRange(hsv, greenLower, greenUpper)
        mask = cv2.erode(mask, None, iterations=2)
        mask = cv2.dilate(mask, None, iterations=2)

        # find contours in the mask and initialize the current
        # (x, y) center of the ball
        cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE)[-2]
        center = None

        # only proceed if at least one contour was found
        if len(cnts) > 0:
                # find the largest contour in the mask, then use
                # it to compute the minimum enclosing circle and
                # centroid
                c = max(cnts, key=cv2.contourArea)
                ((x, y), radius) = cv2.minEnclosingCircle(c)
                M = cv2.moments(c)
                center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))

                # only proceed if the radius meets a minimum size
                if radius > 10:
                        # draw the circle and centroid on the frame,
                        # then update the list of tracked points
                        cv2.circle(frame, (int(x), int(y)), int(radius),
                                (0, 255, 255), 2)
                        cv2.circle(frame, center, 5, (0, 0, 255), -1)

        # update the points queue
        pts.appendleft(center)

We start by computing the contours of the object(s) in the image on Line 57. We specify an array slice of -2 to make the

cv2.findContours

function compatible with both OpenCV 2.4 and OpenCV 3. You can read more about why this change to

cv2.findContours

is necessary in this blog post. We’ll also initialize the

center

(x, y)-coordinates of the ball to

None

on Line 59.

Line 62 makes a check to ensure at least one contour was found in the

mask

. Provided that at least one contour was found, we find the largest contour in the

cnts

list on Line 66, compute the minimum enclosing circle of the blob, and then compute the center (x, y)-coordinates (i.e. the “centroids) on Lines 68 and 69.

Line 72 makes a quick check to ensure that the

radius

of the minimum enclosing circle is sufficiently large. Provided that the

radius

passes the test, we then draw two circles: one surrounding the ball itself and another to indicate the centroid of the ball.

Finally, Line 80 appends the centroid to the

pts

list.

The last step is to draw the contrail of the ball, or simply the past N (x, y)-coordinates the ball has been detected at. This is also a straightforward process:

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
        help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=64,
        help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "green"
# ball in the HSV color space, then initialize the
# list of tracked points
greenLower = (29, 86, 6)
greenUpper = (64, 255, 255)
pts = deque(maxlen=args["buffer"])

# if a video path was not supplied, grab the reference
# to the webcam
if not args.get("video", False):
        camera = cv2.VideoCapture(0)

# otherwise, grab a reference to the video file
else:
        camera = cv2.VideoCapture(args["video"])

# keep looping
while True:
        # grab the current frame
        (grabbed, frame) = camera.read()

        # if we are viewing a video and we did not grab a frame,
        # then we have reached the end of the video
        if args.get("video") and not grabbed:
                break

        # resize the frame, blur it, and convert it to the HSV
        # color space
        frame = imutils.resize(frame, width=600)
        blurred = cv2.GaussianBlur(frame, (11, 11), 0)
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

        # construct a mask for the color "green", then perform
        # a series of dilations and erosions to remove any small
        # blobs left in the mask
        mask = cv2.inRange(hsv, greenLower, greenUpper)
        mask = cv2.erode(mask, None, iterations=2)
        mask = cv2.dilate(mask, None, iterations=2)

        # find contours in the mask and initialize the current
        # (x, y) center of the ball
        cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE)[-2]
        center = None

        # only proceed if at least one contour was found
        if len(cnts) > 0:
                # find the largest contour in the mask, then use
                # it to compute the minimum enclosing circle and
                # centroid
                c = max(cnts, key=cv2.contourArea)
                ((x, y), radius) = cv2.minEnclosingCircle(c)
                M = cv2.moments(c)
                center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))

                # only proceed if the radius meets a minimum size
                if radius > 10:
                        # draw the circle and centroid on the frame,
                        # then update the list of tracked points
                        cv2.circle(frame, (int(x), int(y)), int(radius),
                                (0, 255, 255), 2)
                        cv2.circle(frame, center, 5, (0, 0, 255), -1)

        # update the points queue
        pts.appendleft(center)

        # loop over the set of tracked points
        for i in xrange(1, len(pts)):
                # if either of the tracked points are None, ignore
                # them
                if pts[i - 1] is None or pts[i] is None:
                        continue

                # otherwise, compute the thickness of the line and
                # draw the connecting lines
                thickness = int(np.sqrt(args["buffer"] / float(i + 1)) * 2.5)
                cv2.line(frame, pts[i - 1], pts[i], (0, 0, 255), thickness)

        # show the frame to our screen
        cv2.imshow("Frame", frame)
        key = cv2.waitKey(1) & 0xFF

        # if the 'q' key is pressed, stop the loop
        if key == ord("q"):
                break

# cleanup the camera and close any open windows
camera.release()
cv2.destroyAllWindows()

We start looping over each of the

pts

on Line 84. If either the current point or the previous point is

None

(indicating that the ball was not successfully detected in that given frame), then we ignore the current index continue looping over the

pts

(Lines 86 and 87).

Provided that both points are valid, we compute the

thickness

of the contrail and then draw it on the

frame

(Lines 91 and 92).

The remainder of our

ball_tracking.py

script simply performs some basic housekeeping by displaying the

frame

to our screen, detecting any key presses, and then releasing the

camera

pointer.

Ball tracking in action

Now that our script has been coded it up, let’s give it a try. Open up a terminal and execute the following command:

$ python ball_tracking.py --video ball_tracking_example.mp4

This command will kick off our script using the supplied

ball_tracking_example.mp4

demo video. Below you can find a few animated GIFs of the successful ball detection and tracking using OpenCV:

Figure 3: An example of successfully performing ball tracking with OpenCV.

For the full demo, please see the video below:

Finally, if you want to execute the script using your webcam rather than the supplied video file, simply omit the

--video

switch:

$ python ball_tracking.py

However, to see any results, you will need a green object with the same HSV color range was the one I used in this demo.

Summary

In this blog post we learned how to perform ball tracking with OpenCV. The Python script we developed was able to (1) detect the presence of the colored ball, followed by (2) track and draw the position of the ball as it moved around the screen.

As the results showed, our system was quite robust and able to track the ball even if it was partially occluded from view by my hand.

Our script was also able to operate at an extremely high frame rate (> 32 FPS), indicating that color based tracking methods are very much suitable for real-time detection and tracking.

If you enjoyed this blog post, please consider subscribing to the PyImageSearch Newsletter by entering your email address in the form below — this blog (and the 99 posts preceding it) wouldn’t be possible without readers like yourself.