You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
Embedded High Precision Control and Corn Stand
Counting Algorithms for an Ultra-Compact 3D
Printed Field Robot
Erkan Kayacan* , Zhongzhong Zhang* , and Girish Chowdhary
Coordinated Science Laboratory
Distributed Autonomous Systems Laboratory
Department of Agricultural and Biological Engineering
University of Illinois at Urbana-Champaign
Urbana, IL 61801 USA
Emails: erkank@mit.edu, zzhang52@illinois.edu, girishc@illinois.edu
* Both authors contributed equally to this work.
Abstract—This paper presents embedded high precision control and corn stands counting algorithms for a low-cost, ultracompact 3D printed and autonomous field robot for agricultural
operations. Currently, plant traits, such as emergence rate,
biomass, vigor and stand counting are measured manually.
This is highly labor intensive and prone to errors. The robot,
termed TerraSentia, is designed to automate the measurement
of plant traits for efficient phenotyping as an alternative to
manual measurements. In this paper, we formulate a Nonlinear
Moving Horizon Estimator (NMHE) that identifies key terrain
parameters using onboard robot sensors and a learning-based
Nonlinear Model Predictive Control (NMPC) that ensures high
precision path tracking in the presence of unknown wheel-terrain
interaction. Moreover, we develop a machine vision algorithm to
enable TerraSentia to count corn stands by driving through the
fields autonomously. We present results of an extensive field-test
study that shows that (i) the robot can track paths precisely with
less than 5 cm error so that the robot is less likely to damage
plants, and (ii) the machine vision algorithm is robust against
interferences from leaves and weeds, and the system has been
verified in corn fields at the growth stage of V4, V6, VT, R2, and
R6 from five different locations. The robot predictions agree well
with the ground truth with countrobot = 0.96 × counthuman + 0.85
and correlation coefficient R = 0.96.

I. I NTRODUCTION
Phenotypic traits are measured manually by field technicians
in the field to determine physical differences between plant
genotype and the influence of environmental conditions. Frequent and accurate measurement of these phenotypic traits can
be utilized to breed improved crops that have more nutritional
value, yield, and resilience to weather anomalies. However,
manual phenotyping is expensive due to its labor-intensive
nature, and prone to human measurement errors. This has
led to the so-called phenotyping bottleneck preventing rapid
advances in plant breeding [1], [2]. On the other hand, these
crop characteristics can also be measured by exteroceptive
sensors such as hyperspectral cameras, stereo cameras, thermal
cameras, lidar, etc. These sensors can be integrated on a mobile
robotic scanning and data processing system. Such robotic
phenotyping systems have a tremendous potential to relieve

the phenotyping bottleneck by significantly reducing labor cost
and eliminating errors due to human-subjective information.
Unmanned robots have potential to capture extremely detailed data of plants without the need of an expert operator.
A four-wheeled autonomous rover with variable chassis clearance and width is equipped with 3D time-of-flight cameras,
spectral imaging, light curtain, and a laser distance sensor
in [3], [4]. The rover has many promising features such as
increased battery power, battery capacity, and continuous drive
torque; however, its wide tracks and limited clearance are
serious limitations for plant phenotyping. A new architecture
consisting of two mobile robotic platforms, i.e., a phenotyping
rover and a mobile observation tower, has been introduced for
high-throughput field phenotyping [5]. The phenotyping rover
has been built on a Husky A-200 from Clearpath Robotics
and collects data from individual plants, while the observation
tower identifies specific plants for further inspections. Thus,
it eliminates the need of air vehicles. However, a Husky A200 is too wide (66 centimeters) and heavy (50 kilograms). A
ground-based agricultural robot for high-throughput crop phenotyping was developed in [6]. It is capable of autonomously
navigating below the canopy of row crops and deploying a
manipulator to measure plant stalk strength and gathering
phenotypic data. However, like a Husky A-200, it is too
wide (56 centimeters) and heavy (140 kilograms). TerraSentia
designed in this paper is much narrower (30.5 centimeters)
and light (6.6 kilograms) when compared to aforementioned
agricultural robots as shown in Fig. 1. Thus, it easily fits
between crop-rows and does not compact the soil, which is
highly desirable for improving yield.
This paper describes an ultra-compact (30 centimeters
wide), ultra-light (6.6 kilograms), very low-cost and autonomous field-phenotyping robot that can navigate in a variety of field conditions to overcome the aforementioned limitations with existing field-based phenotyping systems. A wide
variety of systems, e.g., tractors, unmanned aerial vehicles,
rovers, and gantry systems, were developed for field-based

need gigabytes of corn-stand data to be utilizable in practice.
This is achieved through adopting transfer learning. A support
vector machine (SVM) classifier is trained to classify the
features extracted from a convolutional neural network with
pre-trained weights. The result is a robust stand-counting
algorithm where the only sensor needed is a low-cost (< $30)
RGB camera. Our extensive field trials show that the detection
is robust against noises such as corn leaves, weeds, varying
lighting conditions, and residues from the previous year. Our
system achieves high accuracy and reliability throughout the
growing season.
This paper is organized as follows: The developed NMHENMPC framework for high precision control of the ultracompact mobile robot in off-road terrain is described in
Section II. The developed real-time machine vision algorithm
for corn stand counting is given in Section III. Field results
are given in Section IV, and Section V concludes the paper.
II. H IGH P RECISION C ONTROL A LGORITHM
Fig. 1. The ultra-compact 3D printed field robot, termed TerraSentia, in
soybean breeding plots in Energy Farm, Urbana, IL, USA.

phenotyping. However, these systems have many limitations,
including high operational and maintenance costs, low coverage area, safety, logistically difficult to maintain internal
combustion engines or the need for experienced operators
[7]. On the other hand, the ultralight robot described here
mitigates many of these challenges, yet, it leads to a uniquely
challenging control problem in uneven and unstructured terrain
in crop fields. The difficulties in control arise from complex
and unknown wheel-terrain interaction (described in detail
in Section II-A) and wheel slip. In this paper, a Nonlinear
Moving Horizon Estimator (NMHE) that identifies key terrain
parameters using onboard robot sensors, and a learning-based
Nonlinear Model Predictive Control (NMPC) that enables high
accurate tracking are developed to ensure autonomous highprecision robot mobility in off-road terrain.
The other significant contribution of this study is a novel
real-time machine vision algorithm that can estimate plant
stand count from image sequences obtained from a side-facing
camera on an ultra-compact ground robot. This algorithm is
demonstrated on the challenging problem of counting corn
(Zea mays or Maize) plants in field-conditions; however, the
algorithm may also be re-purposed to count other plants,
including Sorghum, Wheat, Soybean, or vegetables. The algorithm leverages a cutting-edge convolution neural network
architecture that runs efficiently on mobile platforms. Deep
learning methods have been shown capable of recognizing
complex structures and features in the presence of heavy
noise. Today, deep neural networks are approaching human
level at image recognition on Internet data [8]–[11]. However,
no machine-vision based (whether utilizing deep-learning or
not) corn stand counting algorithm currently exists that is
robust to real-world noise, varying lighting conditions, and
implementable in real-time on an autonomous ground robot.
Moreover, the algorithm is data-efficient so that it does not

A. System Model
The effect of terrain characteristics on robot performance
and wheel require special attention in field-robot design because the wheels are the unique connections between the
ground and robot, and nearly all forces and moments applied to
the robot are transmitted through the wheels [12], [13]. Due to
the wheel-terrain interaction dynamics, soil characteristic plays
a vital role in determining vehicle speed and steering, which
in turn are utilized for developing traction control algorithms.
The knowledge of soil parameters of unknown terrain is then
advantageous for improving vehicle performance [14]–[18]. In
particular for lightweight robots meant for persistent field use,
like TerraSentia robot described here, careful design of control
algorithms is necessary to ensure that the robot does not tread
over plants due to loss of precision due to unaccounted for
wheel-soil interaction.
A system model including traction parameters is developed
theoretically and used to establish an effective control law for
the 3D printed field robot traveling on rough terrain in this
paper. An adaptive nonlinear kinematic model is derived for
the ultra-compact 3D printed field robot as an extension of the
traditional kinematic model as follows:

 

ẋ
µv cos θ
 ẏ  =  µv sin θ 
(1)
κω
θ̇
where x and y are the position of the field robot, θ is the
yaw angle, v is the wheel speed, ω is the yaw rate, and µ
and κ are the traction parameters. The difference between the
traditional and developed model above is two slowly changing
traction parameters. These parameters provide us a learning
mechanism that can be used to minimize deviations between
the real-time system and ultra-compact 3D printed field robot
with an online parameter estimator. It is noted that they must
be between zero and one. If the traction parameters are equal
to 1, i.e., µ = κ = 1, there exist no longitudinal or side (lateral)
slips. The percentages of the longitudinal and side slips can

be respectively found as 1 − µ and 1 − κ. It is assumed that
only a fraction of the realized speed and yaw rate translates
into actual vehicle motion. These fractions are given by µ and
κ, and determine the effective speed µv and yaw rate κω. To
avoid bias, these parameters should be estimated along with
the full system state in each iteration based on a number of
past measurements.
In this paper, the nonlinear system and measurement models
are represented by the following equations:




ξ˙ (t) = f ξ (t), u(t), p(t) , z(t) = h ξ (t), u(t), p(t)
(2)
where ξ ∈ Rnξ is the state vector, u ∈ Rnu is the control
input, p ∈ Rn p is the system parameter vector, z ∈ Rnz is the
measured output, f (·, ·, ·) : Rnξ +nu +n p −→ Rnξ is the continuously differentiable state update function and f (0, 0, p) = 0 ∀t,
and h : Rnξ +nu +n p −→ Rnz is the measurement function. The
derivative of ξ with respect to t is denoted by ξ˙ ∈ Rnξ .
The state, parameter, input and output vectors are respectively
denoted as follows:

T

T
x y θ
ξ =
, p= v µ κ

T
u = ω , z= x y v ω
(3)
B. Nonlinear Moving Horizon Estimation
Although model-based controllers need full state and parameter information to generate a control signal, the number
of sensors is less than the number of measurable states and
parameters in practice. Therefore, state estimators are required
to estimate immeasurable states and parameters. Extended
Kalman filter is the most well-known state-estimation method
for nonlinear systems. However, EKFs are not capable of
dealing with constrained nonlinear systems [19]. As can be
seen in (2), the traction parameter estimates play a vital role
for the ultra-compact 3D printed robot, and there exist constraints on these parameters, which makes Extended Kalman
filter inconvenient for our system. Nonlinear Moving Horizon
Estimation (NMHE) approach has a capability of dealing with
constraints on state and parameter and is formulated for the
ultra-compact 3D printed robot as follows:

2
1
ξˆ (tk−Ne +1 ) − ξ (tk−Ne +1 )
min
p̂ − p
2
ξ (t),p,u(t)
HN

k
+ ∑ kzm (ti ) − z(ti )k2Hk
(4)
i=k−Ne +1

s. t.
ξ˙ (t) = f ξ (t), u(t), p

z(t) = h ξ (t), u(t), p
0 ≤ µ and κ ≤ 1

∀t ∈ [tk−Ne ,tk ]

where the deviations of the state and parameter estimates
before the estimation horizon are minimized by a symmetric
positive semi-definite weighting matrix HN , and the deviations
of the measured and system outputs in the estimation horizon
are minimized by a symmetric positive semi-definite weighting
matrix Hk [20]. The estimation horizon is represented by Ne ,
and lower and upper bounds on the traction parameters µ κ

are respectively defined as 0 and 1. The objective function
in the NMHE formulation consists of two parts: the arrival
and quadratic costs. The arrival cost stands for the early
measurements t = [t0,k−Ne ], and the quadratic cost stands for
the recent measurements t = [tk−Ne +1,k ].
The measurements have been perturbed by Gaussian noise
with standard deviation of σx = σy = 0.03 m, σω = 0.0175
rad/s, σv = 0.05 m/s based on experimental analysis. Therefore, the following weighting matrices Hk and HN are used in
the NMHE:
Hk

= diag(σx2 , σy2 , σv2 , σω2 )−1
= diag(0.032 , 0.032 , 0.52 , 0.352 )−1

HN

= diag(x2 , y2 , θ 2 , v2 , µ 2 , κ 2 )−1
= diag(10.02 , 10.02 , 0.12 , 1.02 , 0.252 , 0.252 )−1 (5)

The inputs to the NMHE algorithm are the position values coming from the global navigation satellite system, the
velocity values coming from the encoders mounted on the
DC motors and the yaw rate values coming from the gyro.
The outputs of NMHE are the position in x- and y-coordinate
system, the yaw angle, the wheel velocity and the traction
coefficients. The estimated values are then fed to the NMPC.
C. Nonlinear Model Predictive Control
Nonlinear Model Predictive Control (NMPC) approach is
the most well known control method for system with fast
dynamics. The reason is that NMPC has the capability of
dealing with hard constraints on state and input, and online
optimization allows updating cost, model parameters, constraints [21], [22]. The following finite horizon optimal control
formulation for the ultra-compact 3D printed robot is solved
to obtain the current control action by using current states and
parameters of the system as initial state:

o
1 n k+Nc −1
kξr (ti ) − ξ (ti )k2Qk + ku(ti )k2R
min
∑
2
ξ (t),u(t)
i=k+1

2
+ kξr (tk+Nc ) − ξ (tk+Nc )kQN
s. t.

ξ (tk ) = ξˆ (tk )
p = p̂(tk−Ne +1 )

ξ˙ (t) = f ξ (t), u(t), p

− 0.1 rad/s ≤ ω(t) ≤ 0.1 rad/s t ∈ [tk ,tk+N−1 ]
(6)
where Qk ∈ Rnξ ×nξ , R ∈ Rnu ×nu and QN ∈ Rnξ ×nξ are symmetric and positive semi-definite weighting matrices, ξr is the state
reference, ξ and u are the states and inputs, tk is the current
time, Nc is the prediction horizon, ξˆ (tk ) is the current state
estimates and p̂(tk−Ne +1 ) is the current parameter estimates.
The first term in the cost function is the stage cost and it is
the cost throughout the prediction horizon. The second term
in the cost function is the terminal penalty and it is the cost
at the end of the prediction horizon. The terminal penalty is
stated for stability reasons [23]. The weighting matrices Qk ,

R and QN are selected as follows:
Qk = diag(1, 1, 1),

R=1

and QN = 10 × Qk

(7)

The first element of the optimal control sequence is applied
to the system:
u(tk+1 , ξk+1 ) = u∗ (tk+1 )
(8)
and then the procedure is repeated for future sampling times
by shifting prediction horizon for the subsequent time instant.
It is important to remark that the control input u∗ (tk+1 ) is
precisely the same as it would be if all immeasurable states
and parameters acquire values equal to their estimates based
on the estimation up to current time tk due to the certainty
equivalence principle.
The state reference for the 3D printed robot is dynamically
changing online and defined as follows:
ξr = [xr , yr , θr ]T

(9)

where xr and yr are the position references, ωr is the yaw rate
reference calculated from the position references as follows:
θr = atan2 (ẏr , ẋr ) + λ π

(10)

where λ describes the desired direction of the 3D printed field
robot ( λ = 0 for forward and λ = 1 for backward).
D. Solution Methods
NMHE and NMPC methods for systems require online
solutions of nonlinear least square optimization problems at
each sampling time [24]. In this study, a single solution
method, consisting of a fusion between multiple shooting and
generalized Gauss-Newton methods, has been used to solve
both optimization problems [25]. This approach is valid because the formulation of NMPC problem is akin to that NMHE
problem. The generalized Gauss-Newton method derived from
the classical Newton method was developed for least-squared
problems. This method is advantageous because it does not
require difficult computations of the second derivatives; however, it is challenging to foreknow the number of iterations
to reach a solution of the desired accuracy. To overcome this
challenge, the solution proposed in [26] has been used where
the number of Gauss-Newton iterations is restricted to 1, and
the initial value of each optimization problem takes on the
value of the previous one intelligently. Hence, this improves
the convergence of the Gauss-Newton method.
In this paper, the Gauss-Newton iteration is divided into
two parts: preparation and feedback parts. The preparation
parts are executed before the feedback parts, and the feedback
part is executed after measurements for NMHE and estimates
for NMPC are available. In the preparation part, the system
dynamics are integrated with the previous solution, and objectives, constraints, and corresponding sensitives are evaluated.
In the feedback part, a single quadratic program is solved
with the current measurements for the NMHE and the current
estimates for the NMPC. Thus, new estimates for the NMHE
and new control signals for the NMPC are obtained. Compared
to the classical method, this method minimizes feedback

delay and produces similar results with higher computational
efficiency.
III. C ORN S TAND C OUNTING A LGORITHM
Conventional machine learning techniques require considerable domain-specific expertise to carefully design a feature
extractor to transform raw data (e.g., pixel values of an image)
into appropriate feature space where classifiers can detect
patterns in the input. In contrast, Deep learning is a set of
methods that allow end-to-end training and prediction. Deep
learning models take raw data input and automatically learn
the representations from the data’s internal structure. Typically
a deep learning model consists of multiple modules each of
which slightly increases the level of abstraction from the previous representation. When enough such modules are used, very
intricate structures can be learned such that relevant patterns
are recognized whereas irrelevant variations are suppressed.
Deep learning has made tremendous progress in areas that
have confounded traditional machine learning for many years
including image recognition [9], [11]. However, deep learning
requires a lot of human-labeled images, unfortunately, this is
something that is rather expensive to obtain. In this paper,
we combine convolutional neural network (Section III-A) and
support vector machine (Section III-B) via transfer learning to
recognize corn plants using very few labeled images.
Recognition alone is not sufficient to count plants on a
moving robot since it is difficult to distinguish different corn
plants to avoid double-counting. To address this issue, we
utilize motion estimation techniques to determine the number
of plants that have passed through a fixed Region Of Interest
(ROI). Details of this algorithm are given in Section III-D.
A. Convolutional Neural Network
It is important to maintain a good balance between performance and efficiency on algorithm to be implemented on
mobile platforms, such as robots. MobileNets [27] is a family
of networks specially developed for mobile and embedded
applications. Due to the use of depthwise separable convolution that requires between 8 to 9 times less computation
than standard convolution, the model runs significantly faster
than its more complicated counterparts with only slight accuracy compromise. The network parameters are pre-trained
on the ImageNet Large Scale Visual Recognition Competition
(ILSVRC) dataset [28] with fully connected and softmax layer
in the original model replaced by SVM for classification.
B. Support Vector Machine
Support Vector Machine (SVM) is a widely used model for
classification [29]–[34]. The model constructs a hyperplane
that maximizes the distance between the decision boundaries and any data point. The convexity of its loss function
gives SVM desirable advantage over other methods such as
feedforward neural network that suffers from the existence
of multiple local minima and the need for a large dataset
to ensure convergence [35], [36]. Therefore, we chose to
replace the fully connected layer in the MobileNet with SVM

Algorithm 1: Counting algorithm
Input
: I1 , . . . , In := image frames
Parameter: k := average window size;
w := ROI width in pixels;
d := image translation in pixels
Output : C := Corn stand count
Initialize : C = 0;
d=0
foreach It do
Extract features by ConvNet → fi ∈ R1024 ;
Classify by SVM → yi ∈ {−1, 1};

TABLE I
N UMBER OF TRAINING AND TESTING SAMPLE AT EACH GROWTH STAGE .

Training
Testing

for better performance on limited data. The kernel functions
of the SVM transform input data from the latent feature
space of the deep network to a higher dimensional smooth
Reproducing Kernel Hilbert Space. This enables the SVM to
classify non-linearly separable features despite being a linearin-the-parameter classifier.
C. Training
Images are captured by the side facing camera on the robot
throughout the growing season. The camera slightly points
downward so that only the closest row is visible. Patches
are cropped from the images and labeled according to the
presence or absence of corn as positive and negative samples,
respectively. The number of training and testing samples listed
in Table I. We also employ data augmentation to increase the
training data further. Images are rotated (±10◦ ), zoomed (88 %
to 112 %), vertically shifted (±15 %), and horizontally flipped.
Each image is augmented by 16-fold via randomly drawing
transformations from the list.
The weights for the MobileNet are kept constant during
transfer learning. The hyper-parameter for SVM is determined
by grid search and cross-validation implemented in Python
package scikit-learn [37]. Linear kernel and Radial Basis
Function (RBF) kernel are investigated. The values of hyperparameters that we consider in this study are listed Table
II . All combinations are exhausted to identify the optimal
parameters.

V4
348
416
89
105

VT
328
304
83
77

R2
519
310
130
78

R6
263
291
66
73

TABLE II
SVM HYPER - PARAMETERS TESTED IN THE GRID SEARCH .

∑i+k yi

i−k
Average with neighboring images ŷi ← 2k+1
;
if ŷi > 0 then
Estimate translation from It−1 to It → dt ;
d ← d + dt ;
end
else
if ŷi−1 > 0 then
multiplicity M ← wd ;
C ← C + M;
d ← 0;
end
end
end

positive
negative
positive
negative

C
γ

Linear
1, 10, 100, 1000
N/A

RBF
1, 10, 100, 1000
10−2 , 10−3 , 10−4 , 10−5

D. Motion Estimate
Instead of classifying the whole image, we focus on a
fixed region of interest (ROI) whose size is slightly less than
the average plant spacing. In the majority of the time, only
single corn plant appears in the ROI as in Figure 2(a). The
classification results then form a time series of alternating step
function where each plateau represents a corn plant. However,
when the gap between two neighboring plants is smaller than
the ROI, we compute the horizontal translation T between
two consecutive frames I1 and I2 when a positive signal of the
binary classification is present. If the ROI finishes scanning
through a corn before encountering the next one as in most
cases, T should approximately equal to ROI width w. In the
case that adjacent corn plants are close, T = (n − 1)d + w,
where n is the number of plants, d is the distance between
neighboring plants as shown in Fig. 2(b). If we further assume
that d is not much smaller than w, then n ≈ Tw . In real-time
implementations, taking the ratio results in a floating point
number that is rounded towards the nearest integer. In other
words, the inequality (11) must be held. For instance n = 2, the
algorithm can correctly count two nearby corn plants as long
as their distance is not smaller than half of the ROI width. The
lower bound on d increases as the number of adjacent corn
plants n. But based on our observation, in most cases n is less
than 5.
n − 0.5 ≤

T
n − 1.5
d
n − 0.5
≤ n + 0.5,
≤ ≤
w
n−1
w
n−1
IV. F IELD R ESULTS

(11)

A. Real-time Results for Navigation
Linear controllers cannot achieve good trajectory tracking
performance on uneven terrain if the system starts off-track
[38]. As shown in Fig. 3(a), after the 3D printed field robot
is started off-track, it is capable of reaching to the reference
trajectory and staying on-track. One of the major problems of
such GNSS-based navigated outdoor robots is the missing data
values from the satellites. This drawback is inevitable for RealTime Kinematic (RTK) GNSSs, and the control algorithm has

ROI

ROI

w

w

d

d

(a) Single corn appears in the ROI (b) Two corn plants appear in the
at a time (w < d)
ROI at the same time (w ≥ d)
Fig. 2. Examples of: (a) normal cases where a single corn appears in the ROI.
The relative motion between the camera and corn T ≈ w; (b) exceptions where
neighboring corn plants are too close to be separated In this case, T ≈ d + w.

to be robust to cope with missing data points. As can be
observed from Fig. 3(a), although there were 35 missing data
points within 1850 data points throughout the experiment, high
accurate trajectory tracking performance has been achieved by
the NMHE-NMPC framework.
The Euclidean error is shown in Fig. 3(b). The mean value
of the Euclidean error after staying on-track is 0.0423 m.
The NMPC benefits from traction parameters estimated by the
NMHE and results in highly accurate tracking performance so
that the Euclidean error is less than 10 cm. The number of
violations is shown in Fig. 3(c). The results of multiple experiments indicate that the NMHE-NMPC framework does not
violate this error constraint. This demonstrates the capability
of the NMHE-NMPC framework.
The traction parameters estimates are shown in 3(d) and
within the upper and lower bounds specified in (6). It is
clearly seen that they consistently stay close the upper bound
and stabilize at values which ensure stable trajectory. It is
to be noted that if the soil conditions are unchanging, then
hard-coding might result in more accurate trajectory tracking
performance. However, the soil conditions are dynamically
changing in practice.
The measured and estimated speeds of the ultra-compact
3D printed field robot are shown in Fig. 3(e). As can be
seen, the robot has a constant speed, and the NMHE is
capable of dealing with noise on the measurements. It is
to be noted that some estimated speed values between 300
and 350 seconds are larger than measurements. The reason
is that there are missing GNSS measurements as shown in
Fig. 3(a). If GNSS measurement is invalid, previous valid
measurement is fed to the NMHE instead of an outlier. There
are 14 invalid points in the GNSS measurements between
318.8 and 321.6 seconds and these result in the peak values
in the speed estimates between 322.8 and 325.6 seconds.
The effect on the speed estimates arises seen subsequently
because the speed in the system model (2) is considered
as a parameter and the parameter estimation is achieved in
the arrival cost of the NMHE while the parameters in the
estimation horizon are assumed to be time-invariant. Moreover,
the longitudinal traction parameter estimate in Fig. 3(d) has

nadir values between 322.8 and 325.6 seconds similar to the
speed estimate. Consequently, the effective speed µv is equal
to the speed of the robot despite unavailable GNSS points.
The same explanation is the case for the two peak points the
speed estimate between after around 340 seconds.
The output of the NMPC, i.e., the desired yaw rate, and
yaw rate measurements are shown in Fig. 3(f). The generated
control signal by the NMPC is within the lower and upper
bounds specified in (6). The performance of the low-level
controller is sufficient to track the reference signals despite
the high noisy measurements.
B. Real-Time Results for Corn Stand Counting
We collected data from various corn fields. The dates
and growth stages when data were collected span the entire
growing season. In each field, we randomly selected a row and
drove the robot at a constant velocity for an arbitrary number
of plots. The true populations were counted manually. For data
collections prior to October, a small portion of the images were
annotated. We collected two additional data-sets in October to
test the generalization ability of the model on unseen data from
different fields conditions. Table III summarizes the conditions
of the data collection. We follow the widely used staging
method by Abendroth et al. [39]. The method divides the
growth of corn into vegetative phase and reproductive phase.
The vegetative phase covers stages from emergence (VE) to
tasseling (VT), where stages in-between are identified by the
number of collar leaves (e.g. V4). The reproductive phase
includes silking (R1), blister (R2), milk (R3), dough (R4),
dent (R5), and maturity (R6).
1) SVM training: The best hyper-parameters for the SVM
found in the grid search are listed in Table IV with their
performance metrics on the test data. The hyper-parameters
are almost the same for the four growth stages except that the
C-value is different for V4, largely due to the difference in
appearance of corn plants at V4 from VT, R2, and R6. The
RBF kernel consistently works better than the linear kernel,
and the optimal γ-value for the kernel is identical at four
stages.
The performance metrics are also consistently high for
different growth stages. It is noted the training samples exclude
leaves and only focus on stems and stalks. The metrics demonstrate that the deep learning model is capable of distinguishing
subtle difference in images and thus reduces the inference of
leaves and weeds. The mean accuracy for all stages is 91.67%.
2) Validation: In-Field Corn counting: Figure 4 shows the
corn plant population per plot by robot vs human for each
dataset. The robot predictions agree well with the ground
truth. The least-square fitted line through all data is given
as countrobot = 0.96 × counthuman + 0.85 and a correlation
coefficient R = 0.96.
Figure 5 shows the box-and-whisker plot of the relative
accuracy as in Equation (12) for each dataset. The algorithm achieves consistently high accuracy for all locations
and growth stages. It is noted that data from Martinsville
and Lebanon were processed without using the new data

0.08
Error (m)

X axis (m)

30
20
10

3

0.06
0.04

2

0.02

1

0
0
5

10

2

15
20
25
30
Y axis (m)
Estimate

0

100

200

300

Speed (m/s)

(.)

0.5
Bounds
0

100

200

300

1
(.)

200

100

200
Time (s)

100

200
Time (s)

80
60
40
20
0

300

6

7
8
9
Accuracy (cm)

0.4

0.1

0.3
0.2

0

300

Measurement
Estimate
0

100

(d)

200
Time (s)

10

(c)
0.2

Bounds
0

100

0.5

0.1

0.5
0

300

120

(b)

1

0

100

Reference

(a)

0

0

0

Yaw rate (rad/s)

Yaw angle (rad)

0

40

140

0.1

4

Number of violations

Reference trajectory
Actual trajectory
GPS unavailable

40

0
-0.1
Measurement
Reference
Bounds

-0.2

300

0

100

(e)

200
Time (s)

300

(f)

Fig. 3. a) Reference and actual trajectories, and unavailable Global Navigation Satellite System (GNSS) signals on the trajectory. b) Euclidean error. The
mean value of Euclidean error is less than 5 cm after being stayed on-track. c) The number of violations. The available space on either side of the field robot
is restricted to 0.12 m. It does not violate this accuracy limit. d) Reference and estimated yaw angles, and estimated traction parameters µ, κ. e) Measured
and estimated speeds. f) Reference and measured yaw rates.

TABLE III
F IELD CONDITIONS FOR EACH DATA COLLECTION , INCLUDING DATE AND
LOCATION OF THE FIELD , WHETHER DATA ARE SAMPLED FOR TRAINING
IMAGE RECOGNITION MODEL , THE GROWTH STAGE OF CORN PLANTS AT
COLLECTION TIME , AND A NUMBER OF PLOTS COLLECTED .

to train SVM. The results demonstrate that the recognition
model generalizes well to unseen data and handles real-world
variations effectively.
accuracy = 1 −

Date
Jun 6th , 2017
Jul 6th , 2017
Aug 2nd , 2017
Sep 21st , 2017
Oct 25th , 2017
Oct 26th , 2017

Location

Training

Growth stage

Plots

Assumption, IL
Champaign, IL
Champaign, IL
Ivesdale, IL
Lebanon, ID
Martinsville, IL

Yes
Yes
Yes
Yes
No
No

V4
VT
R2
R6
V6
R6

16
10
28
10
15
8

TABLE IV
B EST HYPER - PARAMETERS FOUND IN THE GRID SEARCH AND THEIR
CORRESPONDING PERFORMANCE METRICS ON THE TEST DATA .

kernel
C
γ
accuracy
precision
recall
F1-score

V4

VT

R2

R6

rbf
10
0.001
91.75%
0.91
0.95
0.95

rbf
100
0.001
94.38%
0.91
0.94
0.93

rbf
100
0.001
92.79%
0.91
0.95
0.94

rbf
100
0.001
87.77%
0.89
0.88
0.88

|countrobot − counthuman |
counthuman

(12)

V. C ONCLUSIONS
The embedded high precision control and corn stand counting algorithms for a low-cost, ultra-compact 3D printed, autonomous field robot for off-road terrain have been elaborated
in this paper.
An NMHE that identifies key terrain parameters using onboard robot sensors and a learning-based NMPC that enables
highly accurate tracking have been developed to be capable
of autonomously and safely navigating through row based
crops. The developed framework for high precision control
that enables reliable field robot path tracking in off-road terrain
provides less than 5 cm accuracy in the presence of slip
and does not violate the 12 cm accuracy limit to avoid crop
damage.
The developed vision algorithm based on color imagery
recognizes corn stalks robustly in the presence of interference
from leaves and weeds, yet only requires relatively small
amount of data for training to estimate corn plant population.
This was achieved by fusing a leading deep learning model

(ARPA-E), U.S. Department of Energy, under Award Number
DE-AR0000598. The views and opinions of authors expressed
herein do not necessarily state or reflect those of the United
States Government or any agency thereof.
Dr. Erkan Kayacan is a postdoctoral researcher at Massachusetts Institute of Technology since February 2018. Any
questions or comments should be directed to Girish Chowdhary. We thank EarthSense Inc. for valuable suggestions.
The authors would like to thank Benjamin Thompson and
others at Distributed Autonomous Systems Laboratory for
valuable contributions to the design and construction of the
TerraSentia robot.

8 0

A s s u m p tio n ( V 4 )
C h a m p a ig n ( V T )
C h a m p a ig n ( R 2 )
Iv e s d a le ( R 6 )
M a r tin s v ille ( R 6 )
L e b a n o n (V 6 )

7 0
6 0

R o b o t

5 0
4 0
3 0
2 0
1 0

E q u a tio n

y = a + b *x

In te r c e p t

0 .8 4 6 8 7 ± 0 .8 2 0 2 3

S lo p e

0 .9 6 2 3 4 ± 0 .0 3 0 5 2

P e a r s o n 's r

0 .9 5 9 8 2

R -S q u a re (C O D )

0 .9 2 1 2 5

A d j. R - S q u a r e

0 .9 2 0 3 2

R EFERENCES

0
0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

H u m a n

Fig. 4. Scatter plot of corn plants per plot counted by robot vs human for each
dataset. The line represents a linear fit through all data. The robot predictions
agree well with the ground truth with the correlation coefficient R = 0.96.

1 1 0 %

A s s u m p tio n ( V 4 )

C h a m p a ig n ( V T )

C h a m p a ig n ( R 2 )

Iv e s d a le ( R 6 )

M a r tin s v ille ( R 6 )

L e b a n o n (V 6 )

A c c u ra c y

1 0 0 %

9 0 %

8 0 %

7 0 %

6 0 %

D a ta s e t

Fig. 5. Box plot of the accuracy as in Equation (12) for each dataset. The
accuracy stays consistent for different locations and growth stages even for
datasets from Martinsville and Lebanon where no new training data are used.

with Support Vector Machine Classifiers. Our method was
tested at five different growth stages and locations. The robot
predictions agreed well with the ground truth with countrobot =
0.96×counthuman +0.85. Least-square regression shows strong
correlation (R = 0.96) between populations given by robot and
human. In addition, the accuracy stays consistently high for
all growth stages and locations. These results establish that
ultra-compact, lightweight, autonomous agricultural robots
equipped with exteroceptive sensors and artificial intelligence
could be utilized to relieve the phenotyping bottleneck.
ACKNOWLEDGMENTS
The information, data, or work presented herein was funded
in part by the Advanced Research Projects Agency-Energy

[1] R. T. Furbank and M. Tester, “Phenomics–technologies to relieve the
phenotyping bottleneck,” Trends in plant science, vol. 16, no. 12, pp.
635–644, 2011.
[2] J. L. Araus and J. E. Cairns, “Field high-throughput phenotyping: the
new crop breeding frontier,” Trends in plant science, vol. 19, no. 1, pp.
52–61, 2014.
[3] E. J. van Henten, D. Goense, and C. Lokhorst, Precision agriculture ’09,
D. Goense, E. J. van Henten, and C. Lokhorst, Eds. The Netherlands:
Wageningen Academic Publishers, 2009.
[4] P. Biber, U. Weiss, M. Dorna, and A. Albert, “Navigation system of the
autonomous agricultural robot bonirob,” in Workshop on Agricultural
Robotics: Enabling Safe, Efficient, and Affordable Robots for Food
Production (Collocated with IROS 2012), Vilamoura, Portugal, 2012.
[5] A. Shafiekhani, S. Kadam, F. B. Fritschi, and G. N. DeSouza, “Vinobot
and vinoculer: Two robotic platforms for high-throughput field phenotyping,” Sensors, vol. 17, no. 1, 2017.
[6] T. Mueller-Sim, M. Jenkins, J. Abel, and G. Kantor, “The robotanist: A
ground-based agricultural robot for high-throughput crop phenotyping,”
in 2017 IEEE International Conference on Robotics and Automation
(ICRA), May 2017, pp. 3634–3639.
[7] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Distributed nonlinear model predictive control of an autonomous tractortrailer system,”
Mechatronics, vol. 24, no. 8, pp. 926 – 933, 2014.
[8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, 2016, pp. 770–778.
[9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[10] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
in Proceedings of the IEEE conference on computer vision and pattern
recognition, 2015, pp. 1–9.
[11] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[12] K. Iagnemma, S. Kang, H. Shibly, and S. Dubowsky, “Online terrain
parameter estimation for wheeled mobile robots with application to
planetary rovers,” IEEE Transactions on Robotics, vol. 20, no. 5, pp.
921–927, Oct 2004.
[13] L. E. Ray, “Estimation of terrain forces and parameters for rigid-wheeled
vehicles,” IEEE Transactions on Robotics, vol. 25, no. 3, pp. 717–726,
June 2009.
[14] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Modeling and
identification of the yaw dynamics of an autonomous tractor,” in 2013
9th Asian Control Conference (ASCC), June 2013, pp. 1–6.
[15] S. U. Lee and K. Iagnemma, “Robust motion planning methodology
for autonomous tracked vehicles in rough environment using online slip
estimation,” in 2016 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), Oct 2016, pp. 3589–3594.
[16] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Nonlinear modeling
and identification of an autonomous tractor-trailer system,” Computers
and Electronics in Agriculture, vol. 106, pp. 1 – 10, 2014.
[17] S. U. Lee, R. Gonzalez, and K. Iagnemma, “Robust sampling-based
motion planning for autonomous tracked vehicles in deformable high
slip terrain,” in 2016 IEEE International Conference on Robotics and
Automation (ICRA), May 2016, pp. 2569–2574.

[18] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Robust tubebased decentralized nonlinear model predictive control of an autonomous tractor-trailer system,” IEEE/ASME Transactions on Mechatronics, vol. 20, no. 1, pp. 447 – 456, Feb 2015.
[19] E. L. Haseltine and J. B. Rawlings, “Critical evaluation of extended
kalman filtering and moving-horizon estimation,” Industrial & Engineering Chemistry Research, vol. 44, no. 8, pp. 2451–2460, 2005.
[20] H. Ferreau, T. Kraus, M. Vukov, W. Saeys, and M. Diehl, “High-speed
moving horizon estimation based on automatic code generation,” in
Decision and Control (CDC), 2012 IEEE 51st Annual Conference on,
2012, pp. 687–692.
[21] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Towards agrobots:
Identification of the yaw dynamics and trajectory tracking of an autonomous tractor,” Computers and Electronics in Agriculture, vol. 115,
pp. 78 – 87, 2015.
[22] E. Kayacan, E. Kayacan, I.-M. Chen, H. Ramon, and W. Saeys, On the
Comparison of Model-Based and Model-Free Controllers in Guidance,
Navigation and Control of Agricultural Vehicles.
Cham: Springer
International Publishing, 2018, pp. 49–73.
[23] D. Mayne, J. Rawlings, C. Rao, and P. Scokaert, “Constrained model
predictive control: Stability and optimality,” Automatica, vol. 36, no. 6,
pp. 789 – 814, 2000.
[24] E. Kayacan, J. M. Peschel, and E. Kayacan, “Centralized, decentralized
and distributed nonlinear model predictive control of a tractor-trailer
system: A comparative study,” in 2016 American Control Conference
(ACC), July 2016, pp. 4403–4408.
[25] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Learning in
centralized nonlinear model predictive control: Application to an autonomous tractor-trailer system,” IEEE Transactions on Control Systems
Technology, vol. 23, no. 1, pp. 197–205, Jan 2015.
[26] M. Diehl, H. Bock, J. P. Schlder, R. Findeisen, Z. Nagy, and F. Allgöwer,
“Real-time optimization and nonlinear model predictive control of processes governed by differential-algebraic equations,” Journal of Process
Control, vol. 12, no. 4, pp. 577 – 585, 2002.
[27] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint
arXiv:1704.04861, 2017.
[28] J. Pruegsanusak and A. Howard, https://github.com/tensorflow/models/
blob/master/slim/nets/mobilenet v1.md, accessed: 2017-08-12.
[29] N. Cristianini and J. Shawe-Taylor, “An introduction to support vector
machines,” 2000.
[30] B. Scholkopf, K.-K. Sung, C. J. Burges, F. Girosi, P. Niyogi, T. Poggio,
and V. Vapnik, “Comparing support vector machines with gaussian
kernels to radial basis function classifiers,” IEEE transactions on Signal
Processing, vol. 45, no. 11, pp. 2758–2765, 1997.
[31] A. J. Smola and B. Schölkopf, “On a kernel-based method for pattern
recognition, regression, approximation, and operator inversion,” Algorithmica, vol. 22, no. 1-2, pp. 211–231, 1998.
[32] A. J. Smola, B. Schölkopf, and K.-R. Müller, “The connection between
regularization operators and support vector kernels,” Neural networks,
vol. 11, no. 4, pp. 637–649, 1998.
[33] B. Scholkopf, S. Mika, C. J. Burges, P. Knirsch, K.-R. Muller, G. Ratsch,
and A. J. Smola, “Input space versus feature space in kernel-based
methods,” IEEE transactions on neural networks, vol. 10, no. 5, pp.
1000–1017, 1999.
[34] B. Schölkopf, C. J. Burges, and A. J. Smola, Advances in kernel
methods: support vector learning. MIT press, 1999.
[35] J. A. Suykens, J. De Brabanter, L. Lukas, and J. Vandewalle, “Weighted
least squares support vector machines: robustness and sparse approximation,” Neurocomputing, vol. 48, no. 1, pp. 85–105, 2002.
[36] C. M. Bishop, Neural networks for pattern recognition.
Oxford
university press, 1995.
[37] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,
O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine
Learning Research, vol. 12, pp. 2825–2830, 2011.
[38] E. Kayacan, H. Ramon, and W. Saeys, “Robust trajectory tracking
error model-based predictive control for unmanned ground vehicles,”
IEEE/ASME Transactions on Mechatronics, vol. 21, no. 2, pp. 806–814,
April 2016.
[39] L. J. Abendroth, R. W. Elmore, M. J. Boyer, and S. K. Marlay, “Corn
growth and development,” 2011.
</reference>

<statements>
1. The evidence supports three load-bearing conclusions, each developed below from quantified pipeline results and control results.
2. nonlinear moving-horizon estimation plus nonlinear model predictive control achieves field tracking below 5 cm under unknown wheel–terrain slip
3. recursive platform and optical state estimation
4. For terrain and wheel–slip identification, a nonlinear moving-horizon estimator identifies unknown traction parameters online, constrained between 0 and 1, and feeds a learning-based nonlinear model predictive controller
5. The controller achieves mean Euclidean tracking error of 0.0423 m after staying on track and remains accurate despite 35 missing GNSS points out of 1850
6. The authors note that hard-coding could be more accurate if soil conditions were static, implying that the estimator is most valuable under changing terrain
7. A suitable architecture is hierarchical because the cited systems separate platform control, acquisition control and supervisory scheduling.
8. The navigation layer should combine EKF fusion, adaptive visual servoing and predictive control because these methods address pose failure, uncalibrated camera and robot parameters, and unknown wheel–terrain slip.
9. NMHE+NMPC handles unknown wheel–terrain interaction and slip, achieving <5 cm tracking error and mean Euclidean error of 0.0423 m after staying on track.
10. Stability and robustness must be stated carefully
11. NMHE+NMPC provides constrained online optimization and field tracking performance
12. The available evidence does not establish formal frequency-domain robustness margins for a grain-imaging loop, so the design should require empirical validation under texture, illumination, motion and calibration perturbations
13. Does the controller perform? tracking error, in-focus accuracy, fault response time
14. NMHE+NMPC achieved mean Euclidean error 0.0423 m and <5 cm field tracking
15. The evidence supports each component separately: scan speed [2], acquisition geometry [6], focus [5], illumination power [4], image count [11], environmental risk [3] and pose estimation [7][9]
16. The modern-control contribution is to connect these layers: EKF and adaptive visual servoing estimate pose and camera parameters [7][8], NMHE+NMPC handles terrain and slip [9]
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.