Development of an online Nigella sativa inspection system equipped with machine vision technology and artificial neural networks

INTRODUCTION

Nigella sativa L., as a popular medicinal plant, has one hundred different chemicals including essential acids (Majdalawieh & Fayyad, 2016). Although N. sativa does not have a significant weight in the global market, its important medicinal properties increase its usage constantly (Gharby et al., 2015). The raw seed and the extracted oil are broadly used in food and pharmaceutical applications. Separation, cleaning, and detection of contaminants are the most essential issues in grains post-harvest processing. Due to the small size of N. sativa and its impurities, the separation process has some challenges (Copeland & McDonald, 2012). Machine vision has been one of the most powerful, inexpensive, non-destructive, accurate, and reliable measurement tools in grain technology for the past two decades (Vithu & Moses, 2016). Advances in image processing and machine learning techniques have greatly improved this tool, so this technology with new controllers can classify grains with similar physical characteristics, eliminating the disadvantages of mechanical cleaning systems such as low accuracy, grain damage, and contamination of the product with other impurities such as oil vapor (Rad et al., 2011). Seeds have a broad distribution of visual characteristics. So various morphological, color, and texture properties are needed for their distinction and grading. Also, advanced analytics approaches such as artificial intelligence techniques are required for the analysis of these properties (Orlandi et al., 2018; Qadri et al., 2021).

Machine vision have been used for separation of grains, diseased grains, moldy grains and impurities. Rad et al. (2011) classified five rice cultivars by a back-propagation (BP) artificial neural network (ANN) with 97% accuracy. Determining the number of hidden layers and neurons is very important to achieve a suitable model for mining the data. However, it is a difficult task and requires trial and error. Han et al. (2014) reported that the performance of a support vector machine (SVM) with genetic algorithms (GA) for the selection of superior features from wheat seed images achieved better results than linear discriminant analysis (LDA) and principal components analysis (PCA). The optimum population size of the GA was 24. The results showed that GA-SVM successfully selected and classified the features. Two features, namely, red in RGB space and core area, were sufficient for the online classifying of wheat grains. Ebrahimi et al. (2014) presented an algorithm based on ANN with independent component analysis (ICA) to classify the wheat seeds and determine the degree of purity. In their imaging system, the seeds were separated from each other by hand to avoid contact with each other. Their algorithm was able to accurately detect wheat grains, wheat cultivars, and weed seeds with an accuracy of 96.25%, 87.5%, and 77.22%, respectively. In another study, Liu et al. (2015) showed that using only texture features were not sufficient to show differences in damage, breaks, and partially incomplete skin of soybeans. Shrestha et al. (2016) measured the contrast, correlation, energy, and homogeneity of wheat grains in the horizontal and vertical directions to extract texture characteristics. To identify three Iranian rice species, Fayyazi et al. (2017)reports indicating the possibility of mixing different varieties on the market. Applying machine vision techniques to classify rice varieties is a method which can increase the accuracy of classification process in real applications. In this study, several morphological and textural features of rice seeds’ images were examined to evaluate their efficacy in identification of three Iranian rice varieties (Tarom, Fajr, Shiroodi) extracted morphological and texture features and used the PCA method to select and rank the most important ones; the rice classification was performed using multilayer perceptron (MLP) neural networks. For the detection of discolored peppercorn grains, Kanjanawanishkul et al. (2018) used two processors, a PC-based controller processed images and detected discolored seeds, and an Arduino controller started and stopped the rotation of the circular plate containing the seeds and controlled the pneumatic valves. This machine accurately classified 90.9% of the input grains.

The impurity affects the oil extraction process of N. sativa and influence consumer acceptance in both raw seeds and oil market. Accordingly, the automation of the cleaning process is essential for performing high-quality products. The mechanical separation of N. sativa from impurities based on the pneumatic, friction, and sieve mechanisms has not been implemented due to their similar physical properties (geometric characteristics, static friction coefficient, and terminal velocity). Therefore, in this study, an intelligent cleaning system based on novel non-destructive image processing techniques and machine learning algorithms was designed and developed. The device automatically detected impurities by combining the machine vision system and the ANN model.

MATERIAL AND METHODS

Figure 1 shows a scheme of the developed automatic system for sorting impurities from N. sativa. Python platform was used to develop the algorithms. Further information is presented in the following subsections.

Figure 1. The proposed system for separating Nigella sativa and its impurities. CFS: correlation-based feature selection, PCA: principal component analysis, NS: Nigella sativa, FB: brown seed, YF: yellow flax, PS: soil particles, GS: grey seed.

The N. sativa seeds were collected directly from Kurdistan province, Iran. The grains were transferred to the Department of Agricultural Machinery Engineering, University of Tehran. The product contained several impurities such as weed seeds and soil particles (Fig. 2).

Figure 2. Electric diagram: S1, displacement from left to right; S2, displacement from right to left; S3 and S4, sensors that inform the initial and final position, respectively.

Components of the designed system

This device consists of five main parts: a conveyor belt, an imaging box, an electronic part, a separator, and a personal computer (Fig. 1). In this research, a white conveyor belt was used to achieve better segmentation. The conveyor belt transferred the product under the camera and the separator. The length, width, and thickness of the belt were selected as 1.5 m, 20 cm, and 1 mm, respectively. To move the conveyor belt with a constant speed and with high torque, a DC motor (12 volts with a maximum speed of 150 rpm) with two conical gears was selected. Motor speed can be easily adjusted by controlling the voltage volume. Lighting is one of the most important parts of a machine vision system. To control the light and receive homogenous light, a black rectangular box with four LEDs was used to create uniform lighting conditions and maximize the reduction of shadows. A 64-megapixel mobile camera with the specifications of the Xiaomi Note 8 Pro was used for capturing. The images were transferred wirelessly to the computer for further processing through the IP Webcam software installed on the mobile phone. Image enhancement, segmentation, noise cancellation, feature extraction, superior feature selection, and classification were performed on the computer (Core i7-5500U CPU; 2.4 GHz; 8 GB RAM; Windows 10, 64 bits). The commands were sent to the separator via an ATmega2560 Arduino board.

A servomotor model MG90s separated the impurities from N. sativa seeds, by deflecting the grains from their original path. It has a lever that moves a flexible object in the grain flow path, and can be rotated from zero degrees to the 180-degree angle. It was located at the angle of 45 degrees for N. sativa and the angle of 135 degrees for impurities.

The area of the image where the grain is located was cropped after receiving it. The cropping operation reduces the volume of calculations and speeds up the grading process. To remove noise and fill cavities, the dilation and erosion operation were applied.

Image segmentation

Images in RGB color space were decomposed into three channels (R, G and B), and a binary threshold was applied to the image to separate the grains from the background. In the thresholding method, the whole image was divided into windows with dimensions of 3×3, 5×5, and 7×7. Two thresholding methods, namely, inverted binary thresholding (IBT) and inverted to zero thresholding (IZT) were utilized for segmentation. The threshold value was determined by computing the mean and standard deviation of the brightness of pixels for each category. So all areas of the same intensity were considered as one object. In this way, the seed and the background could be completely separated (Fig. 3).

Figure 3. Image segmentation using the inverted binary thresholding (IBT) and inverted to zero thresholding (IZT) methods. a) Initial image, b) IBT on R channel, c) IBT on G channel, d) IBT on B channel, e) IZT on R channel, f) IZT on G channel, g) IZT on B channel.

Feature extraction

Color, texture, size, and shape are the most important visual features of an object in the image (Nouri-Ahmadabadi et al., 2017). A total of 106 features (9 morphology features, 36 color features, and 61 texture features) were extracted from each sample. N. sativa seeds and their impurities differ slightly in color, so the extraction of all morphological, color, and texture properties is necessary for accurate classification.

The extracted morphology and color features with their formulas are listed in Tables 1 and 2, respectively. Due to the small color difference between some impurities and N. sativa, the four Bao et al.´s (2019) statistical descriptors were computed:

where h(x) represents the grey level of pixels in the image with a pixel position of x, x can take any value between 1 and z = m×n, m and n being the number of rows and columns of the image matrix, respectively.

For texture classification, the features used were roughness, smoothness, or regularity and irregularity of the grains surface (Nouri-Ahmadabadi et al., 2017; Bao et al., 2019). In some cases, the images of impurities and N. sativa had similar color and shape, but the areas with different color intensities and greyscale levels created different tissue structures and could distinguish each class. Texture features including contrast, energy, homogeneity, correlation, dissimilarity, and entropy were extracted (Table 3) using the Grey level co-occurrence matrix according to Eq. (5) (Mollazade, 2017):

where C is the grey level co-occurrence matrix; m and n are the values related to the brightness; i and j are the image sizes; x and y are the spatial coordinates; and ∆x and ∆y are the deviation parameters. In this study, the distance parameter was chosen as 1, 2, and 3 pixels, and the angle value was selected as 0, 45, 90, and 135 degrees (Table 3).

Feature selection

A total of 100 samples for each class of N. sativa and impurities (brown seed, yellow flax, soil particles, and grey seed) were selected and 106 features were extracted from each image. The dataset normalized before feature selection and classification. In this research, the Z-score normalization method, as the most common technique, was utilized. The performance of the developed algorithm depends on the obtained features. The superior and efficient features increase the performance of the proposed algorithm and reduce the execution time. In data mining, there are many ways to select the superior features, such as sequential forward selection (SFS), factor analysis, correlation-based feature selection (CFS), genetic algorithm, and sensitivity analysis (Mollazade et al., 2012a; Valiente-Gonzalez et al., 2014). In this study, the CFS method was used to select important and efficient features from the feature set. The method was implemented in WEKA software version 3.9.

PCA is a simple and efficient linear analysis method. The main purpose of PCA is to identify patterns in the data. PCA intends to identify the correlations between variables. In this method, a large number of independent variables can be replaced with a limited number of new dependent variables called principal components. PCA reduces the dimension of the dataset (X) in the flowing five main steps (Jolliffe & Cadima, 2016; Shen et al., 2020):

1. Decentralization of the values of features (value of each feature minus its average).
2. Computing the covariance matrix for X.
3. Eigenvalues and eigenvectors of the covariance matrix are obtained by eigenvalue decomposition or singular value decomposition.
4. Sorting the eigenvalues from large to small, selecting the k largest eigenvectors, and then taking the corresponding k eigenvectors as column vectors to form the eigenvector matrix P.
5. Transforming the data into the new space constructed by the k eigenvectors, i.e., Y=PX.

In this research, the CFS method reduced the dimension of extracted features and PCA reduced the size of top attributes while retaining all of the data information (Mollazade et al., 2012b). In this research, the scikit-learn open-source library was utilized for the implementation of machine learning algorithms. Also, 75% of the data were used during the model building step including training and validation (375 samples), and the remaining data (125 samples) were selected as test set for final evaluation.

Classification

— Linear discriminant analysis (LDA). LDA was used to classify N. sativa and its impurities. LDA is a statistical method utilized in machine learning and pattern recognition to find the linear combination of properties that best distinguish two or more classes of objects. LDA maximizes the ratio of between-group variance and within-group variance. When the value of between-group ratio is at its maximum, then the samples within each group have the smallest possible scatter (Stanimirova et al., 2013). The purpose of this method is mapping the features, so the different classes are further separated based on the combination of raw input features. To make a prediction, LDA estimates the probability that a new set of inputs belongs to every class. The output class is the one that has the highest probability.

— Support vector machines (SVM). Support vector machines are commonly used as one of the classification methods. SVM showed very good performance in various classification problems. The basis of SVM is the linear classification of data. SVM maps the input vector to a higher dimensional space using a kernel function and then finds a hyperplane in N-dimensional space (N is the number of features) that classifies the data. To separate the two classes of data points, there are many possibilities. A plane that has the maximum distance between data points of classes (maximum margin) is selected as a separating hyperplane. The following steps were implemented to select the most appropriate SVM model:

Several SVM models were developed with different kernel functions (linear, quadratic, polynomial, and radial basis), and SVM with a polynomial kernel was selected as the superior. Finally, the best model was selected based on the highest accuracy in the test phase while having the minimum number of support vectors. SVM with the minimum number of support vectors will increase processing speed and make the algorithm more suitable for use in the online mode.

— Artificial neural network (ANN). In this work, multilayer perceptron (MLP) neural networks were used for predicting the outputs. MLP model has three main layers: the input layer, the hidden layers, and the output layer. The input layer acts as a receiver and contains the input features to be fed into the neural model. Hidden layers compute the data undergoing iterations to reduce the overlap between patterns. The final layer predicts the output. The number of neurons in the output layer depends on the application. If the data were complex, higher-order features and a suitable ANN structure must be designed. To find the optimum structure of the ANN, the following steps were performed.

Various structures were evaluated in terms of the number of neurons and hidden layers. Thus, the network was evaluated in two modes, a single hidden layer, and a double hidden layer. The number of neurons with a single hidden layer was selected as 4, 8, 10... 16, and the number of neurons in the two hidden layers were selected as (4, 4), (8, 8), (10, 10), ..., (20,20), (6,8) and (8,10). In this research, two different activation functions were tested in the hidden layers including hyperbolic tangent and logistic activation functions. Also, the network was tested with three powerful optimizers, namely, adaptive moment estimation (Adam), stochastic gradient descent (SGD), and limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS). The Adam solver performs very well in large datasets (with thousands of training samples) in terms of training time and validation score. However, for small datasets, L-BFGS can converge faster. In the last step, the structures were repeated several times and the accuracy parameter was used to evaluate the results. The tests showed that logistic (sigmoid) activation functions and the Adam optimizer enriched the results. Finally, the accuracy index of LDA, SVM, and ANN were compared and the best classifier was selected for the online separation of N. sativa.

Performance evaluation

To evaluate the performance of the proposed method, three statistical indicators were used in the classification procedure. These parameters are sensitivity, specificity and accuracy. These statistical parameters were calculated from the following equations:

where TP is the impurity correctly detected in the impurity class; TN is the N. sativa correctly detected; FP is N. sativa that has been misdiagnosed as impurity; and FN is the impurity that has been misidentified as N. sativa.

RESULTS AND DISCUSSION

An Arduino board was used to send the commands to separator, and the Python programming language was used to develop the image processing algorithm and ANN classification. The flexibility of Python programming and the power of Arduino hardware provided a high-quality platform for the implementation of grain automatic cleaning projects.

The proposed lighting system was very effective in reducing object shadows which is one of the main problems in previous research (Quan et al., 2018). There is also no need for shadow removal steps in image processing algorithms, which may have influenced on the results of other investigations (Mollazade et al., 2012a; Pearson et al., 2012; Teimouri et al., 2014). As shown in Fig. 2, the shadow around the seeds was greatly reduced. In the capturing of yellow flax, some light was reflected due to the smooth surface of the grain; this reflection caused that some areas of yellow flax were considered as the background in the segmentation. The images were segmented in RGB color space with thresholding on R, G and B channels. Two thresholding methods, IBT and IZT, with a constant threshold value of 90, were utilized. Figure 3 indicates that the best result for both approaches were related to thresholding on channel B. IZT is one of the most popular segmentation methods and other researchers have utilized this approach in similar studies (Rad et al., 2011).

At the feature extraction and selection stages, CFS selected 25 features from a total of 106 features as the superior features. The results are shown in Table 4. The CFS selected four features from morphology, 17 features from color, and four features from texture data. The CFS has also been used in similar investigations for feature selection (Banakar et al., 2017). Selecting the optimal number of PCs in PCA is important to compress the size of the features. The first five components were selected as optimal PCs for keeping more than 95% of the input information. Therefore, the execution of machine learning algorithms could accelerate. Figure 4 shows a two-dimensional data map. It is indicated that the classes are successfully separated using only two PCs. Figure 4 shows that the N. sativa (red dots) was well distinguished from its impurities. However, the separation of soil particles from other impurities was relatively low. This may be due to the uniform color and texture of the N. sativa compared to other classes.

Figure 4. Visitation of Nigella sativa and its impurities data using the first two components of PCA. NS: Nigella sativa, FB: brown seed, YF: yellow flax, PS: soil particles, GS: grey seed.

The first five PCs were given as inputs to the classifiers (LDA, SVM and ANN). The accuracy index in the offline mode for LDA, SVM and ANN were 71.8%, 88%, and 100%, respectively. According to the results, ANNs showed better performance than the other models. The LDA modeling provides an initial features transformation to compute the simple univariate variables that give maximum group separation in the transformed data (Wakholi et al., 2018). This approach could be precisely used in linear solutions. Also, SVM performance is attributable to the use of the kernel function, which maps the information to a higher dimensional space, which makes the SVM a feasible choice on non-linear nature of problem. SVM requires a lower number of input data as compared to ANN due to the size of the structure. The ANN structure employs multi-layer connection and activation for dealing with non-linear problems and computing complex patterns for describing the data. ANN could identify the seeds and quality using machine vision systems (Granitto et al., 2005; Rahman & Cho, 2016; Madhavan et al., 2020).

Therefore, ANN was used as a classifier in the online phase. These results are in line with Teimouri et al. (2018). The accuracy parameter was calculated separately for the training data and test data to avoid over-fitting. If the ANN performance is very good for the training data and poor for the test data, this means that overfitting occurs. The best structure of ANN was 5-16-5, with an accuracy of 100% for offline test data. The same ANN structure was used for the online mode and the results are as follows: To evaluate the developed device’s performance, 50 new grains of N. sativa samples and each of the impurities were randomly selected and evaluated. The confusion matrix for N. sativa and its impurities is shown in Table 5. Compared to results of offline mode, the errors of correct detection rates were increased in online mode, due to the movement of the grains and the very small size of the N. sativa and its impurities. The sensitivity, accuracy, and specificity of each class are shown in Table 6. The proposed model detects the impurities (other seeds) with high accuracy. This proves that the lightning structure could effectively reduce the shadows and distinguish the objects in the image. Results of ANN showed that the proposed model precisely detected the N. sativa class. The accuracy of ANN associated with feature selection for N. sativa samples was 97.04% in online mode.

The ANN model was successfully utilized for grading of seeds. Rad et al. (2011) classified five rice cultivars by ANN with 97% accuracy. Ebrahimi et al. (2014) detected wheat grains, cultivars and seeds using machine vision techniques and ANN algorithms with an accuracy of 96.25%, 87.5%, and 77.22%, respectively. Qadri et al. (2021) classified eight canola seed varieties using a multi-featured machine vision approach, including first-order histogram, second-order statistical texture, binary and spectral features from the bulk images of canola seeds; the accuracy of the ANN classifier for the eight varieties was 95-98%.

Due to the long response time of the servomotor, the distance between the samples on the conveyor was increased; therefore, the limiting factors of the system performance were related to the hardware components of the separator. Similar results were obtained in the fabrication of a machine for peeled pistachios with machine vision by Nouri-Ahmadabadiet al. (2017).

As a result of the first detection of N. sativa contaminations by ANN, the prediction accuracy could also constitute an objective tool for non-destructive and rapid quality evaluation with the preparation of automation of the cleaning process to remove the impurities. Moreover, the proposed method could lead to the development of automated systems for real-time cleaning of other similar products.

In summary, the present study showed that the combination of machine vision and artificial neural networks (ANN) could be effective for separation of N. sativa seeds from impurities. The overall accuracy of the suggested ANN model for the online classification of the N. sativa class was 97.04%. The impurities were distinguished with average sensitivity of 95.57%. The proposed algorithm accurately separated the N. sativa class from its impurity. For industrial applications, the number of grain flow lines could be increased to reach an acceptable capacity. In addition, high-speed pneumatic or electromechanical actuators could be used for adaptation of the suggested approach in industrial applications. The developed methodology can also be used for non-destructive separation of other similar seeds.