Soft Computing Fundamentals: Algorithms and Networks

Characteristics of Soft Computing

• Biological Inspiration: Soft computing often draws inspiration from natural processes, such as the human brain (neural networks) and evolution (genetic algorithms).
• Human Expertise: It can incorporate human knowledge and expertise in the form of fuzzy rules or initial model structures.
• Model-Free Learning: Many soft computing methods, like neural networks, can build models directly from data without requiring explicit mathematical formulations.
• Fault Tolerance: Some soft computing systems, like neural networks and fuzzy systems, can continue to function even if parts of the system fail.
• Goal-Driven: Soft computing aims to achieve specific goals, and the path to the solution is less critical than reaching a satisfactory outcome.

Boltzmann Machine

• These are stochastic learning processes having a recurrent structure and are the basis of early optimization techniques used in Artificial Neural Networks (ANN).
• *Stochastic* means randomly determined; having a random probability distribution or pattern that may be analyzed statistically but may not be predicted precisely.
• The main feature of this network is that it uses only locally available information. The change of weight depends only on the behavior of the two units it connects, even though the change optimizes a global measure.

Rough Set Theory

• The notion of Rough Sets was introduced by Z. Pawlak in 1982.
• Rough sets represent a different mathematical approach to vagueness and uncertainty.
• It is a new mathematical method for processing imprecise, uncertain, and incomplete data.
• Hidden knowledge and laws can be uncovered. Rough set theory has been a methodology for database mining or knowledge discovery in relational databases.
• In its abstract form, it is a new area of uncertainty mathematics closely related to fuzzy theory.
• We can use the rough set approach to discover structural relationships within imprecise and noisy data.

Soft Computing: Representing and Reasoning

• Handling Imprecision and Vagueness: Human knowledge is often expressed in natural language, which is inherently imprecise and vague.
• Modeling Uncertainty and Doubt: Human reasoning often involves uncertainty. We might be "somewhat sure" or have a "strong feeling" about something.
• Capturing Heuristics and Rules of Thumb: Human expertise often manifests as a collection of rules of thumb, heuristics, and experiential knowledge that are not always easily formalized into precise mathematical equations.
• Learning from Experience and Adapting: Humans learn from experience and adapt their understanding over time.

Binary Hopfield Network

The Binary Hopfield Network, a recurrent neural network in soft computing, acts as an associative memory. Its neurons have binary states (-1/+1 or 0/1) and are fully interconnected with symmetric weights. Patterns are stored via Hebbian learning. Given a partial or noisy input, the network iteratively updates neuron states based on weighted sums and a threshold, minimizing an energy function. It converges to a stable state, ideally retrieving the closest stored pattern. This content-addressable memory capability, inspired by human memory, demonstrates robustness to noise, a key feature of soft computing.

Importance of Fuzzy Sets

• Handles Imprecision: Models real-world vagueness and linguistic terms.
• Mimics Human Reasoning: Enables systems to reason with approximate information.
• Bridging Symbolic & Numeric: Connects qualitative linguistic descriptions with quantitative values.
• Robust to Noise: Less sensitive to noisy or incomplete data.
• Facilitates Expert Knowledge: Allows direct incorporation of human expertise through fuzzy rules.
• Simplifies Complex Systems: Provides a higher-level, intuitive way to model complex behavior.

Soft Computing and Traditional Computing

Soft Computing

• Tolerance for Imprecision: Designed to handle uncertainty, vagueness, and approximate information.
• Approximate Solutions: Seeks "good enough" or feasible solutions, often more efficient for complex problems.
• Multi-valued Logic: Employs fuzzy logic with degrees of truth between 0 and 1.
• Parallel Processing: Often utilizes parallel computation for efficiency.

Traditional Computing

• Precision and Certainty: Relies on formal logic, precise mathematical models, and deterministic algorithms.
• Exact Solutions: Aims for exact, verifiable solutions based on accurate input data.
• Binary Logic: Operates on strict true/false (0/1) logic.
• Requires Well-Defined Models: Needs a clearly stated analytical model to solve problems.

Neocognitron Neural Networks

The Neocognitron is a hierarchical, multilayered artificial neural network, considered a pioneering model in deep learning and a precursor to modern Convolutional Neural Networks (CNNs). Developed by Kunihiko Fukushima in the late 1970s and early 1980s, its architecture was inspired by the visual cortex of mammals. It's a significant concept in soft computing for its ability to perform robust visual pattern recognition, particularly being invariant to shifts in position, scale, and some degree of distortion of the input patterns. It models aspects of the biological visual system. Its early versions showcased effective learning without labeled data.

Radial Basis Function (RBF) Networks

• In the field of mathematical modeling, a radial basis function network is an artificial neural network that uses radial basis functions as activation functions.
• Since Radial Basis Functions (RBFs) have only one hidden layer, the convergence of the optimization objective is much faster, and despite having one hidden layer, RBFs are proven to be universal approximators.
• An RBF is a supervised, fast-response network. An RBF network is an artificial neural network with an input layer, a hidden layer, and an output layer. The hidden layer of RBF consists of hidden neurons, and the activation function of these neurons is a Gaussian function. The hidden layer generates a signal corresponding to an input vector in the input layer, and corresponding to this signal, the network generates a response.

Restricted Coulomb Energy (RCE) Networks

• A Restricted Coulomb Energy (RCE) network is one of the competitive learning networks that are able to classify input data, together with the Self-Organizing Map (SOM) and Learning Vector Quantization (LVQ).
• In the RCE network, there is no need for setting the number of required neurons before learning because the RCE network automatically creates new neurons to classify input data into correct categories.
• All hidden layers are fully connected with all input layers.

Probabilistic Reasoning

Probabilistic reasoning in soft computing uses probability theory to handle uncertainty inherent in real-world data and knowledge. It allows for degrees of belief rather than absolute truth. Techniques like Bayesian networks and inference model dependencies and update beliefs based on evidence. This enables systems to make informed decisions under uncertainty, learn from noisy data, and model complex, random phenomena. It's crucial in machine learning, AI, and various applications dealing with real-world complexities where precise information is often lacking.

Competitive Learning

Competitive learning is an unsupervised learning technique where neurons in a network compete to become specialized in responding to specific input patterns. The winning neuron, the one that best matches the input, is updated while others remain unchanged. This process leads to a specialization of the network's nodes, allowing them to effectively cluster or categorize data. Competitive learning is crucial for tasks like data clustering, dimensionality reduction, and pattern recognition in scenarios where labeled data is scarce or the underlying structure of the data needs to be discovered.

Chaos Theory

Chaos theory in soft computing leverages the principles of sensitive dependence on initial conditions and complex dynamics from nonlinear systems. It enhances optimization in evolutionary algorithms by promoting exploration and escaping local optima. Chaotic maps aid parameter tuning. It inspires novel neural network architectures with complex behaviors. Furthermore, chaotic dynamics are applied in secure communication, time series analysis, and modeling intricate real-world systems. By introducing controlled "randomness," chaos theory bolsters the problem-solving capabilities of soft computing techniques in complex environments.

Kohonen Maps (Self-Organizing Maps)

Kohonen Maps, also known as Self-Organizing Maps (SOMs), are a type of unsupervised neural network in soft computing used for dimensionality reduction and clustering of high-dimensional data. Developed by Teuvo Kohonen, they aim to project data onto a lower-dimensional (typically 2D) grid while preserving the topological relationships of the input space. The network consists of an input layer and a competitive output layer arranged in a grid. During training, each input data point is compared to all the neurons in the output layer, and the neuron with the closest weight vector (the Best Matching Unit - BMU) is selected. The BMU and its neighboring neurons then have their weight vectors adjusted to be more similar to the input vector.

Supervised and Unsupervised Learning

Supervised Learning

• Supervised learning algorithms are trained using labeled data.
• Supervised learning models take direct feedback to check if they are predicting correct output or not.
• In supervised learning, input data is provided to the model along with the output.
• The goal of supervised learning is to train the model so that it can predict the output when it is given new data.
• Supervised learning can be categorized into Classification and Regression problems.

Unsupervised Learning

• Unsupervised learning algorithms are trained using unlabeled data.
• Unsupervised learning models do not take any feedback.
• In unsupervised learning, only input data is provided to the model.
• The goal of unsupervised learning is to find hidden patterns and useful insights from the unknown dataset.
• Unsupervised learning does not need any supervision to train the model.

Learning Vector Quantization (LVQ)

• Learning Vector Quantization is a competitive network that uses supervised learning. We may define it as a process of classifying patterns where each output unit represents a class.
• As it uses supervised learning, the network will be given a set of training patterns with known classification along with an initial distribution of the output class. After completing the training process, LVQ will classify an input vector by assigning it to the same class as that of the output unit.
• The LVQ algorithm is related to the Self-Organizing Map, which is in turn inspired by the self-organizing capabilities of neurons in the visual cortex.

Bidirectional Associative Memory (BAM)

• Bidirectional Associative Memory (BAM) is a type of recurrent neural network, introduced by Bart Kosko in 1988.
• BAM is based on hetero-associative memory, meaning given a pattern it can return another pattern which is potentially of a different size.
• It is similar to the Hopfield network in that they are both forms of associative memory. However, Hopfield nets return patterns of the same size.
• A BAM contains two layers (input and output) of neurons, which we shall denote X and Y. Layers X and Y are fully connected to each other using a bidirectional path.

Counter Propagation Network (CPN)

• Defined by Robert Hecht-Nielsen in 1986, CPN is a network that learns a bidirectional mapping in hyper-dimensional space.
• CPN learns both forward mapping (from n-space to m-space) and, if it exists, the inverse mapping (from m-space to n-space) for a set of pattern vectors.
• A Counter Propagation Network (CPN) is an unsupervised winner-take-all competitive learning network.

Sparse Distributed Memory (SDM)

Sparse Distributed Memory (SDM) began in 1974 as a paper written for a class on human memory given by Gordon Bower of Stanford's psychology department. Kanerva's SDM can be regarded as a generalized random-access memory wherein the memory addresses and data words come from high-dimensional vector spaces. As in a conventional random-access memory (RAM), there exists an array of storage locations, each identified by a number (the address of the location) with associated data being stored in these locations as binary words.

Optimization Problems

• An optimization problem aims to find the optimal solutions to an objective function under constraints.
• In mathematics, computer science, economics, or management science, optimization is the selection of the best element (with regard to some criteria) from some set of available alternatives.
• In the simplest case, an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function.
• It is a technique for selecting the most feasible solution for a given problem.
• Nature-inspired algorithms such as GSA, PSO, and GA are often used.

Genetic Algorithms

• Genetic Algorithms (GAs) are adaptive heuristic search algorithms based on the evolutionary ideas of natural selection and genetics.
• Genetic algorithms (GAs) are a part of Evolutionary Computing, a rapidly growing area of artificial intelligence. GAs are inspired by Darwin's theory of evolution: *survival of the fittest*.
• GAs represent an intelligent exploitation of a random search used to solve optimization problems.
• GAs, although randomized, exploit historical information to direct the search into the region of better performance within the search space.
• In nature, competition among individuals for scanty resources results in the fittest individuals dominating over the weaker ones.
• GAs solve problems by mimicking natural processes: selection, crossover, mutation, and acceptance, to evolve a solution.
• The GA looks for the best solution among a number of possible solutions represented by one point in the search space.

Applications of Genetic Algorithms

• Optimization: Genetic Algorithms are most commonly used in optimization problems wherein we have to maximize or minimize a given objective function value under a given set of constraints.
• Economics: GAs are also used to characterize various economic models like the cobweb model, game theory equilibrium resolution, asset pricing, etc.
• Neural Networks: GAs are also used to train neural networks, particularly recurrent neural networks.
• Parallelization: GAs also have very good parallel capabilities, and prove to be very effective means in solving certain problems, and also provide a good area for research.

Genetic Programming (GP)

Genetic Programming (GP) in soft computing is an evolutionary technique that automatically evolves computer programs to solve problems. It starts with a population of random programs (tree structures) and iteratively improves them using genetic operators like crossover and mutation, guided by a fitness function that evaluates their performance. GP excels at tasks like symbolic regression, classification, and control, automatically discovering effective algorithms and complex solutions without explicit human design.

Applications of Neural Networks

• Clustering: A clustering algorithm explores the similarity between patterns and places similar patterns in a cluster. Best known applications include data compression and data mining.
• Classification / Pattern Recognition: The task of pattern recognition is to assign an input pattern (like a handwritten symbol) to one of many classes. This category includes algorithmic implementations such as associative memory.
• Function Approximation: The task of function approximation is to find an estimate of an unknown function subject to noise. Various engineering and scientific disciplines require function approximation.
• Prediction Systems: The task is to forecast some future values of time-sequenced data. Prediction has a significant impact on decision support systems.

Stochastic Neural Networks

A stochastic neural network is a type of neural network that incorporates randomness into its architecture and operation, as opposed to deterministic neural networks. This randomness can be introduced in various ways, such as through stochastic weights, transfer functions, or even in the training process itself. The introduction of stochasticity can improve optimization by helping the network escape local minima and potentially improve generalization and robustness.

Delta Learning Rule

Developed by Bernard Widrow and Marcian Hoff, the Delta Learning Rule is a supervised learning rule with a continuous activation function. The main aim of this rule is to minimize error in the training patterns and thus, it is also known as the least mean square method. The principle used here is that of an infinitesimal gradient descent, and the changes in weight are equal to the product of the error and the input.

Perceptron Learning Rule

Developed by Rosenblatt, the Perceptron Learning Rule is an error correction rule, used in a single-layer feedforward network. Like Hebbian learning, this is also a supervised learning rule. This rule works by finding the difference between the actual and desired outputs and adjusts the weights according to that. Naturally, the rule requires a set of input vectors, along with weights, to produce an output.

ADALINE (Adaptive Linear Neural)

• A network with a single linear unit is called ADALINE. In ADALINE, there is only one output unit, and output values are bipolar (+1, -1). Weights between the input unit and output unit are adjustable.
• The learning rule is found to minimize the mean square error between activation and target values. ADALINE consists of trainable weights; it compares the actual output with the calculated output, and an error training algorithm is applied based on this comparison.
• In ADALINE, all input neurons are directly connected to the output neuron with a weighted connected path. There is a bias *b* of activation function 1 present.

Error Backpropagation Algorithm

Error Backpropagation Algorithm works based on a mathematically derived formula. Criteria or conditions must be found to stop this continuous loop. Local minima are intended for this purpose. At each loop, errors are scaled based on their gradient weight. The error with minimum gradient weight, referred to as a local minimum, is chosen and included in readings. Errors cannot be entirely avoided, as they will always arise. Therefore, this algorithm's idea is to choose the minimum gradient weight error to improve the algorithm's efficiency.

Self-Organizing Map (SOM)

• SOMs are unsupervised neural networks that cluster high-dimensional data (e.g., various crop data in a state, clustering millions of different color neurons into a graphical vision).
• They transform complex inputs into easy-to-understand two-dimensional outputs.
• A Self-Organizing Map (SOM), developed by Teuvo Kohonen in 1980, provides a data visualization technique that helps understand high-dimensional data by reducing its dimensions to a map.
• They provide a low-dimensional view of high-dimensional data.
• SOMs also represent the clustering concept by grouping similar data together. Therefore, it can be said that SOMs reduce data dimensions and display similarities among data.