Artificial Neural Networks
by Andrea Kulakov
Abstract
The general problem of data management in Wireless Sensor Networks (WSNs) is to provide efficient aggregation of different sensor’s data taking into account the problems of the limited energy of the nodes and their unpredictable failures. Generally, this is solved by reducing the communication among nodes. In order to have an efficient data aggregation performance, a preprocessing is needed which would reduce the amount of data being sent over the communication channels.
As an outcome of this research, we propose two similar architectures for data aggregation of sound and video signals. These classification architectures have the same core consisted of a modified FuzzyART neural network and a modified SEQUITUR algorithm used previously only for analysis of symbolic sequences. The proposed architectures have been tested in a prototype implementation using Pocket PCs having microphones and cameras as sensors.
Furthermore, in the pursuit for efficient data management methods in wireless sensor networks, this research has tackled the problem of pattern recognition over time in general. In our belief, the central parts of the proposed architectures can represent a core of a general learning system for any signal patterns that spread over time. This general learning system can be viewed as an extension to all Artificial Neural Networks proposed so far, which only dealt with static patterns, and also it can be viewed as an alternative to the Recurrent Neural Networks.
1 Introduction
A motivation for this research was the fact that a fully centralized data collection approach in Wireless Sensor Networks (WSNs) is inefficient given that sensor data are with high dimensionality and have significant redundancy over time, over space and over different types of sensor inputs, which is all due to the nature of the phenomena being sensed. In such cases when the application demands detecting correlations and finding similar patterns, the use of an Artificial Neural-Network (ANN) algorithm seems a reasonable choice for the problem of data management in WSNs: to provide efficient aggregation of different sensor’s data taking into account the limited energy problem and nodes failures. Usually, this is solved by reducing the communication among nodes by some sort of preprocessing including compression or in our case classification.
A centralized method where data are collected from sensors in a predetermined way and transmitted to a server for storage and later querying is not appropriate for query processing in sensor networks because valuable resources are used to transfer large quantities of raw data to the central system, much of which is often redundant. Actually, sensor networks must save energy in order to extend the lifetime of the sensor nodes, because they are generally powered by batterieswith a small capacity. It is known that wireless communication is much more expensive than data processing. Instead of transmitting all the data to a central node, part of the processing can be distributed from the central server into the sensor networks, thus decreasing power dissipation.
Up until recently, the only application of neural-networks algorithms for data processing in the field of sensor networks was the work of Catterall et al. [1], in which the authors have slightly modified the Kohonen Self Organizing Maps model in a rather rudimentary approach. Recently, we have proposed the adaptation of another types of neural-networks combined with wavelet preprocessing, which has been accepted by the wider scientific community as an interesting and promising endeavor ([2], [3], [4], [5], [6], [7] and [8]).
We have adopted one model of an ANN called FuzzyART [9] deriving from Adaptive Resonance Theory (ART) models ([10] and [11]). The inputs from the sensors in the general scheme are first preprocessed in few layers using either a Discrete Wavelet Transform or Fast Fourier Transform. Then the FuzzyART network classifies the analog input data and the classification IDs are used as symbol inputs to a modified version of the so called SEQUITUR algorithm [12] which is used for analysis of a sequence of signal patterns. The SEQUITUR algorithm generates so called rules out of the reappearance of the symbol patterns in a sequence.
The modifications of the SEQUITUR algorithm include calculation of an activation function for each symbol and rule obtained from the SEQUITUR algorithm, which is later used for correct recognition of the signal pattern over time, turning the whole SEQUITUR data structure into one big evolving neural network. For that purpose, for each rule its level of abstractness is calculated and its length in number of symbols at the lowest level is updated constantly. To some selected rules, we attach a so called annotation label which is sent over the communication channel whenever some rule is recognized from the signal input or whenever the level of activation of some rule exceeds a certain high threshold.
While proposing two similar architectures for data aggregation of sound and video signals, we believe that the core of these architectures, consisted of a modified ART neural network and a modified SEQUITUR algorithm, represents a core of a general learning system for any signal patterns that spread over time. This general learning system can be viewed as an extension to all ANNs proposed so far, which only dealt with static patterns, and also it can be viewed as an alternative to the Recurrent Neural Network models.
The idea behind this research was that the most compressed information sent over the communication channel would be if we could recognize the whole situation at hand and send only a description of that situation. For example, for sound or speech sensory input it would be to actually recognize the sounds or words and send only the letters of those words, or in the case of a video signal it’s when we could recognize the objects in front of the camera and send only a textual description of those objects, or trigger an alarm which would turn on a live video transmission etc.
This research has not succeeded to build such an intelligent data aggregation system, but it was not meant to produce such significant results anyway. Still, what was achieved is a step toward that kind of surveillance systems.
2 A general learning system
Most of the ANNs are created for pattern recognition of static inputs. Exceptions are the so-called Recurrent Neural Networks, where the current outputs are fed back to the input layer where they are combined with the future inputs. Generally, their output is function not only oftheir current inputs, but also they are function of all of their past inputs. Still the number of the recurrent neurons determines the memory capacity of the whole neural network.
As an alternative, we have used a FuzzyART neural network, which classifies the input signals with certain granularity of the input space and later analyze these classification IDs, as if they were symbols fed into the SEQUITUR algorithm specialized for sequence analysis.
Figure 1. A general learning system
In what follows we will briefly describe the classical algorithms of FuzzyART and SEQUITUR and later explain the modifications that we have made in order to prepare them for use in a broader scheme.
2.1 FuzzyART
ART has been developed for pattern recognition primarily. FuzzyART is a model of unsupervised learning for analog input patterns. Generally, ART networks develop stable recognition codes by self-organization in response to arbitrary sequences of input patterns. They were designed to solve the so-called stability-plasticity dilemma: how to continue to learn from new events without forgetting previously learned information. ART networks model several features such as robustness to variations in intensity, detection of signals mixed with noise, and both short- and long-term memory to accommodate variable rates of change in the environment.
Figure 2. Architecture of the ART network.
In Figure 2 a typical diagram of an ART Artificial Neural Network is given. L2 is the layer of category nodes. If the degree of category match at the L1 layer is lower than the sensitivity threshold Θ, originally called vigilance level, a reset signal will be triggered, which will deactivate the current winning L2 node that has maximum incoming activation, for the period of presentation of the current input.
An ART network is built up of three layers: the input layer (L0), the comparison layer (L1) and the recognition layer (L2). The input and the comparison layers have N neurons, while the recognition layer is not limited in the number of neurons. The input layer stores the input pattern, and each neuron in the input layer is connected to its corresponding node in the comparison layer via one-to-one, non-modifiable links. Nodes in the L2 layer represent categories into which the inputs have been classified so far. The L1 and L2 layers interact with each other through weighted bottom-up and top-down connections that are modified when the network learns. There are additional gain control signals in the network, which are not shown in Figure 2 but regulate its operation and they will not be detailed here. The learning process of the network can be described as follows: At each presentation of a non-zero analog input pattern p (p j ∈[0, 1]; j = 1, 2, …, N ), the network attempts to classify it into one of its existing categories based on its similarity to the stored prototype of each category node. More precisely, for each node i in the L2 layer, the bottom-up activation A i is calculated, which can be expressed as
==+=N
j i
N j i i p A 11w w
ε for i = 1, …, M
(1) where w i is the weight vector or prototype of category i (w i j ∈[0, 1]; i ∈ [0, ∝); j ∈ [1, N ], and
ε > 0 is a parameter. Then the L2 node C that has the highest bottom-up activation, i.e. A C = max{A i | i = 1, …, M }, is selected. The weight vector of the winning node (w C ) will then be compared to the current input at the comparison layer. If they are similar enough, i.e. if they satisfy the matching condition:
N j N j C ≥ ==1
1
p
p w
(2)
where Θ is the sensitivity threshold (0 < Θ ≤ 1), then the L2 node C will capture the current input and the network learns by modifying w C :
old C old C new C w )1()p w (w γγ−+= (3)
where γ is the learning rate (0 < γ ≤ 1). All other weights in the network remain unchanged. The case when γ =1 is called “fast learning” mode. If, however, the stored prototype w C does not match the input sufficiently, i.e. if the condition (2) is not met, the winning L2 node will be reset for the period of presentation of the current input. Then another L2 node (or category) is selected with the highest A i , whose prototype will be matched against the input, and so on. This “hypothesis-testing” cycle is repeated until the network either finds a stored category whose prototype matches the input well enough, or allocates a new L2 node, in which case learning takes place according to (3). We have modified the ART cycle of testing and learning in a way that it is not necessary to load a certain set of input patterns on which the learning will take place, but rather it is done after each new signal pattern has been given to the inputs.
One of the limitations of the ART neural networks is the fixed number of inputs to the network. The problem is twofold: there is no possibility to add new input signals to the total signal vector and secondly, there is no possibility to have some inputs undefined for certain input patterns. We have solved the second limitation by letting some of the inputs to have an undefined value and these inputs are left out of the calculations in the equations (1) to (3).
2.2 SEQUITUR
SEQUITUR [12] is an algorithm that infers a hierarchical structure from a sequence of discrete symbols by replacing recurring groups of symbols with a rule, and continuing this process recursively while treating the rules as another symbols. The result is a hierarchical representation of the original sequence. The algorithm is driven by two constraints that reduce the size of system of rules, also called the grammar, and produce structure as a by-product.
These constraints are:
p1: no pair of adjacent symbols appears more than once in the whole SEQUITUR system; and p2: every rule is used more than once.
At the left of Figure 3a is shown a sequence that contains the repeating string bc called digram. To compress it, SEQUITUR forms a new rule A
abcdbc to replace both halves of the sequence. Further gains can be made by forming rule BThe constraint p1 requires that every digram in the system of rules should be unique, and will be referred to as digram uniqueness. The constraint p2 ensures that each rule is useful, and will be called rule utility. These two constraints exactly characterize the system of rules that SEQUITUR generates.
Figure 3c shows what happens when these properties are violated. The first SEQUITUR system of rules contains two occurrences of bc, so p1 does not hold. This introduces redundancy because bc appears twice. In the second system of rules, the rule B is used only once, so p2 does not hold. If it were removed, the system of rules would become more concise. The systems of rules depicted in Figures 3a and 3b are the only ones for which both properties hold for each sequence. However, there is a possibility that different system of rules can be built that satisfy both constraints, as in Figure 3d, which is dependent on the history of sequences upon which the system of rules was constructed. Repetitions cannot overlap, so the sequence aaa does not give rise to any rule, despite containing two digrams aa.
The algorithm operates by enforcing the constraints on the system of rules: when the digram uniqueness constraint is violated, a new rule is formed, and when the rule utility constraint is violated, the useless rule is deleted. Details about the operation of the SEQUITUR algorithm can be found in [12].
Ultimately what is formed out of the SEQUITUR’s operation on some sequence can be depicted as in Figure 4, where the rules are represented with circles while the symbols are represented with squares. Although the real structure that is formed is actually a network since a rule can be reused many times as a sub-rule to other rules. Anyway, the only limitation in the number of rules that can be formed is the capacity of the memory itself, which is not a major concern.
Figure 4. Hierarchical structures in SEQUITUR consisted of rules and symbols describing some
sequence of symbols
It should be mentioned that SEQUITUR is successfully used for compression of long sequences such as DNA genomes, and that it can be used reversibly to get the original sequence.
What we have modified in the original SEQUITUR algorithm is that we have added several new properties to rules and symbols, like the level of abstractness of the rule, or total number of symbols at the bottom level of the rule. We have also included level of activation to each rule and symbol and an activation spreading mechanism described next.
2.3 The general learning system revisited
Since the number of different categories into which the FuzzyART module classifies the sensory inputs tends to saturate, it is a finite number depending on the sensitivity threshold. The idea behind this architecture is that the classification identification numbers from the FuzzyART module can be treated as symbols and entered into the SEQUITUR algorithm in order to analyze their pattern over time. With the activation spreading mechanism added to the SEQUITUR rules and symbols, the SEQUITUR rules can be considered an ensemble of evolving neural networks specialized to activate on certain sensory stimuli.
The winning category node from the FuzzyART module is added as the next symbol in the sequence of incoming symbols in the SEQUITUR module. The symbols at the bottom level of the hierarchy in SEQUITUR receive different level of activation from the corresponding category nodes of the FuzzyART module – the winning category casts a maximal activation to the layer of symbols at SEQUITUR, but the rest of the FuzzyART categories also transmit some activation to the symbols at SEQUITUR. This mechanism provides the necessary flexibility during the recognition process - which rule has gained the maximum activation, because similar categories are given similar activations in the FuzzyART module.
Figure 5. The category nodes of the FuzzyART module serve as input symbols to the SEQUITUR module in this unsupervised version of a general learning system
The purpose of the spreading of the activation is to determine the relevance of each particular piece of knowledge (in our case the rules), bringing relevant ones into the working memory [13]. The associative mechanism used in our architecture is a modified version of the Grossberg activation function [14]. Activation level of all rules is always non-negative and is bounded by some maximum value M . There is a threshold value π t hat clips small activation levels to zero. If we neglect the threshold for the moment and if we discretize the function using some small time step ∆t , the activation value a(t) of any single rule is governed by the following equation:
()()()()()()[]{}t t a M )t (net E t a L d t a t t a a t a ∆⋅−⋅⋅+⋅−−+=∆+=λ00 (4)
where a(t) is the activation level as a function of time, net(t) is the net input to the node, M = const is the maximal activation value, d and E are parameters that control the rate of decay and excitation respectively, and the decay of the activation linearly depends on the level of abstractness of the rule L , modified by some constant λ.
Whenever the activation falls bellow some predefined minimal value π, it is momentarily forced to zero. On the other hand, when the activation of a node is zero and the activation net input is bigger than some critical value n π, the activation level of the node jumps instantaneously to the threshold level π. This critical value n π is determined from the following equation:
()L M E d n πλππ
π−−⋅= (5)
From the last equation we can notice that the critical value of more abstract rules (bigger L ) is linearly smaller than that of more concrete rules (λπ is a small constant). Hence abstract rules are harder to deactivate but also become easier to activate. In the calculation of the activation that is propagated from each rule, the rule length, expressed in number of symbols at the bottom underlying level, is used in order to calculate the fan-in effect from each symbol or rule toward the super-rule encompassing them. Also, the fan-out effect is calculated for each rule, as the inverse value of the number of times each rule is used in other super-rules. The first rule, denoted in Figure 3 as S, represents the history of all symbols entered into SEQUITUR. Since it is not psychologically plausible to have a memory of all signal representations, it is shortened from the beginning after certain number of signal inputs were sensed. As such, it can be viewed as a short-term memory, which can be kept with constant length. In Figure 6 is shown a supervised version of the general learning system in which there is a layer of nodes with annotations, separate from the rules. For each different input from the teacher, a new node with annotation is created which is linked to the rules and symbols, which appeared in the last continuous sensing activity. The links are weighted according to the proportional participation of each corresponding rule or symbol in the overall length of this last continuous sensing activity.
Figure 6. A supervised version of the general learning system
A continuous sensing activity is defined differently for sound processing and for video processing. A continuous sensing activity for sound processing spans from the beginning of the detection of non-zero input signals until it ceases back to zero input signals. While in case for
Whenever some rule is expanded, i.e. deleted from the SEQUITUR rules, it is checked whether it has links to some nodes with annotations, and in that case, new links are created among these nodes and all subordinate rules and symbols. The weights of the original links toward the nodes with annotations are proportionately divided among these subordinate rules and symbols according to their length. Probably the addition of inhibitory links among the nodes with annotations would help the winning node to gain maximal activity, but that remains to be checked in future research.
3 System for sound processing
The general learning system can be easily adopted for sound processing applications. As can be seen from Figure 7, the raw input from the microphone is preprocessed using the Fast Fourier Transform (FFT) and then the output frequencies are logarithmically transformed using a variation of the Mel frequency transform [15]:
M( f ) =1125log10 (1 + f / 700) (6)
Many experiments have shown that the ear’s perception to the frequency components in the speech does not follow the linear scale but the Mel-frequency scale, which should be understood as linear frequency spacing below 1kHz and logarithmic spacing above 1kHz.
After that, a FuzzyART neural network classifies initially the current frequency response of the sound input and the result of that classification is the identification number of a certain category. This number is used as an input symbol for the SEQUITUR algorithm, which analyses the stream of such symbols.
Figure 7. Possible application of the general learning system in a sound processing application The result of the analysis with SEQUITUR is not given until the continuous sensing activity ceases back to the resting state. This is because there is no meaning to give as an output a rule, which can also be replaced by some other super-ordinate rule, as a result of the subsequent input symbols to SEQUITUR.
Finally this stream of output rules or symbols from the SEQUITUR algorithm, gathered in the last continuous sensing activity is sent over the transmission channel to the Clusterhead collecting such compressed sound information also from other nodes in the same cluster of the WSN.
In Figure 8a is given a screenshot from the sample sound processing application written in Embedded C++ for Pocket-PCs. In the lower black rectangle is given the Frequency response of the sound input where vertically are given different frequencies while with the color is shown the intensity of each frequency. Time is represented horizontally. Three continuous sensing activities can be easily noticed in the same figure and are highlighted in Figure 8b. These continuous periods of sensing activity are calculated using the moving average of the frequency response. In that way, the system can adopt to environments with different levels of noise.
a b
Figure 8. a) Screenshot from the sample application for sound processing with a Pocket-PC.
b) Enlarged view of the spectrogram of the sound input where the periods of continuous sensing
activities are circumscribed
4 System for video processing
We have adopted the Behavioral Model of Visual Perception (BMVP) [16] where among other things we have replaced the sensory memory with a FuzzyART neural network instead of Hopfield neural network, and instead of fixed motor memory we have used the SEQUITUR module (see Figure 9). BMVP develops representations about the visual objects based on the responses from a fixed number of edge detecting sensors during saccadic movements of a small Attention Window (AW). Instead of using these edge-detecting sensors as inputs to the sensory memory like in BMVP, we have experimentally deduced that using oriented Gabor wavelet responses yields better and faster results. Another difference with BMVP is that it was primarily used for recognition of fixed images, while we have used it on life video captured sequences.
Figure 9. A sample architecture for video processing having the general learning system in its
coreIn Figure 9, the thicker arrows denote the information flow, while thinner arrows denote control flow. The movement detection also controls the shift of the AW and we have found out that, unlike in BMVP, there is no need for human intervened determination of so called “interesting” zones around the eyes in some picture, since the blinks and other movements of the head make them interesting by only including the movement detection to influence the decision about the shift of AW, i.e. about the position of the next focus.
Figure 10, taken from [16], explains the content of one Attention Window (AW). The relative orientation of each context point (ϕ) is calculated as a difference between the absolute angle of the edge at the center of the AW (ϕ0) and the absolute angle of the edge at that context point (ϕc). This relative orientation is used to get the oriented Gabor wavelet response at that context point in a small window. This response is then used as an input to the FuzzyART neural network, which plays the role of a sensory memory.
Figure 10. Schematic of the Attention Window (AW). The next possible focal points are located at the intersections of sixteen radiating lines and three concentric circles. XOY is the absolute coordinate system. The relative coordinate system X1OY1 is attached to the basic edge at the center of the AW. The absolute parameters of the edge at one possible next focal point, ϕc and ψc, are shown as well as its relative parameters, ϕ and ψ.
The SEQUITUR is used as a motor memory by providing alternating inputs once from the sensory memory and once from the vector selector that determines the next focal point of the AW. As can be seen from the Figure 10, there are 48 different possible saccadic movements. These are represented by 48 different symbols, which are entered as input symbols to the SEQUITUR module. The SEQUITUR rules in this architecture are of form Percept-Saccade-Percept-Saccade-…-Percept, taken from the FuzzyART and from the “Shift of AW” modules.
The relative calculation of the orientation of the edges at the next focal points, according to the orientation of the edge at the current focus of AW, gives rise to the possibility for recognition of objects independent of orientation. The relative calculation of the distance between the current and the next focal point allows recognition of objects independent of size.
The selection of the next saccade is relatively simply solved and can be further improved. Yet the results, as discussed later, are promising. In Figure 11 there are two examples of the saccadic movements over two different images.
Figure 11. Two examples of the saccadic movements shown with red-white lines which are oriented as the approximately detected edges at these points. Eighty saccadic movements are made over one video sequence. The small yellow circle in the middle shows the estimated center of the movement activity.
The estimated center of the movement activity is used as an influence towards which the saccade jumps at each new video sequence and also when the saccades tend to exit outside the image frontiers.
5 Communication among the nodes in the prototype WSN
The communication among the Pocket-PCs used in our prototype WSN is built upon the available Microsoft SDK for such applications, found at the MSDN Library. This software platform is called MultiCommFramework [17] and allows an easy inclusion of communication capabilities to applications developed for Pocket-PC devices. The MultiCommFramework utilizes connectionless UDP datagrams to avoid the problems associated with trying to maintain a connection-oriented protocol in the loosely connected environment of WiFi. All information is sent via UDP datagrams, because connection-oriented protocols such as TCP do not work well if network connectivity is intermittent as is often the case with wireless networks. There should be one listener application active in the network and several client applications have to be registered at the listener for a conversation. Then the UDP datagrams containing the passed data can be sent to the listener. Because datagrams are not reliably delivered (although they rarely fail unless there are network connectivity problems), the client watches for the listener to acknowledge receipt of the message. If not acknowledged, the client will resend the message up once more (which can be increased up to 3 times if necessary, depending on the network connectivity problems).
6 Experiments
We have built a small testing environment in order to try the functionality of data management strategy discussed so far.
6.1 Experimental setup
We have used 5 hp iPAQ h4000 series Pocket-PCs equipped with a built-in microphone and wireless network card and an add-on camera as nodes in this small prototype WSN. As a Clusterhead we have used a laptop PC with a built in wireless network card. All of them were first connected to the same network through the wireless access point which can be seen in the upper middle part of Figure 12a, mounted at the wall.
At the beginning of the experiment, all Pocket-PCs and the Clusterhead laptop PC were connected successfully to the same network and registered to the listener started at the laptop PC.
a b
Figure 12. a) The full experimental setup situated in our multimedia and video-conferencing lab, including 5 Pocket-PCs equipped with cameras (highlighted with yellow circles), a wireless access point and a laptop PC acting as a Clusterhead in this prototype WSN; b) closer look of two
Pocket-PCs equipped with cameras.
6.2 Results from the sound processing application
The current level of development of the sound processing application can be used only as a proof of existence for the possibility of the general learning system to recognize sound or even speech of any language. We believe that further development would result in much more reliable recognition rates. Also, the application needs to be trained and tested over a larger corpus of different sound and speech samples.
At this stage, besides the use for compression which is discussed further, the only possibility to use the sound processing application in a real task could be as a detector of different sounds, for example in natural world observations, for traffic monitoring or even for preservation of forests from illegal timber cut, where it could be used for detection of specific bird songs, of sounds of the cars passing by, or for detecting the sounds of the motor saws cutting trees.
6.3 Results from the video processing application
We have measured different parameters during several runs of our video processing application. For example, on a single Pocket-PC, the speed of the whole image processing is around 1 image in 4 seconds, while the speed of only capturing raw images with the SDIO cameras for Pocket-PCs is around 2 images per second. Just for comparison and testing purposes we have built a testbed application for a desktop PC and the speed of image processing is around 10 images in 1 second, which is dependent entirely from the speed of the used web camera.
Important information which proves that the system is learning progressively during its functioning is the average length and the average level of the SEQUITUR rules it creates. It shows that the system actually takes saccadic samples from the image and categorizes them in a consistent and consecutive manner.
Since the application deals with images from a video, which further introduces novelty and diversity in means of different objects orientations and shadows that these objects cast, this reduces the possibilities for detecting regularities in the saccadic streams and in that way for creating new SEQUITUR rules.
That’s why, for comparison reasons, we have made a test run where the application has dealt with a static image. The results show that the average and the maximal lengths and levels of
the rules are remarkably bigger in the case of static images, which was expected. These results are shown in Figures 13 and 14, where one iteration involves 80 saccades over one image frame.
The number of confirmed expectations from SEQUITUR rules after each selection of a
shift of the AW was also higher for around 67% in the case of a static image processing over video image processing.
All of this confirms us that, after sufficient number of iterations, the video processing
architecture, built around the general learning system, would be able to create structured, stable, and growingly longer (complete) and abstract recognition codes for objects, regardless of their size or orientation.
Figure 13. Comparison of the maximal and the average lengths of the SEQUITUR rules when
processing video or a static image taken from the same video sequence.
Figure 14. Comparison of the maximal and the average levels of the SEQUITUR rules when
processing video or a static image taken from the same video sequence.
7 Discussion about the compression and cryptographic aspects
This general learning system when utilized in a WSN application can be used as a lossless
or lossy compression method before sending the measurement information over the communication channel.
For a sound processing system, decompression can be obtained by reversing the processes
involved in the transformation of the information. For SEQUITUR the inverse transformation means decomposing the rules back to the constituting symbols, which can be done easily by traversing the rule-trees from left to right in a recursive manner. For the Mel Frequency transform and for the Fast Fourier Transform there are easily computable inverse transformations.
The only problematic transform never discussed before is the inverse transform of the
FuzzyART neural network, seen as a transformation from the input space [0, 1]N , where N is the number of inputs, to the space of integers i ∈ [0, ∝). In the case of a lossy compression, the weight vector of the category nodes (w i ) also called a prototype of category i , as defined before, can be used as the inverse transform of the symbol i. This can be explained as if we take away the
In the case for lossless compression, we need a vector of the original input values to be remembered along with the creation of each new category node. Here we does not need to use the full supervised version of the FuzzyART neural network, called FuzzyARTMAP [18], but simply to attach the original input vectors to the category nodes, a possibility mentioned in [18] as well.
For a video processing system, only a lossy compression can be obtained, since we have created a processing system, which does not deal with the whole image taken from a video sequence, but only with a small fragment of it, considered interesting for processing. What could be obtained after inversing the transformation processes would be similar to the images obtained after similar transform with wavelet neural networks, as shown in Figure 15, taken from [19]. Modifying the number of saccades per image, as well as the number of wavelet decompositions on each saccade, we would obtain different quality of the decompressed video stream at the receiver.
Figure 15. The image (at the farmost right) represented with increasing number of Gabor
wavelets.
Concerning the cryptographic aspects of our signal processing architectures, there are two possible scenarios. In the first scenario, after each creation of a new ART category or rule node their definitions can be sent to the Clusterhead together with the original measured signals. In that way any intruder in the communication lines has to know all the prehistory of the communications in order to be able to decipher this evolving code, which is also a unique code liable only to that ongoing communication.
In the second scenario, all learned ART categories and SEQUITUR rules could be transferred off-line to all nodes in the WSN, and after that, each client nodes should only detect but not learn anymore, which could be justified only after a long and extensive training.
8 Conclusion
In order to have an efficient data aggregation performance, a preprocessing is needed which would reduce the amount of data being sent over the communication channels. The most compressed information sent over the communication channel would be if the system could actually recognize the whole situation at hand and send only a description of that situation.
Two similar architectures for data aggregation of sound and video signals were proposed having the same core, consisted of a modified FuzzyART neural network and a modified SEQUITUR algorithm. The proposed architectures have been tested in a prototype implementation using Pocket PCs having microphones and cameras as sensors.
The modifications of the SEQUITUR algorithm include calculation of an activation function for each symbol and rule obtained from the SEQUITUR algorithm turning the whole SEQUITUR data structure into one big evolving neural network with possibilities for both supervised and unsupervised learning.
The core of these signal-processing architectures represents a general learning system for any signal patterns that spread over time. This general learning system can be viewed as anextension to all Artificial Neural Networks proposed so far, which only dealt with static patterns, and also it can be viewed as an alternative to the Recurrent Neural Network models.
Although at present the units used in Wireless Sensor Networks have to a great extent less processing power than the current Pocket-PCs, soon this gap will be overcome and we hope that the prototype implementations developed during this research would have an impact toward the promotion of the Artificial Neural Networks into the field of Wireless Sensor Networks.
9 References
[1] Catterall, E., Van Laerhoven, K., and Strohbach, M., Self-Organization in Ad Hoc Sensor Networks: An
Empirical Study, in Proceedings of Artificial Life VIII, The 8th International Conference on the Simulation and Synthesis of Living Systems, University of New South Wales, Sydney, NSW, Australia, 9-13 December, 2002.
[2] Kulakov, A., Davcev, D., Tracking of unusual events in wireless sensor networks based on artificial neural-
networks algorithms, in Proceedings of International Conference on Information Technology: Coding and Computing, ITCC 2005, Las Vegas, NV, USA, April 4-6, 2005
[3] Kulakov, A., Davcev, D., Trajkovski, G., Implementing artificial neural-networks in wireless sensor networks,
in Proceedings of IEEE Sarnoff Symposium on Advances in Wired and Wireless Communications, Princeton, NJ, USA, April 18-19, 2005
[4] Kulakov, A., Davcev, D., Data mining in wireless sensor networks based on artificial neural-networks
algorithms, in Proceedings of Workshop on Data Mining in Sensor Networks in conjunction with 2005 SIAM International Conference on Data-Mining, Newport Beach, CA, USA, April 21-23, 2005
[5] Kulakov, A., Davcev, D., Intelligent data aggregation in sensor networks using artificial neural-networks
algorithms, Poster in Proceedings of Nanotech 2005, Anaheim, CA, U.S.A, May 8-12, 2005
[6] Kulakov, A., Davcev, D., Trajkovski, G., Application of wavelet neural-networks in wireless sensor networks,
in Proceedings of 6th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, Towson, MD, USA, May 23-25, 2005
[7] Kulakov, A., Davcev, D., Distributed data processing in wireless sensor networks based on artificial neural-
networks algorithms, in Proceedings of The 10th IEEE Symposium on Computers and Communications, Cartagena, Spain, June 27-30, 2005
[8] Kulakov, A., Davcev, D., Stojanov, G., Learning patterns in wireless sensor networks based on wavelet neural-
networks, in Proceedings of the First International Workshop on Heterogeneous Wireless Sensor Networks held in conjunction with the 11th IEEE International Conference on Parallel and Distributed Systems, Fukuoka, Japan, July 20 - 22, 2005
[9] Carpenter, G.A., Grossberg, S., and Rosen, D.B., Fuzzy ART: Fast stable learning and categorization of analog
patterns by an adaptive resonance system, Neural Networks, vol. 4, pp. 759-771, 1991.
[10] Grossberg, S. Adaptive pattern classification and universal recoding: I. parallel development and coding of
neural feature detectors, Biological Cybernetics, vol. 23, pp. 121-134, 1976.
[11] Grossberg, S. Adaptive Resonance Theory in Encyclopedia of Cognitive Science, Macmillan Reference, 2000.
[12] Nevill-Manning, C.G., Witten, I.H., Identifying Hierarchical Structure in Sequences: A linear-time algorithm,
Journal of Artificial Intelligence Research, 1997, 7, 67-82.
[13] Anderson, J., The Architecture of Cognition, Harvard University Press, Cambridge, MA, 1983.
[14] Grossberg, S., A Theory of Visual Coding, Memory, and Development, in Leeuwenberg, L., and Buffart, J.
(eds.), “Formal Theories of Visual Comparative Analysis”, Wiley, New York, 1978.
[15] Davis, S. B., Mermelstein, P., Comparison of Parametric Representation for Monosyllabic Word Recognition in
Continuously Spoken Sentences, IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol 28(4), pp. 357-366.
[16] Rybak, I.A., Gusakova, V.I., Golovan, A.V., Podladchikova, L.N., Shevtsova, N.A., A model of attention-
guided visual perception and recognition, Vision Research (38), 1998, pp. 2387–2400
[17] http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnnetcomp/html/MultiCommFramework.asp,
as of 14. Noe. 2005.
[18] Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, J.H., and Rosen, D.B., Fuzzy ARTMAP: A neural
network architecture for incremental supervised learning of analog multidimensional maps, IEEE Transactions on Neural Networks, vol. 3, pp. 698-713, 1992.
[19] Krüger, V., Bruns, S., Sommer, G., Efficient Head Pose Estimation with Gabor Wavelet Networks, in
Proceedings of the 11th British Machine Vision Conference, September, 2000.下载本文
