Accurate Dependability Analysis of CAN-based Networked

Accurate Dependability Analysis of CAN-based Networked SystemsJ. Pérez1, M. Sonza Reorda2, M. Violante21Universidad de la República, Facultad de Ingeniería,Instituto de Ingeniería Eléctrica, Montevideo, Uruguay2Politecnico di Torino, Dipartimento di Automatica e Informatica, Torino, ItalyAbstract*Computer-based systems where several nodes exchange information via suitable network interconnections are today exploited in many safety-critical applications, like those belonging to the automotive field. Accurate dependability analysis of such a kind of systems is thus a major concern for designers. In this paper we present an environment we developed in order to assess the effects of faults in CAN-based networks. We developed an IP core implementing the CAN protocol controller, and we exploited it to set-up a network composed of several nodes. Thanks to the approach we adopted we were able to assess via simulation-based fault injection the effects of faults bothin the bus used to carry information and inside each CAN controller, as well. In this paper, we report a detailed description of the environment we set-up and we present some preliminary results we gathered to assess the soundness of the proposed approach.1. IntroductionIn recent years, there has been a rapid increase in the use of computer-based systems in safety-critical applications such as railway traffic control, aircraft flight and telecommunications, where it is quite common to find several computing nodes connected by a proper network sub-system.This trend has led to concerns regarding the validation of the fault tolerance properties of these systems and the evaluation of their reliability. The continuous increase in the integration level of electronic systems is indeed making more difficult than ever to guarantee an acceptable degree of reliability due to the occurrence of soft errors that can dramatically affect the behavior of systems. As an example, the decrease in the * This work has been partially supported by European Commission, through the Alfa II-0086-FI TOSCA Project. Contact author: Massimo Violante, Politecnicodi Torino, Dip. Automatica e Informatica, C.so Duca degli Abruzzi 24, 10129 Torino, Italy, E-mail: massimo.violante@polito.it magnitude of the electric charge used to carry and store information is seriously raising the probability that highly energized particles hitting the circuit may induce transient errors (often modeled as Single Event Upsets) [1].To face the above issue, mechanisms are required to increase the robustness of electronic devices and systems with respect to possible errors occurring during their normal function, and for these reasons on-line testing is now becoming a major area of research. No matter the level these mechanisms work at (hardware, system software, or application software), there is a need for techniques and methods to debug and verify their correct design and implementation.Fault injection [2] is commonly adopted to this purpose, and several techniques have been proposed and practically experimented. They can be grouped into simulation-based techniques (e.g., [3][4]), software-implemented techniques (e.g., [5][6][7]), and hardware-based techniques (e.g., [8][9]).As pointed out in [2], physical fault injection (hardware- and software-implemented fault injection approaches) is more suited when a prototype of the system is already available, or when the system itself is too large to be modeled and simulated at an acceptable cost. Conversely, simulation-based fault injection is very effective in allowing early analysis of designed systems, since it can be exploited when a prototype is not yet available. The main disadvantage of simulation-based fault injection versus physical fault injection is the high CPU time required to simulate the model of the system (provided that it is available).As the complexity of the system under analysis grows, it is thus important to define which is the most suitable approach to be exploited during fault injection in order to match two conflicting requirements. On the one hand designers should adopt system models that are as close as possible to system physical implementations in order to precisely reflect the effects of real faults. At the same time, designers should be able to minimize the time needed for performing injection experiments, thus to be able to analyze sets of faults that are wide enough for providing statistically meaningful information.In this paper we addressed the aforementioned problems while devising a fault injection environment tostudy the effects of soft errors on CAN networks. In particular, given a system where several nodes exchange information via a CAN network, we are interested in studying the effects of soft errors inside the controller implementing the CAN protocol (i.e., in the memory elements that each CAN controller embeds), and in the network as well.Due to the nature of the fault locations we are interested in, and in particular due to the need for injecting faults inside the memory elements each CAN controller embeds, we developed a synthesizable IP-core (modeled at the Register Transfer level) implementing the features of a CAN protocol controller; we then modeled the analyzed network resorting to the developed IP-core. Finally, we exploited simulation-based fault injection for inoculating faults in the locations we selected.The IP-core we developed is fully compliant with the CAN protocol specifications, it thus allows obtaining an accurate model of the physical implementation of the system. As a result, we were able to gather meaningful information about the effects of soft errors in the real system.To assess the soundness of the environment we developed, we gathered some preliminary results by considering a simple network composed of four nodes, one of them acting as a master that periodically polls the remaining three nodes. When a commercial VHDL simulator is exploited, the simulation of the whole network (modeled at the RT-level) takes about 0.3 seconds for each transmitted data frame (each one composed of 51 bits).This preliminary result suggests that the environment is suitable for debugging purpose and that it can be exploited even for validation purposes, when a limited amount of packets are sent over the network.The remainder of the paper is organized as follows. Section 2 reports an overview of the CAN protocol, while Section 3 describes the network model we developed, while Section 4 discusses the techniques we adopted for supporting fault injection. Section 5 reports the experimental results we gathered and finally section 6 draws some conclusions.2. BackgroundThe Computer Area Network (CAN) protocol was initially developed by Bosch in the middle eighties and published later as the ISO 11898 standard [14] [15]. It has been used in the automotive industry for on board networks since 1992.CAN is a multiple access, non-destructive collision detection contention protocol. The specification defines two possible values for the bus state: recessive and dominant. If more than one node is driving the bus at the same time, the dominant value prevails over the recessive value. Usually the recessive value is associated with logical value “1” and the dominant value with a logical “0”, so the bus can be seen as a wired and.Data are exchanged in frames (Data and Remote Frames) containing an ID and 0 to 8 bytes of data. The addressing of the nodes is not specified in the protocol. The ID field on each frame identifies the message, and itis defined by the application generating the transmitted information.Two formats are available for Data and Remote Frames: the standard format (11 bit ID) and the extended format (11 bit basic ID + 18 bit extended ID).If a collision takes place between two nodes that started the transmission of a frame at the same time, a priority arbitration based on the arbitration field1 value is used to resolve the contention. At the first difference detected in the arbitration field, the node driving a recessive value must retire and become a receiver.The error detection mechanisms provided by CAN include bit error (each node monitors the bus while driving it, and it expects to read the same value it sent), stuff error, CRC error, form error (errors detected when the fields of the frame do not comply with the standard) and acknowledgement error (the transmitter expects a dominant value driven by the receivers at the end of each frame).In order to assure data consistency across all the nodes, any node detecting an error will drive the bus to a dominant value for six bit periods thus forcing a stuff error on all the other nodes. In this way transmission will not be completed and the transmitter must resend the frame.Suitable fault confinement mechanisms are implemented in order to avoid completely blocking the bus in presence of permanent faults. Each node maintains two counters totalizing the number of recent transmission and reception errors. The normal condition of each node is the error active status. If the error counter exceeds a given threshold, the node moves to the error passive status in which it cannot drive the bus during an error frame. If even more errors are detected, the node enters the bus off status and it only returns to error active after having monitored the bus idle for a long period of time.3. Network modelFor the purpose of performing fault injection experiments in both the internal memory elements of a CAN protocol controller and in the network where it is1 The arbitration field is formed by the ID, the IDE flag specifying the ID format and the RTR flag specifying the kind of frame.exploited, a model of the CAN protocol controller is needed. In particular, access to the source code of the model is required to effectively support fault injection. As the model is intended for being used also in emulation-based fault injection experiments, the model should be fully synthesizable, too.At the beginning of the project there was no open source IP core available, so a VHDL core was developed starting from the scratch (an open source Verilog IP core was later released with a VHDL version announced [16]). For this purpose, some results from a previous work [17] tutored by one of the authors were used.The first version of the core has been completed and validated (it accounts for about 2,400 lines of synthesizable VHDL code) and it has been used for the first fault injection experiments. It still lacks the fault confining mechanism, meaning that it always behaves as an error active node.The main block of the core is the can_interface, which implements one network node. It is provided with an interface to the CAN bus (via the tx_data and rx_data lines) and to the application using the core (via rx_msg, tx_msg and handshake signals).An additional synthesizable block was developed, can_bus, which is used to model the behavior of the CAN bus and it essentially an AND gate.A top-level model of a generic network composed of N controllers communicating via one CAN bus was finally coded, where N is a parameter freely customizable by the model final user.The obtained network can be either simulated at the RT level or emulated via FPGA.4. Supporting fault injectionThe fault tolerance community is increasingly concerned by the occurrence of soft errors resulting from the perturbation of storage cells caused by ionization, known as Single Event Upset (SEU). SEUs are random events and thus they may occur at unpredictable times. For example, they may corrupt the content of a register during any clock cycle.As far as SEUs in CAN controllers are considered, we adopted the fault model called upset or transient bit-flip, which results in the modification of the content of a storage cell during any clock cycle. Possible fault locations are thus memory elements, e.g. flip-flops, inside each CAN controller in the considered network.As far as the bus connecting the CAN controllers is considered, we adopted two different fault models: alongside the SEUs, we considered the permanent stuck-at fault model. The former is used to mimic the effects of transient failure such as the ones induced by highly-energized particles or electromagnetic interferences on the bus, the latter is conversely exploited to account for permanent damage.In order to support injection experiments, we enriched the model with features that simplify the injection of the aforementioned fault models.For this purpose, we equipped each CAN controller with a saboter module [8] that allows to complement or stuck the content of the bus at any clock cycle. Moreover, the memory elements in the CAN controller module were equipped with the architecture described in [18], which allows injecting bit-flips in all the internal memory elements of a device with low intrusiveness and high accuracy.The obtained fault injection-enriched model can then be used for assessing the effects of faults either via simulation or FPGA-based emulation.No matter which approach is exploited, the effects of faults were classified according to the following categories:• No effect: all the data frame in the given workload were transmitted and correctly received.• Incorrect answer: at least one of the nodes in the network received corrupted data and the protocoldid not detect the presence of the fault.• Performance degradation: after some error frames and retransmissions due to the error detection andcorrection mechanisms the protocol provides, thesystem recovered its normal behavior and the dataframe in the given workload were transmitted andcorrectly received. However, all the operationswere completed in a time greater than the expectedone.• Network degradation: due to the effects of faults the network was partitioned in two sets, oneproperly working (i.e., transmitting and receivingas expected), and one not working. CAN controllers belonging to the latter set did notproduce any network activity.• Timeout: the operations in the workload were not completed after a given time limit was reached.5. Experimental resultsIn order to assess the soundness of the proposed environment, we performed some experiments on a simple network configuration encompassing four nodes. One of the nodes behaves as a master and all the others as slaves. Each slave implements a sequence generator (e.g., a counter) that is monitored by the master.After reset, the master (whose behavior is described in figure 1) initiates a polling loop over the slaves (whose behavior is described in figure 2) asking for the next value in the sequence, and it waits for the answer from the slave. Upon reception of one data frame fromany slave, the master checks the correctness of the answer. In case a value out of the expected sequence is received, the fault effect is classified as incorrect answer.In our experiments a workload consisting in the transmission of 12 data frames, each composed of 51 bits, was considered. We simulated the whole network by means of the Modelsim VHDL simulator on a Sun Enterprise 250 machine running at 400 MHz and equipped with 2GB of RAM.Figure 1: Behavior of the master nodeFigure 2: Behavior of the slave node As far as the CPU time for the execution of the workload was considered, we measured a simulation time of 4 seconds, which corresponds to an average CPU time of 0.3 seconds for each data frame. Please note that the simulation we run were cycle accurate: they thus allowed observing all the details of each CAN protocol controller during workload execution.During our experiments, we considered two possible scenario, a debug one presented in sub-section 5.1 and a validation one, which is reported in sub-section 5.2.5.1. Debug scenarioIn this scenario the user is still developing the networked system, and he needs to quickly evaluate the effects of just a limited number of faults, corresponding to the design corner cases. From the experience learned in developing our IP-core, we observed that in this case a very simple workload composed of one packet, only, is typically exploited. As a result, the performance we obtained is compatible with the simulation of some thousands of faults per hour.As far as fault effects are concerned, we observed that:• When the injection of permanent faults in the CAN bus was considered, we observed several timeoutconditions provoked by the bit error detection/correction mechanisms the protocolembeds: the sender node identified a mismatchbetween the value read from the bus and thetransmitted one, and thus it continuously retried toresend the same data frame. Another condition weobserved when injecting permanent faults in therx_data line of any node was the networkdegradation one. The affected node did notproduce any network activity and thus it acted asdisconnected from the bus.• When the CAN bus was affected by transient faults, we observed either no effects, when thefaults modified meaningless bits in the data frame,or performance degradation when the fault wasdetected and a retransmission performed.• As far as SEUs in the memory elements embedded in the CAN protocol controller are considered, forthe sake of this paper we injected faults in thecounter registers that control the length of eachframe field, both for transmission and for receptionpurposes. The results we gathered showed that faultare classified either as no effects or performancedegradation.5.2. Validation scenarioIn this scenario the network is ready, and the designer needs to validate the application that is using the network. A workload encompassing the transmission of several packets may be required, while the fault list is composed of several thousands of faults. In this case, in order to gather statistical evidence of the application robustness, faults are normally randomly selected. For the considered application, given a workload composed of 12 packets, we estimated a simulation performance of about 900 faults per hour.No effect 241Incorrect answer 4Performance degradation 351Network degradation 0Timeout 4Table 1: Injection of SEUs in the master nodemaster(){while(1) {foreach slave {send request(slave ID)wait data frame from(slave ID)compare with counter(slave ID)increment counter(slave ID)}}}slave(){count = 1while(1) {wait request from(master ID)new data frame = countsend data frame(master ID)count++}}As far as fault effects are concerned, we performedthe injection of 600 SEUs inside the counters embeddedin the CAN controllers. For each fault we selectedrandomly the time, the network node and the register bitwhere the SEU was injected. The figures we measuredare reported in table 2.We found some faults classified as Timeout andIncorrect answer . After analyzing in detail these faults itwas found that all of them correspond to situationswhere a frame is not consistently accepted or rejected byall the nodes. For example, on one of these faults theEndOfFrame field of a transmitting node was made longer by a fault and thus this node rejected the frame because of an error detected at the end of the longer EndOfFrame field, when all the receivers have already accepted the frame. To illustrate the possibility offered by our environment of emulating different types of serviceinterruption, we performed a further set of experimentsduring which we perturbed the CAN bus with SEUscharacterized by a duration of several clock cycles. Thefigures we measured are reported in table 2, where fault effect classification is reported for SEU durations ranging from 1 to 20 clock cycles. Fault duration [Clock cycles]1 5 10 20No effect 97 60 17 15Incorrect answer 0 0 0 0 Performance degradation 3 40 82 84 Network degradation 0 0 0 0 Timeout 0 0 1 1Table 2: Injection of SEUs in the master node The observed variation of fault effects with faultduration is due to the peculiarity of the selected fault location: the CAN bus is sampled at any clock cycle but its value is used only once during each bit interval (10 clock cycles with the can bus timing parameters used). As a result, any fault that lasts 10 clock cycles or more is likely to affect the data transmission for at least one bit, while faults lasting for less than 10 clock cycles have lower probability of producing any visible effect on the system.6. ConclusionsIn this paper, we reported a detailed description of an environment we set-up for assessing the effects of faults in a networked system based on the CAN protocol.Our environment is based on a synthesizable IP-core implementing the CAN protocol, which is used to layouta generic network comprising N nodes, where N is a parameter fully customizable by the model end user. The network model was enriched with fault injection-oriented features allowing evaluating the effects of permanent and transient faults in both the CAN bus and the memory elements each CAN controller embeds. The cycle-accurate simulation experiments we performed allowed us to experimentally verify that our environment can be exploited for both debug and validation purposes of a simple network composed of four nodes. References [1] M. Nikoladis, “Time Redundancy Based Soft-Error Tolerance to Rescue Nanometer Technologies”, IEEE 17th VLSI Test Symposium, April 1999, pp. 86-94[2] R. K. Iyer and D. Tang, “Experimental Analysis of Computer System Dependability”, Chapter 5 of Fault-Tolerant Computer System Design, D. K. Pradhan (ed.), Prentice Hall, 1996 [3] E. Jenn, J. Arlat, M. Rimen, J. Ohlsson, J. Karlsson,“Fault Injection into VHDL Models: the MEFISTO Tool2, Proc. FTCS-24, 1994, pp. 66-75 [4] T.A. Delong, B.W. Johnson, J.A. Profeta III, “A Fault Injection Technique for VHDL Behavioral-LevelModels”, IEEE Design & Test of Computers, Winter1996, pp. 24-33 [5] G.A. Kanawati, N.A. Kanawati, J.A. Abraham, “FERRARI: A Flexible Software-Based Fault andError Injection System”, IEEE Trans. on Computers,Vol 44, N. 2, February 1995, pp. 248-260 [6] J. Carreira, H. Madeira, J. Silva, “Xception: Software Fault Injection and Monitoring in Processor FunctionalUnits”, DCCA-5, Conference on Dependable Computing for Critical Applications, Urbana-Champaign, USA, September 1995, pp. 135-149[7] A. Benso, P. Prinetto, M. Rebaudengo, M. SonzaReorda, “EXFI: a low cost Fault Injection System for embedded Microprocessor-based Boards”, ACM Transactions on Design Automation of Electronic Systems, Vol. 3, Number 4, October 1998, pp. 626-634[8] J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.C. Fabre,J.-C. Laprie, E. Martins, D. Powell, “Fault Injection for Dependability Validation: A Methodology and some Applications”, IEEE Transactions on Software Engineering, Vol. 16, No. 2, February 1990[9] J. Karlsson, P. Liden, P. Dahlgren, R. Johansson, U.Gunneflo, “Using Heavy-Ion Radiation to Validate Fault-Handling Mechanisms”, IEEE Micro, Vol. 14, No. 1, pp. 8-32, 1994[10] D. Gil, R. Martinez, J. V. Busquets, J. C. Baraza, P. J.Gil, “Fault Injection into VHDL Models: Experimental Validation of a Fault Tolerant Microcomputer System”, Dependable Computing EDCC-3, September 1999, pp. 191-208[11] J. Boué, P. Pétillon, Y. Crouzet, “MEFISTO-L: AVHDL-Based Fault Injection Tool for theExperimental Assessment of Fault Tolerance”, Proc.FTCS´98, 1998[12] A. Benso, M. Rebaudengo, L. Impagliazzo, P. Marmo,“Fault-List Collapsing for Fault InjectionExperiments”, RAMS'98: Annual Reliability andMaintainability Symposium, January 1998, pp. 383-388[13] K. H. Huang and J. A. Abraham, “Algorithm-BasedFault Tolerance for Matrix Operations”, IEEETransactions on Computers, vol. 33, June 1984, pp.518-528[14] Bosch’s Controller Area Network Homepage at/[15] Can in Automation (CiA) Can Protocol introduction athttp://www.can-cia.de/can/protocol/[16] Opencores Can protocol controller project./projects/can/[17] J. Juárez, P. Bustamante, “Interfase Bus Can”, CursoDiseño Lógico 2. FinalProject. .uy/ense/asign/dlp/proyectos/2002/can/[18] P. Civera, L. Macchiarulo, M. Rebaudengo, M. SonzaReorda, M. Violante, “An FPGA-based approach forspeeding-up Fault Injection campaigns on safety-critical circuits”, Journal of Electronic Testing:Theoryand Applications, Vol. 18, No. 3, June 2002, pp. 261-271。