Statistics
| Branch: | Revision:

## root / thesis / ch4_experiment_setup.tex @ c6e2ce5a

 1 %!TEX root = main.tex  \chapter{Experiment} \label{sec:experiment}   The previous chapter presents two algorithms to calculate Betweenness Centrality (BC). One algorithm was found out by \citeauthor{Brandes01afaster} in \cite{Brandes01afaster}. For the ease of notation, from now on it is called \emph{BBC}. The other heuristic version to calculate betweenness centrality was devised by \citeauthor{PuzisZEDB12} \cite{PuzisZEDB12}, and its short-form name from this point is \emph{HBC}.   This chapter presents all the setup information to compare the running time of two algorithms. In order to perform the analysis, there are few things needed to address first. To analyze the running time of the algorithm, we need to know about all the factors that can influence the running time. Those factors are the implementation, the type and size of graph, the system that runs the algorithms. The following section addresses those factors in depth. \cref{sec:experiment_implementation} presents the implementation of BBC and HBC were run on the physical router and server. \cref{sec:experiment_hardware} provides the hardware specifications for the system we use. \cref{sec:experiment_openwrt} concerns with the technical side of porting the program to embedded device, in our case is the Ubiquiti router. This section contains a lot of technical note, which might be served as a guideline on how to make a custom package for embedded system. \cref{sec:experiment_input} presents about two types of input that we use in the experiment: the randomly generated graphs, and the real topologies of the WCNs. Lastly, there are fluctuation when measuring the running time of the algorithms. \cref{sec:experiment_design} presents the method to get the average running time, how the experiment was carried out, e.g. how many graphs was generated, how many times the algorithm was run for one specific input.  \section{Implementation} \label{sec:experiment_implementation}   The BBC is implemented in the Boost Graph Library\footnote{http://www.boost.org/doc/libs/1_60_0/libs/graph/doc/index.html} (BGL). That is the generic library with implementation for many classical graph algorithms.   From \cref{sec:computation_heuristic}, the implementation for HBC contains four main steps. The first step involves dividing the main graph into some bi-connected components $B_i$. And then for each articulation points in all $B_i$, the link weight is computed. To calculate the link weight, we need to use the information from the original graph. Second, for each $B_i$, create the traffic matrix $h^i$ to represent the \emph{communication intensity} between all pair of vertices in each $B_i$.   The third, and the final part is the procedure to calculate betweenness for each $B_i$. Since the original graph is divided into some bi-connected components $B_i$, performing the original BBC for each $B_i$ will not generate the correct result. Actually, the accumulation step, as shown in \cref{eq:BC_for_v_heuristic} involves multiplying the pair-dependency $\delta_{st}(v)$ with its corresponding communication intensity value $h_{st}^{i}$. For an ease of comparison, the pseudo code of the accumulation step for BBC and HBC are presented in \cref{algo:accummulation_brandes} and \cref{algo:accumulation_heuristic}, respectively.   \begin{algorithm}[h]   \caption{Accumulation procedure for BBC} \label{algo:accummulation_brandes}   \begin{algorithmic}   \Procedure{accumulation}{}   \State $c_B[s] \leftarrow c_B[s] + (|S| - 1)$   \For{$v \in V$}   $\delta[v] \leftarrow 0$   \EndFor   \While{$S$ not empty}   \State pop $w \leftarrow S$   \For{$v \in Pred[w]$}   $\delta[v] \leftarrow \delta[v] + \frac{\sigma[v]}{\sigma[w]} (1 + \delta[w])$   \EndFor   \If{$w \neq s$}   $c_B[w] \leftarrow c_B[w] + \delta[w] + 1$ \Comment{$w$ is target of $s$ once}   \EndIf   \EndWhile   \EndProcedure   \end{algorithmic}   \end{algorithm}   \begin{algorithm}   \caption{Accumulation procedure for HBC}\label{algo:accumulation_heuristic}   \begin{algorithmic}   \Procedure{accumulation}{}   \For{$v \in V$}   $\delta[v] \leftarrow 0$   \EndFor   \While{$S$ not empty}   \State pop $w \leftarrow S$   \State $h \leftarrow$ get_the_communication_intensity_between $(w, s)$   \State $c_B[s] += h$   \For{$v \in Pred[w]$}   $\delta[v] \leftarrow \delta[v] + \frac{\sigma[v]}{\sigma[w]} (h + \delta[w])$   \EndFor   \If{$w \neq s$}   $c_B[w] \leftarrow c_B[w] + \delta[w] + h$   \EndIf   \EndWhile   \EndProcedure   \end{algorithmic}   \end{algorithm}  \section{Hardware specifications} \label{sec:experiment_hardware}   \subsection*{Server}   The experiment is run on the machine provided by Advanced Networking Systems (ANS) lab\footnote{https://ans.disi.unitn.it/}, with the hardware components as in \cref{tab:specification_computer}   \begin{table}   \caption{Specification of a computer}   \label{tab:specification_computer}   \centering   \begin{tabular}{l l}   \toprule   CPUs: & 2 $\times$ Intel(R) Xeon(R) CPU E5-2630 v3 \@ 2.40GHz \\   Memory: & 4 $\times$ 16GB \@ 1866 MHz \\   \bottomrule   \end{tabular}   \end{table}   The machine was running GNU/Linux with kernel 3.16.   \subsection*{WiFi Router}   We use Ubiquiti NanoStation M2 router to test the performance. The specification is in \cref{tab:specification_router}. Ubiquiti router is flashed with a firmware obtained from OpenWrt as described in the next section.   \begin{table}   \caption{Specification of Ubiquiti NanoStation M2 router}   \label{tab:specification_router}   \centering   \begin{tabular}{l l}   \toprule   Platform: & Atheros AR7240 \\   CPU MHz: & 390 \\   Flash MB: & 8 \\   RAM MB: & 32 \\   \bottomrule   \end{tabular}   \end{table}  \section{Porting a program to OpenWrt} \label{sec:experiment_openwrt}   OpenWrt\footnote{https://openwrt.org/ -- the most widely used open-souce operating system for wireless routers.} is described as a Linux distribution for embedded devices.''. It allows people to build custom firmware to run on many supported devices. And it aids with cross compiling the package which can be installed on those devices with custom firmware.   Even though we can build the firmware image with OpenWrt toolchain. However, there is the firmware image available on the Download page  The source code of the program running on the machine with GNU/Linux kernel 3.16, and the program running on Ubiquiti router are the same. To make it possible to run a program in the Ubiquiti router, OpenWrt helps with compiling the source code into a package that can be installed on Ubiquiti router.  \section{Input data} \label{sec:experiment_input}   The experiment involves testing with 2 main sources of input: the artificially generated graphs, and the real topologies from the Wireless Community Networks.   \subsection{Synthetic graphs}   For randomly generated graphs, we are interested in three types: Erdos, Power Law, and Community Graph. All the graphs were generated following the open-source code made available by \citeauthor{Baldesi2016} \cite{Baldesi2016} in a framework called \emph{NePA TesT}. NePA TesT provides a way to generate synthetic graph for different graph type such as Erdos, Power Law, and Community Network.   The reason why we wanted to test on synthetic graphs is because we want to know how BBC and HBC would perform with different graph structures, and graph sizes. And as will see later in \cref{sec:experiment_result}, performance of BBC and HBC depends on the structure and property of each graph type.   \begin{table}   \caption{All synthetic graphs generated for the experiment}   \label{tab:graph_generator}   \centering   \begin{tabular}{l|l|l}   \toprule   Graph type & Number of nodes & Edges / Nodes Ratio \\   \midrule   CN & from [100, 1000] with the step 100 & 1.2 \\   ER & from [100, 1000] with the step 100 & 3 \\   PL & from [100, 1000] with the step 100 & 1.2 \\   \bottomrule   \end{tabular}   \end{table}   For every line in \cref{tab:graph_generator}, 30 different graphs were generated per size. So for CN graphs, we have 10 different sizes multiplies by 30 graphs, and we have 300 CN graphs in total. For all three graph types, we have 900 graphs. We use the notation \emph{CN100}, \emph{PL100}, \emph{ER100} to denotes the Community Network, Power Law and Erdos graphs with 100 nodes, respectively. The same format of notation is used for graphs with larger number of nodes.   \subsection{Real topologies of WCNs}   In conjunction with the synthetic networks, we also want to evaluate BBC and HBC on the real networks. Since in the end, what matters is which algorithm would work better in reality. In this thesis, we focus on three WCNs: Funk Feuer Wien (FFW)\footnote{https://www.funkfeuer.at/} and Funk Feuer Graz (FFG)\footnote{graz.funkfeuer.at/en/index.html} in Austria and Ninux (NNX)\footnote{wiki.ninux.org/} in Italy.   The WCNs are ever evolving with some current node leaving the network, while some new nodes might join it. \cite{Maccari2015175} provides a depth and throughout analysis on these networks for a week \cite{Maccari2015175}. He found out that the fluctuation in the number of nodes and the number of links are small. According to him, those small changes are from a small subset of leaf nodes leaving and entering network sporadically. Since that fluctuations happen at peripheral nodes, the shortest paths for pair of nodes did not vary much with other time-periods. The reason is that those peripheral leaf nodes does not lies in the shortest paths when routing for other nodes. Hence the performance of shortest-path based betweenness measurement varies slightly. There are other measurements taken as well, but the conclusion from the paper is that for a week, the network features of these real WCNs are stable. With the above analysis, we limit the performance testing of BBC and HBC on one snapshot for each WCNs in consideration.  \section{Experiment Design} \label{sec:experiment_design}   Those previous subsections present all the component of the experiment. We have the input data composing of synthetic and real graphs. We have 2 algorithms needed to evaluate: BBC and HBC. We specified the target system to evaluate the running time of the 2 mentioned algorithms. We start the section with definition of running time. The execution step of the experiment is presented after, aka how many times one single algorithm (BBC or HBC) is evaluated for each input graph.   Every time we mention about the running time, or execution time of BBC or HBC algorithm, we are talking about the time that a processor spent on calculating result for each algorithm. In C++, there is the function call in standard library to help measure the number of CPU clocks consumped by the program.   \begin{figure}   \begin{lstlisting}   #include   clock_t begin = clock()   // Run BBC algorithm / HBC algorithm for one graph   clock_t end = clock()   double running_time = double (end - begin) / CLOCKS_PER_SECOND   \end{lstlisting}   \caption{Code to calculate the CPU time}   \end{figure}   The experiment consists of running each algorithm 10 times for each input graph. In this experiment, the measurement we want to get is the running time for each algorithm. And we want that measure to be as close to the actual value as possible. When running for one time, there are various things which might affect the result. And we need to repeat the experiment multiple times to make sure the measurement we get is not a coincidence.   \textbf{Note:} Though we run the program on the machine with CPU containing 8 cores and supporting hyper threading, the program does not support parallelism. One program only run on one single thread from the beginning to the end.   For example, for the Ninux graph, we will run BBC 10 times with it, and then we will run HBC 10 times. The running time were all noted down for the analysis.   \begin{table}   \centering   \begin{tabular}{l|l|l}   \toprule   Graph type & On the computer & On Ubiquiti router \\   \midrule   CN with nodes in the range [100, 1000] & Yes & Yes \\   ER with nodes in the range [100, 1000] & Yes & No \\   PL with nodes in the range [100, 1000] & Yes & No \\   Ninux & Yes & Yes \\   FFG & Yes & Yes \\   FFW & Yes & Yes \\   \bottomrule   \end{tabular}   \caption{Summary of the number of runs (repetitions) for different input graph}   \label{tab:experiment_design}   \end{table} `