Advanced Functional Programming in Scala

Editor's Notes

• #2 Context of the presentation: The transition from Java and Python to Scala is not that easy: It goes beyond selecting Scala for its obvious benefits.- support functional concepts- leverage open source libraries and framework if needed- fast, distributed enough to handle large data setsScala was the most logical choice. Scientific programming may very well involved different roles in a project: Mathematicians for formulas Data scientists for data processing and modeling Software engineering for implementation Dev. Ops and performance engineers for deployment in productionIn order to ease the pain, we tend to to learn/adopt Scala incrementally within a development team.. The problem is that you end up with an inconsistent code base with different levels of quality and the team developed a somewhat negative attitude toward the language.The solution is to select a list of problems or roadblocks (in our case Machine learning) and compare the solution in Scala with Java, Python ... (You sell the outcome not the process).PresentationA set of diverse Scala features or constructs the entire team agreed that Scala is a far better solution than Python or Java.
• #3 Disclaimer…
• #4 Being an object oriented and functional language, Scala has a lot of features and powerful constructs to choose from…. Here is a list of some of the features of Scala that are particularly valuable for writing scientific workflows, machine learning algorithms and complex analytics solutions.
• #13 Geometric entities on a differential (Riemann) manifold are defined as tensors. Tensor can be Covariant Contra variant A bilinear form such as Tensor product Inner product n-differential forms …
• #16 Some useful references, on the theory of categories and monads .. 1 One Div Zero Monads are Elephants Part 2 J. Iry Blog 2007 http://james-iry.blogspot.com/2007/10/monads-are-elephants-part-2.html 2. Monad Design for the Web §7 A Review of Collections as Monads L.G. Meredith Artima 2012
• #17 For better understanding of kernel functions in Machine learning Introduction to Machine Learning §Nonparametric Regression: Smoothing Models. E. Alpaydin MIT Press 2007 A Short Introduction to Learning with Kernels B. Scholkopt, Max Planck Institut f ̈ur Biologische Kybernetik A. Smola Australian National University 2005 http://alex.smola.org/papers/2003/SchSmo03c.pdf The purpose here is to generate and experiment with any kind of explicit kernels by defining and composing two g and h functions A function h operates on each feature or component of the vector A function g is the transformation of the dot product of the two vectors. The dot product is computed by applying the function to all the elements and compute the sum. The “dot” product K is computed by traversing the two observations (vector of features), computing the sum and finally applying the g transform. The variable type is the type of the function g (F1 = Double => Double)
• #18 Once the functor is defined, the monad is created by adding the flatMap method. The monad, KFMonad which take a kernel function as argument is defined as an implicit class so kernel functions “inherits” the monadic methods. The map and flatMap transformation applies to the g function or transformation on the inner product. The flatMap method is implemented by creating a new Kernel and applying the transformation to only one of the component of the Kernel function: function h (in red). This “partial” monadic operation is good enough for building Kernel functions on the fly.
• #19 Kernel functions that project the inner product to the manifold for non-linear models belong to the family of exponential functions Polynomial functions and Radius basis functions are two of the most commonly used kernel functions. Note: The source code is shown here to illustrate the fact that the implementation in any other language would be a lot more messy and won’t be able to fit in any of those slides.
• #20 Finally we can chain flatMap to map to compose two kernel function and compute the dot/inner product of the resulting kernel function. Note that composed kernel function used the h function of the last invocation in the for instruction. The method does not expose the functor or KF classes that wraps the components of the kernel function.
• #22 Real-time streaming is becoming popular (i.e. Apache Spark streaming library, Akka reactive streams….). Short of using one of these frameworks, you can create a simple streaming mechanism for large data sets that require. Streams vs. Iterator: Iterator does not allow to dynamically select the chunk of memory or preserve it if necessary for future computation. It is not uncommon to have to train a model as labeled data becomes available (online training). In this case, the size of the data set can be very large and unknown. The processing of the data would result in high memory consumption and heavy load on the Garbage collector. Finally, the traversing the entire data sets (~ allocating large memory chunk) may not even needed as some computation may abort if some condition is met. Scala’s streams can help!
• #23 In order to minimize the memory footprint, two actions have to take place Allocate slice of the data set (memory chunk) from the heap using take method. Release the memory chunk back to the Garbage collection through a drop method. This example is taking from “Scala for Machine Learning” Packt Publishing. Most of machine learning algorithms consists of minimizing a loss function or maximizing a likelihood. For each iteration (or recursion) the cumulative loss is computed and passed to the optimizer (i.e. Gradient descent or variant) to update the model parameters. Each slice of n observations is allocated, processed by the loss function then released back to the Garbage collector.
• #24 The Loss class has a single method, exec, that traverses the stream. Once again, the loss is computed using a tail recursion. An observation is defined as y = f(x) where x the feature set containing for instance, age, ethnicity of patient, body temperature and y is the label value such as the diagnosed disease.. The tail recursion allocates the next slice of STEP observations through the take method, computes the lost, nextLoss, then drop the slice. The reference is recursively redefined as the reference to the remaining stream. The problem is that the garbage collector cannot reclaim the memory because the first reference to the stream is created outside the recursion. The solution is to declare the reference to the stream as weak so it chunks of memory associated to the slices/batches of observations already processed can be reclaimed.
• #25 The reference of the stream is created as a Weak java reference to an instance created by the stream constructor, Stream.iterate. In this case, the weak reference has been used to show Java concepts are still relevant.
• #26 Let’s compare the memory consumption of three strategies to compute the loss function on a very large dataset. A list A stream with standard reference A stream with weak reference In the first scenario, and as expected, the memory for the entire data set is allocated before processing. The memory requirement for the stream with a strong reference increases each time a new slice is instantiated, because the memory block is held by the reference to the original stream. Only the stream with a weak reference guarantees that only the memory for a slice of STEP observations is needed through the entire execution.
• #35 The first thing to come to mind in creating complex system from existing objects (or classes) is a factory design pattern. Design patterns have been introduced by the “Gang of four” in the eponymous “Design Patterns: Elements of Reusable Object-Oriented software” some 20 years ago… The list of factory design patterns includes Builder, Prototype, Factory method, Composite, Bridge and obviously Singleton. Those patterns are not very convenient for weaving data transformation (these transformation being defined as class or interface). This is where dependency injection popularized by the Spring framework comes into play. Beyond composition and inheritance, Scala enables us to implement and chain data transformations/reductions by stacking the traits that declare these transformations or reductions.
• #36 The implementation in Scala matches perfectly the universal mathematical formalism Here is another example Declaration variable 𝑥∈ℝ, 𝑦∈ℝ Declaration of model f(x,y)=𝑥+𝑦 Instantiation of variable 𝑥=5, 𝑦=7; 𝑓 5,7 =12
• #38 The first thing to come to mind in creating complex system from existing objects (or classes) is a factory design pattern. Design patterns have been introduced by the “Gang of four” in the eponymous “Design Patterns: Elements of Reusable Object-Oriented software” some 20 years ago… The list of factory design patterns includes Builder, Prototype, Factory method, Composite, Bridge and obviously Singleton. Those patterns are not very convenient for weaving data transformation (these transformation being defined as class or interface). This is where dependency injection popularized by the Spring framework comes into play. Beyond composition and inheritance, Scala enables us to implement and chain data transformations/reductions by stacking the traits that declare these transformations or reductions.
• #39 We briefly mentioned that the for comprehension can be used to chain/stack data transformation. Dependency injection provides a very flexible approach to create workflows dynamically, sometimes referred as the Cake pattern. Note: As far the 3rd point, deployment of tasks, it usually involves a actor-based (non blocking) distributed architecture such as Akka and Spark. We will mention it briefly later in this presentation is introducing mailbox back-pressure mechanism.
• #40 Let’s look at the Training module/task as an example. The task of training a model is executed by a Supervisor instance that can be either a support vector machine or a multi-layer perceptron, in this simplistic case. Each of these two “supervisors” can have several implementations (single host, distributed through a low-latency network,…) Once defined, the modules are to be weaved/chained by making sure that output of a module/tasks matches the input of the subsequent task. Notes: The training module can be broken down further into generative and discriminative models. Real-world applications are significantly more complex and would include REST service, DAO to access relational database, caches…. The terms “module”, “tasks” or “computational tasks” are used interchangeably in this section.
• #42 Let’s consider the Preprocessing module (or task) implemented as trait Preprocessing of a data set is performed by a processor of type Preprocessor that is defined at run time. Therefore the preprocessor has to be declared as an abstract value. The three preprocessors defined in the preprocessing modules are Kalman filter, Moving Average (MovAv) and Discrete Fourier filter (DFTF). Those 2 inner classes act as adapter or stub to the actual implementation of those algorithm.
• #43 Here is an implementation of the Kalman and Discrete Fourier transform band-pass filter. The2 inner classes, Kalman and DFTF act as adapter or stub to the actual implementation of those algorithm. It allows the implementation may consist of multiple version. For instance filtering.Kalman is a trait with several implementation of the algorithm (single host, distributed, using Spark…) Such design allows to Select the type of preprocessing algorithm within the Preprocessing module or namespace Select the implementation of this particular type of algorithm/preprocessor in the filtering package
• #45 From the data management perspective, Clustering implements two consecutive data transformations: preprocessing and dimension reduction.
• #46 Modeling workflow is created by chaining an implementation of the filter, training and validator, all selected at run-time. Modeling is therefore implemented as a stack of 3 traits, each representing a transformation or reduction on data sets.
• #47 Computational tasks related to machine learning can be complex and lengthy. The process should be able to select the appropriate date flow (or sequence of data transformation or reduction) at run time, according to the state of the computation. In this simple case, a clustering task is triggered if anomalyDetection is needed, training a model is launched otherwise. These conditional path execution are important for complex analysis or lengthy computation that require unattended execution (i.e. overnight or over the week-end). Note: The overriding of the abstract value for the Modeling workflow are omitted here for the sake of clarity Summary: This factory pattern operates on 3 level of componentization: Dynamic selection of 1- Workflow or sequence of tasks according to the objective of the computation (i.e. Clustering => Preprocessing) 2- Task processing algorithm according to the data (i.e. Preprocessing => Kalman filter) 3- Implementation of task processing according to the environment (i.e. Kalman filter => Implementation on Apache Spark)
• #63 The objective is to avoid bottleneck in the computation data flow which would result in overflowing actors’ mail box/local buffers. A strategy to control the flow (or back-pressure flow) is needed to regulate the data flow across all modules. This example use a back-pressure handling mechanism that consists of monitoring bounded mail boxes. This is a simplistic approach to flow control described for the sake of illustrating the concept. As we will see later, there is a far more effective mechanism to deal with back-pressure.
• #64 We mentioned earlier that a learning platform requires implementing, wiring and deploying tasks. Akka or framework derived from Akka, are commonly used to deploy workflow for large datasets because of the immutability, non-blocking and supervision characteristic for actors. Scala/Akka actors are resilient because that are defined with hierarchical context for which an actor because a supervisor to other actors. In this slide, a router is a supervising actor to the workers and, depending on the selected strategy, is responsible of restarting a worker in case of failure. But, what about the case for which the load (number of messages in the
• #65 In this example, an actor, Controller loads chunk of data, partitions and distributed across multiple Worker actors, along with a data transformation. Upon receiving the message ‘Compute’ the workers process data given a transformation function. The workers returns the processed data through a Completed message. The purpose of the watch dog actor, ‘Watcher’ is to monitor the utilization of mailbox and report it to the Controller This is a simple feedback control: 1- Watcher monitors the utilization of the mailbox (average length) 2- The controller adjust the size of each batch in the load message handler (throttling) 3- The workers process the next batch
• #66 Let’s start with the Worker actor. The load on a worker depends on three variables 1- The amount of data to process 2- The complexity of the data transformation 3- The underlying system (cores, memory..) The controller provides the slice of data to be processed by the workers msg.xt as well as the data transformation msg.fct.
• #67 Let’s look at our watch dog actor, Watcher: It computes the load as the average mailbox utilization and send it back to the controller through a Status message.
• #68 As its name implies, the controller configure and manage the dynamic workflow and control the back pressure from the worker actors. As far as the configuration is concerned, the Controller generates a list of workers, the bounded mailboxes, msgQueues, for the workers and ultimately the watcher actor. The worker and watcher are created within the Controller context using the Akka actorOf constructor.
• #69 As far as the management of data flow and feedback control loop, the Controller loads partition and distribute batch of data points to the worker actors (message: Load) processes the results of the computation in workers (message: Completed) - Throttle up or down the flow upon receiving the status on utilization of mail boxes from the watcher (message: Status)
• #70  The composition of the messages processed by the controller are self-explanatory. It Adjusts the size of the next batch if required (throttle method) Extracts the next batch of data from the input stream Partition and distribute the batch across the worker actors Send the partition along with the data transformation to the workers.
• #71 The implementation of the throttle method is rather simple. It takes the load computed by the watcher actor and the current batch size (number of data points to be processed) as input. It update the batch size using a simple ratio relative to the watermark. For instance if load is below the watermark, the batch size is increased.
• #72 The bottom graph describes the throttle action and the complexity of data transformation. The complexity of the data transformation and has an impact on the load on workers. It varies from 0 (simple map operation) to 2 (complex data processing involving recursion or iterations). The throttle intensity ranges between -6 (rapid decrease of size of batches) and +6 (rapid increase of size of batches of data) The top graph displays the actual utilization of the mail boxes with capacity of 512 messages as regulated by the feedback control loop (executed by the controller).
• #73 The deployment of a reactive data flow in production would require significant improvement on our Naïve model. The feedback control loop could be smoothed with a moving average technique or Kalman filter to avoid erratic behavior We would need to provide a larger range of options for control actions beside adjusting the size of data batches: increase of number of workers, mail box capacity, caching strategy, .. A fine grained set of actions reduces also the risk of instable systems. The watch dog should be able to handle dead letters in case of failure (mailbox overflowing) Reactive streams control the flow back pressure at the TCP connection level. It is far more accurate and responsive that mailbox utilization.