Sunday, 12 November 2017

Programming Microservices

I remember watching the movie Avatar, by James Cameron at IMAX. Seeing the richly built world of Pandora on a larger than life screen was awe inspiring. Like every true fan, I looked beyond the state of the art CGI and was hooked on to Na’vi, the language of the natives, that James created for his movie.



Reviewers from movie magazines asked James the need of creating a new language. James explained that he had many requirements which necessitated it. First was for the symbolism, the metaphorical references to oppression, colonialism and annihilation of the indigenous communities and the philosophy of the nourishing, all providing, omnipresent female. James used voiceless consonants in the language, as used by many primordial languages of Asia and Americas and modeled Na’vi based on the Mayan and Polynesian dialects to achieve this.

He also wanted the language to allow his actors to emote expressively, in addition to it being nice to hear and easy to speak. The language uses varying tones and vowels of varying length to achieve an expressive yet pleasing intonation. The new language helped him to not to limit his creativity based on the conventions of English.

When developing Microservices, it is relatively easy to do what is required. Components and RESTful interfaces are some of the low hanging fruits. As we develop, we discover that it is difficult not to do what is not required. The hard-to-avoid antipatterns are structured composition, centralised data and about managing mutability.

Microservices need to be composed functionally, not structurally. Teams fall back on structural composition as an incidental effect of using a programming language with a procedural/imperative style. The most popular programming languages - C, C++ and Java are imperative and coding in them let you compose structurally, usually in a pattern of some central providers and many contextual users. The problem with structured composition is that it creates dependencies (on the providers) which makes it difficult in creating independent and loosely coupled Microservices. A litmus test for your composition would be to look at the module names and check whether they are in English or geekish. For e.g. module names like iNode, checkpoint etc. suggests a structured composition where as names like replication, backup suggests a functional approach.

Centralised data is a consequence of dependencies created by building your application structurally. Decentralising requires intuitive frameworks, which may need to be created for the popular programming languages like C, C++ and Java, using some reactive programming techniques.

Most of the popular programming languages have mutable data types, which creates a need for managing mutability to enable concurrency. This is an incidentally complex problem in C, C++ and Java, requiring locks and identifying critical sections. Even the best implementations have deadlocks and race conditions, in spite of these synchronised methods. Is there a simpler alternative?

In the next part of this blog, we will look at how to avoid these antipatterns and the features of some programming languages that would help us to do so.

Wednesday, 3 February 2016

Iterative Design

Nature is a great designer. The world around us is full of amazing lessons in design.

Take the weaver bird, for instance. The bird gets its name from the elaborately woven nest (most elaborate among birds) it builds.

The weaver bird designs its nest pragmatically, by building it.

Pragmatic Design © Paul & Paveena Mckenzie, Getty Images






















After building some part, the bird would check the increment for strength and stability.

Strength check © Tim Jackson, Getty Images



















The bird will reinforce the structure if not satisfied with its strength or stability. Then it will continue to build the nest further.

Building further © Manoj Shah, Getty Images






















Thus, the bird iteratively switches between building further, and strengthening what is built. Eventually, it would complete the main chamber.

Chamber completed © Anup Shah, Getty Images


















The bird doesn't stop building, although the nest is functionally complete. It will secure the nest's entrance with a narrow passage, guarded with some sharp twigs, to protect it from the predators.

Securing the entrance © James Warwick, Getty Images


























What lessons, if any, can we learn from this, and use in the world of software design?

Any software has three sets of characteristics; the functional part which is about the features, the operational part which deals with aspects like security, performance and concurrency and the developmental aspects of being readable, easy to debug and extensible. These three characteristics are orthogonal, i.e. changing one will not usually affect the other. Since they are orthogonal, it is not very intuitive to consider them together, say security and being extensible, at the same time.

A good design is one where these three characteristics are visited iteratively, and one at a time.

In my last blog about incremental coding (http://agileiq.blogspot.in/2015/12/code-like-sloth.html), we saw how to build the functionality incrementally.

Our encrypting function is as shown below:

...
public static void encrypt(String s)
{
      s = s.replaceAll("\\s", "");
      int rows = (int) Math.floor(Math.sqrt(s.length()));
        int cols = (int) Math.ceil(Math.sqrt(s.length()));
        if(rows*cols < s.length())
              rows++;
        for(int n=0;n<cols;n++)
        {
              for(int m=0;m<rows;m++)
              {
                     if(n+m*cols >= s.length())
                           break;
                     System.out.print(s.charAt(n+m*cols));
              }
              if(n<cols - 1)
                     System.out.print(" ");
        }
            
        System.out.println("");
}
...

Let us continue with the same example to do some iterative design for the developmental and operational aspects.

A good programmer once told me that loops should do nothing, but only loop. Let us refactor the loop code by extracting it as a method for better readability.

...
for(int n=0;n<cols;n++)
{
      printColumn(s, rows, cols, n);
}
...

Next, we look at the concurrency design, and find that the print functions cannot assure it. We can use some indirection here and achieve concurrency as follows:

...
public static void encrypt(String s)
{
       encryptThreadSafe(s, new PrintStreamThreadSafe(System.out));
}

public static void encryptThreadSafe(String s, PrintStream p)
{
       

        for(int n=0;n<cols;n++)
        {
              printColumn(s, rows, cols, n, p);
        }
            
        p.println("");
}
...

We will look at another operational aspect now, by securing our code. In my experience, tools are an invaluable asset in secure design. You may use a combination of code scanning tools, like Coverity and application scanning tools, like Nessus for this.

In our code example, let us take care of a common security flaw, handling the null input, as follows

...
public static void encrypt (String s) throws IllegalArgumentException
{
       if(s == null)
       {
             throw new IllegalArgumentException("Null input");
       }
...

In our small example, after two or three iterations of handling the operational and developmental parts, our iterative design is satisfactory.

In bigger implementations, we would have to do more number of iterations to be satisfied with our design.

We can visualize the state of our design, any time, as below:

Design Radar





















Initially, the design would be skewed towards functionality. After some iterations of addressing the operational & developmental concerns, we would have a more well-rounded(!) design.

We can configure our CI system to regularly generate such snapshots of our design; where the functional indices may be based on the function test passing rate and suitable tools, like Coverity, Structure-101, Nessus, JMeter, Valgrind etc. may be used to suggest the operational and developmental health.

When we find that our operational and developmental design is poor, it would be better to pause adding new functions and have a Zero-Feature sprint to catch up.


Monday, 14 December 2015

Code like a sloth

Code like a sloth


A programmer’s guide to incremental coding


Set a timer (in your mobile) for 21 minutes.

We will do a programming puzzle now. It is from HackerRank, a popular competitive programming site. While we are at it, when the timer fires; we will stop what we are doing (bang!) and come back to this blog.

Get on with this puzzle; https://www.hackerrank.com/challenges/encryption. You may solve it in the editor in the page or use the development tool of your choice.

***

You are back.

Answer the following questions. If the answer to any question is “No”, skip the rest of the questions and read on.
1.     Does your code compile (as applicable)? Any runtime error?
2.     Do you know what all cases/scenarios are handled by your code? Are you highly confident about it?
3.     Do you know how your code may behave for any unhandled case/scenario? Are you highly confident about it?
4.     Does the code pass the test cases when you click the “Run Code” button?
5.     Go ahead and submit your code. The site will run more tests on your code now. Are the results as expected?

Coding in a sprint/iteration is also somewhat like this exercise. Software product development is essentially problem solving in nature. Coding is like solving a puzzle. I love solving puzzles and that is why I am a programmer. In the sprint, the programmer starts developing an increment, assuming a solution. The assumption can be tested only by coding it; thus the solution evolves while coding. Somewhere during this process, the time box of the iteration ends. At this time, the only question that matters is how well done is your increment? We would understand later in this discussion that it is not really about how much of the planned work we completed or what percentage (ugh!) of the feature we coded.

Did the titular reference to the sloth get you curious? Let me explain the use of this metaphor.

The sloth is a much misunderstood animal. Many would think about it as a lazy dozer. It may be surprising to know that they don’t sleep that much, but only for 8 or 9 hours a day. Being still for most part of the day is simply, the defense against their worst predator – the eagle.
The eagle is a very formidable predator, swift and with a keen eye for a prey. Being inconspicuous is probably your best defense against something that can dive bomb and snatch you off a branch, just like that. They have some adaptations that help them to be unseen, like their fur which is very suitable for algal growth. From the sky, they would probably look like some dull branch or fruit. Their lazy, yet amazingly simple defense has been so effective that the sloths has been around for 100 million years or so (to put that in perspective, some of our earliest known ancestors are only 6 million years old) and did not require any further adaptation for natural selection all this time.

Incremental coding is like the sloth; mostly misunderstood, although amazingly simple in its scheme. 

Let us attempt the same puzzle by doing incremental coding. I have grouped the activities in a sequence of six steps as follows:

1.    Fail loud and clear


Make clear decisions about what scenarios are included and what are excluded. In incremental development, it is more important to decide what not to do than what to; be it for the product, the iteration or the increment.

Apple decided not to support SD cards in iPhone. Contrast it against the multiple usability issues reported on android versions till date related to SD card support. The android community also agrees that their support is not (yet) full and promises some improvements in the Marshmallow version.

Once we decide about the exclusions, define our behavior it to it clearly. It is suggested to fail loud and clear, like throwing an exception in Java.

As we start, let us exclude everything and fail with a special string, "_E". Don't forget to test it with a simple test case like encrypt("xyz") and check for the expected failure.

public static void encrypt(String s)
    {
             System.out.println("_E");
}

2.    Start with the simplest case


We will start with the simplest case. In this case, a two letter string would be good enough to start. We will implement it in the simplest way (as shown below) and yes! we would test it by calling encrypt("ab") and expecting "a b" as the output.

if(s.equals("ab"))
        System.out.println("a b");
    else
        System.out.println("_E");
    …

3.    Build upon it


Now let us build upon our "early win".

Let us implement encrypt("cd"), again with a simple hack.

public static void encrypt(String s)
    {
             System.out.print(s+" ");
         if(s.equals("ab"))
                    System.out.println("a b");
         else if(s.equals("cd"))
                    System.out.println("c d");
         else
                    System.out.println("_E");
}

We can use the same approach for any other tow-lettered string. However, we know there are a large number of such combinations to be handled in if-else. So we refactor the code as shown below. (Note: I have retained the commented code to illustrate the guideline of don't burn the bridges. We can fallback to the commented code if we get stuck.)

/*if(s.equals("ab"))
             System.out.println("a b");
       else if(s.equals("cd"))
             System.out.println("c d");*/
       if(s.length() == 2)
       {
             System.out.println(s.charAt(0)+" "+s.charAt(1));
       }
       else
             System.out.println("_E");
       …

A 3 lettered string would be encoded in a 2x2 matrix. Thus the first column would be char[0] followed by char[2] and the next column will have only char[1]. We can reuse the code for the two lettered string and extend it as follows:

       …
       else if(s.length() == 3) 
       {
             System.out.println(s.charAt(0)+""+s.charAt(2)+" "+s.charAt(1));
       }
       …

And so on for a four-lettered input ...

       …
else if(s.length() == 4)    
       {  
System.out.println(s.charAt(0)+""+s.charAt(2)+""+s.charAt(1)+""+s.charAt(3));
       }
       …

... and a five lettered input ...

       …
       else if(s.length() == 5) 
       {
 System.out.println(s.charAt(0)+""+s.charAt(3)+" "+s.charAt(1)+""+s.charAt(4)+" "+s.charAt(2));
       }
       …


... and don't forget to test each case.

We notice that the code handling the five-lettered input is a rather long line and we could refactor it so as to print the output column-wise, as shown below:

             …
             System.out.print(s.charAt(0)+""+s.charAt(3));
             System.out.print(" ");
            
             System.out.print(s.charAt(1)+""+s.charAt(4));
             System.out.print(" ");
            
             System.out.print(s.charAt(2));
            
             System.out.println("");
             …

4.    Observe the patterns


Let us reuse the code which handles the five lettered string and enhance it to handle the six lettered string.
else if(s.length() == 6)    
    {
        System.out.print(s.charAt(0)+""+s.charAt(3));
        System.out.print(" ");
            
        System.out.print(s.charAt(1)+""+s.charAt(4));
        System.out.print(" ");
            
        System.out.print(s.charAt(2)+""+s.charAt(5));
            
        System.out.println("");
 }
 …

We observe that the above code prints each column. The characters in each column also follows a pattern; viz. char[0] and char[0+3] for the 0th column, char[1] and char[1+3] for the 1st column etc. We refactor the code based on this pattern:


for(int n=0;n<3;n++)
    {
        System.out.print(s.charAt(n)+""+s.charAt(n+3));
        if(n<2)
            System.out.print(" ");
    }
    …


We also observe the pattern that the maximum number of characters in a column is equal to the number of rows (2 in this case). We refactor again based on this pattern.

      
       for(int n=0;n<3;n++)
       {
             for(int m=0;m<2;m++)
             {
                    System.out.print(s.charAt(n+m*3));
             }
             if(n<2)
                    System.out.print(" ");
       }

When we reuse this code for a seven lettered input, we would get an OOB exception as shown below. This is because the code would try to access charAt(7).

for(int n=0;n<3;n++)
     {
           for(int m=0;m<3;m++)
           {
                  System.out.print(s.charAt(n+m*3)); --> OutOfBounds exception!!
           }
           if(n<2)
                  System.out.print(" ");
}

We can avoid the exception with a simple bounds check.


for(int m=0;m<3;m++)
    {
          if(n+m*3 >= 7)
                 break;
          System.out.print(s.charAt(n+m*3));
    }


5.    Combine logically


To handle an eight-lettered input, we would refactor the bounds check as below.

for(int m=0;m<3;m++)
             {
                    if(n+m*3 >= s.length())
                          break;
                    System.out.print(s.charAt(n+m*3));
}

We could see that strings of 7 <= length < 10 will be encoded using a 3x3 matrix and we can refactor the code for all these cases as follows:
    {
             int rows = 3;
             int cols = 3;
             for(int n=0;n<cols;n++)
             {
                    for(int m=0;m<rows;m++)
                    {
                          if(n+m*cols >= s.length())
                                 break;
                          System.out.print(s.charAt(n+m*cols));
                    }
                    if(n<cols - 1)
                          System.out.print(" ");
             }
            
             System.out.println("");
       }
       …  

       We will reuse the code to handle strings of 10 <= length < 13 (which needs a 3x4 matrix) by only changing the number of columns.
else if(s.length() < 13)    
       {
             int rows = 3;
             int cols = 4;
       …

       Now let us compute the rows and columns as suggested in the problem statement. 
       …
       int rows = (int) Math.floor(Math.sqrt(s.length()));
       int cols = (int) Math.ceil(Math.sqrt(s.length()));
       …
   
      However, to handle a string of 13 characters, we will have adjust the rows as follows: 
       …
       if(rows*cols < s.length())
             rows++;
       …

      Thus the logic of our encrypt function would have evolved as follows:

      
       int rows = (int) Math.floor(Math.sqrt(s.length()));
       int cols = (int) Math.ceil(Math.sqrt(s.length()));
       if(rows*cols < s.length())
             rows++;
       for(int n=0;n<cols;n++)
       {
             for(int m=0;m<rows;m++)
             {
                    if(n+m*cols >= s.length())
                          break;
                    System.out.print(s.charAt(n+m*cols));
             }
             if(n<cols - 1)
                    System.out.print(" ");
       }
            
       System.out.println("");
       …


     And yes! we would handle the spaces also ...

       s = s.replaceAll("\\s", "");

6.    Reflect


The programmer may code only for half of the time; in the other half, one should reflect. Reflection may refer to any activity in which the programmer re-looks at the code including testing, refactoring or pairing.

In our example above, let us reflect by adding some more test cases, including some long strings and some special strings (only white spaces etc.).

Let us also refactor, the loop looks like a good place to start with.

To test your understanding of incremental coding, take another programming puzzle. Set the timer and start coding. When the timer fires, answer the set of questions which we have seen in the beginning of this blog. And ask yourself, is your increment well done?