Thursday, February 4, 2016

LoopBack for ScrumTime? - Not at this time

Quick Update:

In the previous post, I began documenting my findings on using LoopBack as a possible REST API layer for the ScrumTime 1.0 release.  The question:

Should I use LoopBack for the ScrumTime 1.0 release?

The answer is no.  I have decided to go with the original plan of using .NET Web API for the REST layer for ScrumTime 1.0.

My research on LoopBack indicates that it is very promising but the ScrumTime 0.9 Beta code base has a functioning business logic layer implemented in C#.  So, in the interest of time and simplicity of design, I am going to complete ScrumTime 1.0 with .NET Web API 6 and then pursue the use of LoopBack or a custom Javascript REST API in a new open source project, FeatureTrack, planned after ScrumTime 1.0 is complete.


Wednesday, December 30, 2015

LoopBack for ScrumTime?

Should I use LoopBack instead of roll-your-own API for ScrumTime?

This post and at least one more in the future are intended to document my findings on this subject.

First of all, what is LoopBack?

LoopBack is an open source Node.js API framework created by StrongLoop, Inc.  It provides a quick and easy way to stand up a REST API that is backed by an in-memory database, Oracle, MySQL, SQL Server, MongoDB, PostgreSQL, Redis, or Cloudant.

Quick and Easy, Famous Last Words, Right?

While the 'quick and easy' aspect is appealing, I also want a framework that is open, secure, flexible, and scalable.  Based on my first impressions with LoopBack and the free tools from StrongLoop, I am pretty happy with the 'quick and easy' promise.  I spent only one hour getting the following API stood up against the ScrumTime 0.9 Beta1 SQL Server database.



The products endpoint is as follows:

http://localhost:3000/api/Products

[{"productid": 0,"name": "Test Product","description": "Just a test"},{"productid": 1,"name": "Sample Inventory","description": "A sample product for demonstration purposes."},{"productid": 7,"name": "Android GeoCache","description": "A sample product for demonstration purposes."}]

Is it really open?  Really?

Well, the source may be found at https://github.com/strongloop/loopback.  It appears this is the entire set of source required to build it.  They also use waffle.io to manage the project at: https://waffle.io/strongloop/loopback. Therefore, it seems pretty open to me.  

Is it secure?

So far I have only scratched the surface of what the capabilities are documented to be in terms of authentication, authorization, and auditing.  I will need to cover this further in my next post as ScrumTime will be using all three.

The questions that I will address in the weeks to come include:

  • Is it possible to put business logic in the API layer?
  • How can role-based access controls be implemented in the API?
  • Is it possible to do push notifications and if so, how?
  • How can 3rd party login providers be used such as Facebook, Google, or GitHub?

Wednesday, January 22, 2014

DevOps and You

What is DevOps?

It is a concept that seeks to remove the communication barriers between development, QA, and IT (operations).  When all teams are communicating openly and effectively, the following benefits may be achieved:

  • shorter release cycle for new features
  • lower percentage of bugs that make it to production
  • better utilization of the organization's hardware and cloud resources

Is there a precedent for this concept?

In the 1990's, Toyota implemented a philosophy of "lean manufacturing" that reduced their production costs while simultaneously increasing their production line flexibility and their product quality.  This is the same goal of the DevOps concept.  Any software organization that has competitors (not all do) must constantly focus on staying competitive by reducing costs, increasing the product innovation, and reducing the relative percentage of bugs released into production.

What are specific issues that need to be addressed?

First I must point out that it is very difficult to change the behavior of groups of people when the behaviors have been entrenched for many years.  For example:

If you are a developer, have you ever been frustrated with QA because they never seem to find your bugs unless it is 5pm on that same night that you plan to see your four year old daughter's ballet recital?

Or, if you are in QA, have you ever been frustrated with development because they never seem to give you a build until it is 5pm on the same night that you plan to reconcile with your girlfriend/boyfriend at her/his special restaurant?

Better yet if you are a system administrator, have you ever been frustrated with everyone because no one tells you of a major release until 5pm on the same night that you plan to go night hang gliding with a bright flashlight?  (I ran out of ideas on that one...I hope you thought it was funny.)

The point is that most software development organizations are set up to keep the teams focused on their task at hand.  Often, there is not much inter-team communication.  Once the teams do communicate, it is often incidental or emergency event driven.  Therefore, the fundamental change required to begin thinking in terms of "lean" or DevOps is to open communication between teams.

The goal of opening up the communication is to allow team members from each team to understand the challenges faced by each team.  In other words, each team member should begin to understand that their team is only a part of the greater whole, the company/organization itself.

How can behavior be changed?

In order to bring about change in behavior within an organization, there must be a high ranking sponsor that can champion the change.  There must also be a technical leader that is viewed as objective among the teams.  This is often an architect, a release manager, or a technical director.  The technical leader would work with the teams to identify tools and procedures that facilitate communication between the teams and visibility within the teams.

Beyond leadership, what else can drive change to behavior?

Tools.  Tools act as the catalyst to change the behavior of the groups within the organization.  The tools chosen must be capable of showing measurable results.  For example:

Conclusion

DevOps may not be new to you if your organization has already been communicating across teams effectively and releasing products continuously with fewer bugs.  However, there are still plenty of companies that have not had a need to focus on efficiency of product development.  So, if you are a developer, a tester, a system administrator, a support engineer, a release manager, an architect, a manager, a director, a product owner, a business analyst, a sales person, or an executive in a company that does not focus on DevOps by name or otherwise, then you need to consider it.  DevOps, the term, has been around since 2009 (as far as I know).  DevOps the practice has been around since much longer.  However, there are many tools that are now showing their value in the real world.  This means that your competitors may be slowly converting to the DevOps approach or converting all at once.

It is my opinion that DevOps is here to stay.  It is possible that the term may change over time, but the tools that facilitate the process are not going away.  Furthermore, the companies that choose to make use of these tools will continue to evolve over time.  Their competitors that do not follow the DevOps approach will likely not be able to keep up and potentially lose customers as a result.  

Thursday, January 2, 2014

Software Design Patterns...Why Are They Important?

Background.

In 1991, I was just a co-op at IBM in Charlotte, NC.  If you had asked me then about software design patterns, I would have thought you were talking about algorithms or data structures.  At that time I was writing C and providing QA for a large banking image recognition project.  Since then I have been exposed to 6 years of C++, 14 years of Java, 8 years of C#, 15 years of HTML, and 8 years of JavaScript.  My employers have ranged from 4 different multi-billion dollar corporations and 6 different start-up companies.

Around the year 2000, I was working at a start-up where "speed to market" was the name of the game.  However, I learned that if I keep my source code organized and easy to read, then I will be able to change it more quickly when the investors come around to tell us what the new direction for company is going to be (every six months, haha).  I latter learned that I was following a practice known as separation of concerns (SoC).

By 2004, I was working on a large contract for the Department of Defense.  This required a huge amount of interaction with many talented developers from all over the country.  I was force fed the software design pattern of Service-Oriented Architecture (SOA) that brought 19 different DoD organizations together to share their data in order to fight global terrorism.  At the time, there were aspects of it that felt like overkill, but I was happy to be able to follow the concepts and jump right in to help in the coding effort.

In 2007, I was leading my own team at Martin Marietta to design and implement a web-based product that would expose billing information to customers in a highly scalable manner.  Based on my experience of the past, I decided to focus on using design patterns so that each member of the team would have a baseline for understanding my expectations for them.  Therefore, I realized that design patterns would help me create a common dialog for which to build discussions about the problem domain.  Furthermore, by using the patterns in the implementation, it was easier to split up the work among different developers.  The project became very successful and is known as www.erocks.com.  Erocks uses Spring MVC, a repository pattern (Hibernate underneath), and Inversion of Control (IoC).

The Point.

I could go on and on about projects that I have seen using patterns that have been successful.  I could also tell you about a few projects that have been unsuccessful using patterns...but it was always due to a problem with funding or with business aspects outside of the control of development.  Of course, there is the occasional misuse of patterns that cause great confusion and disarray in projects, but I have had only encountered that a few times in my career (thank goodness).

The point is that I am a huge fan of Software Design Patterns.  Not because they sound cool (well, maybe only nerds think they sound cool).  I believe that patterns allow us to get to what matters the most about creating software, understanding the problem domain and doing something about it.

Imagine that you are a gifted developer that always works alone and always remembers the frame of mind that you were in for every line of code you ever created.  You would not need any design patterns in this case.  You could always just say, "Hey, it works doesn't it!"  However, if you are like me and can't always look at a line of code you wrote last year and recall the reason for it immediately, then you probably will benefit by having patterns.

Patterns also help with identifying the overall design strategy when architecting a solution.  Odds are that there is another developer in the world that has already created your design but maybe it was not applied to your business domain.  Why not take advantage of that fact and learn from the problems that have already been solved and resolved by hundreds or even thousands of others.  This is what Patterns are.

Why Are Software Design Patterns Important?

Software Design Patterns are important because they facilitate a baseline of communication among teams using common terms and concepts, they provide a foundation of solutions to be used in solving problems in new domains, and they allow implementations to be assessed in terms of meeting an established expectation or standard.  The bottom line is that these benefits save money for the company and save time for the developer.

Friday, June 15, 2012

Grails 2.0 vs ASP.NET MVC 4 (Conclusion)


Background
This is the conclusion to two previous posts that considered the pros and cons of using Grails 2.0 vs ASP.NET MVC 4 for open source projects.  I have used both technologies for a few years apiece.  However, their latest incarnations have offered benefits over their respective predecessors.  Therefore, it was due time for me to reevaluate.


If you read my last post on this topic, ScrumTime - Grails 2.0 vs ASP.NET MVC 4, you saw that I came up with seven questions to use for the comparison.  I answered each question to the best of my ability at the time.  Since that post, I have researched both frameworks with two different open source projects: ScrumTime and BumbleBee.


ScrumTime source code on GitHub:
BumbleBee source code on GitHub:
If you are bored and actually want to view the source of all four projects, you will notice that the project with the most active work is the BumbleBee Grails 2 implementation.


What is BumbleBee?
BumbleBee allows project managers to track the progress of large software integration projects.  It provides a single view of Mantis bugs, test scripts, unit tests, design documents, and third party vendor issues.  The default security is currently single-sign on with active directory.  The SSO with AD is a post all its own since it is implemented in a java web stack.


What happened to ScrumTime?
I have only put ScrumTime to the side for the moment because I have a business "need" for the BumbleBee application immediately.  It seemed like an excellent candidate to finish my evaluation of Grails vs MVC.


What is the final web-stack decision and why?










I decided to complete the BumbleBee project in Grails 2.0 because of many factors:

  • velocity of implementation is quicker for me due to less source code and the following
    • repository finders are dynamically generated automatically (MVC does not have this)
    • pre-request, post-request, and post-view filters are much simpler to implement than MVC.
    • the default in-memory database supports live changes to the domain models without a server restart (MVC does not have this)
    • nearly all aspects of the web application can be changed without requiring a server restart (MVC has limited support for this)
  • the end result is a war file that will run on any hardware that can run jdk 6+ (MVC can only run on windows based computers (Mono would require significant EF changes)
  • more database vendor options after the product is developed
    • Because Grails is based on Gorm which is built on Hibernate, changing the back-end database vendor after the entire application is complete does not pose a challenge.  This can be done by changing the properties in the DataSource.groovy file.  However, the story with MVC is pretty bad.  The only options in MVC are to use nHibernate or create your own ORM from scratch.  Even in these cases, there is no automatic integration between the model objects of the MVC application based on nHibernate as there is with model objects based on the Entity Framework.  I can only guess that Microsoft prefers to make it easy for developers to connect to their own databases and not so easy to connect to other vendor databases.
These factors were too important to the success of an open source project for me to ignore.  

I hope to complete the first release of BumbleBee within the next few weeks.

Wednesday, May 2, 2012

"NoSQL" Databases - (MongoDB)

I have recently become a big fan of the "NoSQL" databases.  When I first read about them, I thought that the entire concept seemed a bit like fantasy land because of its simplicity and elegance.  It seemed too good to be true.  However, there are already thousands of world-class applications using these very real databases.  In fact, companies like Netflix, Facebook, and Twitter (to name a few) make extensive use of NoSQL.  These three alone make use of Apache Cassandra in particular.


ThoughtWorks has an excellent introduction to NoSQL databases that I recommend to anyone that wants to learn more about the subject.  The most important question to ask of any new tool or technology is: 'How is this thing going to help me do my job better or faster?'  


Taken directly from the ThoughtWorks post...

Why would you want to use a NoSQL database?
One of fundamental drivers is that you have challenges in your business that are difficult to solve using traditional relational database technology. If you have an excellent relational model running on a mature database that provides all the features you need then there is probably little need to change your data storage mechanism. Here are some use cases where it is sub-optimal to use a conventional database: -
  • Your relational database will not scale to your traffic at an acceptable cost
  • Your data is supplied in small updates spread over time so the number of tables required to maintain a normal form has grown disproportionally to the data being held. Informally if you can no longer print your ERD on an A3 piece of paper you may have hit this problem or you are storing too much in a single database.
  • Your business model generates a lot of temporary data that does not really belong in the main data store. Common examples include shopping carts, retained searches, site personalisation and incomplete user questionnaires.
  • Your relational database has already been denormalised for reasons of performance or for convenience in manipulating the data in your application.
  • Your dataset consists of large quantities of text or images and the column definition is simply a Large Object (CLOB or BLOB).
  • You need to run queries against your data that do not involve simple hierarchical relations; common examples are recommendations or business intelligence questions that involve an absence of data. For the latter consider "all women in Paris who do have a dog and whose ex sister-in-laws have not yet purchased a paperback this year" as a contrived example, "all people in a social network who have not purchased a book this year who are once removed from people who have" is a real one if you want to target advertising on a site that says "Fred bought X".
  • You have local data transactions that do not have to be very durable. For example "liking" items on websites: creating transactions for these kind of interactions are overkill because if the action fails the user is likely to just repeat it until it works. AJAX-heavy websites tend to have a lot of these use-cases.


There are two of these bullet points that I find interesting in the context of ScrumTime 1.0.  The next release of ScrumTime will have extensive support for attaching artifacts to nearly all object types.  For example, a Release may have a Statement of Work document uploaded and associated with it.  Also, a Story may be associated with a few photographs of a whiteboard discussion for which it relates.  Therefore, the ScrumTime dataset will consist of large quantities of binary objects.  Also, the collaborative nature of ScrumTime is an excellent fit for AJAX as was shown in release 0.9.  The 1.0 release will use AJAX in a similar way that Facebook and Google use it to promote team collaboration and communication in so far as indicating another user is online.

Due to the fact that the purpose of ScrumTime is to provide a usable open source agile project management tool to the masses, it stands to reason that providing a simple setup and configuration to get running quickly is a high value requirement.  Therefore, when considering the use of both a relational database and a NoSQL database to solve all of the data storage needs of ScrumTime, I have to think about the impact to the guy or girl that deploys ScrumTime.  If that person has to setup two databases, they are not going to be happy with me.  I could write a setup application to configure both databases, but that effort takes away from adding value to the ScrumTime product itself.  Therefore, if all of the features of ScrumTime may be implemented in one database, all the better.


In the past week, I have implemented the ScrumTime domain models in a NoSQL implementation called MongoDB.  I also tried CouchDB and RavenDB.  I like all three of them, but there is no standard querying syntax between them.  This had me wondering which of the three would be the best choice for ScrumTime since none of them seem to follow a standard convention.  But, what is a standard convention...well, it is usually dictated by the most popular choice.  As a result, I decided to research the number of open jobs on the market in the United States that mentioned either of the product names.  The winner by far was MongoDB which tells me that it is currently the most popular.  I was also able to find numerous options for language specific database drivers and lots of documentation for MongoDB.  So, MongoDB is the database choice for ScrumTime 1.0.

If you have not had a chance to look at NoSQL, please make time.  It is not all hype.

Friday, April 20, 2012

ScrumTime - Grails 2.0 vs ASP.NET MVC 4

In my previous post, I documented the process of considering the various "Web Stacks" that are widely available for the implementation of ScrumTime 1.0.  This post is a continuation of that post.


Here are the factors that go into considering these two "Web Stacks" as viable open source platforms:

  1. Does the platform support multiple databases in such a way that does not require a major refactoring for each database supported?
  2. Does the platform have native support for a multilingual user interface?
  3. Does the platform support a modular view solution that can support various screen sizes and capabilities?
  4. Does the platform allow the business logic layer to be decoupled and reused in other non-web application capacity?
  5. Is there a large development community that will be able to look at the code base to add to the project at a future date?
  6. Is the platform significantly cheaper to develop and deploy than the competition?
  7. Is the platform stable over a long period of time? (do security patches to the OS threaten the features of the deployment?)

Last week, I began implementing ScrumTime 1.0 in ASP.NET MVC 4.  The source may be found here.  This week I began implementing ScrumTime 1.0 with Grails 2.0.  The source may be found here.


Here are the answers to the questions above:


Grails Answers

  1. Yes. Grails uses hibernate under the covers.  Therefore, Grails has support for all major database vendors and the JDBC drivers are widely available.
  2. Yes. Grails has a large focus on multilingual user interfaces with the inclusion of the i18n folder with each newly created project.  This folder holds the resource files that contains the different strings for the different languages supported.
  3. Yes. Grails supports a concept of "layouts" and "templates" that allows a developer to create and reuse view components.
  4. Yes. Grails compiles down to a set of Java classes that may be referenced from any other Java application running on the same VM.
  5. No. Grails is built on Groovy which is a dynamic language that compiles into byte-code that runs on the Java virtual machine.  Unfortunately, the number of Groovy developers in the world are far outnumbered by the number of C, C++, VB, C#, and Java developers.
  6. Yes. Grails is much cheaper to develop and deploy.  It is conceivable that the cost of development and deployment is free since it can be done with open source tools and platforms.
  7. Yes. Since Grails runs in a JVM, there is little to no relationship between the JVM and the operating system.  
ASP.NET MVC 4 Answers
  1. Yes, well sort of. MVC 4 supports the latest Entity Framework 4.1 that includes something known as "Code First".  Code First support all Microsoft databases, Oracle, MySQL, and supposedly any database that has an ADO.NET provider.  Another option is to use NHibernate instead of EF at all.  NHibernate supports all of the major database vendors.
  2. Yes. MVC 4, like all .NET applications, may contain "resource files."  There are several options for making use of the resource strings, but the most clear option is to use a reference in the model's constraint annotations.
  3. Yes. MVC 4 allows developers to define "layouts" and "partial views" that follow the view reuse best practices of other web frameworks like Grails and Ruby on Rails.
  4. Yes. The most straightforward way to implement a reusable Business Logic Layer in MVC 4 is to implement the BLL in a separate .NET DLL.  Then the DLL may be used by the MVC 4 web application.
  5. Yes. MVC 4 may be built on .NET VB or .NET C#.  The usage of C# seems to be growing versus VB according to indeed.com job trends.  Therefore, in the comparison with Grails, I have focused on MVC 4 .NET C#.  The C# open source community is very large.  It is not quite the size of Java or C/C++, but it is global.
  6. No. The cost of development is basically free if Visual Studio 2011 Express is considered sufficient (I have been using Visual Studio Professional).  The cost of deployment is likely more than zero...it depends on the amount of data that is required to be stored and the number of users that would be connecting. 
  7. No.  The .NET framework is affected by system updates to the Windows operating system.  While it is possible to separate the .NET specific updates from the rest of the operating system updates, that is a hassle.  Also, due to the success of the .NET framework for Microsoft over past ten years, it has become more intertwined with the operating system.  As a result, it is not always clear where .NET begins and and the operating system ends.  Therefore, at the very least, ASP.NET MVC 4 requires Windows operating system knowledge and research for each patch deployment...that is an ongoing cost that is arguably greater than that of maintaining a JVM. 

Based on the criteria above, the ASP.NET MVC 4 "Web Stack" seems to be the prudent choice because of the large development community.  However, the development cost and deployment cost is not as positive. By April 23, 2012 June 15, 2012, I will make the final decision and proceed with the full implementation at that time.


*Update (04/23/2012) - ASP.NET MVC 4 is the first to be heavily researched.  The source may be found here on GitHub. 


*Update (05/14/2012) - Grails 2.0 heavy research begins.  The source may be found here on GitHub.