Shashi Velur’s blog

One of the projects that I am currently working on is a Dual-API SDK. This “Framework” has two APIs (or “sides”) - Client API and the SPI (Service Provider Interface) API.

The Client API provides the ability for the Client to deal with various kinds of Artifacts (Requirements, Tasks, CRs, etc.) without paying attention to which “Repository” they came from. Meanwhile, the SPI API provides the ability to ”implement” and “register” the “Providers” (corresponding to various “Repositories” that implement a subset of the SPIs).

Once these SPIs are “registered”, it becomes possible for the “Framework” to provide the “CRUD” operations for the appropriate Artifacts from the “Repositories” that support the given Artifact type(s).

The following requirements decided the backbone-technology to be used for this “Framework” product:

• Ability to dynamically register and un-register SPIs (corresponding to various “Repositories”).
• Ability to support multiple versions of the same SPIs co-existing in the same runtime environment.
• Support modularity of SPI implementations and loose coupling between various modules.
• A low cost (or possibly free) server with a license that allowed us to bundle it with our “Framework” product.
• A Java-based technology.

The most suitable technology for this turned out to be OSGi. This was a suprisingly easy choice. The only other option that came close was Mule 2.0.

Technorati Tags: Java Versioning, OSGi, SDK Development

I had an opportunity to attend a one-day lean manager’s workshop (by Mary and Tom Poppendieck) organized by my employer at my workplace.

It was so much different than the other similar workshops I attended in the past:

• The whole talk was so cohesive - Mary and Tom constantly focused on a few core principles approaching from various perspectives. They arrived at the very same conclusions by using examples and data from various Industries/Verticals in a very lucid style.
• Another thing that resonated with me was the application of Scientific principles from Queuing theory, Fluid dynamics and emphasis on using actual measurements as data to improve. Also, they used some examples/directives from the Military/Army manual (on communication) and applied them to Lean/Agile Software development.

I will be blogging more about what I learnt and how I plan to apply them at work.

Technorati Tags: lean software development

This is awesome. I am excited to see the new browser wars. This reminds me of Netscape Navigator/IE 3.0 days..!

I hope this brings some much needed out-of-the-box-thinking of the whole browser-to-server interactions. Here is a good list of the feature requests by OpenAJAX Alliance.

In my career so far, I have written various types Rich-Client applications on Browser using various forms of HTTP tunneling for server interaction:

• Java AWT/Swing Applets: Custom Java serialization
• Flash/Flex: Flash Remoting/AMF/ASTranslator
• AJAX: XML-RPC, JSON

All of the above suffer from at least one issue that I know of. The browser client application has to poll to get server updates to the Clients. There is no server-HTTP-push. This leads to unnecessary complexity in the Client-side code (usually in JavaScript or ActionScript).

I hope there will be some sort of an effort towards raw sockets (or something similar but more powerful than XHR) for the browser with proper sandboxing for the plugins to reduce the security issues due to raw sockets. This means that JavaScript language might have to include some support for network programming too.

Another thing I really wish is a simple/powerful API on the Google browser to write new plugins and widgets.

Technorati Tags: browser wars, Google browser

All the individual products in our organization use continuous integration development practice. However, all the products are built by different teams (with different skill-sets) under Sprints that do not share the same boundaries.

Some of these are complete products by themselves. They also are bundled together in a suite and so require integration testing. How do we apply continuous integration practice on a grand scale to this problem?

We ended up making these decisions:

1. The individual product team (with “inward” dependency) shall take the responsibilities of writing/maintaining the integration test cases.
2. We decided to push the individual product builds into this environment every week, mostly and also have the flexibility of being able to push the builds on demand, if necessary.
3. Start with a basic set of integration test cases and build upon them to keep it simple/Agile.
4. Start with manual steps to deploy these individual product builds into the integration environment and then slowly build a set of scripts to automate this.
5. Eventually, we should have a “push-button” integration environment where individual teams can push a button or set up rules to promote their builds in to this integration environment.
6. Decide that all teams should eventually move to a single source repository.
7. Initially, to keep it simple, all of the individual product teams will push only binaries into this environment.
8. We agreed upon a simple defect tracking process for reporting/escalating integration testing bugs.

Technorati Tags: Continuous Integration, Product Development

There are many real-world scenarios where Hibernate can be introduced on a legacy system that could be using straight JDBC for various reasons.

Also, there could be some pre-existing custom Java types to Database type mapping in the legacy application. Hibernate has it’s default mappings for each database excapsulated in the appropriate SQLDialect classes.

Fortunately, there is a very simple way to customize/override these SQL Types to Database types mappings by extending the SQLDialect class implementation corresponding to the Database in question.

The org.hibernate.dialect.SQLDialect abstract class has these following methods that are provided for overloading the SQL Types to Database types mapping. Also, there is a method to override the standard Database function name mappings:

• protected void registerColumnType(int code, int capacity, String name)
• protected void registerColumnType(int code, String name)
• protected void registerFunction(String name, SQLFunction function)

Here is an example of the usage of one of the above methods to override a mapping of a double (Java SQL type) to a “real” MS SQLServer type.

package com.x.y.common.dao;
import java.sql.Types;
import org.hibernate.dialect.SQLServerDialect;
public class MyMSSQLServerDialect extends SQLServerDialect {
  public MyMSSQLServerDialect() {
    super();
    //Override the default mappings for MS SQLServer
    registerColumnType( Types.DOUBLE, "real" );
  }
}

Also, there is a way to add custom value types in Hibernate by implementing either org.hibernate.UserType or org.hibernate.CompositeUserType.

Technorati Tags: custom Types mapping, Hibernate, Java SQL Types

I am working on a product that uses Hibernate that required DB (schema and data) migration, like most Enterprise Software.

Hibernate’s schema create/update feature (using hibernate.hbm2ddl.auto property) has it’s limitations. Also, certain complex DB migration requirements will definitely require custom coding.

So, I had to set hibernate.hbm2ddl.auto=validate due to the following known limitations with this Hibernate feature:

1. Cannot drop a table or a column
2. Cannot change a constraint on a column
3. Cannot add a column with a not-null constraint to an existing table

The possible approaches/alternatives for DB migration that I could think-of were:

1. Build a custom DB migration framework that uses separate SQL(DDL&DML) scripts and/or ETL scripts (Java/Ruby/Python/Groovy scripts) - flexible, efficient, could become unmanageable.
2. ETLM - lot of work required, process decoupled from the application, custom ETLM process for every upgrade.
3. Third party DB Upgrade/Change management tools:
   • Liquibase - XML based (Not very useful for most applications that do not maintain DB schema in XML)
   • AutoPatch
   • DBMigrate (Is this production-ready?)

The other design decision was to make the application “smart” so that it:

• Maintains the version and executed script info. so that the same old scripts are not executed again for every upgrade.
• Performs the necessary incremental upgrades based on the version number of the newly installed upgrade (read from the Jar manifest files) and the last installed version (read from the DB Version table).
• Requires no effort from the end-user of the application for upgrading the application.

I chose approach #1 listed above. The following section details the framework for Database Data and Schema Migration:

• This will require at least 2 new tables, “Version” and “MigrationScript” tables to hold the last upgraded version and the set of all migration scripts successfully executed so far.
• Before making any changes to the Schema or data for migration, the Database needs to be locked. It will be unlocked after all the migration changes are done.
• The main application Jars will have a Manifest entry that indicate the version of the Software that is installed. So, the application needs to migrate the Database from the version table in the database to the installed version and apply the necessary scripts.
• Designed File and Directory naming conventions for the script names:
  • File extensions:
    • SQL DDL –> .ddl
    • SQL DML –> .dml
    • Java/Ruby/Python/Groovy ETL Scripts –> .txt (This text file will have the fully qualified Java Class name or the name of the Ruby/Python/Groovy script to be executed. The scripts themselves are assumed to be on the classpath either directly or inside a Jar file).
  • File and Directory Naming conventions: The directories (that hold the migration script names) are based on the Major versions of the Software and the target database. There will be a “common” directory for all database-agnostic scripts (typically simple DML SQLs). The Files in those directories are named using a 4-digit number prefix that will help sorting. So, the DB migration module in the application will execute the scripts after sorting the scripts in alphabetical order. The number prefix gives the flexibility to name the script whatever the developer decides (for example, 0001_create_tables_for_baseline.ddl). This provides the ability to mix all kinds of the above scripts (DDL, DML & ETLM scripts) which makes this approach very flexible and powerful.
• All of the scripts are packaged in a Jar (or a Zip) file and deployed such that it is available in the classpath (as a resource).
• The DB migration module shall update the db version table with the new version/revision number and migration scripts table with all the successfully executed script names.
• The Java Class names specified in the Java ETLM script will have to implement a “Migrator” interface which will be called from within the DB migration framework. This is necessary for complex migrations.
• The framework will also need to take into consideration of the “Dev” builds vs “Production” builds. The “Dev” builds (typically executed from an Eclipse launcher) may not have the correct Manifest entries. So, the build version is set to ”-1″. The “Production”/QA builds will be run using ANT which will have the right Version info.

One of the major problems with this above approach is that the Developers will have to check-in the Scripts (if DB-specific) for all of the supported databases separately (in to separate directories). I created an ANT task that generates the DDL from a given hibernate mapping configuration. This serves as a starting point for new tables. On the flip side, this forces the developers to understand the tables that they are creating and also force them to make decisions. Also, testing migration scripts for several databases can be challenging, but, a continuous integration build setup for each DB should take care of this problem.

Next, I shall blog about the implementation to force Hibernate to use certain Database mappings and override the default Hibernate SQL type mappings.

Technorati Tags: Database Migration, Database upgrade design, DB migration, Hibernate

I have come across several articles on the best practices for Agile/SCRUM.

I read this somewhere and really liked it:
There is not and never will be a list of “Scrum Best Practices” because team and project context trump all other considerations.

Here are 5 practices that may be considered as “not-so-agile” by purists. But, making the following changes to our Agile/SCRUM process actually made us a very effective Agile Organization..!

Dedicated Testers & Writer on the SCRUM Team:
This could be against the “No specialization” or “No fixed Roles” Agile principle.

Minispecs:
This could be against “use the working code as documentation”. We use MiniSpecs very effectively to document key requirements and design/implementation decisions in a concise manner.

Our SCRUM Master is our Sr. Director/Manager:
This is self-explanatory. Our Manager is actually a certified SCRUM Master with two decades of development experience. He demonstrates great balance.

Team members shared across teams:
We have a UI specialist and an Architect/Developer shared across 2 teams. I guess it might make sense to share the UI specialist. The Architect/Developer shared across teams has helped us immensely improve communication and to effectively sort out integration issues across the products in our Product Suite.

Multiple backlogs for the same product suite:
Okay. This one might be up for debate. Our Product suite consists of at least 5 products that have their own SCRUM Teams. There is also a Product Owner with individual Product backlog for each of these products. Also, there is a common backlog for infrastructure/common functionality.

Technorati Tags: Agile, Agile Best practices, SCRUM

I have been using SCRUM and XP principles/practices since early 2003 for various software projects.

Here is my take on how SCRUM/Agile software development has affected the typical roles in Software development for the Organizations adopting SCRUM/Agile.

Management, Leadership:

Customers/Clients:

Product Owners:

SCRUM Team/Developers:

Technorati Tags: Agile, Agile Development Roles, Agile Management, SCRUM

I started blogging in 2004 and stopped after a few entries. Since then, every time I solve a difficult design/coding problem or find something interesting to share, I have been thinking about making a commitment to blog at least once a week.

It just took me about 4 years to make that commitment.

Technorati Tags: Add new tag