Search This Blog

Friday, November 13, 2015

Continuous Deployment Using Jenkins, ANT and GitHub for IBM Integration Bus

In my earlier post we dealt with having to use Jenkins, ANT and SVN for Continuous Integration for IBM Integration Bus. In this post lets take it forward and do Continuous Deployment across different environment. Just for some change I decided to use GitHub as the source repository instead of SVN.

So with Jenkins already running we need to add the plugin for GitHub and also need to setup the security. To do that select 'Manage Jenkins' on the left of the screen to open the panel -

Click on the 'Manage Plugins' link and search for 'promoted plugin' with the 'available' highlighted and select 'Install without restart'

Next move to have the security setup. For this click on the 'Manage Jenkins' on the left of the screen as in the first image and click the 'Setup Security' blue button.Setup the security with 'Jenkins' own user database' and 'Allow users to sign up' selected.

Next up set the users that we are going to use for this. I have created the user 'admin' as the admin user for Jenkins and 'intuser' is the developer role to deploy the code to INT, 'testuser' will approve the deployment to TEST and mark it QA Build. 'produser' will approve the deployment to PROD after it has been set to 'GA Release'

Click on 'Apply' and save it by clicking on 'Save'. Next up is to create the users that we have mentioned in the user. Click on the 'Create an account' to create the users.
For simplicity sake I have kept the details simple just to the user id. Create the other users similar to this. 
Now that all the users are created login using user 'intuser' and lets work through the process to deploy the code across different environment with process approval. In this case I have the code in a GitHub repository and will pull the code to a local workspace and build it and deploy to the IIB runtime.

Before we start building the job in Jenkins, lets create the build file that we are going to use to create the bar file and deploy in INT and when promoted run the script to deploy the bar file to TEST and PROD.

First up is the build file to build the bar file and deploy to INT - 

<?xml version="1.0"?>
<project name="project" default="run">
<target name="run" description="">
<antcall target="mqsideploybar" />
</target>

<!--Target to build the broker archive using mqsipackagebar-->
<target name="mqsicreatebar.buildbar">
<echo message="Building Broker Archive file: ${APP.NAME}-${BUILD.NUMBER}.bar " />
<exec executable="${IIB.CREATEBAR}\mqsipackagebar.bat" spawn="false">
<arg value="-w" />
<arg value="${IIB.WORKSPACE}" />
<arg value="-a" />
<arg value="${BARDIR}\${APP.NAME}-${BUILD.NUMBER}.bar" />
<arg value="-k" />
<arg value="${APP.NAME}" />
</exec>
<echo message="Completed building Broker Archive file - ${APP.NAME}-${BUILD.NUMBER}.bar " />
</target>

<!--Target to deploy the broker archive using mqsideploybar-->
<target name="mqsideploybar" depends="mqsicreatebar.buildbar">
<echo message="Deploying Broker Archive file: ${APP.NAME}-${BUILD.NUMBER}.bar " />
<exec executable="${IIB.DEPLOYBAR}\mqsideployscript.bat" spawn="false">
<arg value="${IIB.RUNTIME}" />
<arg value="-e" />
<arg value="${IIB.EG}" />
<arg value="-a" />
<arg value="${BARDIR}\${APP.NAME}-${BUILD.NUMBER}.bar" />
</exec>
<echo message="Completed building Broker Archive file - ${APP.NAME}-${BUILD.NUMBER}.bar " />
</target>
</project>

I have parameterized the build file to accept - 
${IIB.CREATEBAR} - The application name we are building.
${BUILD.NUMBER} - The build number-the build number of the job.
${BARDIR} - The location of the bar file after it has been packaged.
${IIB.RUNTIME} - The IIB Runtime to deploy the bar file to.
${IIB.EG} - The IIB Execution Group to deploy the bar file to.
${IIB.CREATEBAR} - The location of IIB's mqsipackagebar.bat file location.
${IIB.DEPLOYBAR} - The location of IIB's mqsideployscript.bat file location.
The last two properties are the same for IIB10 but anyone using IIB9 and before it would be different.

Moving on to the the build file to promote the bar file to TEST and PROD - 
<?xml version="1.0"?>
<project name="project" default="run">
<target name="run" description="">
<antcall target="mqsideploybar" />
</target>

<!--Target to deploy the broker archive using mqsideploybar-->
<target name="mqsideploybar" >
<echo message="Deploying Broker Archive file: ${APP.NAME}-${PROMOTED.NUMBER}.bar " />
<exec executable="${IIB.DEPLOYBAR}\mqsideployscript.bat" spawn="false">
<arg value="${IIB.RUNTIME}" />
<arg value="-e" />
<arg value="${IIB.EG}" />
<arg value="-a" />
<arg value="${BARDIR}\${APP.NAME}-${PROMOTED.NUMBER}.bar" />
</exec>
<echo message="Completed building Broker Archive file - ${APP.NAME}-${PROMOTED.NUMBER}.bar " />
</target>
</project>

Note that most of the properties for this script remains the same except that PROMOTED.NUMBER takes the position of BUILD.NUMBER.

Create a new job in Jenkins logged in with user intuser -

In this instance I am creating the project 'MFA_AggregationStockQuote_GitHub' as a Freestyle project and click 'OK'. The build file needs 3 properties that we are going to pass on in here. They are - 
IIB.RUNTIME - IIB.INT
IIB.EG - default
APP.NAME - MFA_AggregateStockQuote
Specify the location of the project. In this case we are using GitHub as the repository. Jenkins GitHub plugin allows it to retrieve the files to the local workspace. For simplicity sake I am getting the full repository that includes the sub flow projects I will need. 
Scroll down to the build section and specify the build.xml to push the bar file to the INT environment in this case IIB.INT. Also specify the properties that are needed in the build file as shown.
Now that we have configured the build file to push the bar file to INT environment, we would like to push that bar file to TEST environment if the job is successful. To do this search for the term 'Promote builds when' on the same page and select the check box. Name the promotion process which will be helpful to indicate that it is to be promoted to TEST-in this case 'MFA_AggregationStockQuote_GitHub-ToTEST'. Setup the approvers, in this case it is going to be 'testuser'. Setup the parameters for deployment to TEST i.e. IIB.RUNTIME - IIB.TEST and IIB.EG-default . Enter the ANT script to promote the bar file to TEST and specify the parameters.

Next up, add another step to deploy the bar file to PROD after it has been successfully deployed to TEST. In this case the job will become active after the deployment to INT is successful and promotion to TEST has been done. The job details remain the same as TEST promotion except that the IIB runtime we are changing to IIB.PROD. Also not that the promotion to PROD is reliant on manual approval as well as the promotion to TEST being a success.
Save the job and lets run the build as intuser to initiate the deployment to INT environment. Click on the 'Build with Parameters' link on the left. Make sure that the parameters are as we envisioned and click on 'Build'

This will start the build and the progress bar would be visible in the Build History as shown. 
Click on the date time on the progress bar and then click on the 'Console Output' link on left to view how it is progressing. 
In the end the build was successful and it was deployed to the IIB.INT to execution group default.
A confirmation of the same cane be viewed in here.
 Click on the Promotion Status and this will present the status of the promotion.

Before we move on further click on the workspace link on the left and it will show you the artefacts that have been used to create the bar file - 

Now log off as the user intuser and login as user testuser and click on the job 'MFA_AggregationStockQuote_GitHub'. Click on the Promotion Status and it will show you if any of the builds have been done for the mentioned job as in screen below.
Click on the build number on the left in this case '#8' and click on the Promotion Status. Scroll down to note that this build is now available for TEST deployment because the INT build succeeded and manual approval is requested which we do by clicking on the 'Approve' button. Confirm the parameters before clicking the 'Approve' button.
This will start the promotion to TEST environment and a link will appear to show the log info for the progress. Click on the link to check the console output.
The console output shows that the promotion to TEST was a success and the clarification can be noted from the messages.
Click on the build number and the promotion status and you will notice the update. Now the promotion to TEST job has been updated to show when the last promotion was done and a Silver start appears next to the build number to indicate the success.
Confirmation of the successful deployment of the bar file to IIB.TEST
Next up log off as user testuser and login as the user produser and click on the job and next the build number. The 'Approve button is available to be clicked to deploy the bar file to PROD environment as the TEST promotion was successful and we are going to manually approve the promotion. Click on the Approve button and immediately a link will appear to show the progress.
Click on the link to check the progress.
It will show that the job has been successful and the promotion number will be adjacent.
Click on the job number and now we have the gold star beside the silver star to indicate that the promotion to PROD has been successful. The permalinks show the time when it was done.
Confirmation of the successful promotion to PROD.

















Saturday, March 15, 2014

Configuring DSN for IBM Integration Bus in Fedora for Oracle

Configuring DSN for IBM Integration Bus in Fedora for Oracle

I had a very troubling time setting up a DSN for IIB in Fedora. To make sure that others don't have to waste a precious weekend and long time on this rather menial subject I thought I would document it for all.

Assumption :- The software is installed at the location - /opt/ibm/mqsi/9.0.0.0

Here are the steps -

  • A sample odbc.ini and odbcinst.ini file are in the 'install_dir/ODBC/unixodbc/' in my case it is /opt/ibm/mqsi/9.0.0.0/ODBC/unixodbc. Copy the files to /var/mqsi/odbc directory. 
           cp /opt/ibm/mqsi/9.0.0.0/ODBC/unixodbc/odbc.ini /var/mqsi/odbc/odbc.ini
           cp /opt/ibm/mqsi/9.0.0.0/ODBC/unixodbc/odbcinst.ini /var/mqsi/odbc/odbcinst.ini

  • Change the owner of the files to mqm:mbrkrs using the following command

           chown mqm:mqbrkrs /var/mqs/odbc/odbc.ini
           chown mqm:mqbrkrs /var/mqs/odbc/odbcinst.ini

  • Open the '/var/mqsi/odbc/odbc.ini' file. Copy the following lines and paste them just above the copied part- 
           ;# Oracle stanza
           [ORACLEDB]
           Driver=<Your Broker install directory>/ODBC/V7.0/lib/UKora26.so
           Description=DataDirect 7.0 ODBC Oracle Wire Protocol
           HostName=<Your Oracle Server Machine Name>
           PortNumber=<Port on which Oracle is listening on HostName>
           ServiceName=<Your Oracle Service Name>
           CatalogOptions=0
           EnableStaticCursorsForLongData=0
           ApplicationUsingThreads=1
           EnableDescribeParam=1
           OptimizePrepare=1
           WorkArounds=536870912
           ProcedureRetResults=1
           ColumnSizeAsCharacter=1
           LoginTimeout=0


           Make the changes as follows for your DSN. In my case I create a DSN as XE for my XE database. The                driver path may be different as per your installation.

           [XE]
           Driver=/opt/ibm/mqsi/9.0.0.0/ODBC/V7.0/lib/UKora26.so
           HostName=localhost
           PortNumber=1521
           ServiceName=XE

           Not to forget, at the end of the file mention the install directory.


           [ODBC]
           InstallDir=/opt/ibm/mqsi/9.0.0.0/ODBC/V7.0
           UseCursorLib=0
           IANAAppCodePage=4
           UNICODE=UTF-8

           Save the file.
  • Download the IE02 support pac from the following location - http://www-01.ibm.com/support/docview.wss?uid=swg24026935 . In my case I used the 64 bit version 2.0.1 and the file name is 'ie02_amd64_linux_2.tar'. Extract the archive file and it create a folder-in my case 'amd64_linux_2'- that will contain 'install-ie02.bin' file. Run the .bin file and install it. I had it installed in the location '/opt/ibm/IE02'
  • Now that we have all the files in place we need to setup some environment variables in the .profile file of the user that would control the broker runtime. I have the following variables added to end of my .bash_profile 

           export ODBCINI=/var/mqsi/odbc/odbc.ini
           export ODBCSYSINI=/var/mqsi/odbc/
           export IE02_PATH=/opt/ibm/IE02/2.0.1/


    • Use the mqsisetdbparms command to associate the user id and password to the ODBC. The following example command will prompt you for the password and then set the user id and password -
               mqsisetdbparms WBRK9 -n XE -u system
    • Restart the broker to allow it to absorb the setting and then issue the command to check if the broker runtime can access the DSN. 
              mqsicvp -n XE -u system -p yourpassword

    If the command runs with success you should see the output of the command as follows - 

               BIP8290I: Verification passed for the ODBC environment. 

    BIP8270I: Connected to Datasource 'XE' as user 'SYSTEM'. The datasource platform is 'Oracle', version '11.02.0000 Oracle 11.2.0.2.0'. 
    ===========================
    databaseProviderVersion      = 11.02.0000 Oracle 11.2.0.2.0
    driverVersion                = 07.01.0097 (B0099, U0067)
    driverOdbcVersion            = 03.52
    driverManagerVersion         = 03.52.0002.0002
    driverManagerOdbcVersion     = 03.52
    databaseProviderName         = Oracle
    datasourceServerName         = localhost
    databaseName                 = N/A
    odbcDatasourceName           = XE
    driverName                   = UKora26.so
    supportsStoredProcedures     = Yes
    procedureTerm                = PL/SQL
    accessibleTables             = Yes
    accessibleProcedures         = Yes
    identifierQuote              = "
    specialCharacters            = None
    describeParameter            = Yes
    schemaTerm                   = User Name
    tableTerm                    = Table
    sqlSubqueries                = 31
    activeEnvironments           = 0
    maxDriverConnections         = 0
    maxCatalogNameLength         = 128
    maxColumnNameLength          = 30
    maxSchemaNameLength          = 30
    maxStatementLength           = 0
    maxTableNameLength           = 30
    supportsDecimalType          = Yes
    supportsDateType             = No
    supportsTimeType             = No
    supportsTimeStampType        = No
    supportsIntervalType         = No
    supportsAbsFunction          = Yes
    supportsAcosFunction         = No
    supportsAsinFunction         = No
    supportsAtanFunction         = No
    supportsAtan2Function        = No
    supportsCeilingFunction      = Yes
    supportsCosFunction          = Yes
    supportsCotFunction          = No
    supportsDegreesFunction      = No
    supportsExpFunction          = Yes
    supportsFloorFunction        = Yes
    supportsLogFunction          = Yes
    supportsLog10Function        = Yes
    supportsModFunction          = Yes
    supportsPiFunction           = No
    supportsPowerFunction        = Yes
    supportsRadiansFunction      = No
    supportsRandFunction         = No
    supportsRoundFunction        = Yes
    supportsSignFunction         = Yes
    supportsSinFunction          = Yes
    supportsSqrtFunction         = Yes
    supportsTanFunction          = Yes
    supportsTruncateFunction     = Yes
    supportsConcatFunction       = Yes
    supportsInsertFunction       = Yes
    supportsLcaseFunction        = Yes
    supportsLeftFunction         = Yes
    supportsLengthFunction       = Yes
    supportsLTrimFunction        = Yes
    supportsPositionFunction     = No
    supportsRepeatFunction       = Yes
    supportsReplaceFunction      = Yes
    supportsRightFunction        = Yes
    supportsRTrimFunction        = Yes
    supportsSpaceFunction        = Yes
    supportsSubstringFunction    = Yes
    supportsUcaseFunction        = Yes
    supportsExtractFunction      = No
    supportsCaseExpression       = No
    supportsCastFunction         = No
    supportsCoalesceFunction     = No
    supportsNullIfFunction       = No
    supportsConvertFunction      = Yes
    supportsSumFunction          = Yes
    supportsMaxFunction          = Yes
    supportsMinFunction          = Yes
    supportsCountFunction        = Yes
    supportsBetweenPredicate     = Yes
    supportsExistsPredicate      = Yes
    supportsInPredicate          = Yes
    supportsLikePredicate        = Yes
    supportsNullPredicate        = Yes
    supportsNotNullPredicate     = Yes
    supportsLikeEscapeClause     = Yes
    supportsClobType             = No
    supportsBlobType             = No
    charDatatypeName             = CHAR
    varCharDatatypeName          = VARCHAR2
    longVarCharDatatypeName      = CLOB
    clobDatatypeName             = N/A
    timeStampDatatypeName        = N/A
    binaryDatatypeName           = RAW
    varBinaryDatatypeName        = RAW
    longVarBinaryDatatypeName    = BLOB
    blobDatatypeName             = N/A
    intDatatypeName              = NUMBER
    doubleDatatypeName           = BINARY_DOUBLE
    varCharMaxLength             = 0
    longVarCharMaxLength         = 0
    clobMaxLength                = 0
    varBinaryMaxLength           = 0
    longVarBinaryMaxLength       = 0
    blobMaxLength                = 0
    timeStampMaxLength           = 0
    identifierCase               = Upper
    escapeCharacter              = \
    longVarCharDatatype          = -1
    clobDatatype                 = 0
    longVarBinaryDatatype        = -4
    blobDatatype                 = 0

    BIP8273I: The following datatypes and functions are not natively supported by datasource 'XE' using this ODBC driver: Unsupported datatypes: 'DATE, TIME, TIMESTAMP, INTERVAL, CLOB, BLOB' Unsupported functions: 'ACOS, ASIN, ATAN, ATAN2, COT, DEGREES, PI, RADIANS, RAND, POSITION, EXTRACT, CASE, CAST, COALESCE, NULLIF' 
    Examine the specific datatypes and functions not supported natively by this datasource using this ODBC driver.  
    When using these datatypes and functions within ESQL, the associated data processing is done within IBM Integration Bus rather than being processed by the database provider.  
      
    Note that "functions" within this message can refer to functions or predicates. 


    BIP8071I: Successful command completion. 




    References -

    ftp://public.dhe.ibm.com/software/integration/support/supportpacs/individual/ie02_v2.pdf

    http://pic.dhe.ibm.com/infocenter/wmbhelp/v9r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbk58060_.htm