search by tags

for the user

adventures into the land of the command line

jenkins pipeline idiosyncrasies

Some useful resources

PIPELINE SYNTAX
GROOVY SYNTAX REFERENCE
GROOVY / BASH ESCAPE SEQUENCE RIDICULOUSNESS
PIPELINE BEST PRACTICES
JENKINS PIPELINE DIRTY SECRETS PART ONE & TWO

Keywords

pipeline (required) -  contains the entire Pipeline definition
agent (required)- defines the agent used for entire pipeline or a stage
any - use any available agent
none - do not use a node
node -  allocate a specific executor
label - existing Jenkins node label for agent
customWorkspace - use a custom workspace directory on agent
docker - requires docker-enabled node
image - run inside specified docker image
label -  existing Jenkins node label for agent
registryUrl - use a private registry hosting image
registryCredentialsId - id of credentials to connect to registry
reuseNode - (Boolean) reuse the workspace and node allocated previously
args -  arguments for docker container.
customWorkspace - use a custom workspace directory on agent
dockerfile - use a local dockerfile
filename - name of local dockerfile
dir - subdirectory to use
label -  existing Jenkins node label
reuseNode - (Boolean) reuse the workspace and node allocated previously
args -  arguments for docker container
customWorkspace - use a custom workspace directory on agent
stages (required) - contains all stages and steps within Pipeline
stage (required) - specific named  “Stage” of the Pipeline

steps (required) - build steps that define the actions in the stage. Contains one or more of following:

any build step or build wrapper defined in Pipeline. e.g. sh, bat, powershell, timeout, retry, echo, archive, junit, etc.

script  - execute Scripted Pipeline block
when  - executes stage conditionally
branch - stage runs when branch name matches ant pattern
expression - Boolean Groovy expression
anyOf - any of the enclosed conditions are true
allOf - all of the enclosed conditions are true
not - none of the enclosed conditions are true
parallel -
stage - stages are executed in parallel but
agent, environment, tools and post may also optionally be defined in stage           
environment - a sequence of “key = value” pairs to define environment variables
credentials(‘id’) (optional) - Bind credentials to variable.
libraries - load shared libraries from an scm
lib - the name of the shared library to load
options - options for entire Pipeline.
skipDefaultCheckout - disable auto checkout scm
timeout - sets timeout for entire Pipeline
buildDiscarder - discard old builds
disableConcurrentBuilds - disable concurrent Pipeline runs
ansiColor - color the log file output
tools - Installs predefined tools to be available on PATH
triggers - triggers to launch Pipeline based on schedule, etc.
parameters - parameters that are prompted for at run time.
post - defines actions to be taken when pipeline or stage completes based on outcome. Conditions execute in order:
always - run regardless of Pipeline status.
changed - run if the result of Pipeline has changed from last run
success - run if Pipeline is successful
unstable - run if Pipeline result is unstable
failure - run if the Pipeline has failed

Pass variable from one stage to another

stages {
    stage("1") {
        agent any
        steps {
            script {
                my_app_CHANGED = true
                def SOMETHING = true
            }
        echo "${my_app_CHANGED}" // true
        echo "${SOMETHING}" // true
        }
    }
    stage("2") {
        agent any
        steps {
            script {
                echo "${my_app_CHANGED}" // true
                echo "${SOMETHING}" // build will fail here, scope related
            }
        }
    }
}

Omitting the “def” keyword puts the variable in the bindings for the current script and groovy treats it (mostly) like a globally scoped variable.

Pass a variable from bash to groovy

stages {
    stage("Determine What To Build") {
        agent any
        steps {
            sh '''#!/bin/bash
                echo true > my_app_CHANGED.txt // pipe something into a text file in your working directory. oooooo so0o0o0o0o diiiiiirty.
            '''
            script {
                try {
                    my_app_CHANGED_UNSAFE = readFile('my_app_CHANGED.txt') // assign contents of file to groovy variable here
                    my_app_CHANGED = "${my_app_CHANGED_UNSAFE.trim()}" // need to trim the newline in file from the variable's value.
                }
                catch (exception) {
                    my_app_CHANGED = false // in case the bash command failed.
                }
            }
            echo "${my_app_CHANGED}" // true
        }
    }
}

Pass a variable from groovy to groovy

Ensure the entire thing is encased in “”. ${blah} or $blah can both be used. GOOD - echo “Deploying my_app to ${DEPLOYMENT_GATEWAY}-0, ${ENVIRONMENT}, ${PACKAGE}” BAD - echo Deploying my_app to “${DEPLOYMENT_GATEWAY}”-0, “${ENVIRONMENT}”, “${PACKAGE}”

stages {
    stage("Determine What To Build") {
        agent any
        steps {
            script {
                if ("${DEPLOY_TO}" == 'Somewhere' ) {
                    DEPLOYMENT_GATEWAY = 'abc.somewhere.com'
                    ENVIRONMENT = 'my_app-prod'
                    PACKAGE = "my_app-${BRANCH_NAME}.tgz"
                    echo "${DEPLOYMENT_GATEWAY}"
                    echo "${ENVIRONMENT}"
                    echo "${PACKAGE}"

                    if ("${my_app_CHANGED}" == 'true') {

                        switch("${BRANCH_NAME}") {
                            case ".*":
                                echo "Deploying my_app to ${DEPLOYMENT_GATEWAY}-0, ${ENVIRONMENT}, ${PACKAGE}"
                                //sh gui_deployer.sh "${DEPLOYMENT_GATEWAY}" 8022 "${ENVIRONMENT}" "${PACKAGE}"
                                echo "Deploying my_app to ${DEPLOYMENT_GATEWAY}-1, ${ENVIRONMENT}, ${PACKAGE}"
                                //sh gui_deployer.sh "${DEPLOYMENT_GATEWAY}" 8023 "${ENVIRONMENT}" "${PACKAGE}"
                                break
                        }

                    } else {
                        echo "Nothing was deployed because the commit didn't include any changes to my_app."
                    }

                } else {
                    echo "Nothing was deployed because no environment was selected."
                    return
                }
            }
        }
    }
}

Pass a variable from groovy to a single line shell script

Ensure the entire thing is encased in “”, not just the ${variable}.

script {
    def SOMETHING = "https://some.thing.com"
    sh "echo ${SOMETHING}"
    sh "SOMETHING=${SOMETHING}; echo SOMETHING"
}

Pass a variable from groovy to a multiline line shell script

Make sure the “”“ are with double quotes. ”’ sucks. pass the variable inside double quotes: “${variable}” if there is a bash $ being used, such as when referencing a bash variable $MYVAR, you need to escape it: \$MYVAR

script {
    def SOMETHING = "https://some.thing.com"
    sh """
        eval \$(docker-machine env somenode)
        echo "${SOMETHING}"

    """
}

Assign groovy string to groovy variable which contains another groovy variable

Once again, ensure the entire thing is encased in “”, not just the ${variable}.

script {
    PACKAGE = "my_app-${BRANCH_NAME}.tgz"
}

Choosing which parallel stages to run based on conditions or parameters previously set

stages {
    stage("1") {
        agent any
        parallel (
            "Package Consumer" : {
                echo "${my_app_1_CHANGED}"
                script {
                    if ("${my_app_1_CHANGED}" == 'true') {
                        sh '''
                            case "${BRANCH_NAME}" in
                                *)
                                    echo Compressing Consumer GUI Package
                                    rm -f my_app_1-"${BRANCH_NAME}".tgz || true
                                    tar -czf my_app_1-"${BRANCH_NAME}".tgz -C my_app_1/web .
                                ;;
                            esac
                        '''
                    } else {
                        echo "Nothing was tarballed because the commit didn't include any changes to my_app_1."
                    }
                }
            },
            "Package Manager" : {
                echo "${my_app_2_CHANGED}"
                script {
                    if ("${my_app_2_CHANGED}" == 'true') {
                        sh '''
                            case "${BRANCH_NAME}" in
                                *)
                                    echo Compressing Manager GUI Package
                                    rm -f my_app_2-"${BRANCH_NAME}".tgz || true
                                    tar -czf my_app_2-"${BRANCH_NAME}".tgz -C my_app_2/web .
                                ;;
                            esac
                        '''
                    } else {
                        echo "Nothing was tarballed because the commit didn't include any changes to my_app_2."
                    }
                }
            },
            "Package Operator" : {
                echo "${my_app_3_CHANGED}"
                script {
                    if ("${my_app_3_CHANGED}" == 'true') {
                        sh '''
                            case "${BRANCH_NAME}" in
                                *)
                                    echo Compressing Operator GUI Package
                                    rm -f my_app_3-"${BRANCH_NAME}".tgz || true
                                    tar -czf my_app_3-"${BRANCH_NAME}".tgz -C my_app_3/web .
                                ;;
                            esac
                        '''
                    } else {
                        echo "Nothing was tarballed because the commit didn't include any changes to my_app_3."
                    }
                }
            }
        )
    }
}

Escaping $ in “”

sh '''
    SOMETHING=hello
    echo $SOMETHING buddy
'''
// hello buddy

sh """
    SOMETHING=hello
    echo $SOMETHING
"""
// buddy

sh """
    SOMETHING=hello
    echo \$SOMETHING buddy
"""
// hello buddy

Junit stupidity in the post section

post {
    always {
        script {
            junit "my_app/coverage/junit/*.xml"
        }
    }
}

Error stating that the time the test was run was older than some current time. Apparently if its more than something like 4 seconds, you’ll get this error.

post {
    always {
        script {
            sh "sudo touch ${WORKSPACE}/my_app/coverage/junit/*.xml"
            junit "${WORKSPACE}/my_app/coverage/junit/*.xml"
        }
    }
}

Touch the damn file right before. Not a solution, a bandaid, like everything about jenkins, one giant ball of bandaids. However this will also error because of the ${WORKSPACE} variable in the junit command. I dont know why.

post {
    always {
        script {
            sh "sudo touch ${WORKSPACE}/my_app/coverage/junit/*.xml"
            junit "my_app/coverage/junit/*.xml"
        }
    }
}

This one works then.

Disabling concurrent builds on a multistage pipeline doesn’t work if you use agents in the stages

pipeline {
    agent none
    options {
        buildDiscarder(logRotator(numToKeepStr: '10'))
        disableConcurrentBuilds()
    }
    stages {
        stage("1") {
            agent any
            steps {
                ...
            }
        }
        stage("2") {
            agent any
            steps {
                ...
            }
        }
    }
}

Results in:

my_build-TWRKUBXHW7FVLDXTUXR7V4NSBGZMX4K65ZYM6WHW3NCJK5DECL5Q
[email protected]

For 2 concurrent builds even when we clearly specified in the pipeline options to disableConcurrentBuilds()

pipeline {
    agent any
    options {
        buildDiscarder(logRotator(numToKeepStr: '10'))
        disableConcurrentBuilds()
    }
    stages {
        stage("1") {
            steps {
                ...
            }
        }
        stage("2") {
            steps {
                ...
            }
        }
    }
}

Use the global agent.