这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script-白红宇

这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script

阅读量：6284 次

发布时间：2019-06-22

本文共 27208 字，大约阅读时间需要 90 分钟。

这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script

##本文同一时候发表在 ##转载注明出处

部门用持续集成已经非常久了，但事实上使用起来还是非常麻烦的。每当要给一个新项目set up持续集成的环境，尽管是Copy一些现有的jobs, 可是很多參数，变量须要去改，然后还有调试，少说3,4天搞一下，非常不方便。

近期比較空，就把现有的持续集成系统升级改造下，job用一套模板，所有參数化，仅仅要改动配置文件，就能够为新项目配置好环境。

本文的重点是一些经验，想法的分享，并非一篇手把手教你搭建持续集成环境的教程，并且阅读本文须要一定的Jenkins, perl, shell脚本的基础知识，所用到的知识有：Jenkins+Maven+Git+Sonar+Perl+Shell script

下图非常好的说明了我们现有的Job流，以往是每一个项目都有一套这种job流，如今我改成这个Template_service job流，供全部的项目使用，项目名也是作为參数传递进来的。每一个job都是用Jenkins Execute shell这个组件来实现功能的，像check out code，Email Notification, Publish Junit test results, code coverage report 等Jenkins自带组件都没有使用。

执行环境： apache-maven-3.0.5, jdk1.7.0_15, git 1.8.2.1, SonarQube 4.0, perl v5.8.8

##转载注明出处：

接下来简单的说明一下各个Job的功能，以及贴出具体的脚本供大家參考。

除了上图里面的job, 另一个Config job, export一些环境变量

0)Template_Service_Config (one time run)

用来Export 环境变量的，比方Project list, project git 地址，

export PROJECT_LIST=PROJECTA,PROJECTB
 export PROJECTA_GIT_PROJECT_URL=ssh://git@stash.xxx.com:7999/Project/ProjectA.git
 export PROJECTA_BRANCH=development
 export PROJECTA_SERVICE_NAME=China_Template_Service
 export PROJECTA_EMAIL_LIST=wadexu@xxx.com,xxx@xxx.com
 export PROJECTA_EMAIL_CC_LIST=wadexu@xxx.com
......
export PROJECTB_GIT_PROJECT_URL=ssh://git@stash.xxx.com:7999/abc/ProjectB.git
 export PROJECTB_BRANCH=perf
 export PROJECTB_SERVICE_NAME=China_ProjectB_DEV 
 export PROJECTB_EMAIL_LIST=wadexu@xxx.com,xxx@xxx.com 
 export PROJECTB_EMAIL_CC_LIST=wadexu@xxx.com
......

Build 这个job 的时候须要Parameters

代码例如以下：主要作用就是将输入的Parameters写入到一个文件。以便兴许全部job 一上来source一下

TEST=${WORKSPACE}/testCONFIG_FILE=Project_Config.txtrm -rf ${TEST}git clone ssh://git@stash.xxx.com:7999/project/test.gitcd ${TEST}cd Configif [ ! -f "${CONFIG_FILE}" ]; then  touch "${CONFIG_FILE}"fi##########Rewrite the config file base on the project list########### CURRENT_TIME=`date +"%m/%d/%Y/%T"` echo -e "export CONFIG_TIME=${CURRENT_TIME}\n export PROJECT_LIST=${PROJECT_LIST}\n export SHARED_FOLDER=~/jenkins/common\n" > ${CONFIG_FILE} ###########List the project1 parameters########### echo -e "${PROJECTA_PARAMETERS}" >> ${CONFIG_FILE} ###########List the project2 parameters########### echo -e "${PROJECTB_PARAMETERS}" >> ${CONFIG_FILE} git add ${CONFIG_FILE} git commit -m "config file" git push origin master cp ${CONFIG_FILE} ~/jenkins/common/Config cd .. cp Scripts/* ~/jenkins/common/Scripts chmod 755 ~/jenkins/common/Scripts/*

还有就是将一些perl写的脚本从git clone下来放到某一个公共文件夹下。以便兴许job使用，有例如以下脚本，比方发送email, Unit test analyzer

部分脚本像gridService, packageService是操作公司一个网格云计算平台。安装rpm包，重新启动service用的，不方便贴出来。

[wadexu@vm10226 common]$ cd Scripts/[wadexu@vm10226 Scripts]$ lltotal 404-rwxr-xr-x 1 wadexu wadexu  37853 Mar 27 09:12 gridService-rwxr-xr-x 1 wadexu wadexu   4991 Mar 27 09:12 packageService-rwxr-xr-x 1 wadexu wadexu   1628 Mar 27 09:12 queryDB.pl-rwxr-xr-x 1 wadexu wadexu 317183 Mar 27 09:12 Reporter.jar-rwxr-xr-x 1 wadexu wadexu   2190 Mar 27 09:12 report.pl-rwxr-xr-x 1 wadexu wadexu   3415 Mar 27 09:12 sendEmail.pl-rwxr-xr-x 1 wadexu wadexu   1210 Mar 27 09:12 ServiceInfoProvider.pl-rwxr-xr-x 1 wadexu wadexu   3418 Mar 27 09:12 UnitTestAnalyzer.pl

1)Template_Service_Initial

遍历Project list 列出的项目，使用curl 调用下一个job (Commit_Auto_Build) 的Rest API， Build job with parameter接口

每一个项目的Job流都是通过Project name + Initial job的Build number 来贯穿的, 这点非常重要，多项目一起执行时各项目之间不会错乱，比方写Log文件。分析Log文件生成report

rm -rf ${WORKSPACE}/*####### Import config file #######cd ~/jenkins/common/Config. Project_Config.txtcd ~/jenkins/common/Temp####### Get the Project list and traverse the list ######              id=1          list_size=`echo $PROJECT_LIST | awk -F "," '{print NF;}'`          let list_size=list_size+1          while [ $id -lt $list_size ]            do                 projectName=`echo "${PROJECT_LIST}"|awk -v id=$id -F "," '{print $id}'`                      if [ "$projectName " != ' ' ];  then                            curl -X POST --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Commit_Auto_Build/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'$projectName'"},{"name":"PIPELINE_NUM","value":"'$BUILD_NUMBER'"}]}'                                            id=`expr $id + 1`                 else                    break                fi            done

2)Template_Service_Commit_Auto_Build

这个Job的作用是用来比較代码版本号有没有更新，假设有更新（或者是第一次run这个job）就会调用兴许的-- Build job

还有就是列出这次Build和上次Build之间修改过代码的作者。

假设将Initial job 设置成每几分钟run一下，事实上这个job就起到了每当有人提交代码，则自己主动Build的作用了。

### Import config file ###cd ~/jenkins/common/Config. Project_Config.txt### Initial parameters ###GIT_PROJECT_URL=${PROJECT_NAME}"_GIT_PROJECT_URL" GIT_PROJECT_URL=$(eval echo \${$GIT_PROJECT_URL})BRANCH=${PROJECT_NAME}"_BRANCH"BRANCH=$(eval echo \${$BRANCH})LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG_FOLDER=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMEmkdir $BUILD_LOG_FOLDERGIT_PROJECT_NAME=${PROJECT_NAME}"_GIT_PROJECT_NAME"GIT_PROJECT_NAME=$(eval echo \${$GIT_PROJECT_NAME})#### setup ####export CURRENT_RESOURCE_DIR=${WORKSPACE}/${GIT_PROJECT_NAME}export CURRENT_MD5_FILE=${WORKSPACE}/${PROJECT_NAME}"_Code_Current"export BASE_MD5_FILE=${WORKSPACE}/${PROJECT_NAME}"_Code_Base"export CURRENT_AUTHOR_LIST=${WORKSPACE}/${PROJECT_NAME}"_Author_Current"export BASE_AUTHOR_LIST=${WORKSPACE}/${PROJECT_NAME}"_Author_Base"#### cd resource,get log commit MD5 ####cd ${WORKSPACE}rm -rf ${CURRENT_RESOURCE_DIR}git clone ${GIT_PROJECT_URL}cd ${CURRENT_RESOURCE_DIR}git checkout -b ci origin/${BRANCH}if [ ! -f "${BASE_AUTHOR_LIST}" ]; then    touch "${BASE_AUTHOR_LIST}"fiif [ ! -f "${BASE_MD5_FILE}" ]; then    touch "${BASE_MD5_FILE}"  ########call build job######    curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Build/build --data json='{"parameter":[{"name":"PROJECT_NAME","value":"'${PROJECT_NAME}'"},{"name":"PIPELINE_NUM","value":"'${PIPELINE_NUM}'"}]}'  git log --pretty=format:"Author: %an" > ${BASE_AUTHOR_LIST}  sort -u  ${BASE_AUTHOR_LIST}  >  ${BUILD_LOG}  git log --max-count=1 --pretty=format:"%H" > ${BASE_MD5_FILE}  exit 0else  git log --max-count=1 --pretty=format:"%H" > ${CURRENT_MD5_FILE}  git log --pretty=format:"Author: %an" > ${CURRENT_AUTHOR_LIST}fi#### compare the md5 file, and run the job.###### -a means && ##    if cmp -s ${CURRENT_MD5_FILE} ${BASE_MD5_FILE}       then          echo "same,no changes!"     else      mv ${CURRENT_MD5_FILE} ${BASE_MD5_FILE}      diff -b $CURRENT_AUTHOR_LIST $BASE_AUTHOR_LIST | grep "<" | sed 's/^< //g' > Update_Author.txt      mv ${CURRENT_AUTHOR_LIST} ${BASE_AUTHOR_LIST}      sort -u Update_Author.txt > ${BUILD_LOG}        ########call build job######       curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_service_Build/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'${PROJECT_NAME}'"},{"name":"PIPELINE_NUM","value":"'${PIPELINE_NUM}'"}]}'    fi

这里用到了--pretty=format 这个參数， %H就是打印出哈希字串， %an是打印出Author， --max-count=1 取近期的一条log 记录

##转载注明出处：

3) Template_Service_Build

这个job是最核心的job。执行rpm build, 将rpm包丢到公司内部网格云计算平台的Repository下，分析单元測试结果，执行Sonar, 从Sonar的首页上拿到这个项目的一些信息，比方Sonar上 blocker, critical, major 等issues, 还有code coverage，所有打印到log里，以便兴许的脚本生成report, 另外就是Sonar issue, code coverage 不达标。则发送对应的email 给对应的project email list

rm -rf ${WORKSPACE}/*###### Import config file ######cd ~/jenkins/common/Config. Project_Config.txt###### Initial parameters ######GIT_PROJECT_NAME=${PROJECT_NAME}"_GIT_PROJECT_NAME"GIT_PROJECT_NAME=$(eval echo \${$GIT_PROJECT_NAME})LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMEBRANCH=${PROJECT_NAME}"_BRANCH"BRANCH=$(eval echo \${$BRANCH})GIT_PROJECT_URL=${PROJECT_NAME}"_GIT_PROJECT_URL" GIT_PROJECT_URL=$(eval echo \${$GIT_PROJECT_URL})REPOSITORY=${PROJECT_NAME}"_REPOSITORY" REPOSITORY=$(eval echo \${$REPOSITORY})INSTANCES=${PROJECT_NAME}"_INSTANCES"INSTANCES=$(eval echo \${$INSTANCES})COVERAGE_URL=${PROJECT_NAME}"_COVERAGE_URL"COVERAGE_URL=$(eval echo \${$COVERAGE_URL})SONAR_MIN_COVERAGE=${PROJECT_NAME}"_SONAR_MIN_COVERAGE"SONAR_MIN_COVERAGE=$(eval echo \${$SONAR_MIN_COVERAGE})MAX_MAJOR_ISSUE_NUM=${PROJECT_NAME}"_MAX_MAJOR_ISSUE_NUM"MAX_MAJOR_ISSUE_NUM=$(eval echo \${$MAX_MAJOR_ISSUE_NUM})MAX_CRITICAL_ISSUE_NUM=${PROJECT_NAME}"_MAX_CRITICAL_ISSUE_NUM"MAX_CRITICAL_ISSUE_NUM=$(eval echo \${$MAX_CRITICAL_ISSUE_NUM})SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"SERVICE_NAME=$(eval echo \${$SERVICE_NAME})EMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"EMAIL_LIST=$(eval echo \${$EMAIL_LIST})author_list=`cat $BUILD_LOG | grep "Author"` || true######### SET ENVIRONMENT VARIABLE ##############export BASE_DIR=${WORKSPACE}/${GIT_PROJECT_NAME}export SCRIPTS_HOME=${SHARED_FOLDER}/Scriptscd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`####### Build Job ######cd ${WORKSPACE}START_TIME=`date +"%m/%d/%Y %T"`echo "overview->startTime: ${START_TIME}" >> ${BUILD_LOG}echo "overview->logURL: ${BUILD_URL}" >> ${BUILD_LOG}echo "overview->serviceType: ${PROJECT_NAME} Service" >> ${BUILD_LOG}git clone ${GIT_PROJECT_URL}BUILD_FILE=rpmbuild.shcd ${BASE_DIR}git checkout origin/${BRANCH}let BUILD_NUMBER=100+${BUILD_NUMBER}sed -i -r "s/^RELEASE.+/RELEASE=${BUILD_NUMBER}_Dev/g" ${BUILD_FILE}sh ${BUILD_FILE}cd ${WORKSPACE}rm -rf *.rpmmv `find . -type f -name "*.rpm" | egrep -v '(\.src\.)' | egrep -v 'BUILD'` .RPM=`ls *.rpm`packageService --addPackages --repository ${REPOSITORY} --packages "${RPM}"############# Unit test analyze  ###############UnitTestAnalyzer.pl --path "${BASE_DIR}/.rpm/BUILD" --sufix xml >> ${BUILD_LOG}################Run Sonar################cd ${BASE_DIR}mvn clean install -f ${BASE_DIR}/pom.xml -DPASSWORD=$ENV{PASSWORD} -DUSERNAME=$ENV{USERNAME} -e -B sonar:sonar################Get project in Sonar ################sonarProjectName=`sed -ne '/name/{s/.*
       
        \(.*\)<\/name>.*/\1/p;q;}' pom.xml`lowerProjectName=`echo $sonarProjectName | awk '{print tolower($0)}'`sonarId=`queryDB.pl --query="select * from resource_index where kee = '$lowerProjectName' order by resource_id;"`echo $sonarIdsonar_path=http://vm10686.global.xxx.net:9000/dashboard/index/${sonarId}echo "SonarPath: ${sonar_path}" >> ${BUILD_LOG}################Function for get Sonar home page info##################Key for what you want to find##column to printfunction get_html_value() {html=$1key=$2column=$3echo `cat $html | sed -n '/'$key'/p' | awk -v column=$column -F "[><]" '{print $column}'`}##############Sonar issues analyze ###############curl ${sonar_path} > html.txtblocker_num=`get_html_value html.txt m_blocker_violations 3`critical_num=`get_html_value html.txt m_critical_violations 3`major_num=`get_html_value html.txt m_major_violations 3`minor_num=`get_html_value html.txt m_minor_violations 3`info_num=`get_html_value html.txt m_info_violations 3`echo "SonarIssues->Blocker: ${blocker_num}" >> ${BUILD_LOG}echo "SonarIssues->Critical: ${critical_num}" >> ${BUILD_LOG}echo "SonarIssues->Major: ${major_num}" >> ${BUILD_LOG}echo "SonarIssues->Minor: ${minor_num}" >> ${BUILD_LOG}echo "SonarIssues->Info: ${info_num}" >> ${BUILD_LOG}##############Sonar Code Coverage analyze ###############code_coverage=`get_html_value html.txt m_coverage 7`code_coverage_integer=`echo $code_coverage | sed -n 's/%//p'`line_coverage=`get_html_value html.txt m_line_coverage 5`branch_coverage=`get_html_value html.txt m_branch_coverage 5`echo "CoverageInfo->UnitTestsCoverage: ${code_coverage}" >> ${BUILD_LOG}echo "CoverageInfo->LineCoverage: ${line_coverage}" >> ${BUILD_LOG}echo "CoverageInfo->BranchCoverage: ${branch_coverage}" >> ${BUILD_LOG}echo "CoverageInfo->CoverageThreshold: ${SONAR_MIN_COVERAGE}%"  >> ${BUILD_LOG}################Send Email Code Coverage < Threshold ################if (( $(echo "$code_coverage_integer < $SONAR_MIN_COVERAGE" | bc -l) ))thenmail_subject=${PROJECT_NAME}"_Service_Build Job - Build # "${BUILD_NUMBER}" Failure"'!' mail_body="Hi, ${PROJECT_NAME}"" project member"$'\n'$'\n'"Alerts : Unit Tests Coverage: ${code_coverage} < CoverageThreshold: ${SONAR_MIN_COVERAGE}%, the latest build package cannot be allowed to install to service - ${SERVICE_NAME} [$INSTANCES]"'!'$'\n'$'\n'"For details please refer to SonarQube - ${sonar_path}"$'\n'$'\n'"The latest code author list as below:"$'\n'"${author_list}"sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --mailbody "$mail_body" --msg_type "text"exit 0fi################Send Email when Sonar has major above issue################if [ ${blocker_num} -gt 0 ] || [ ${critical_num} -gt ${MAX_CRITICAL_ISSUE_NUM} ] || [ ${major_num} -gt ${MAX_MAJOR_ISSUE_NUM} ]thenmail_subject=${PROJECT_NAME}"_Service_Build Job - Build # "${BUILD_NUMBER}" Failure"'!' mail_body="Hi, ${PROJECT_NAME}"" project member"$'\n'$'\n'"Alerts : Sonar major above issues != 0, the latest build package cannot be allowed to install to service - ${SERVICE_NAME} [$INSTANCES]"'!'$'\n'$'\n'"Blocker issues: ${blocker_num}"$'\n'"Critical issues: ${critical_num}"$'\n'"Major issues: ${major_num}"$'\n'"Minor issues: ${minor_num}"$'\n'"Info issues: ${info_num}"$'\n'$'\n'"For details please refer to SonarQube - ${sonar_path}"$'\n'$'\n'"The latest code author list as below:"$'\n'"${author_list}"sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --mailbody "$mail_body" --msg_type "text"exit 0fi####### Call Deploy Job ######curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Deploy/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'${PROJECT_NAME}'"},{"name":"PIPELINE_NUM","value":"'${PIPELINE_NUM}'"}]}'

mvn clean install -f ${BASE_DIR}/pom.xml -DPASSWORD=$ENV{PASSWORD} -DUSERNAME=$ENV{USERNAME} -e -B sonar:sonar

执行Sonar 的一些数据库配置信息，是放在maven/conf/settings.xml文件中的。

另外一点：每一个项目在Sonar里的名称是项目pom文件的<name></name>标签里的值。依据这个值去数据库里搜出id, 有了这个id就能够直接訪问到Sonar上的这个项目比方http://vm10686.global.xxx.net:9000/dashboard/index/1234

4)Template_Service_Deploy

Build完之后就是要Deploy了。将rpm包装到对应Service的instance上。然后重新启动service

rm -rf ${WORKSPACE}/*    ###### Import config file ######cd ~/jenkins/common/Config. Project_Config.txt###### Initial parameters ######SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"SERVICE_NAME=$(eval echo \${$SERVICE_NAME})INSTANCES=${PROJECT_NAME}"_INSTANCES"INSTANCES=$(eval echo \${$INSTANCES})PATTERN=${PROJECT_NAME}"_RPM_GREP_PATTERN"PATTERN=$(eval echo \${$PATTERN})LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAME##########SET ENVIRONMENT VARIABLE##############SCRIPTS_HOME=~/jenkins/common/Scriptscd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`InstallTime=`date +"%m/%d/%Y %T"`echo "Installation->InstallTime: ${InstallTime}" >> ${BUILD_LOG}############ Deploy ###############gridService --add-service --service ${SERVICE_NAME}  --instance ${INSTANCES}sleep 60gridService --start-service --service ${SERVICE_NAME}Endpoint="${INSTANCES}:8080"echo "Installation->Endpoint: ${Endpoint}" >> ${BUILD_LOG}echo "Installation->LogURL: ${BUILD_URL}" >> ${BUILD_LOG}UpdateTime=`date +"%m/%d/%Y %T"`echo "ServiceInfo->UpdateTime: ${UpdateTime}" >> ${BUILD_LOG}ServiceInfoProvider.pl --service ${SERVICE_NAME} --pattern ${PATTERN} >> ${BUILD_LOG}###### Call Polling Job ######curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Polling/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'$PROJECT_NAME'"},{"name":"PIPELINE_NUM","value":"'$PIPELINE_NUM'"}]}'

5)Template_Service_Polling

接下来通过这个Polling job 轮询service有没有启动好，每1分钟推断一下，timeout时间10分钟。脚本事实上非常easy，就是调一个接口，查看状态是否status=200

rm -rf ${WORKSPACE}/*        ######## Import config file #########cd ~/jenkins/common/Config. Project_Config.txt####### SET ENV #############export SCRIPTS_HOME=${SHARED_FOLDER}/Scriptscd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`###### Initial parameters #############SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"SERVICE_NAME=$(eval echo \${$SERVICE_NAME})LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMEecho "pollingjob->logURL: ${BUILD_URL}" >> ${BUILD_LOG}###### Polling service whether start #############sleep 30gridService --ping --timeout 600 --interval 60 --service ${SERVICE_NAME}###### Call API Testing Job ######curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_APITesting/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'${PROJECT_NAME}'"},{"name":"PIPELINE_NUM","value":"'${PIPELINE_NUM}'"}]}'

6)Template_Service_APITesting

Polling之后就是调用借口測试的内部測试系统。假设export 的变量有 TestPlan 就会运行，否则直接调用最后一个job--Report

rm -rf ${WORKSPACE}/*###### Import config file ######cd ~/jenkins/common/Config. Project_Config.txt###### Initial parameters ######LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMETESTPLAN_ID=${PROJECT_NAME}"_TESTPLAN_ID"TESTPLAN_ID=$(eval echo \${$TESTPLAN_ID})SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"SERVICE_NAME=$(eval echo \${$SERVICE_NAME})EMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"EMAIL_LIST=$(eval echo \${$EMAIL_LIST})EMAIL_CC_LIST=${PROJECT_NAME}"_EMAIL_CC_LIST"EMAIL_CC_LIST=$(eval echo \${$EMAIL_CC_LIST})###########Set Env #############export SCRIPTS_HOME=${SHARED_FOLDER}/Scripts/cd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`################Function##################Key for what you want to find##num occurrence of value if it multiple timefunction jsonValue() {KEY=$1num=$2awk -F"[,:}]" '{for(i=1;i<=NF;i++){if($i~/'$KEY'\042/){print $(i+1)}}}' | tr -d '"' | sed -n ${num}p}############### Main ######################if [[ $TESTPLAN_ID == "" ]]; thenecho "TestPlan ID is blank"###### Call Report Job ######curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Report/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'$PROJECT_NAME'"},{"name":"PIPELINE_NUM","value":"'$PIPELINE_NUM'"}]}'elseprocessID=$(curl -X GET http://vm10765.global.xxx.net:8080/tms/project/testPlan/run.spring?

ids=$TESTPLAN_ID) if [[ $processID != TestPlan-* ]]; then mail_subject=${PROJECT_NAME}"_Service_TMS_APITesting Job - Build # "${BUILD_NUMBER}" Failure"'!' mail_body="TMS Server is not available now, please contact adminstrator." echo $mail_body sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --emailcclist "${EMAIL_CC_LIST}" --mailbody "$mail_body" --msg_type "text" else while [ 1 ] do response=$(curl -X GET

http://vm10765.global.xxx.net

:8080/tms/project/getProcessResult.spring?processID=$processID) totalCounts=`echo $response | jsonValue totalCounts` notStartCounts=`echo $response | jsonValue notStartCounts` processingCounts=`echo $response | jsonValue processingCounts` faildCounts=`echo $response | jsonValue faildCounts` comparedCounts=`echo $response | jsonValue comparedCounts` comparedFaildCounts=`echo $response | jsonValue comparedFaildCounts` unComparedCounts=`echo $response | jsonValue unComparedCounts` if [ $notStartCounts == 0 ] && [ $processingCounts == 0 ]; then echo "End of run." break else echo "API Testing is still running, please wait..." sleep 5 fi done ############# Print Log ################### echo "APITestingInfo->TotalCounts: $totalCounts" >> ${BUILD_LOG} echo "APITestingInfo->ErrorCounts: $faildCounts" >> ${BUILD_LOG} echo "APITestingInfo->ComparedPassedCounts: $comparedCounts" >> ${BUILD_LOG} echo "APITestingInfo->ComparedFailedCounts: $comparedFaildCounts" >> ${BUILD_LOG} echo "APITestingInfo->UnComparedCounts: $unComparedCounts" >> ${BUILD_LOG} ###### Call Report Job ###### curl -d --user "jadmin:71103407" -s

http://vm10686.global.xxx.net

:8080/view/Template/job/Template_Service_Report/build --data json='{"parameter": [{"name":"PROJECT_NAME","value":"'$PROJECT_NAME'"},{"name":"PIPELINE_NUM","value":"'$PIPELINE_NUM'"}]}' fi fi

##转载注明出处：

7)Template_Service_Report

这里主要是生成Report而且发送Email。用Java写的一个jar放到linux上运行的，用html直接贴在邮件里发送email的。简单的事实上能够用Perl写个excel,统计下測试情况，做个报表。

rm -rf ${WORKSPACE}/*###### Import config file ######cd ~/jenkins/common/Config. Project_Config.txt###### Initial parameters ######LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG_FOLDER=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMESERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"SERVICE_NAME=$(eval echo \${$SERVICE_NAME})INSTANCES=${PROJECT_NAME}"_INSTANCES"INSTANCES=$(eval echo \${$INSTANCES})EMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"EMAIL_LIST=$(eval echo \${$EMAIL_LIST})EMAIL_CC_LIST=${PROJECT_NAME}"_EMAIL_CC_LIST"EMAIL_CC_LIST=$(eval echo \${$EMAIL_CC_LIST})TESTPLAN_ID=${PROJECT_NAME}"_TESTPLAN_ID"TESTPLAN_ID=$(eval echo \${$TESTPLAN_ID})export SCRIPTS_HOME=${SHARED_FOLDER}/Scripts/cd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`export XML_REPORT=${WORKSPACE}/interface.xmlreport.pl  --log-path="$BUILD_LOG_FOLDER" --output ${XML_REPORT}REPORT_PATH=${WORKSPACE}/Report.htmlecho ${REPORT_PATH}java -Dxmlpath="${XML_REPORT}" -Dreport="${REPORT_PATH}" -jar Reporter.jarreport_context=`cat $REPORT_PATH`errorCounts=`cat $BUILD_LOG | grep "APITestingInfo->ErrorCounts" | awk -F ": " '{print $2}'`comparedFailedCounts=`cat $BUILD_LOG | grep "APITestingInfo->ComparedFailedCounts" | awk -F ": " '{print $2}'`sonarPath=`cat $BUILD_LOG | grep "SonarPath" | awk -F ": " '{print $2}'`############## Send Email ##################if [ ${errorCounts} -gt 0 ] || [ ${comparedFailedCounts} -gt 0 ]thenmail_subject=${PROJECT_NAME}"_Service_Report Job - Build # "${BUILD_NUMBER}" Failure"'!' mail_body="Hi, ${PROJECT_NAME} project member 
The latest build package has been installed to service - ${SERVICE_NAME} [$INSTANCES], but the API testing has some test case failed.
Errored Test Case: ${errorCounts}
Failed Test Case: ${comparedFailedCounts}
For details please refer to Test Management System - http://vm10765.global.xxx.net:8080/tms/login.spring
SonarQube - $sonarPath
Report as below:
""$report_context"sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --emailcclist "${EMAIL_CC_LIST}" --mailbody "$mail_body" --attached ${REPORT_PATH}elseif [[ $TESTPLAN_ID == "" ]]; thenmail_subject=${PROJECT_NAME}"_Service_Report Job - Build # "${BUILD_NUMBER}" Success"'!'" [No TestPlan for API testing]"mail_body="Hi, ${PROJECT_NAME} project member 
The latest build package has been installed to service - ${SERVICE_NAME} [$INSTANCES].
SonarQube - $sonarPath
Report as below:
""$report_context"elsemail_subject=${PROJECT_NAME}"_Service_Report Job - Build # "${BUILD_NUMBER}" Success"'!'mail_body="Hi, ${PROJECT_NAME} project member 
The latest build package has been installed to service - ${SERVICE_NAME} [$INSTANCES] and API testing has passed.
For detailed test results, please refer to Test Management System - http://vm10765.global.xxx.net:8080/tms/login.spring
SonarQube - $sonarPath
Report as below:
""$report_context"fisendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --emailcclist "${EMAIL_CC_LIST}" --mailbody "$mail_body" --attached ${REPORT_PATH}fi########### Tear Down ###########rm -rf ${BUILD_LOG_FOLDER}

我设置的是每天执行一次全部的project的持续集成，收到的Email例如以下图所看到的：

Report中的非常多信息都是来源于Sonar

##转载注明出处：

到这里就结束了吗。还没有，假设Build job,单元測试不通过或者其它原因导致Build Failure, 那怎么办？

再加一个Post Build job, 来监控Build job，假设挂了，发email通知项目人员。

下图为Build job 加一个Action, 当Job Failed时触发Template_Service_Post_Build job

Template_Service_Post_Build 脚本例如以下：

rm -rf ${WORKSPACE}/*        ######## Import config file #########cd ~/jenkins/common/Config. Project_Config.txt####### SET ENV #############export SCRIPTS_HOME=${SHARED_FOLDER}/Scriptscd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`###### Initial parameters #############LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMEEMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"EMAIL_LIST=$(eval echo \${$EMAIL_LIST})EMAIL_CC_LIST=${PROJECT_NAME}"_EMAIL_CC_LIST"EMAIL_CC_LIST=$(eval echo \${$EMAIL_CC_LIST})author_list=`cat $BUILD_LOG | grep "Author"` || true###### Monitor Service Build Job ########build_job_url=`cat $BUILD_LOG | sed -n '/overview->logURL/p' | awk -F ": " '{print $2}'`build_job_build_number=`echo $build_job_url | awk -F "/" '{print $(NF-1)}'`build_job_log_url=$build_job_url"logText/progressiveText?start=0"build_job_console_output=$build_job_url"console"mail_subject=${PROJECT_NAME}"_Service_Build Job - Build # "${build_job_build_number}" Failure"'!' mail_body="Hi, ${PROJECT_NAME}"" project member"$'\n'$'\n'"Build Failure"'!'$'\n'$'\n'"For details please refer to Console Output - ${build_job_console_output}"$'\n'$'\n'"The latest code author list as below:"$'\n'"${author_list}"sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --emailcclist "${EMAIL_CC_LIST}" --mailbody "$mail_body" --msg_type "text"

同理还须要加一个Template_Service_Post_Polling job来监控Template_Service_Polling job，假设轮询Service的结果是重新启动失败，则邮件通知项目组人员。

脚本例如以下：

rm -rf ${WORKSPACE}/*        ######## Import config file #########cd ~/jenkins/common/Config. Project_Config.txt####### SET ENV #############export SCRIPTS_HOME=${SHARED_FOLDER}/Scriptscd ${SCRIPTS_HOME}chmod 755 *export PATH=$PATH:`pwd`###### Initial parameters #############LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAMEEMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"EMAIL_LIST=$(eval echo \${$EMAIL_LIST})EMAIL_CC_LIST=${PROJECT_NAME}"_EMAIL_CC_LIST"EMAIL_CC_LIST=$(eval echo \${$EMAIL_CC_LIST})author_list=`cat $BUILD_LOG | grep "Author"`###### Monitor Service Polling Job ########polling_job_url=`cat $BUILD_LOG | sed -n '/pollingjob->logURL/p' | awk -F ": " '{print $2}'`polling_job_build_number=`echo $polling_job_url | awk -F "/" '{print $(NF-1)}'`polling_job_log_url=$polling_job_url"logText/progressiveText?start=0"polling_job_console_output=$polling_job_url"console"mail_subject=${PROJECT_NAME}"_Service_Polling Job - Build # "${polling_job_build_number}" Failure"'!' mail_body="Hi, ${PROJECT_NAME}"" project member"$'\n'$'\n'"Service Restart Failure"'!'$'\n'$'\n'"For details please refer to Console Output - ${polling_job_console_output}"$'\n'$'\n'"The latest code author list as below:"$'\n'"${author_list}"sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --emailcclist "${EMAIL_CC_LIST}" --mailbody "$mail_body" --msg_type "text"

##转载注明出处：

如今我们来看一下更新后的CI 流程图

那么，这个flow的前提是全部的项目都是按这个流程来的，万一有些特别的项目呢? 比方一个项目，它在Build之前须要Pre-Build一个CommonUtils包，那怎么办呢。简单一点就是为了这个项目定制一下。

为了更具通用性，扩展性，方便的插入一些special job, 我们能够在每一个job之间增加 Exit Point, 以适应一些特别的项目想在随意点Plug in一些特别的job。

除了要在Config job 加上一些规则，比方 export ProjectX_EXIT_POINT=Template_Service_Commit_Auto_Build|Special ProjectX-PreBuild job

还要在每一个job里加一些推断语句，比方当前run的job name 等于ProjectX_EXIT_POINT以竖线切割的第一个值。则表明要在这个job退出，第二个值表明要去调的special job name，当然 special job 里做完了自己的事之后还是要回调原来flow的下一个job

流程图更新例如以下所看到的：