My Blog Home Has Moved

For those who follow my blogs, my blog home has moved.

My new blog home is here now:

http://blogs.perficient.com/integrate/author/ymeng/

Thanks!

Installing On-Prem Mule ESB and Gateway as Windows Service on the Same Machine

This is a quick note on installing both ESB and Gateway as Windows service on the same machine.

To install as windows service for ESB and Gateway alone is straight forward, just go to the bin directory and run "mule install" or "gateway install" respectively.

However, if you want install both of them as windows service on the same machine, you'll get an error says service name already exists. A quick solution is to edit either mule.bat or gateway.bat, search for "mule_ee", change either one or two of them to a new name. In my case, i changed gateway from "mule_ee" to "mule_gw". That's it.

Connecting Mule API Gateway and ESB in Hybrid Mode

This is the abstract. To  read the details, please follow my new blog page http://blogs.perficient.com/integrate/author/ymeng/

APIs are driving the digital economy. One of the key players in the API field is MuleSoft. Instead of a labyrinth of overlapping products offerings, MuleSoft provides the Anypoint Platform with Mule API gateway and a nimble Mule ESB that can run in the cloud, on-prem or in hybrid mode. Mule promotes a wholesome API approach called API-led Connectivity.

In this post, I will demonstrate an end to end example on how to connect the front-end Mule API gateway to the back-end Mule ESB server in hybrid mode. I’ll also show how easy it is to apply various policies to the API proxy. The final section illustrates applying LDAP security policy to the API interface.

This post assumes you have the basic working knowledge of Anypoint Studio.

The main steps:

1. Install Mule ESB server, install Mule API gateway server
2. Add local servers to the AMC (Anypoint Management Console)
3. Create "Hello World" RAML file with API management console
4. Create Mule application based on the RAML
5. Deploy Mule application to local ESB server from AMC console (Hybrid mode)
6. Deploy API proxy to local API gateway
7. Manage the API proxy with policies (simple rating policy, http basic authentication, and LDAP)

To read the details, please follow my new blog page http://blogs.perficient.com/integrate/author/ymeng/

Wireshart: NPF driver not running

quick note so i don't have to search all over the place...

when i installed wireshark, i chose not to start winpcap automatically on windows startup, so I'll get npf driver not running error when i use wireshark,

solution: on windows 7, start cmd as "Administrator", run net start npf

Configure Jenkins on Tomcat

I installed Jenkins under Tomcat 7. Initially, it appears so easy. After I copied war file under webapps directory, I can access Jenkins from the browser right away. However, that's only the beginning. It has been a couple of days, I'm still fighting with endless problems.

1.Where is the configure file for Jenkins.

I got my clue from this post http://stackoverflow.com/questions/9640030/where-does-jenkins-store-job-and-node-configuration

The installer username for Jenkins is "tomcat", but "tomcat" home directory is not in the standard place like my normal login user.

run this:

$ grep tomcat /etc/passwd
tomcat:x:496:10002::/opt/apache-tomcat-7.0.63:/bin/bash

that tells me "tomcat" home directory is actually /opt/apache-tomcat-7.0.63.

Do a "ls -a" in the directory reveals Jenkins configuration is stored under ".jenkins". That solved my puzzle since a week ago.

It would be too easy if that's the end of the story. So the question is:

2. Who is running Jenkins' processes?

Since Jenkins is installed as an web app, when Jenkins executes various processes, which user name is it using? It turned out, it has huge significance as I start to configure build jobs.

I finally realized that the user who starts "tomcat" will be the user who owns all Jenkins processes. I wasn't paying close attention to this before. I was using either "root' or "tomcat" to run "startup.sh" for tomcat. Imagine, when you run "startup.sh" as "root", then Jenkins configure directory will end up  under "/root/.jenkins", that can create so many unintended consequences!
1). First, it switches ".jenkins" directory. Whatever, you have configured under "tomcat" suddenly disappear from you. 2). Then there is the user permission issue, A command runs fine with "root" user may not run under "tomcat, such as "ssh-keygen" command. 3). finally there is a "credential" or identity problem. It took me forever to figure out I am fighting with two SSH keys (one is for "root" and one is for "tomcat", when I was configuring "git" integration. That alone deserves another blog that I'll post after this one.

3. Environment variables: I created a .bash_profile for "tomcat":

JAVA_HOME=/opt/jdk1.8.0_45 export JAVA_HOME M2_HOME=/opt/apache-maven-3.3.3 export M2_HOME M2=$M2_HOME/bin export M2 PATH=$PATH:$JAVA_HOME/bin PATH=$PATH:$M2 export PATH

otherwise, "mvn", java may not be available inside Jenkins.
I think I may need to set JENKINS_HOME later as well, I'm worried I may encounter other issues without proper jenkins home env.

Just too many things to worry about...

Gitlab No authentication methods configured, psql: could not connect to server: No such file or directory

Before jumping into any details, here are two most import directories you need to know for debugging:
    /var/opt/gitlab  -- important data, configuration and repositories
    /var/log/gitlab -- log files

On my new Gitlab install (Redhat/CentOS 6), while messing around with login, I locked myself (everyone else) out!

On the main page, it merely says: "No authentication methods configured."  It cost me nearly a day to finally re-install and get back to square one. 

At issue is that PostgreSQL remembered some configuration and I just couldn't reset it, even by re-run rpm package. 

I first merely ran "gitlab-ctl uninstall", then run "rpm -Uvh --force xxx.rpm". The install obviously re-uses the old DB. 

When I tried to connect DB from command line, I got this error:

 psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

status check "sudo gitlab-rake gitlab:check" didn't reveal any problem either.

I just could figure out how to connect to the DB. After much frustration, here is how i forced the issue:

1. ps auxw |grep post

From here, I found many PostgreSQL processes and much needed information of where the DB data is stored:

root      2508  runsv postgresql
root      2509  svlogd -tt /var/log/gitlab/postgresql
492       2510  /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
492       2517  postgres: checkpointer process                        
492       2518  postgres: writer process                              
492       2519  postgres: wal writer process                          
492       2520  postgres: autovacuum launcher process                
492       2521  postgres: stats collector process                    
492       2613  postgres: gitlab gitlabhq_production [local] idle    

2. I ran "gitlab-ctl uninstall"
3. I ran "kill -9 2508" to stop "runsv" (not sure if this step is necessary)
4. I renamed "/var/opt/gitlab/postgresql" to "postgresql.bak". So old DB is invisible to the installer.
5. I blew away "/opt/gitlab" (i don't think this step is necessary, i just want to do it clean)
6. finally, i run " rpm -Uvh --force gitlab-ce-7.14.1-ce.0.el6.x86_64.rpm".
7. after that, i ran "gitlab-ctl reconfigure".

That brought me back to the fresh starting point.

BTW, the default GitLab login is "root/5iveL!fe".

If you also had previous accounts, you may need to clean up data under "/var/opt/gitlab/git-data/repositories", otherwise, you'll see some strange behavior when you re-create the same user account, and using it to create new project. It may fail and complain gitlab-shell error.

gitlab install::ENOENT (No such file or directory - ssh-keygen)

I'm installing gitlab on RedHat 6, when I posted the ssh key, I got a 500 html error.

In the /var/log/gitlab/gitlab-rails/production.log file, it shows

    Completed 500 Internal Server Error in 433ms (ActiveRecord: 40.8ms)

    Errno::ENOENT (No such file or directory - ssh-keygen):
    lib/gitlab/popen.rb:23:in `popen'

After a whole day trial and error, I found my ugly work around. I'm not much a Linux admin, so there must be good solutions, but I need to move-on. If you want try my hack, proceed at your own risk.

How I found out the problem:

run "sudo gitlab-rails c" (c stands for console)

type in: system("which ssh-keygen"), 

You will get a stupid error that says "ssh-keygen" is not available from the path, /usr/bin, etc, However, you can see clearly "ssh-keygen" is under /usr/bin.

Then I tried 

system("whoami"), it tells me the user is "git". 

In the terminal console, I run "sudo su git", now I'm "git" user, then I ran "which ssh-keygen", I got the same error. That tells me, user "git" does not have permission to run "ssh-keygen".

So I just run "sudo chmod a+x /usr/bin/ssh-keygen". That allows me to go past that issue temporarily. If you need a permanent solution, please go check with a true Linux admin guru.

I also noticed that the execute permission is turned off "automatically" by the system after a little while. So each time when i post a key, i have to run the temp fix. It's a pain in the neck!

BTW, I did ran into another issue during the install. I followed this post https://gitlab.com/gitlab-org/omnibus-gitlab/issues/361 to "fix" the issue, which essentially is just to bypass the issue, modifying /opt/gitlab/embedded/cookbooks/gitlab/recipes/postgresql.rb file with 

command "/sbin/sysctl -e -p /etc/sysctl.conf"

that additional "-e" just says to ignore the error from sysclt command. 

I initially thought bypassing the error was the cause of my ssh-keygen problem. I finally came to the conclusion it has nothing to do with it. Additionally, I believe it's totally safe to bypass the sysctl error in this case.

My install steps: download rpm, then run
rpm -Uvh --force gitlab-ce-7.14.1-ce.0.el6.x86_64.rpm