Yuan's SOA Blog: 2015

Monday, September 21, 2015

Wireshart: NPF driver not running

quick note so i don't have to search all over the place...

when i installed wireshark, i chose not to start winpcap automatically on windows startup, so I'll get npf driver not running error when i use wireshark,

solution: on windows 7, start cmd as "Administrator", run net start npf

Saturday, August 29, 2015

Configure Jenkins on Tomcat

I installed Jenkins under Tomcat 7. Initially, it appears so easy. After I copied war file under webapps directory, I can access Jenkins from the browser right away. However, that's only the beginning. It has been a couple of days, I'm still fighting with endless problems.

1.Where is the configure file for Jenkins.

I got my clue from this post http://stackoverflow.com/questions/9640030/where-does-jenkins-store-job-and-node-configuration

The installer username for Jenkins is "tomcat", but "tomcat" home directory is not in the standard place like my normal login user.

run this:

$ grep tomcat /etc/passwd
tomcat:x:496:10002::/opt/apache-tomcat-7.0.63:/bin/bash

that tells me "tomcat" home directory is actually /opt/apache-tomcat-7.0.63.

Do a "ls -a" in the directory reveals Jenkins configuration is stored under ".jenkins". That solved my puzzle since a week ago.

It would be too easy if that's the end of the story. So the question is:

2. Who is running Jenkins' processes?

Since Jenkins is installed as an web app, when Jenkins executes various processes, which user name is it using? It turned out, it has huge significance as I start to configure build jobs.

I finally realized that the user who starts "tomcat" will be the user who owns all Jenkins processes. I wasn't paying close attention to this before. I was using either "root' or "tomcat" to run "startup.sh" for tomcat. Imagine, when you run "startup.sh" as "root", then Jenkins configure directory will end up under "/root/.jenkins", that can create so many unintended consequences!
1). First, it switches ".jenkins" directory. Whatever, you have configured under "tomcat" suddenly disappear from you. 2). Then there is the user permission issue, A command runs fine with "root" user may not run under "tomcat, such as "ssh-keygen" command. 3). finally there is a "credential" or identity problem. It took me forever to figure out I am fighting with two SSH keys (one is for "root" and one is for "tomcat", when I was configuring "git" integration. That alone deserves another blog that I'll post after this one.

3. Environment variables: I created a .bash_profile for "tomcat":

JAVA_HOME=/opt/jdk1.8.0_45 export JAVA_HOME M2_HOME=/opt/apache-maven-3.3.3 export M2_HOME M2=$M2_HOME/bin export M2 PATH=$PATH:$JAVA_HOME/bin PATH=$PATH:$M2 export PATH

otherwise, "mvn", java may not be available inside Jenkins.
I think I may need to set JENKINS_HOME later as well, I'm worried I may encounter other issues without proper jenkins home env.

Just too many things to worry about...

Gitlab No authentication methods configured, psql: could not connect to server: No such file or directory

Before jumping into any details, here are two most import directories you need to know for debugging:
/var/opt/gitlab -- important data, configuration and repositories
/var/log/gitlab -- log files

On my new Gitlab install (Redhat/CentOS 6), while messing around with login, I locked myself (everyone else) out!

On the main page, it merely says: "No authentication methods configured." It cost me nearly a day to finally re-install and get back to square one.

At issue is that PostgreSQL remembered some configuration and I just couldn't reset it, even by re-run rpm package.

I first merely ran "gitlab-ctl uninstall", then run "rpm -Uvh --force xxx.rpm". The install obviously re-uses the old DB.

When I tried to connect DB from command line, I got this error:

psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

status check "sudo gitlab-rake gitlab:check" didn't reveal any problem either.

I just could figure out how to connect to the DB. After much frustration, here is how i forced the issue:

1. ps auxw |grep post

From here, I found many PostgreSQL processes and much needed information of where the DB data is stored:

root 2508 runsv postgresql
root 2509 svlogd -tt /var/log/gitlab/postgresql
492 2510 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
492 2517 postgres: checkpointer process
492 2518 postgres: writer process
492 2519 postgres: wal writer process
492 2520 postgres: autovacuum launcher process
492 2521 postgres: stats collector process
492 2613 postgres: gitlab gitlabhq_production [local] idle

2. I ran "gitlab-ctl uninstall"
3. I ran "kill -9 2508" to stop "runsv" (not sure if this step is necessary)
4. I renamed "/var/opt/gitlab/postgresql" to "postgresql.bak". So old DB is invisible to the installer.
5. I blew away "/opt/gitlab" (i don't think this step is necessary, i just want to do it clean)
6. finally, i run " rpm -Uvh --force gitlab-ce-7.14.1-ce.0.el6.x86_64.rpm".
7. after that, i ran "gitlab-ctl reconfigure".

That brought me back to the fresh starting point.

BTW, the default GitLab login is "root/5iveL!fe".

If you also had previous accounts, you may need to clean up data under "/var/opt/gitlab/git-data/repositories", otherwise, you'll see some strange behavior when you re-create the same user account, and using it to create new project. It may fail and complain gitlab-shell error.

Thursday, August 27, 2015

gitlab install::ENOENT (No such file or directory - ssh-keygen)

I'm installing gitlab on RedHat 6, when I posted the ssh key, I got a 500 html error.

In the /var/log/gitlab/gitlab-rails/production.log file, it shows

Completed 500 Internal Server Error in 433ms (ActiveRecord: 40.8ms)

Errno::ENOENT (No such file or directory - ssh-keygen):
lib/gitlab/popen.rb:23:in `popen'

After a whole day trial and error, I found my ugly work around. I'm not much a Linux admin, so there must be good solutions, but I need to move-on. If you want try my hack, proceed at your own risk.

How I found out the problem:

run "sudo gitlab-rails c" (c stands for console)

type in: system("which ssh-keygen"),

You will get a stupid error that says "ssh-keygen" is not available from the path, /usr/bin, etc, However, you can see clearly "ssh-keygen" is under /usr/bin.

Then I tried

system("whoami"), it tells me the user is "git".

In the terminal console, I run "sudo su git", now I'm "git" user, then I ran "which ssh-keygen", I got the same error. That tells me, user "git" does not have permission to run "ssh-keygen".

So I just run "sudo chmod a+x /usr/bin/ssh-keygen". That allows me to go past that issue temporarily. If you need a permanent solution, please go check with a true Linux admin guru.

I also noticed that the execute permission is turned off "automatically" by the system after a little while. So each time when i post a key, i have to run the temp fix. It's a pain in the neck!

BTW, I did ran into another issue during the install. I followed this post https://gitlab.com/gitlab-org/omnibus-gitlab/issues/361 to "fix" the issue, which essentially is just to bypass the issue, modifying /opt/gitlab/embedded/cookbooks/gitlab/recipes/postgresql.rb file with

command "/sbin/sysctl -e -p /etc/sysctl.conf"

that additional "-e" just says to ignore the error from sysclt command.

I initially thought bypassing the error was the cause of my ssh-keygen problem. I finally came to the conclusion it has nothing to do with it. Additionally, I believe it's totally safe to bypass the sysctl error in this case.

My install steps: download rpm, then run
rpm -Uvh --force gitlab-ce-7.14.1-ce.0.el6.x86_64.rpm

Wednesday, August 26, 2015

Posting to HTTP listener with Firefox poster vs html form submit

I have a simple Mule application listens for http on port 8088. When I use Firefox poster to post up data, it sends in pure payload (in this case text/xml). And the application is happy to process the data.

Naturally I was thinking why go through Firefox poster each time, so I created a simple html file with a form post, something like

<form method="post" action="myserver:8088/mypath">
enter data below
<textarea cols=20 row=20>
</textarea>
<input type='submit'>
</form>

I ran into several issues, and decided to give up. I may come back later to see there is an actual work around. For now, this post just reminds what NOT to do. Because I have already attempted this twice in 6 weeks. Need to remind myself stopping making the same mistake.

Issue 1: as it appears above, <textarea> has no name, so I don't know what got posted up to the server, Mule application picked up a "null" payload. I believe it posted something, i just didn't spend time to track it. Maybe I'll track it some other day.

Issue 2: I added <textarea name="foo"...>
Now the application picked up the payload, except the payload is "foo=<xml....>", I need the payload to be pure xml like "<xml ...>" so no go :(

Issue 3: I tried to give it a name with just a space like <textarea name=' '...>, didn't go very far either, the payload shows up as " =<xml ...>".

couple other things I looked into:

apparently, firefox poster posts the content up as "content-type=text/xml", but I don't seem to have control over that with HTML form. "enctype" only has 3 options, the closest is "text/plain", no cigar here.

Until I have too much time on my hand to tinker around it, I'm staying with Firefox poster for now.

Wednesday, August 19, 2015

Encyrpt Mule Properties with Jasypt with Complete Sample Code

Mule ESB uses Java properties file to set the dynamic properties for application. We all know we need to encrypt passwords in the property files. I know two ways to encrypt properties with Mule ESB.

One is to use Mule Enterprise Security module

https://developer.mulesoft.com/docs/display/current/Installing+Anypoint+Enterprise+Security I have posted step by step instructions here: http://blogs.perficient.com/integrate/2016/05/16/mule-enterprise-security-password-encryption-steps/

The other one is Jasypt

http://blogs.mulesoft.com/dev/mule-dev/encrypting-passwords-in-mule/

This post will follow the above blog and show an end to end example of Jasypt with full sample code. I’ll cover the other method in a different post.

Let’s get started.

Step 1. Create a Hello World Application

<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>

<http:listener config-ref="HTTP_Listener_Configuration" path="/enc" doc:name="HTTP"/>

<expression-filter expression="#[message.inboundProperties.'http.request.uri' != '/favicon.ico']" doc:name="Filter favicon" />

<set-payload value="Hello World" doc:name="Set Payload"/>

</flow>

Do a quick test to make yourself feel good.

Step 2. Mavenize the project

Right click on the project name, move to “Maven support in studio”, then select “Mavenize”

This will create a pom.xml for you.

I would like to point out, if you are not happy at any point with your Maven pom.xml, you can chose to delete pom.xml (backup if needed), and then re-run the above step to Mavenize it again.

Step 3. Add a property file to the project

The default property file is my-app.properties. But you should create your own under "src/main/resources". Let’s create “enc.properties” over there.

p1=hello

p2=world

p3=welcome1

p4=foo

Let’s change the flow

<set-payload value="Here are the parameters p1=${p1} p2=${p2} p3=${p3} p4=${p4}" doc:name="Set Payload"/>

If you test the program, it may not pick up those properties. If that’s the case, add this line right above the flow

<context:property-placeholder location="enc.properties" />

Test it again, you should see “Here are the parameters p1=hello p2=world p3=welcome1 p4=foo”. If not, try to re-generate the pom.xml again. Hope that solves your problem.

Step 4. Add Jasypt dependency to pom

<groupId>org.jasypt</groupId>

<artifactId>jasypt-spring3</artifactId>

</dependency>

Step 5. Add the spring bean section from this post http://blogs.mulesoft.com/dev/mule-dev/encrypting-passwords-in-mule/

<spring:beans>

<spring:bean id="environmentVariablesConfiguration" class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig">

<spring:property name="algorithm" value="PBEWithMD5AndDES"/>

<spring:property name="password" value="mypass"/>

</spring:bean>

<spring:bean id="configurationEncryptor" class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor">

<spring:property name="config" ref="environmentVariablesConfiguration"/>

</spring:bean>

<spring:bean id="propertyConfigurer" class="org.jasypt.spring.properties.EncryptablePropertyPlaceholderConfigurer">

<spring:constructor-arg ref="configurationEncryptor"/>

<spring:property name="locations">

<spring:list>

<spring:value>enc.properties</spring:value>

</spring:list>

</spring:property>

</spring:bean>

</spring:beans>

Please note 1) I commented out the above “property-placeholder” line. Spring bean we are adding should be sufficient to make the applicaiton include the property file. 2) I have put enc.properties in the “locations” section. 3) I commented out “passwordEnvName”, and added “password” property. I’ll explain 3) later.

If the compiler whines, you may consider re-generating the pom again, remember to put Jasypt dependency back in if you ended up regenerating the POM file.

If everything works, your application should work exactly like before. If not, then you may need to

stop the application if you are running inside the studio, and re-run it again. Sometimes, the property file change is not picked up properly without restarting the application.

Step 6. Encrypt the Parameters

In command window, run this (you need to pick your own path to the “jasypt” jar file):

java -cp C:\m2\repository\org\jasypt\jasypt\1.9.1\jasypt-1.9.1.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="hello" password="mypass" algorithm=PBEWithMD5AndDES

Run the same command for “world”, “welcome1”.

You should have 3 encrypted string for “hello”, “world”, “welcome1” . Put them in the property file like below, the encrypted strings are inside ENC():

p1=ENC(jDkJK7Ns+OJCRXAZ+7kcUA==)

p2=ENC(eMgHrLD5bPQrVEhu4yOaZw==)

p3=ENC(ennXknKsiEUhAO1a4rpuMXonJ94Nooa1)

p4=foo

Now re-run the program again, you should get the same result. This illustrates that the encryption/decryption is working properly.

Step 7. Wrap it up (using environment variable to keep your master password)

I have copied the final finished source code at the end.

A few notes here:

1 In the previous step, I used “password” property to test the encryption/decryption without using the environment variable

2 In the final code, I have removed (commented out) the “password” property and put back in “passwordEnvName” property, in this case, it is called “MULE_ENCRYPTION_PASSWORD”. This is telling you that you need to setup an environment variable “MULE_ENCRYPTION_PASSWORD” with the value “mypass”.

3 After you setup environment variable, you may need to restart the studio, so it can pick up the environment variable (if you are running standalone Mule ESB, you may need to close that command window and start a new command window, then restart Mule ESB)

Is it safe to use the env varialbe to keep the password? Well, password need be specified by the admin, it has to be somewhere. As explained in the other posts, you can setenv right before you start the server,then unset the env varialbe right after the server is started.

That should be it!

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:context="http://www.springframework.org/schema/context"

xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"

xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.7.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="

http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-current.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd

http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd

http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">

<spring:beans>

<spring:bean id="environmentVariablesConfiguration" class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig">

<spring:property name="algorithm" value="PBEWithMD5AndDES"/>

<spring:property name="passwordEnvName" value="MULE_ENCRYPTION_PASSWORD"/>

</spring:bean>

<spring:bean id="configurationEncryptor" class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor">

<spring:property name="config" ref="environmentVariablesConfiguration"/>

</spring:bean>

<spring:bean id="propertyConfigurer" class="org.jasypt.spring.properties.EncryptablePropertyPlaceholderConfigurer">

<spring:constructor-arg ref="configurationEncryptor"/>

<spring:property name="locations">

<spring:list>

<spring:value>enc.properties</spring:value>

</spring:list>

</spring:property>

</spring:bean>

</spring:beans>

<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>

<http:listener config-ref="HTTP_Listener_Configuration" path="/enc" doc:name="HTTP"/>

<expression-filter expression="#[message.inboundProperties.'http.request.uri' != '/favicon.ico']" doc:name="Filter favicon" />

<set-payload value="Here are the parameters p1=${p1} p2=${p2} p3=${p3} p4=${p4}" doc:name="Set Payload"/>

</flow>

</mule>

For the sake of completeness, I am including the pom.xml as well, which really doesn’t have anything special:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mycompany</groupId>

<artifactId>encrypt</artifactId>

<version>1.0.0-SNAPSHOT</version>

<name>Mule encrypt Application</name>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

<mule.version>3.7.0</mule.version>

<mule.tools.version>1.1</mule.tools.version>

</properties>

<build>

<groupId>org.mule.tools.maven</groupId>

<artifactId>mule-app-maven-plugin</artifactId>

<version>${mule.tools.version}</version>

</configuration>

</plugin>

<artifactId>maven-assembly-plugin</artifactId>

<descriptorRef>project</descriptorRef>

</descriptorRefs>

</configuration>

</plugin>

<groupId>org.codehaus.mojo</groupId>

<artifactId>build-helper-maven-plugin</artifactId>

<id>add-resource</id>

<phase>generate-resources</phase>

<goals>

<goal>add-resource</goal>

</goals>

</resource>

<directory>mappings/</directory>

</resource>

</resources>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

<groupId>org.jasypt</groupId>

<artifactId>jasypt-spring3</artifactId>

</dependency>

<groupId>com.mulesoft.muleesb</groupId>