Thursday, December 19, 2013

Adding an Eclipse Project to Heroku

I'm still picking up the pieces, just want to record what I did, not that I fully understand every step.

Heroku has tutorials on creating Java app from command line and using Eclipse. The Eclipse tutorial shows you how to use the plug-in to create the Heroku project. I want to see how I can create an independent Maven project in Eclipse, then push it up to Heroku.

This writing assumes your Heroku account and git is all setup properly.

1. I created a Maven project inside Eclipse. I used the Spring MVC archetype from here:

I noticed that Git is not happy when I created the project in default workspace. Not sure why, so I picked a separate spot. Since Heroku plugin uses "C:\Users\ymeng\git", so I just picked that spot.

2. I added the Procfile, here is the content
web: java $JAVA_OPTS -jar target/dependency/webapp-runner.jar --port $PORT target/*.war

3. I added the following to the pom.xml
4. In Eclipse, right click the project, select "Team", then follow the options to pick "git" 

5. I set the .gitignore in my project root directory to:
6. Then go to "Team" again, and select "commit", pick the files (i picked all). Now your project is committed.
7. This step, i have to do it from the command line:
go to your project root directory, assuming you logged to Heroku already, run "heroku create yourappname". This step will create the git remote for you and also setup "yourappname" on Heroku site for you.
8. Go to "Team" again, select "remote > push", it will asks you to put in your "master" branch.

If all works fine, then your app is on Heroku now.

Wednesday, November 6, 2013

Oracle Coherence Memory Footprint Tests

In a production environment, we have configured 3 coherence nodes with 1G heap size each. We need to cache about 200~300K user account records with additional room to grow. Naturally, we want to find out if we have enough nodes and have allocated enough heap to hold that many records. I conducted some investigation and came up with some interesting numbers.

I wrote an application that can populate the cache with X number records. Then I tested the Coherence server first with single node and different heap sizes. Then I tested multiple Coherence nodes, each with 512M heap.The results are listed below (1 single account record is about 2K when serialized):

Single node:
of rec
serialized size
(default java format)
in (M) B3*RecNum Server
Heap Size
Heap Size
0 4 0
1 2,236 0.2
25K 55,824,754 55M 55900000 512M 64M
50K 111,649,640 112M 111800000 512M 64M
100K 223,297,020 223M 223600000 512M 64M
200K 446,593,392 447M 447200000 1G 1G
300K 669,886,958 670M 670800000 1G 1G
500K 1,116,488,087 1.1G 1118000000 2G 1G

As it shows, 512M heap size can handle up to 100K records. 1G heap size can handle up to 200K records. Then 2G can handle 500K records.

Multiple nodes:
of rec
serialized size
(default java format)
in (M/G) Num of
Heap Size
Combined Serv
Heap Size
Single Node Heap Size Client
Heap Size
200K 446,593,392 512M 1G 1G out of memory
200K 446,593,392 447M 512M 1.5G 1G 1G OK
300K 669,886,958 512M 1.5G 1G out of memory
300K 669,886,958 670M 512M 2G 1G 1G OK
500K 1,116,488,087 512M 2G 1G out of memory
500K 1,116,488,087 512M 2.5G 1G out of memory
500K 1,116,488,087 512M 3G 1G out of memory
500K 1,116,488,087 1.1G 512M 3.5G 2G 1G OK

In the 2nd test group, each node has 512M heap size. As it shows in the table above, when the records grow, I had to add more nodes. It also shows the each node has memory overhead. A combined heap size needs to be bigger than the heap size in the single node test case.

Finally, here is the server start script:
set COHERENCE_HOME=D:\coherence\weblogic
REM set COH_OPTS=-server -cp %COHERENCE_HOME%\lib\coherence.jar;%COHERENCE_HOME%\lib\coherence-web-spi.war;d:\coherence\myRecordType.jar;

set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.distributed.localstorage=true
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.session.localstorage=true
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.cacheconfig=d:\coherence\cache-config.xml
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.wka=devmachine
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.localhost=devmachine

java %COH_OPTS% -Xms2024m -Xmx2024m

Snippet of tangosol file for node 1:
<cluster-name system-property="tangosol.coherence.cluster">Grid/1_0/DEV</cluster-name>
        <socket-address id="1">
          <address system-property="tangosol.coherence.wka">devmachine</address>
          <port system-property="tangosol.coherence.wka.port">7201</port>
<address system-property="tangosol.coherence.localhost">devmachine</address>
        <port system-property="tangosol.coherence.localport">7201</port>

Snippet of <cache-config> (they are the same for each node)

Here is the snippet of tangosol file for 2nd node:

<cluster-name system-property="tangosol.coherence.cluster">Grid/1_0/DEV</cluster-name>
        <socket-address id="1">
          <address system-property="tangosol.coherence.wka1">devmachine</address>
          <port system-property="tangosol.coherence.wka1.port">7201</port>
<address system-property="tangosol.coherence.localhost">devmachine</address>
        <port system-property="tangosol.coherence.localport">7203</port>
Please note:
cluster name is the same.
wka (well known address) is the same (but it uses wka, wka1, wka2, wka3 etc)
each node runs on a different localport: 7021, 7023, 7024 etc

My understanding of "Well known address" is like a "ring leader", a new node will try to talk to this guy first when joining the node initially.

Finally, customize query.cmd under "coherence/weblogic/bin" directory, add your sample "myRecordType.jar" to the classpath and adjust member size as you need.

The CohQL command i used to dump the objects:

backup cache "TestCacheSize" to "client-test/dump-100k.txt";

You can use query like:
select * from "TestCacheSize" where key()='100008';
to check the individual record.

Wednesday, September 11, 2013

Maven Archetype for OSB Project

This is a maven archetype ( for creating an OSB project that Eclipse and Maven can share.

Two things to note about the archetype:

1. I have a local OSB installed, so the POM references to some jar files are specific. You may need to adjust it.
2. If you just want to create setup an OSB project for Eclipse, you can remove the entire plugin section in the POM file. But then, there is no point of using Maven to create the project. The "value" of Maven in this case is it allows you to build OSB jar from the command line, and also allows you to use Eclipse to work on the project.

To place the archetype in your local repository, you can just run "mvn install" after unzip the file.

To create a new OSB project, run something like "mvn archetype:generate -DgroupId=myOsbGrp -DartifactId=myOsbProj", then select the new archetype. For me the # is 832, yours can be different depends on how many archetypes you already have in your repository.

Once the project is created, it will have a folder structure like below:

│   pom.xml

│   └───main
│       └───resources
│           ├───OSBConfiguration
│           │   │   .project
│           │   │
│           │   └───.settings
│           │           com.bea.alsb.core.prefs
│           │           org.eclipse.wst.common.component
│           │           org.eclipse.wst.common.project.facet.core.xml
│           │           org.eclipse.wst.validation.prefs
│           │
│           └───OSBProject
│               │   .project
│               │
│               ├───.settings
│               │       org.eclipse.wst.common.component
│               │       org.eclipse.wst.common.project.facet.core.xml
│               │       org.eclipse.wst.validation.prefs
│               │
│               ├───BusinessServices
│               │
│               │
│               ├───ProxyServices
│               │       ProxyService1.proxy
│               │
│               └───Resources
│                       helloworld.wsdl

From here, you can use Eclipse to import the existing project (pick "main/resources" directory, Eclipse will sort out the rest). You can check in/check out the source to SVN.

The new pom.xml file allows you to build the OSB project from the command line without using Eclipse, therefore make the automated build (CI) possible.

The archetype can be further improved to add deployment (deploying the OSB jar to a server). I'm a beginner of Maven. That would be a future project at some point.

Thursday, August 29, 2013

Custom JCA Adapter and Dynamic log4j Logging Level Control with Weblogic

Recently I worked on dynamical log4j logging level control in a custom JCA adapter. I still don’t know much about JCA adapter. But I managed to make it work, and in the process I picked up a few more interesting things about Weblogic and log4j configurations.

At the core, the solution is the same as in i.e. use this code snippet in a "central" location of your code:

if (notInitialized) {
   String fileName = System.getProperty("jca.log4j.configFile");
   DOMConfigurator.configureAndWatch(fileName, 3000);

I stumbled upon a few subtle things about log4j and Weblogic before I made it to work. Here are a few things to pay close attention if you want to do it:

1. By default, Weblogic uses its own logging mechanism (it’s based on log4j)
2. script may set a specific Java system variable "-Dlog4j.configuration". Based on my test, this variable is sort of a "reserved" by Weblogic. The value of this variable is "file:yourAbsoluteFilePath". Be extra careful with the variable name and the "file:" prefix. 
3. If you place log4j.xml in the domain home directory, then Weblogic will pick it up and uses that one. It confused me for a while. I only found it out by trial and error.

The common thing about these scenarios is that Weblogic will internally pick up the log4j.xml file. When that happens, Weblogic initializes log4j, and you have no control of the file whatsoever. You can modify the log4j.xml all you want; weblogic is not going to do anything with the changes unless you restart the server.

So the solution is :
1. Do not to place "log4j.xml" in the domain folder, or use a different file name, such as "mylog4j.xml". 
2. Use a different Java system variable name, such as "jca.log4j.configFile" (just make sure do not "log4j.configuration").
3. Since you pick your own variable name, you should not use the "file:" prefix for the variable value. Instead, just use the full path name of you log4j configuration file.

That’s about it. I named my file "my-test-log4j.xml". I picked a variable name like "jca.log4j.confiureFile". I was able to change "my-test-log4j.xml" and watch the logging levels change on the fly.

Also, make sure if you start the server (such as AdminServer) with the script, then you need to update the script to pass the variable in. If you use nodeManager (for managed servers), then you should use weblogic console, server, startup argument field to pass in the variable.

Finally, in the JCA rar file, there is a weblogic-ra.xml that allows you to name your "<log-filename>" element, I tested that, it merely generated a file with that name; nothing gets logged into that file. I’m not sure how to use that file. So I simply skipped this element in the weblogic-ra.xml file.

Thursday, August 15, 2013

OSB 11g XSLT multiple input parameters and the conversion error of$DocumentXobj

OSB 11g XSLT expression editor allows you to enter multiple input parameters. The developer guide (page 4-84) even contain a precise definition of each field with example. However, it doesn't work. You may see errors similar to this:
java.lang.RuntimeException: Invalid conversion from 'node-set' to '$DocumentXobj'

I see many frustrated bloggers posting questions. After my own long frustrating hours, i concluded it's an OSB bug. Good news is i also figured out a solution.

At the core, the issue is OSB passes in the parameter to XLST as$DocumentXobj object (with the exception if the parameter is a simple string). But XSLT is expecting a NodeList.

Here is a quick solution. Compile this class, jar it up and copy it to your domain/lib directory:
package org.example.xslt;
import org.w3c.dom.NodeList;
import org.w3c.dom.Node;
public class XObjUtil
  public NodeList getNodeList(Object o)
  { System.out.println("### getNodeList o class name="+o.getClass().getName());
org.w3c.dom.Document xo = (org.w3c.dom.Document)o;
Node n = (Node) xo;
NodeList nl = n.getChildNodes();
       return nl;
// lazy way:  return (((org.w3c.dom.Document)o)).getChildNodes();

Here is my cheap script to compile, jar and copy the file:

javac -d .
del myxlstext.jar
jar cvf myxlstext.jar org
copy myxlstext.jar C:\Oracle\Middleware\user_projects\domains\osbserver\lib

On the OSB side, here is the XSLT what will parse the input parameter:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0"
exclude-result-prefixes="xsl client ">
<xsl:param name="car" />
<xsl:template match="/">
<xsl:variable name="x" select="XObj:new()" />
<xsl:variable name="obj" select="XObj:getNodeList($x, $car)" />
<xsl:value-of select="$obj/client:car1" />
<xsl:value-of select="$obj/client:car2" />
<xsl:value-of select="/client:barElem/client:bar2" />

My foo/bar/car sample is based on this cheap XSD
<?xml version="1.0" encoding="UTF-8" ?>
<schema attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace=""
     xmlns:this="" xmlns="">
    <complexType name="fooType">
            <element name="input" type="string"/>
            <element name="foo1" type="string"/>
            <element name="foo2" type="string"/>
    <complexType name="barType">
            <element name="input" type="string"/>
            <element name="bar1" type="string"/>
            <element name="bar2" type="string"/>
    <complexType name="carType">
            <element name="input" type="string"/>
            <element name="car1" type="string"/>
            <element name="car2" type="string"/>
    <element name="process" type="this:fooType"/>
    <element name="carElem" type="this:carType"/>
    <element name="barElem" type="this:barType"/>
    <element name="processResponse">
                <element name="result" type="string"/>

Here is a sample output from the OSB test console:

client:process xmlns:client=""xmlns:XObj="">

One interesting fact is this sample shows you a cheap way of extending XSLT without a full blown Xpath function project, see here

BTW, very importantly, SOA and OSB uses different XSLT framework. OSB uses more org.w3 XML, SOA use "Oracle" own XML stuff. When SOA invokes XSLT, it passes in the parameter as type oracle.xml.parser.v2.XMLDocumentFragment, if ever want to play with that, the conversion is this:
 public NodeList getNodeList(oracle.xml.parser.v2.XMLDocumentFragment o)
org.w3c.dom.NodeList nl = ((org.w3c.dom.Node)o).getChildNodes();
return  nl;

Two other things you can explore:

1. instead of tossing the jar file into domain/lib directory, try if you can merely upload the jar to the proxy folder in OSB, treats it like a java callout jar. If that works, then you don't need to reboot OSB when change your jar file.

2. implement this as an Xpath function, then your JDev XSLT editor can pick it up. Of course, your XSLT syntax will be modified slightly as well. Have fun!

Saturday, August 10, 2013

OWSM SAML Policy Tests

This post assumes readers have the basic understanding of OWSM policy framework.
In this post, I’ll explore two examples of OWSM SAML policies. There are more than a dozen SAML policies that come with OWSM. One reason for the numerous SAML policies is OWSM provides two collections of SAML policies for WSS 1.0 (WS Security 1.0) and WSS 1.1. You can explore the difference between the WSS 1.0 and 1.1 that is outside the scope of this discussion. Suffice to say that when you pick policies on the client and service sides, you need to make sure that WSS versions match.

Nearly all of the OWSM SAML policies require some kind of encryption, except two wss10_saml(20)_token_service_policy. We will experiment with the non-encryption policy first, then we’ll discuss wss10_saml(20)_token_with_message_protection_service_policy.

For the experiments, I’ll create two local weblogic domains. “Domain 1” runs on port 7001. “Domain 2” runs on 7011. (Refer to previous post on how to createmultiple domains on your localhost).

For demonstration purpose, I use the following services:
  •         A JavaEE web service – deployed to domain 2 and locked by a SAML security service policy
  •         An OSB business service – on domain 1, it hooks up to the JavaEE web service. It has an attached SAML client policy.
  •         OSB proxy service – on domain 1, locked up by WS name token security, which routes to the business service.
You can find jar files for these services here: javaEESvc and osb-config-jar. You can deploy OSB config jar to domain1, javaEESvc jar to domain2.

The steps below illustrate the test scenarios:

1.      use soap UI or other client tool to call the proxy service with ws name token header (with user name, password)
2.      OSB/OWSM authenticates the user. In this case, it uses Weblogic server’s default security realm. So you need to have a proper user defined with the Weblogic server. Proxy is invoked if the user authentication is successful.
3.      OWSM create a SAML assertion based on the authentication of step #2 above.
4.      OSB attaches the SAML assertion (user name only, no password) to the outbound business service call to the JavaEE service
5.      Weblogic/OWSM authenticates the web service call with SAML header. In this case, the authentication only verifies the user name exists in the security realm. The password doesn’t matter. If the authentication is successful, the JavaEE service is invoked.
6.      JavaEE service response is routed back to SoapUI (via business service and proxy service)

<definitions xmlns=""
  xmlns:wsdl="" xmlns:tns=""
  xmlns:xsd="" xmlns:soap=""
  name="samlPocService" targetNamespace="">
    <xs:schema xmlns:tns="" xmlns:xs=""
      version="1.0" targetNamespace="">
      <xs:element name="sayHello" type="tns:sayHello" />
      <xs:element name="sayHelloResponse" type="tns:sayHelloResponse" />
      <xs:complexType name="sayHello">
          <xs:element name="arg0" type="xs:string" minOccurs="0" />
      <xs:complexType name="sayHelloResponse">
          <xs:element name="return" type="xs:string" minOccurs="0" />
  <message name="sayHello">
    <part name="parameters" element="tns:sayHello" />
  <message name="sayHelloResponse">
    <part name="parameters" element="tns:sayHelloResponse" />
  <portType name="samlPoc">
    <operation name="sayHello">
      <input message="tns:sayHello" />
      <output message="tns:sayHelloResponse" />
  <binding name="samlPocPortBinding" type="tns:samlPoc">
    <soap:binding style="document"
      transport="" />
    <operation name="sayHello">
      <soap:operation soapAction="" />
        <soap:body use="literal" />
        <soap:body use="literal" />
  <service name="samlPocService">
    <port name="samlPocPort" binding="tns:samlPocPortBinding">
      <soap:address location="http://localhost:7011/javaeesvc/samlPocPort" />

JavaEE service:
It’s a no thrill simple hello world java function:
package javaeesvc;
import javax.jws.WebService;
@WebService(targetNamespace = "")
public class samlPoc {
    public samlPoc() { super(); }   
    public String sayHello(String s) {
        System.out.println("######Hello " + s + ", this is saml poc");
        return "######Hello " + s + ", this is saml poc";

JavaEE service is deployed to domain 2:

Biz Service Configuration, no policy attached yet:

Proxy service:

Then attach WS name token policy to the proxy

Experiment 1SAML Policy without Encryption
Testing “oracle/wss10_saml20_token_client_policy
In this experiment, we’ll demonstrate the SAML policy without encryption (plain SAML policies). Without encryption, there will be no additional server configurations involved.  You can merely attach the corresponding SAML policies on both the JavaEE service and OSB business service, that’s it.
In the next section we’ll see when message protection (encryption) is involved, it becomes a different ball game. You need to configure quite a few things before SAML policies can work.
Let’s do the simple one first.

Attach SAML policy to JavaEE service:

Under “Web Service Endpoints” tab, click on endpoint name “samlPocPort”:

Click on “Attach/Detach” icon:

Select “wss10_saml20_token_service_policy” then press “Attach”
Attach SAML policy to OSB business service:

SOAP UI client test:

What has happened?
Part 1: Soap UI calls Proxy Service with WSS name token
<soapenv:Envelope xmlns:soapenv=""
    <wsse:Security soapenv:mustUnderstand="1"
      <wsse:UsernameToken wsu:Id="UsernameToken-2"

Part 2: Business Service Invokes JavaEE service with SAML token

Internally, OWSM first authenticates user “weblogic” with password, then it generate SAML token (assertion). OSB business service attaches the SAML assertion in the header. Business service calls JavaEE service with the SAML assertion.

<soap:Header xmlns:soap="">
  <wsse:Security soapenv:mustUnderstand="1"
    <saml:Assertion MajorVersion="1" MinorVersion="1"
      AssertionID="SAML-xnTew5qgIPuu2kkZGcVqhQ22" IssueInstant="2013-07-29T22:26:43Z"
      Issuer="" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion">
      <saml:Conditions NotBefore="2013-07-29T22:26:43Z"
        NotOnOrAfter="2013-07-29T22:31:43Z" />
        AuthenticationInstant="2013-07-29T22:26:43Z" AuthenticationMethod="urn:oasis:names:tc:SAML:1.0:am:password">
  <urn:sayHello xmlns:urn="urn:examples:helloservice">
    <firstName xsi:type="xs:string" .>test</firstName>

In case anyone wonders how I captured the SAML payload, I used TCPMon tool to trap the business outbound call. With OSB version, you can also see similar payload using the OSB business service test console.

The main take away in this experiment is that OSB business service calls JavaEE with SAML assertion and user name, but no password is provided. JavaEE service is configured with a plain SAML token policy that blindly trusts anyone with a well formatted SAML token, as long as the user name exists in “Domain 2”. The web service request will be accepted.

Well, if you find this behavior questionable, then you are correct. Oracle doesn’t recommend using this “plain” dumb SAML policy. In next section, we’ll test with a SAML policy that requires “message protection”. i.e. payload will be encrypted with key (certificate), so the service requests (from the client) can be trusted with assurance by the web service (server).

SAML policy with message protection (encryption)
We’ll use wss11_saml20_token_with_message_protection_client_policy for our experiment.
This SAML policy requires the payload be encrypted (message protection). As far as OWSM policy configurations are concerned, there is not much difference between this policy and the plain SAML policy we discussed earlier. However, there are major configuration steps involved to setup the domains for the encryption/decryption to work.

Setting up certificate trusts between the two domains

If you never worked with OWSM key stores, it can be a daunting task to go through the steps in this section for the first time. You can reference this previous post to get familiar with the OWSM key stores.

By default, the key store is here {domain_home}/config/fmwconfig/default-keystore.jks. The actual key store location is specified in this file {domain_home}/config/fmwconfig/jps-config.xml

In order for message encryption/decryption to work, the two domains need to exchange their keys. “Domain 2” will import the public key of “Domain 1”. So when “domain 1” sends an encrypted message (encrypted with Domain 1’s private key), “domain 2” will be able to decrypt the message with “domain 1’s” public key. Vice versa, “domain 1” needs to import the public key of “domain 1”.

Here are the keytool commands to generate keys for each domain, and to import the public keys to each other’s key store.
  #### generate default keypair for domain1, valid for 10 years
  keytool -genkeypair -alias d1key  -keyalg RSA -keypass d1pass -keystore domain1-keystore.jks  -storepass welcome1 -validity 3650
  #### generate default keys for domain2, valid for 10 years
  keytool -genkeypair -alias d2key  -keyalg RSA -keypass d2pass -keystore domain2-keystore.jks  -storepass welcome2 -validity 3650

  #### list key stores
  keytool -list -v -storepass welcome1 -keystore domain1-keystore.jks
  keytool -list -v -storepass welcome2 -keystore domain2-keystore.jks

  #### export domain1 public key
  keytool -exportcert -alias d1key -storepass welcome1 -keystore domain1-keystore.jks -file domain1-pubkey.cer
  #### export domain1 public key
  keytool -exportcert -alias d2key -storepass welcome2 -keystore domain2-keystore.jks -file domain2-pubkey.cer

  #### import domain2 public key into domain 1 as "d2impkey"
  keytool -importcert -alias d2impkey -storepass welcome1 -keystore domain1-keystore.jks  -file domain2-pubkey.cer
  #### import domain1 public key into domain 2 as "d1impkey"
  keytool -importcert -alias d1impkey -storepass welcome2 -keystore domain2-keystore.jks  -file domain1-pubkey.cer

  ### if you need to remove a key
  keytool -delete  -alias yourkey  -keystore keystore.jks -storepass welcome1

The screen below shows how to configure the key store for domain 1. Pay close attention to each field in this screenshot and notice how the field values correspond to the values we used in the key commands above. You will need to bounce the servers if you change key store files. Remember the default is default-keystore.jks, which is specified in jps-config.xml.

This screen shows how to configure the key store for domain 1.

Once the OWSM key stores are configured for both domains. You will detach the plain SAML policies from javaEESvc on domain 2 and the business service on domain 1. Then reattach wss10_saml20_token_with_message_protection_service_policy to JavaEESvc, and wss10_saml20_token_with_message_protection_client_policy to the OSB business service.

You also need to configure the security property on the business service as shown below.The recipient alias is the domain 2 public key. We imported into domain 1 with the alias "d2impkey". This key will be actually used to decrypt the response message from domain 2, which is encrypted with domain's private key.

This screen shows how to configure the business service “security” tab:

You will need to configure domain 2 OWSM key store similarly.

Test in OSB console

If everything is configured correctly, you can test the business service from the OSB console (works for OSB, may not work for earlier version)
The test console shows data is encrypted:

Test with SoapUI
The SoapUI test will show the same thing either SAML policy is encrypted or not encrypted.
Final notes:
I used OSB and JavaEE service for the experiments. You can also use SOA composite to test the SAML policies. It is recommended to use JDeveloper to attach policies to SOA composite, but you can also use "em" console to the same.