Import Data into Salesforce

It's such a common topic. It's covered by numerous posts, SF documents, tutorials and even on youtube. However, as I embark on the journey, with the help of all the online resource, I still need to feel my own way through. Here is my lesson while uploading the initial Accounts and Contacts data.

1. Data Loader (locally installed exe) is generally considered more powerful and can load larger amount of data than the Wizard. However, Data Loader can only load one type of records in a data file! Furthermore, it uses SF internal ID (one of those funky long 15 or 18 characters ID's).

The implication is not very pleasant. If you need to import Accounts and Contacts, then you need to import Account first, then retrieve the ID's, then update your Contact data file to correlate the SF internal ID's for Accounts. I think that's too much for my case.

2. Luckily for me, I only need a single piece of data for Account, when I generated the Contacts CSV file, I placed the account name into the "Account" column. Import wizard auto generates the Accounts using that column.

3. Rest of the CSV field names should use "Field Label", not the actual "Field Name", definitely not the xxx__c name, otherwise. You have to manually map any fields if Wizard cannot auto map them.

4. date format uses mm/dd/yyyy (i suppose it's defined by the object field in SF?)

5. boolean takes "true/false" (maybe configurable in SF?)

Use force.com Rest API to upload documents or attachment

It took me hours and hours try use force.com Rest API to upload documents and attachments. It didn't help the developer guide only shows upload the document. I guess you are supposed to figure out how to add attachment. Anyway, here are the hoops I had to jump through.

Let's look at document first.

I follow this doc:
http://www.salesforce.com/us/developer/docs/api_rest/index_Left.htm#CSHID=dome_sobject_insert_update_blob.htm|StartTopic=Content%2Fdome_sobject_insert_update_blob.htm|SkinName=webhelp

Pay close attention to a couple of things:

the URL in the sample is: curl https://na1.salesforce.com/services/data/v23.0/sobjects/Document/ -H "Authorization:
Bearer token" -H "Content-Type: multipart/form-data; boundary=\"boundary_string\""
--data-binary @newdocument.json

* replace your own URL, yours won't be "na1"!
* the last part of the URL is "/Document/" for document. For attachment is "/Attachment/"
* @newdocument.json means the cURL will read in the file (pardon me, not a json guy)
* In the sample jason code, it shows "FolderId" : "00lJ0000000HzyY". Normally, with na1salesforce.com/xyz, xyz is the object ID. But for document folder, it looks like salesforce.com/015?fcf=00lJ0000000HzyY, use the last part after cfc=.
*use this text file to test your upload before you try binary file:

--boundary_string
Content-Disposition: form-data; name="entity_content";
Content-Type: application/json

{"Description":"Marketing brochure for Q1 2013","Keywords":"marketing,sales,update","FolderId":"00lJ0000000HzyY","Name":"Marketing Brochure Q1","Type":"text"}

--boundary_string
Content-Disposition: form-data; name="Body"; filename="fakefile.txt";
Content-Type: text/plain

This is the content of my fake file

--boundary_string--

* trick to create binary json file: create a header.txt (everything before "This is the content of the my fake file", and tail.txt (just the last line). Make sure newlines accounted for. Now run copy header.txt + test.pdf/b+tail.txt newdocument.json
notice there is "/b" after the binary file name, in this case, it's pdf. you need to update header to change to "PDF' where applicable, for example: Content-Type: application/pdf

Now let's move on to attachment:

You need to change two things:

1. the URL, last part needs be "/Attachment".
2. in the json property, i modified it to be {"Description":"Marketing brochure for Q1 2013","ParentId":"a00J0000006ntLCIAY","Name":"Marketing Brochure Q1"}. I removed "Keywords", "Type", replaced with "ParentId", this is the object ID to which you want to add the attachment to.

I'm sure there are other properties you can play with. That's quite enough for me for now.

Adding an Eclipse Project to Heroku

I'm still picking up the pieces, just want to record what I did, not that I fully understand every step.

Heroku has tutorials on creating Java app from command line and using Eclipse. The Eclipse tutorial shows you how to use the plug-in to create the Heroku project. I want to see how I can create an independent Maven project in Eclipse, then push it up to Heroku.

This writing assumes your Heroku account and git is all setup properly.

1. I created a Maven project inside Eclipse. I used the Spring MVC archetype from here: http://maven-repository.com/artifact/co.ntier/spring-mvc-archetype/1.0.2

I noticed that Git is not happy when I created the project in default workspace. Not sure why, so I picked a separate spot. Since Heroku plugin uses "C:\Users\ymeng\git", so I just picked that spot.

2. I added the Procfile, here is the content
web: java $JAVA_OPTS -Dspring.profiles.active=prod -jar target/dependency/webapp-runner.jar --port $PORT target/*.war

3. I added the following to the pom.xml

        <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-dependency-plugin</artifactId>

            <version>2.3</version>

            <executions>

                <execution>

                    <phase>package</phase>

                    <goals><goal>copy</goal></goals>

                    <configuration>

                        <artifactItems>

                            <artifactItem>

                                <groupId>com.github.jsimone</groupId>

                                <artifactId>webapp-runner</artifactId>

                                <version>7.0.40.0</version>

                                <destFileName>webapp-runner.jar</destFileName>

                            </artifactItem>

                        </artifactItems>

                    </configuration>

                </execution>

            </executions>

        </plugin>

4. In Eclipse, right click the project, select "Team", then follow the options to pick "git" 

5. I set the .gitignore in my project root directory to:

/target

.settings

6. Then go to "Team" again, and select "commit", pick the files (i picked all). Now your project is committed.

7. This step, i have to do it from the command line:

go to your project root directory, assuming you logged to Heroku already, run "heroku create yourappname". This step will create the git remote for you and also setup "yourappname" on Heroku site for you.

8. Go to "Team" again, select "remote > push", it will asks you to put in your "master" branch.

If all works fine, then your app is on Heroku now.

Oracle Coherence Memory Footprint Tests

In a production environment, we have configured 3 coherence nodes with 1G heap size each. We need to cache about 200~300K user account records with additional room to grow. Naturally, we want to find out if we have enough nodes and have allocated enough heap to hold that many records. I conducted some investigation and came up with some interesting numbers.

I wrote an application that can populate the cache with X number records. Then I tested the Coherence server first with single node and different heap sizes. Then I tested multiple Coherence nodes, each with 512M heap.The results are listed below (1 single account record is about 2K when serialized):

Single node:

number of rec	serialized size (default java format)	in (M)		B3*RecNum		Server Heap Size	Client Heap Size
0	4	0
1	2,236	0.2
25K	55,824,754	55M		55900000		512M	64M
50K	111,649,640	112M		111800000		512M	64M
100K	223,297,020	223M		223600000		512M	64M
200K	446,593,392	447M		447200000		1G	1G
300K	669,886,958	670M		670800000		1G	1G
500K	1,116,488,087	1.1G		1118000000		2G	1G

As it shows, 512M heap size can handle up to 100K records. 1G heap size can handle up to 200K records. Then 2G can handle 500K records.

Multiple nodes:

n number of rec serialized size (default java format) in (M/G) Num of Nodes Server Heap Size Combined Serv Heap Size Single Node Heap Size Client Heap Size 200K 446,593,392 2  512M 1G 1G out of memory 200K 446,593,392 447M 3  512M 1.5G 1G 1G OK 300K 669,886,958 3  512M 1.5G 1G out of memory 300K 669,886,958 670M 4  512M 2G 1G 1G OK 500K 1,116,488,087 4  512M 2G 1G out of memory 500K 1,116,488,087 5  512M 2.5G 1G out of memory 500K 1,116,488,087 6  512M 3G 1G out of memory 500K 1,116,488,087 1.1G 7  512M 3.5G 2G 1G OK

In the 2nd test group, each node has 512M heap size. As it shows in the table above, when the records grow, I had to add more nodes. It also shows the each node has memory overhead. A combined heap size needs to be bigger than the heap size in the single node test case.

Finally, here is the server start script:
setlocal
set COHERENCE_HOME=D:\coherence\weblogic
REM set COH_OPTS=-server -cp %COHERENCE_HOME%\lib\coherence.jar;%COHERENCE_HOME%\lib\coherence-web-spi.war;d:\coherence\myRecordType.jar;

set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.management.remote=true
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.distributed.localstorage=true
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.session.localstorage=true
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.cacheconfig=d:\coherence\cache-config.xml
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.wka=devmachine
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.localhost=devmachine

java %COH_OPTS% -Xms2024m -Xmx2024m com.tangosol.net.DefaultCacheServer
endlocal

Snippet of tangosol file for node 1:
<member-identity>
<cluster-name system-property="tangosol.coherence.cluster">Grid/1_0/DEV</cluster-name>
</member-identity>
<unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address system-property="tangosol.coherence.wka">devmachine</address>
          <port system-property="tangosol.coherence.wka.port">7201</port>
        </socket-address>
        </well-known-addresses>
<address system-property="tangosol.coherence.localhost">devmachine</address>
        <port system-property="tangosol.coherence.localport">7201</port>

Snippet of <cache-config> (they are the same for each node)

   <caching-scheme-mapping>

      <cache-mapping>

         <cache-name>TestCacheSize</cache-name>

         <scheme-name>distributed</scheme-name>

      </cache-mapping>

   </caching-scheme-mapping>

Here is the snippet of tangosol file for 2nd node:

  <member-identity>
<cluster-name system-property="tangosol.coherence.cluster">Grid/1_0/DEV</cluster-name>
</member-identity>
<unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address system-property="tangosol.coherence.wka1">devmachine</address>
          <port system-property="tangosol.coherence.wka1.port">7201</port>
        </socket-address>
        </well-known-addresses>
<address system-property="tangosol.coherence.localhost">devmachine</address>
        <port system-property="tangosol.coherence.localport">7203</port>

Please note:

cluster name is the same.

wka (well known address) is the same (but it uses wka, wka1, wka2, wka3 etc)

each node runs on a different localport: 7021, 7023, 7024 etc

My understanding of "Well known address" is like a "ring leader", a new node will try to talk to this guy first when joining the node initially.

Finally, customize query.cmd under "coherence/weblogic/bin" directory, add your sample "myRecordType.jar" to the classpath and adjust member size as you need.

The CohQL command i used to dump the objects:

backup cache "TestCacheSize" to "client-test/dump-100k.txt";

You can use query like:

select * from "TestCacheSize" where key()='100008';

to check the individual record.

Maven Archetype for OSB Project

This is a maven archetype (osb-archetype-blog.zip) for creating an OSB project that Eclipse and Maven can share.

Two things to note about the archetype:

1. I have a local OSB 11.1.1.6 installed, so the POM references to some jar files are 11.1.1.6 specific. You may need to adjust it.
2. If you just want to create setup an OSB project for Eclipse, you can remove the entire plugin section in the POM file. But then, there is no point of using Maven to create the project. The "value" of Maven in this case is it allows you to build OSB jar from the command line, and also allows you to use Eclipse to work on the project.

To place the archetype in your local repository, you can just run "mvn install" after unzip the file.

To create a new OSB project, run something like "mvn archetype:generate -DgroupId=myOsbGrp -DartifactId=myOsbProj", then select the new archetype. For me the # is 832, yours can be different depends on how many archetypes you already have in your repository.

Once the project is created, it will have a folder structure like below:

myOsbProj
│   pom.xml
│
├───src
│   └───main
│       └───resources
│           ├───OSBConfiguration
│           │   │   .project
│           │   │
│           │   └───.settings
│           │           com.bea.alsb.core.prefs
│           │           org.eclipse.wst.common.component
│           │           org.eclipse.wst.common.project.facet.core.xml
│           │           org.eclipse.wst.validation.prefs
│           │
│           └───OSBProject
│               │   .project
│               │
│               ├───.settings
│               │       org.eclipse.wst.common.component
│               │       org.eclipse.wst.common.project.facet.core.xml
│               │       org.eclipse.wst.validation.prefs
│               │
│               ├───BusinessServices
│               │       BusinessService1.biz
│               │
│               ├───ProxyServices
│               │       ProxyService1.proxy
│               │
│               └───Resources
│                       helloworld.wsdl

From here, you can use Eclipse to import the existing project (pick "main/resources" directory, Eclipse will sort out the rest). You can check in/check out the source to SVN.

The new pom.xml file allows you to build the OSB project from the command line without using Eclipse, therefore make the automated build (CI) possible.

The archetype can be further improved to add deployment (deploying the OSB jar to a server). I'm a beginner of Maven. That would be a future project at some point.

Custom JCA Adapter and Dynamic log4j Logging Level Control with Weblogic

Recently I worked on dynamical log4j logging level control in a custom JCA adapter. I still don’t know much about JCA adapter. But I managed to make it work, and in the process I picked up a few more interesting things about Weblogic and log4j configurations.

At the core, the solution is the same as in http://yuanmengblog.blogspot.com/2013/08/weblogic-log-files-and-log4j-tricks.html i.e. use this code snippet in a "central" location of your code:

if (notInitialized) {

   String fileName = System.getProperty("jca.log4j.configFile");

   DOMConfigurator.configureAndWatch(fileName, 3000);

}

I stumbled upon a few subtle things about log4j and Weblogic before I made it to work. Here are a few things to pay close attention if you want to do it:

1. By default, Weblogic uses its own logging mechanism (it’s based on log4j)

2. setDomainEnv.sh script may set a specific Java system variable "-Dlog4j.configuration". Based on my test, this variable is sort of a "reserved" by Weblogic. The value of this variable is "file:yourAbsoluteFilePath". Be extra careful with the variable name and the "file:" prefix. 

3. If you place log4j.xml in the domain home directory, then Weblogic will pick it up and uses that one. It confused me for a while. I only found it out by trial and error.

The common thing about these scenarios is that Weblogic will internally pick up the log4j.xml file. When that happens, Weblogic initializes log4j, and you have no control of the file whatsoever. You can modify the log4j.xml all you want; weblogic is not going to do anything with the changes unless you restart the server.

So the solution is :

1. Do not to place "log4j.xml" in the domain folder, or use a different file name, such as "mylog4j.xml". 

2. Use a different Java system variable name, such as "jca.log4j.configFile" (just make sure do not "log4j.configuration").

3. Since you pick your own variable name, you should not use the "file:" prefix for the variable value. Instead, just use the full path name of you log4j configuration file.

That’s about it. I named my file "my-test-log4j.xml". I picked a variable name like "jca.log4j.confiureFile". I was able to change "my-test-log4j.xml" and watch the logging levels change on the fly.

Also, make sure if you start the server (such as AdminServer) with the script, then you need to update the script to pass the variable in. If you use nodeManager (for managed servers), then you should use weblogic console, server, startup argument field to pass in the variable.

Finally, in the JCA rar file, there is a weblogic-ra.xml that allows you to name your "<log-filename>" element, I tested that, it merely generated a file with that name; nothing gets logged into that file. I’m not sure how to use that file. So I simply skipped this element in the weblogic-ra.xml file.

OSB 11g XSLT multiple input parameters and the conversion error of org.apache.xmlbeans.impl.store.Xobj$DocumentXobj

OSB 11g XSLT expression editor allows you to enter multiple input parameters. The developer guide http://docs.oracle.com/cd/E23943_01/dev.1111/e15866/toc.htm (page 4-84) even contain a precise definition of each field with example. However, it doesn't work. You may see errors similar to this:
java.lang.RuntimeException: Invalid conversion from 'node-set' to 'org.apache.xmlbeans.impl.store.Xobj$DocumentXobj'

I see many frustrated bloggers posting questions. After my own long frustrating hours, i concluded it's an OSB bug. Good news is i also figured out a solution.

At the core, the issue is OSB passes in the parameter to XLST as org.apache.xmlbeans.impl.store.Xobj$DocumentXobj object (with the exception if the parameter is a simple string). But XSLT is expecting a NodeList.

Here is a quick solution. Compile this class, jar it up and copy it to your domain/lib directory:
package org.example.xslt;
import org.w3c.dom.NodeList;
import org.w3c.dom.Node;
public class XObjUtil
{
  public NodeList getNodeList(Object o)
  { System.out.println("### getNodeList o class name="+o.getClass().getName());
org.w3c.dom.Document xo = (org.w3c.dom.Document)o;
Node n = (Node) xo;
NodeList nl = n.getChildNodes();
       return nl;
// lazy way:  return (((org.w3c.dom.Document)o)).getChildNodes();
 }
}

Here is my cheap script to compile, jar and copy the file:

javac -d . XObjUtil.java

del myxlstext.jar

jar cvf myxlstext.jar org

copy myxlstext.jar C:\Oracle\Middleware\user_projects\domains\osbserver\lib

On the OSB side, here is the XSLT what will parse the input parameter:

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet version="1.0"

xmlns:client="http://xmlns.oracle.com/JCA_ResourceAdapter/foo/BPELProcess1"

xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

xmlns:XObj="http://www.oracle.com/XSL/Transform/java/org.example.xslt.XObjUtil"

exclude-result-prefixes="xsl client ">

<xsl:param name="car" />

<xsl:template match="/">

<client:process>

<xsl:variable name="x" select="XObj:new()" />

<xsl:variable name="obj" select="XObj:getNodeList($x, $car)" />

<client:input>

<xsl:value-of select="$obj/client:car1" />

</client:input>

<client:foo1>

<xsl:value-of select="$obj/client:car2" />

</client:foo1>

<client:foo2>

<xsl:value-of select="/client:barElem/client:bar2" />

</client:foo2>

</client:process>

</xsl:template>

</xsl:stylesheet>

My foo/bar/car sample is based on this cheap XSD

<?xml version="1.0" encoding="UTF-8" ?>

<schema attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://xmlns.oracle.com/JCA_ResourceAdapter/foo/BPELProcess1"

     xmlns:this="http://xmlns.oracle.com/JCA_ResourceAdapter/foo/BPELProcess1" xmlns="http://www.w3.org/2001/XMLSchema">

    <complexType name="fooType">

        <sequence>

            <element name="input" type="string"/>

            <element name="foo1" type="string"/>

            <element name="foo2" type="string"/>

        </sequence>

    </complexType>

    <complexType name="barType">

        <sequence>

            <element name="input" type="string"/>

            <element name="bar1" type="string"/>

            <element name="bar2" type="string"/>

        </sequence>

    </complexType>

    <complexType name="carType">

        <sequence>

            <element name="input" type="string"/>

            <element name="car1" type="string"/>

            <element name="car2" type="string"/>

        </sequence>

    </complexType>

    <element name="process" type="this:fooType"/>

    <element name="carElem" type="this:carType"/>

    <element name="barElem" type="this:barType"/>

    <element name="processResponse">

        <complexType>

            <sequence>

                <element name="result" type="string"/>

            </sequence>

        </complexType>

    </element>

</schema>

Here is a sample output from the OSB test console:

client:process

xmlns:client="http://xmlns.oracle.com/JCA_ResourceAdapter/foo/BPELProcess1"xmlns:XObj="http://www.oracle.com/XSL/Transform/java/org.example.xslt.XObjUtil">

<client:input>car18</client:input>

<client:foo1>car29</client:foo1>

<client:foo2>bar23</client:foo2>

</client:process>

One interesting fact is this sample shows you a cheap way of extending XSLT without a full blown Xpath function project, see here http://yuanmengblog.blogspot.com/2013/04/break-comma-separated-string-into-array.html

BTW, very importantly, SOA and OSB uses different XSLT framework. OSB uses more org.w3 XML, SOA use "Oracle" own XML stuff. When SOA invokes XSLT, it passes in the parameter as type oracle.xml.parser.v2.XMLDocumentFragment, if ever want to play with that, the conversion is this:
 public NodeList getNodeList(oracle.xml.parser.v2.XMLDocumentFragment o)
 {
org.w3c.dom.NodeList nl = ((org.w3c.dom.Node)o).getChildNodes();
return  nl;
 }

Two other things you can explore:

1. instead of tossing the jar file into domain/lib directory, try if you can merely upload the jar to the proxy folder in OSB, treats it like a java callout jar. If that works, then you don't need to reboot OSB when change your jar file.

2. implement this as an Xpath function, then your JDev XSLT editor can pick it up. Of course, your XSLT syntax will be modified slightly as well. Have fun!