Wednesday, April 24, 2013

Unix file permission when file adatper generates the output file

When use file adapter to write a file on Linux, what is the permission of the output file?

The common assumption is it is determined by the umask of the user who owns the “Weblogic” java process. If a user runs startWeblogic.sh from the command line, then that user is the owner of Weblogic processes (many places use “oracle” or “weblogic” as the user to start webogic processes).

For the discussion, let’s assume the owner of “weblogic” logic process is “webadmin”. We need to generate an output file with “664” permission. However, we keep getting 644.
We double checked “webadmin” user has 002 umask. We also tried to play with the umask settings in startWeblogic.sh script to no avail.

So it’s gloves off time, I added java embed to change the output file permission by calling “setWrtiable(true, false)”, and that, opened another can of worms.
To my surprise, I stumbled on another unexplainable file adapter behavior. After the file adapter finished writing the file (with 644 permission), if I use java embed to open the file, and do any operations on the file (even just read it), then it TRUNCATES the output file. I thought it has to do with the transaction, so I tried to call check_point() before I open the file in java. It made no difference.

In the end, I just skipped the file adapter, use java embedding to generate the output file, then set the permission. Here is the snippet:
        try {                   
                String content = (String)getVariableData("csvOutput");   // grab file content                       
                String fileName = (String)getVariableData("fileName"); 
                fileName = "/output/"+fileName;
                FileWriter fstream = new FileWriter(fileName);
                BufferedWriter out = new BufferedWriter(fstream);
                out.write(content+"\r\n");
                out.close();
                                               
                java.io.File f = new java.io.File(fileName);
                f.setWritable(true, false);  // set permissions
                f.setReadable(true, false);
        }      catch(Exception e)      {
                System.out.println("*****file exception:"+e.getMessage());
        }

 There is an Oracle Note - 1430075.1 that covers the same issue. Oracle recommendation is to use java embedding, then do
   System.exec(“chmod…”);
I didn’t try that. Not sure how it works. Hope it doesn’t truncate the output file.

Saturday, April 13, 2013

No unique bean of type [javax.jms.Queue] is defined

Here is a strange error I ran into, and a even more strange solution in the end.

I created a simple web service that publishes JMS messages to a queue. I used the simple annotations like:

    @Resource(mappedName="foo.jmsprovider.XAConnectionFactory")
    private ConnectionFactory connectionFactory;
    @Resource(mappedName="foo.jms.bar.queue")
    private javax.jms.Queue queue;

when I deployed the project to Weblogic, I got exceptions like:

"Dependency injection failure: can't find the bean definition about class interface javax.jms.Queue; nested exception is com.bea.core.repackaged.springframework.beans.factory.NoSuchBeanDefinitionException:
No unique bean of type [javax.jms.Queue] is defined: No beans of type javax.jms.Queue;"

The error description indicates the 2nd resource has problem. I finessed with various forms of the annotation, and simply couldn't get around it.

At one point, I managed to work around the issue by removing the annotated resource, and hard code the queue like:
               InitialContext ctx = new InitialContext();    
               javax.jms.Queue queue = (Queue) ctx.lookup("foo.jms.bar.queue");

however, I don't like the hard coded work-around. Frustrated, I rebooted Weblogic server. To my surprise, that took care of the problem!

I have no explaination why rebooting would fix the problem. Nor do I know if this may help the same problem for other people.

The moral of the story is - do not always trust those silly error messages. They can lead you to a wild chase. In the end, a mere magic reboot would take care of the problem.

Wednesday, February 20, 2013

WLST and MDS


Developers commonly use JDeveloper to create deployment project to upload files to MDS.
To remove files from MDS, you need to use WLST tool. In fact, you can use WLST tools to import, export, delete, purge and remove MDS directories.

The following are the commonly used WLST “commands” to manipulate the MDS:

·        importMetaData – import directories and documents

·        exportMetaData – export MDS trees and documents

·        deleteMetaData – delete MDS documents (not directories)

·        purgeMetaData

·        sca_removeSharedData – remove directories and documents

These “commands” appear to be straight forward, but you can easily be tripped over if you don’t pay attention to some details.

Not All wlst.cmd Files Are Created Equal

Depends on how WLST is started, and where it’s started, you may have different “commands” available to you. Which WLST “commands” are available to you, depends on what .jar files you load into the classpath when you start WLST tool.
With my locally installed SOA suite, I found 6 different versions of “wlst.cmd”:

1.      C:\Oracle\Middleware\wlserver_10.3\common\bin

2.      C:\Oracle\jdev-mw\wlserver_10.3\common\bin

3.      C:\Oracle\jdev-mw\oracle_common\common\bin

4.      C:\Oracle\Middleware\Oracle_OSB1\common\bin

5.      C:\Oracle\Middleware\oracle_common\common\bin

6.      C:\Oracle\Middleware\Oracle_SOA1\common\bin

Not all wslt.cmd files in this list support MDS commands. I know #2 doesn’t work.
I was able to import and export MDS data using wslt.cmd under #3, C:\Oracle\jdev-mw\oracle_common\common\bin. But I have to use #6 (C:\Oracle\Middleware\Oracle_SOA1\common\bin) for sca_removeSharedData command to work.

Examples Using WSLT to Work with MDS

cd c:\Oracle\jdev-mw\oracle_common\common\bin
connect(‘weblogic’, ‘welcome1’, 't3://localhost:7001')

To export data
exportMetadata(application='soa-infra', server='soa_server1', toLocation='c:/junk/mdsout', docs='/apps/test/**')

Assuming I have some files in MDS under “/apps/test” tree, this command will create the “c:\junk\mdsout\apps\test” directory, this directory will contains all non-empty directories and files in  MDS under ‘/apps/test’ tree.

Notes:
1.      You may need to do: exportMetadata(application='soa-infra', server='soa_server1', toLocation='c:/junk/mdsout', docs='/apps/test/**', remote=’true’) if your server is remote (not localhost).
2.      You may need to do: exportMetadata(application='soa-infra', server='soa_server1', toLocation='c:/junk/mdsout/test.jar', docs='/apps/test/**') if your server is remote (not localhost) depends on which version of wlst.cmd you use.
3.      This commands only dumps out non-empty tree structures in MDS

To import data
importMetadata(application='soa-infra',server='soa_server1',fromLocation='C:/junk/mdsout', docs='/apps/test/**')

This will import data files under c:\mdsout\apps\test into MDS tree “apps/test”.
This command will not import empty directories. Like export command, you may need to import .jar file instead of directories.

To Delete Data
deleteMetadata(application='soa-infra',server='soa_server1',docs='/apps/test/**')
This will only delete all files under MDS /apps/test tree, it will not delete the directories.

To Remove the Directories
sca_removeSharedData('http://localhost:8001', ‘test’) - will remove MDS tree “/apps/test”
sca_removeSharedData('http://localhost:8001', 'test/dvm')  - will remove MDS tree “/apps/test/dvm”

Be careful, do not add / in ‘test’ or ‘test/dvm’, otherwise WLST complains cannot find documents to remove.

 

 

Use JCA file adapter to Parse CSV file with master and detailed records


JCA file adapter can parse single record type CSV file easily. It also has limited support to parse mixed record type CSV files. However, it relies on the record data starts with a fixed values.
In my case, I have a CSV file with master-detailed records that look like this:
Master c1, master c2, master c3
 a, b, c
Det c1, det 2, det 3, det 4, det 5
1, 2, 3,4,5
6,7,8,9,0

Since my detailed records do not start with a fixed value (detail row 1 starts with 1, row 2 starts with 6), I cannot use JCA wizard to parse this file directly. Here is how I managed to do it. My solution is to create the two file adapters and parse the same data file twice. First parse the master record, then parse the detailed records.

For master record:

1.      Create a copy of the sample data file, and remove the detailed records
2.      Remove the spaces in the master header row
3.      Generate XSD with JCA adapter native file wizard, select uniform file, use 1st line as header
4.      After XSD (let’s call it header.xsd) is generated, make these changes in the XSD:

·        nxsd:headerLinesTerminatedBy="${eol}" – make sure it’s like this.
·        nxsd:headerLines="1" – this is a misnomer, it simply means how many lines to skip
·        nxsd:hasHeader="false"     -- sounds contradictory to the line above. But the actual data file has spaces in the header title, this will make the parser skip header line
·        nxsd:dataLines="1"             –  so it only reads 1 line of data

Follow similar steps for detailed-records; remove the master head and records from sample data file, after XSD (call it body.xsd) is generated, make the following changes:

·        nxsd:hasHeader="false" – so it won’t parse header records, because spaces in header cause problems
·        nxsd:headerLines="3" – this will skip 3 lines in the data file
·        nxsd:headerLinesTerminatedBy="${eol}" – make sure the headers line is terminated properly
In the BPEL process, create two JCA file adapters.
The first adapter is called to load (parse) the header record into BPEL, choose header.xsd to when create the adapter. Remember, header.xsd skips the first line of the data file, and parse the 1st line, since we set dataLines=”1”, it will skip rest of the detailed records.
Also make sure modify the .jca file, so after header record is loaded, do not delete the data file:
  <property name="DeleteFile" value="false"/>

Create the 2nd file adapter and select body.xsd. Based on the XSD, the 2nd adapter will skip the first 3 lines (master record header, data and detailed record header.

With this two-pass approach,  you can load both header and detailed records into BPEL.

Thursday, January 17, 2013

You can run managed server without admin server

Just to clarify, technically you can run Weblogic managed server without admin server, if the service on the managed server doesn't access anything from the admin server. Although I have never seen anyone doing this in a prod environment.

e.g. You can start the admin server, then start OSB server, create a proxy call it echoPS which just echoes back your input. Then you shutdown admin server. Test echoPS from soapUI, it would run just fine. You can also simply starts OSB server without admin server, you'll see a bunch of errors, but the managed server will eventually start. If you test your echoPS, it works just fine.

Friday, January 4, 2013

Rolling soa_server.out log file

SOA server keeps many logs. They are mostly WLS logs, such as soa_sever1.log, osb_server1.log etc. All of these files can be configured to auto roll. Except soa_server1.out.

Strictly speaking, soa_server1.out is not really WLS "log" file. If you start SOA server from a command window, by default the standard output spits all the messages in the command Window. When you start SOA server via node manager, the standard output get redirected to soa_server1.out. Since standard output is not really a log file, you cannot configure WLS to roll this file. Over time, soa_server1.out can grow very big and cause problems. 

I'm sure many people have their solutions to deal with the issue. Here is a cheat I figured out that can achieve the effect of "rolling" soa_server1.out.

Write a simply script:

  cp soa_server.out soa_server.out_$(date +%b)_$(date +%d)_$(date +%Y)"
  echo > soa_server.out

the first command backs up the file, 2nd command truncates the file, but still keep the same file. SOA is happy to to dump data into the same "log" file. Use cron to schedule the script to run, that should be it.

One reminder, i used cp in the first step. Then truncate the file in 2nd step. I also attempted in the first step to rename soa_server1.out file, then "touch soa_server1.out" to create a new log file. It doesn't work that way. SOA server obviously has got its mind somewhere, it doesn't like you to rename the log file. Even if you create a new soa_server.out, it won't write to the new file.



OSB cluster synching problem

We ran into an OSB cluster synching problem in our production environment. The key words are the "production environment". Otherwise, it won't be an issue, because you can always restart both nodes, then it would be synched without problem. We need a solution without shutting down the cluster.

The problem happened after we updated an OSB project which contains a collection of proxies, business service etc. To be exact, we used "ant", which in turn uses WLST to run the deployment, but i don't think that's relevant.

We have two OSB nodes, after the deployment we start to see exceptions in the logs of node1 (osb_server1).

<Error> <ConfigFwk> <BEA-000000> <notifyAfterEnd() failed for listener ResourceListenerNotifier
java.lang.NoClassDefFoundError: com/bea/alsb/coherence/impl/RemoveProcessor
at com.bea.alsb.coherence.impl.CoherenceCache.removeAll(CoherenceCache.java:132)
at com.bea.wli.sb.service.resultcache.ResultCache.removeAll(ResultCache.java:254)

and 
<Error> <OSB Kernel> <BEA-382016> <Failed to instantiate router for service ProxyService CIS/proxy/GetAutopayWithdrawOptionByAccountID: com.bea.wli.sb.management.BrokerManagementException: The configurations for the proxy service is in flux.
com.bea.wli.sb.management.BrokerManagementException: The configurations for the proxy service is in flux.
at com.bea.wli.sb.pipeline.RouterContext.getInstance(RouterContext.java:178)
at com.bea.wli.sb.pipeline.RouterManager.processMessagen


we had to shut down node1. Otherwise, it causes nasty service interruption. We logged a Sev-1 SR with Oracle. The response we got from the support borders on irresponsibility: they merely told us to shut down both nodes, and restart up. Hello..., this is production environment! You can't just simply use the "reboot" lame response like those Windows desktop tech support!

Additionally, Oracle support pointed out that we were deploying updates while OSB proxy is running. I am not sure I get it. Does it mean that we need to shut down the server to deploy any updates???

Anyway, I noticed the exceptions indicating some kind of issue with "coherence", so we decided to let node2 running for over 24 hours, hoping "coherence" cache may get cleared. Then we tried to start node1 again and, unfortunately, got the same error.

After poking my heads all over the place, I have a hunch that node1 (osb_server1) might be keeping its cached values somewhere under the <domain home>/servers/osb_server1.

I renamed the directory "osb_server1" to "osb_server1.bak", re-created an empty "osb_server1" directory. I restarted osb_server1 (from the Weblogic console), voila, it started up beatifically. Node1 synched up with node2 without incidents!

Lessons learned:

1. in a production environment, try to deploy OSB update using resources, not the whole project, as the whole project update has a bigger foot print that may affect many proxies and other resources. If you only changed a couple of resources, only update the changed resources so you can minimize the chance of cluster syching problem.

2. Do not just take the tech support's advice blindly. If we did, we would need to shut down a busy client portal site. Shutting down a production environment should only be the last resort, not the first and only advice to the clients.

additional note (4/29/13):

Recently, i noticed another example of OSB server is out synch. We initially added a "report" activity, however, it causes some issue with the OSB report DB. The symptom is OSB will continue try to commit the DB transaction after it fails. It creates an indefinite loop. It fills up the log file quickly and brings the server down. We quickly removed the report activity from the proxy, activated the change, and bounced both nodes of OSB. But it didn't help.

One particular behavior is the error only happens on one OSB node (node 2). My theory is when the original proxy call was made, the request was dispatched to node2, however, the request failed during db transaction, therefore it is stuck only in node2 (attempting to commit DB transaction).

I applied the same trick by blowing away osb_server2 directory. No luck :(

Eventually, I found out this deleted report activity (with one unfinished db transaction) is somehow stuck in the JMS persistent store (go figure!). In our case, the persistent store is .../osb_cluster/jms/FileStore_auto_2. We blow this file away (gee, not sure we should just do that), restarted node2, that took care of the problem.