Custom JCA Adapter and Dynamic log4j Logging Level Control with Weblogic

Recently I worked on dynamical log4j logging level control in a custom JCA adapter. I still don’t know much about JCA adapter. But I managed to make it work, and in the process I picked up a few more interesting things about Weblogic and log4j configurations.

At the core, the solution is the same as in http://yuanmengblog.blogspot.com/2013/08/weblogic-log-files-and-log4j-tricks.html i.e. use this code snippet in a "central" location of your code:

if (notInitialized) {

   String fileName = System.getProperty("jca.log4j.configFile");

   DOMConfigurator.configureAndWatch(fileName, 3000);

}

I stumbled upon a few subtle things about log4j and Weblogic before I made it to work. Here are a few things to pay close attention if you want to do it:

1. By default, Weblogic uses its own logging mechanism (it’s based on log4j)

2. setDomainEnv.sh script may set a specific Java system variable "-Dlog4j.configuration". Based on my test, this variable is sort of a "reserved" by Weblogic. The value of this variable is "file:yourAbsoluteFilePath". Be extra careful with the variable name and the "file:" prefix. 

3. If you place log4j.xml in the domain home directory, then Weblogic will pick it up and uses that one. It confused me for a while. I only found it out by trial and error.

The common thing about these scenarios is that Weblogic will internally pick up the log4j.xml file. When that happens, Weblogic initializes log4j, and you have no control of the file whatsoever. You can modify the log4j.xml all you want; weblogic is not going to do anything with the changes unless you restart the server.

So the solution is :

1. Do not to place "log4j.xml" in the domain folder, or use a different file name, such as "mylog4j.xml". 

2. Use a different Java system variable name, such as "jca.log4j.configFile" (just make sure do not "log4j.configuration").

3. Since you pick your own variable name, you should not use the "file:" prefix for the variable value. Instead, just use the full path name of you log4j configuration file.

That’s about it. I named my file "my-test-log4j.xml". I picked a variable name like "jca.log4j.confiureFile". I was able to change "my-test-log4j.xml" and watch the logging levels change on the fly.

Also, make sure if you start the server (such as AdminServer) with the script, then you need to update the script to pass the variable in. If you use nodeManager (for managed servers), then you should use weblogic console, server, startup argument field to pass in the variable.

Finally, in the JCA rar file, there is a weblogic-ra.xml that allows you to name your "<log-filename>" element, I tested that, it merely generated a file with that name; nothing gets logged into that file. I’m not sure how to use that file. So I simply skipped this element in the weblogic-ra.xml file.

Weblogic log files and log4j tricks

I have relied on weblogic log files for trouble shooting for many years, but never given a serious thought where they are configured and how to customize them. Well, I still don't know the whole story, but I managed to learn a couple of interesting things lately.

1. By default, WebLogic uses Java logging API which is based on log4j, but it is not exactly log4j. Therefore, if you install an out of box domain, there is no direct reference to log4j.

2. Your application can choose to use log4j for logging, then you can control how things are logged and where to log to etc.

3. If you want to system to dynamically pick up log4j.xml changes during run time, you can use https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/DOMConfigurator.html I'll show a full example below.

4. We also need to send certain log messages to sensage, which monitors the syslog. What we need was using log4j "syslogger" to direct log msgs to syslog, then we configured the /etc/syslog.conf to forward the messages to senage like:
# Sensage Configuration
*.notice;mail.none;user.none;auth.none;mark.none;kern.none     @sensage.acme.com
mail.crit;user.crit                                    @sensage.acme.com
kern.info                                              @sensage.acme.com
auth.info                                              @sensage.acme.com
user.info                                               @sensage.acme.com

However, in order to send message to syslog file, we had to modify "syslogd" start up script and add "-r" parameter.

Now, for the sample log4j and DOMConfigurator example. I grabbed some sample file online and modified it with the configurator:

import org.apache.log4j.Logger;
import java.io.*;
import java.sql.SQLException;
import java.util.*;
import org.apache.log4j.xml.DOMConfigurator;
public class logExample{
  static Logger log = Logger.getLogger(logExample.class.getName());
  public static void main(String[] args)
                throws IOException,SQLException{
BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
String file = System.getProperty("Log4jConfigFile");
System.out.println(String.format("Log4jConfigFile=%s",file));
DOMConfigurator.configureAndWatch(file, 3000);

     while (true) {
     log.debug("Hello this is an debug message");
     log.info("Hello this is an info message");
     System.out.println("modify your log4j.xml, then hit enter to continue, monitor the changes take effect.");

String x = null;
try {
x = reader.readLine();
} catch (IOException e) {
e.printStackTrace();
}
  }
  }
}

Here is my local compile and run script:

javac -cp .;C:\Oracle\Middleware\Oracle_OSB1\lib\external\log4j_1.2.8.jar -d . logExample.java

java -cp .;C:\Oracle\Middleware\Oracle_OSB1\lib\external\log4j_1.2.8.jar -DLog4jConfigFile="c:\proj\poc\log4j\config\log4j.xml" logExample

You can use your own log4j.xml or grab a sample from here http://stackoverflow.com/questions/2619160/sample-xml-configuration-for-log4j-have-a-main-java-application-and-want-to-w

Finally, to integrate this sample into your web application, you need to find your initialization function, such as servlet context initializer, and insert the DCOMConfigurator over there. Of course, remember to put in -DLog4jConfigFile in your server startup script if your server is started with the script, or put it in the weblogic console server startup argument field if you use nodeManager to start your server(s).

setup multiple SOA domains on localhost

If you need to create multiple domains on your localhost for testing purpose, here is what I do:

Create 1st domain as normal.

For 2nd (and each additional) domains:

1. use config wizard to set up the domains, until towards the end
2. on "Select Optional Configuration" screen, check "Administration Server", then you can select a different admin port (i believe you can also simply change the port after config, use config.xml with <listen-address>)
3. after domain is created, edit domain/bin/setDomainEnv.cmd, find
if "%DEBUG_PORT%"=="" (
set DEBUG_PORT=8453
)
change the port# to something new, e.g 2nd domain, set it to 8454, 3rd domain set it to 8455 etc...

While i'm at it, here is how to delete a domain:

1. delete "user-projects/domains/your-domain-folder" (or rename it)
2. go to middleware home (c:\oracle\middleware), in domain-registry.xml, delete your-domain entry
3. oh, remember to delete "user-projects/applications/your-domain-name" folder as well.

Java Web Service Resource Injection Fails on JMS Connection Factory When JMS Server Migrates

Here is the problem:
We have a clustered Weblogic environment. Let's say two managed servers M1 and M2. We have JMS server that needs be migrated among the managed servers. The JMS server is initially targeted to M1.

We have web services running on the cluster. One service publishes messages to the JMS server. This web service uses Java resource injection for the JMS connection factory like:
    @Resource(mappedName="ll.jmsprovider.XAConnectionFactory")
    private ConnectionFactory connectionFactory;

When the service is deployed to the cluster. "connectionFactory" is properly initialized, everything is fine. Even when we shutdown M1, the JMS resource is migrated to M2, the "connectionFactory" still works fine on M2. We can still publish messages.

The problem is when we bring back up M1, and shutdown M2. That's when "connectionFactory" doesn't work anymore. I suspect there might be some configurations on migration policy to make "connectionFactory" work seamlessly when the migration happens. But I don't know how. So my kludge is to recreate the "connectionFactory" on the fly. Here is the entire "hack":

package pubmsg;

import javax.annotation.Resource;

import javax.jms.Connection;

import javax.jms.ConnectionFactory;

import javax.jms.JMSException;

import javax.jms.MessageProducer;

import javax.jms.Queue;

import javax.jms.Session;

import javax.jms.TextMessage;

import javax.jws.Oneway;

import javax.jws.WebMethod;

import javax.jws.WebService;

import javax.naming.InitialContext;

import javax.naming.NamingException;

import javax.transaction.SystemException;

@WebService(targetNamespace = "http://pubmsg.org/")

public class msgSender {

    @Resource(mappedName="ll.jmsprovider.XAConnectionFactory")

    private ConnectionFactory connectionFactory;

    @Resource(mappedName="ll.jms.partner.enroll.queue")

    private javax.jms.Queue queue;

      

    public msgSender() {

        super();

    }

    @WebMethod

    @Oneway

    public void sendMessage(String msg) {

        Connection conn = null;

        try {

            try {

                 conn = connectionFactory.createConnection(); // my "hack"

            }

            catch (Exception e) {

                InitialContext ctx = new InitialContext();      

                connectionFactory = (ConnectionFactory) ctx.lookup("ll.jmsprovider.XAConnectionFactory");

                conn = connectionFactory.createConnection();

                queue = (Queue) ctx.lookup("ll.jms.partner.enroll.queue");

            }

   Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);

   TextMessage message = session.createTextMessage(msg);

   MessageProducer messageProducer = session.createProducer(queue);

   messageProducer.send(message);

        }

        catch (NamingException e) {}

        catch (JMSException e) {}

        finally {

                if (conn!=null) {

                        try {

                                conn.close();

                        } catch (JMSException e) {}

                }

        }

    }

}

Not the most elegant solution. Oh, well, i need to move on...

Ironically, the purpose of this blog is keep track what i did and how i did it. But it has been several weeks since I worked on it, I already forgot the steps how I created the project :( So I just clicked through JDeveloper try to re-construct the same thing. I believe this is a sequence that can create a java based web service in Jdeveloper (11.1.1.6):

Part I - Ceate a "Web Project"
 1. select project name
 2. select default "serverlet 2.5..."
 3. chose "None" on the "page flow technology" screen
 4. chose no libraries.
 5. select doc root, app name, context root etc.
 6. finish

Part II - add a Java class,
    add the public methods as you need

Part III - add "Web Services->Java Web Service"
   select your java class, from that point on, pretty much follow the screen. That should just work.

====
hmm, i did find another way to create the Java web service: you can start with a generic application, then select "Web Services" for the default project, then "java" is selected by default. However, this process only works for the 1st default project. Once the application is created, and you try to add new project, and you pick "web services", you'll get different options, talking about confused...

Unix file permission when file adatper generates the output file

When use file adapter to write a file on Linux, what is the permission of the output file?

The common assumption is it is determined by the umask of the user who owns the “Weblogic” java process. If a user runs startWeblogic.sh from the command line, then that user is the owner of Weblogic processes (many places use “oracle” or “weblogic” as the user to start webogic processes).

For the discussion, let’s assume the owner of “weblogic” logic process is “webadmin”. We need to generate an output file with “664” permission. However, we keep getting 644.

We double checked “webadmin” user has 002 umask. We also tried to play with the umask settings in startWeblogic.sh script to no avail.

So it’s gloves off time, I added java embed to change the output file permission by calling “setWrtiable(true, false)”, and that, opened another can of worms.

To my surprise, I stumbled on another unexplainable file adapter behavior. After the file adapter finished writing the file (with 644 permission), if I use java embed to open the file, and do any operations on the file (even just read it), then it TRUNCATES the output file. I thought it has to do with the transaction, so I tried to call check_point() before I open the file in java. It made no difference.

In the end, I just skipped the file adapter, use java embedding to generate the output file, then set the permission. Here is the snippet:

        try {                   

                String content = (String)getVariableData("csvOutput");   // grab file content                       

                String fileName = (String)getVariableData("fileName"); 

                fileName = "/output/"+fileName;

                FileWriter fstream = new FileWriter(fileName);

                BufferedWriter out = new BufferedWriter(fstream);

                out.write(content+"\r\n");

                out.close();

                                               

                java.io.File f = new java.io.File(fileName);

                f.setWritable(true, false);  // set permissions

                f.setReadable(true, false);

        }      catch(Exception e)      {

                System.out.println("*****file exception:"+e.getMessage());

        }

 There is an Oracle Note - 1430075.1 that covers the same issue. Oracle recommendation is to use java embedding, then do

   System.exec(“chmod…”);

I didn’t try that. Not sure how it works. Hope it doesn’t truncate the output file.

Rolling soa_server.out log file

SOA server keeps many logs. They are mostly WLS logs, such as soa_sever1.log, osb_server1.log etc. All of these files can be configured to auto roll. Except soa_server1.out.

Strictly speaking, soa_server1.out is not really WLS "log" file. If you start SOA server from a command window, by default the standard output spits all the messages in the command Window. When you start SOA server via node manager, the standard output get redirected to soa_server1.out. Since standard output is not really a log file, you cannot configure WLS to roll this file. Over time, soa_server1.out can grow very big and cause problems. 

I'm sure many people have their solutions to deal with the issue. Here is a cheat I figured out that can achieve the effect of "rolling" soa_server1.out.

Write a simply script:

  cp soa_server.out soa_server.out_$(date +%b)_$(date +%d)_$(date +%Y)"

  echo > soa_server.out

the first command backs up the file, 2nd command truncates the file, but still keep the same file. SOA is happy to to dump data into the same "log" file. Use cron to schedule the script to run, that should be it.

One reminder, i used cp in the first step. Then truncate the file in 2nd step. I also attempted in the first step to rename soa_server1.out file, then "touch soa_server1.out" to create a new log file. It doesn't work that way. SOA server obviously has got its mind somewhere, it doesn't like you to rename the log file. Even if you create a new soa_server.out, it won't write to the new file.