1. go to SOA ext folder, in my case: C:\Oracle\Middleware\Oracle_SOA1\soa\modules\oracle.soa.ext_11.1.1
2. you have two ways to do it:
2.1 explode your jars under "classes" sub folder here
2.2 copy the jar files in this folder, then in command prompt, run "ant" (this folder should have a default build.xml file), it basically does some magic and your jars will be included in SOA. As for the "magic", I believe it simply updates oracle.soa.ext.jar file and modified the "Manifiest" under "META-INF", set something like "Class-Path: yourCustomer.jar classes/"
Keep in mind, that push to shove, you may also have to add the same jar to "$DOMAIN_DIR/lib", that you just drop the file in.
I experienced problem before that merely putting jar in "racle_SOA1\soa\modules\oracle.soa.ext_11.1.1" works fine for SOA run time (with your BPEL referencing the class in the jar). However, if you don't copy the same file under "$DOMAIN_DIR/lib", you cannot re-deploy your BPEL composite that referencing the custom class. Because BPELCompiler, 2nd pass compilation (during the composite deployment), seems to require the jar to be in "$DOMAIN_DIR/lib" folder. Go figure...