Maintenence Mode in R12

Q.Why we enable maintenance mode while patching?

Maintenance mode provides a clear separation between normal runtime operation of Oracle Applications and system downtime for maintenance. Enabling the maintenance mode feature shuts down the Workflow Business Events System and sets up function security so that no Oracle Applications functions are available to users. Used only during AutoPatch sessions,maintenance mode ensures optimal performance and reduces downtime when applying a patch. For more information, refer to Preparing your System for Patching in Oracle Applications Maintenance Utilities.

Q.what is the use of having two appl_tops (shared appl_top)?

If you have two/three (Multiple) Middle Tier for Large User base then you have to install APPL_TOP, COMMON_TOP & ORA_TOP equal to number of Nodes in your configuration. So if you have three middle tier then you need these three top three times & if there are n nodes you need n time these three top’s .This result in lots of Space Requirement & bigger problem is patching so If you have 5 nodes you need to apply apps patch 5 times.
So Oracle Came Up with solution to have single APPL_TOP which is shared across all nodes in apps. This will save space as well patch timing. This concept of sharing APPL_TOP file system across all Middle Tier Nodes is called as APPL_TOP . You will say if I can share APPL_TOP why Can’t I share COMMON_TOP & ORA_TOP as well which are also part of Application Tier , Yes you can & this concept is called as Shared application Tier.

Q.how can we create context file in 11i and R12?

in 11i we can user adbldxml.pl for both application and database node but in R12 it is changed

 ./adclonectx.pl retrieve-- on Apps tier

./adbldxml.pl---- Dbtier

Q.What will happen When you run Adpreclone and Adcfgclone

This will run in two steps Techstack and database.
Techstack:   It will create following directories in the ORACLE_HOME/appsutil/cloneJlib, db, data where “Jlib” relates to libraries “db” will contain the techstack information, “data” will contain the information related to datafiles and required for cloning.
Creates driver files at ORACLE_HOME/appsutil/driver/instconf.drv
Converts inventory from binary to xml, the xml file is located at $ORACLE_HOME/appsutil/clone/context/db/Sid_context.xml
Prepare database for cloning:  This includes creating datbase control file script and datafile location information file at
$ORACLE_HOME/appsutil/templateadcrdbclone.sql, dbfinfo.lst
Generates database creation driver file at ORACLE_HOME/appsutil/clone/data/driverdata.drv
Copy JDBC Libraries at ORACLE_HOME/appsutil/clone/jlib/classes12.jar and appsoui
When you run “adpreclone.pl appsTier” On Apps Side:
This will create stage directory at $COMMON_TOP/clone. This also run in two steps.
Techstack:  Creates template files for Oracle_iAS_Home/appsutil/template and Oracle_806_Home/appsutil/template
Creates Techstack driver files for IAS_ORACLE_HOME/appsutil/driver/instconf.drv and 806_ORACLE_HOME/appsutil/driver/instconf.drv
APPL_TOP preparation:  It will create application top driver file at $COMMON_TOP/clone/appl/driver/appl.drv-Copy JDBC libraries and $COMMON_TOP/clone/jlib/classes111.zip
Now Shutdown all the services of Application and database to Copy the file System to target location
On Target Machine
===============
Run adcfgclone.pl for dbTier.
On database side:
cd $ORACLE_HOME/appsutils/clone/bin
perl adcfgclone.pl dbTier pwd=apps
This will use the templates and driver files those were created while running adpreclone.pl on source system and has been copied to target system.
Following scripts are run by adcfgclone.pl dbTier for configuring techstack
adchkutl.sh — This will check the system for ld, ar, cc, and make versions.
adclonectx.pl — This will clone the context file. This will ceate a new context file as per the details of this instance.
runInstallConfigDriver — located in $Oracle_Home/appsutil/driver/instconf.drv
Relinking $Oracle_Home/appsutil/install/adlnkoh.sh — This will relink ORACLE_HOME
For data on database side, following scripts are runDriver file $Oracle_Home/appsutil/clone/context/data/driver/data.drv
Create database adcrdb.zipAutoconfig is runControl file creation adcrdbclone.sql
Run adcfgclone.pl for dbTier.
On Application Side:
cd $COMMON_TOP/clone/bin/
perl adcfgclone.pl appsTier pwd=apps
Following scripts are run by adcfgclone.pl:
Creates context file for target adclonectx.pl
Run driver files $ORACLE_HOME/appsutil/driver/instconf.drv and $IAS_ORACLE_HOME/appsutil/driver/instconf.drv
Relinking of Oracle Home $ORACLE_HOME/bin/adlnk806.sh and $IAS_ORACLE_HOME/bin/adlnkiAS.sh
At the end it will run the driver file $COMMON_TOP/clone/appl/driver/appl.drv and then runs autoconfig.

Q: Whether we can apply 2 patches at a time without merging them, why ??

A: We can’t apply 2 patches at a time because when patch started it will create 2 tables in db (AD_DEFERRED_JOBS AND

FND_INSTALL_PROCESSES). If you apply patches at a time both will try to create those 2 tables in db, so both will fail.

Q. After applying patch whether we can revert it back ?

A: No, as it is going to update some tables in db.

Q. Whether we need to run adpreclone everytime when we clone, why ?

A: We have to run adpreclone if there are changes in any customizations or any other tablespaces like if applied any AD patches,

Big Patches, Minipacks, ATG patches, Tech Stack, and AD Patches. If there are no changes, then i don’t think there is any need to

preclone it again.

Example: When you run adpreclone.pl, it essentially does an “alter database backup controlfile to trace” and stores the information from

that in a file within ${ORACLE_HOME}/appsutil. This represents all of the datafiles in the database at that particular time. The information

gathered at this time represents the state of the source at this time. If you does not run adpreclone.pl again when u do clone next time,

Your backup (your target) represents an earlier time (prior to the addition of that datafile), so the “create controlfile” step

that adcfgclone is doing is going to fail.

Q: Which files it will call when we run adcmctl.sh start apps/apps ?

A: It will call FNDLIBR executable, which will be located in $FND_TOP/bin.

Q: What are the main tables related to concurrent manager ?

A: FND_NODES

FND_CONCURRENT_PROCESSES (fcproc)

FND_CONCURRENT_REQUESTS (fcr)

FND_CONCURRENT_QUEUES (fcq)

FND_CONCURRENT_PROGRAMS (fcprog)

FND_EXECUTABLES

FND_CP_SERVICES

FND_CONCURRENT_QUEUE_SIZE

FND_CONCURRENT_QUEUE_CONTENT

FND_CONCURRENT_PROGRAM_SERIAL

FND_CONCURRENT_TIME_PERIODS

FND_CONCURRENT_PROCESSORS

Oracle Application Cloning (11i)

Cloning is the process of creating an identical copy of the oracle application system.There are two types of cloning methods available for Oracle application one is adclone and another one is rapid clone.
Ad clone is an oracle provided utility to clone application system. This utility is used to clone application system for release 11.5.1 to 11.5.5 for systems which are not autoconfig  enabled, Whereas rapidclone is used for those which are autoconfig enabled.
We can find our application is autoconfig enabled or not by opening env files from APPL_TOP APPLSYS.env and APPSORA.env. If the top of the file says that it is maintained by autoconfig, then your system probably using auotconfig.

What happens when you run adpreclone.pl?
adpreclone.pl prepares source system to be cloned by collecting information about the database and creates various templates of files containing source specific hard core values.It collects all the information about the source application system, the port number of the source etc.these templates are stored in the appsutil/templae directory.

Different kinds of Cloning.

1)Single node to multi node clone

Sorce Node-PROD                                                                  Tartget-Dev1

DB Tier                                                                                    DB Tier 
prodb                                                                                       proddb
proddata                                                                                   proddata

                                                                                                Tartget-Dev2
Apps Tier
prodcomn                                                                                Apps tier
prodappl                                                                                  prodcomn,prodappl,prodora
prodora

Some of the reasons to do a cloning are:

• To create a test environment from an existing production environment to test some patches or to reproduce any production issues.
• To keep a test environment with the most current information of a production environment.
•  To move any existing environment to other servers

Cloning prerequisite steps:

Make sure target servers have same component verson and OS as source servers.

• Login as Applications file user & set the environment file on source node.

           -bash-3.2$ su - applmgr
           -bash-3.2$ cd /maildata_test/mcgm/R12/apptier/apps/apps_st/appl
          -bash-3.2$. APPSrdb_mailserver2.env

• Login to database tier as oracle user and set the environment on source node.

            -bash-3.2$ su - oracle
            -bash-3.2$ cd /maildata_test/mcgm/R12/dbtier/db/tech_st/11.1.0/
            -bash-3.2$ . rdb_mailserver2.env

Prepare the source server

1.      Prepare the source system dbtier for cloning 

           -bash-3.2$ cd /maildata_test/mcgm/R12/dbtier/db/tech_st/11.1.0/appsutil/scripts/rdb_mailserver2/
            -bash-3.2$.perl /adpreclone.pl dbTier

     2       Prepare the source system application tier for cloning
               -bash-3.2$  cd $COMMON_TOP/admin/scripts
                                in R12(cd $ADMIN_SCRIPTS_HOME)
              -bash-3.2$cd  /maildata_test/mcgm/R12/apptier/apps/rdb_mailserver2/admin/scripts(R12)
              -bash-3.2$perl adpreclone.pl appsTier

Copy the Source Node File System

1. Log on to the source system application tier nodes as the APPLMGR user
2. Shut down the application tier server processes as shown below

             -bash-3.2$cd $COMMON_TOP/admin/scripts/rdb_mailserver2(11i)
             -bash-3.2$cd  /maildata_test/mcgm/R12/apptier/apps/rdb_mailserver2/admin/scripts(R12)
              -bash-3.2$./adstpall.sh apps/apps

Copy the following application tier directories from the source node to the target application tier node:
copy appltop and commontop and oratop to target servers
scp –pr appl_top applmgr@target_server:/d01/oracle
scp –pr comn_top applmgr@target_server:/d01/oracle
scp –pr ora_top applmgr@target_server:/d01/oracle
Once copied –

Check the ownership as required

Rename the directory on target server
cd /d01/oracle
mv appl_top devappl
mv comn_top devcomn
mv ora_top devora


Copy the database tier file system

stop all database services
Perform a normal shutdown of the source system database
-bash-3.2$cd $RDBMS_ORACLE_HOME/appsutil/scripts/rdb_mailserver2
-bash-3.2$./addbctl.sh stop
Copy the database (DBF) files from the source to the target system
Copy the source database ORACLE_HOME to the target system as shown below:
cd /d01/oracle
scp –pr proaddb oracle@target_server:/d01/oracle
scp –pr proaddata oracle@target_server:/d01/oracle
Once copied –
- Check the ownership as required
- Rename the directory
cd /d01/oracle
mv proddb devdb
mv proddata devdata
Start up the source Applications system database and application tier processes

Configure the Target System

Operating System of Target should be same as Source. Operating system should have all pre-requisite packages required for Oracle R11i before configuring the Target System. Execute the following commands to configure the target system. You will be prompted for the target system specific values (SID, Paths, Ports, etc).
Log on to target node as oracle user and run the following command and input your values to each prompt as shown below
-bash-3.2$ cd /maildata_test/mcgm/R12/dbtier/db/tech_st/11.1.0/appsutil/clone/bin
-bash-3.2$ perl adcfgclone.pl dbTier
Enter APPS Password :
 Configure the target system application tier server nodes
Log on to the target system as the APPLMGR user and type the following commands and
specify your values to each prompt as shown below.
-bash-3.2$ cd /maildata_test/mcgm/R12/apptier/apps/apps_st/comn/clone/bin
$ perl adcfgclone.pl appsTier
Enter the APPS password
All application services are started successfully. Thatmeans, we have done cloning successfully.
Finishing Tasks
Post clone steps vary from client to client, here is the basic change.
Profile Option Name Changes at Site Level after Cloning
Site Name-> PROD, Change it to “DEV – Clone of PROD as of Dec 14 2010”
DEV – Clone of PROD as of 14-Dec-10
Login to Oracle Apps as SYSADMIN
Select System Administrator responsibility


Multinode Sharedappltop to same architechture

If you having shared APPL_TOP then apart from running the adcfgclone.pl you need to run "adclonectx.pl sharedappltop" for sharing appltop from all nodes. In this case you need to run adcfgclone.pl only form the first node and you can create the context file using adclonectx.pl in  other nodes using context file refernece as first node. Then you need to run the txkSOHM.pl from $FND_TOP/patch/115/bin which will create 8.0.6 and IAS config homes for your application system.

Multinode to single node(multinode appltop to single appltop)

1)Maintain snapshot information

Log in as the applications user on each application tier node and run ‘Maintain Snapshot Information’ by using adadmin.

2) Merge existing APPL_TOPs
Log in to the primary node of your application tier as the application user user and run:
$ cd [COMMON_TOP]/admin/scripts/[CONTEXT_NAME]
$ perl adpreclone.pl appsTier merge

This will prompt you with option to merge secondary nodes

Now log in as the applications user to each of the secondary application tier nodes being merged and run:

$ cd [COMMON_TOP]/admin/scripts/[CONTEXT_NAME]
$ perl adpreclone.pl appltop merge

4) Prepare the source system database tier.

Log on to the database tier of the source system as the oracle user and run the following commands:

$ cd [RDBMS ORACLE_HOME]/appsutil/scripts/[CONTEXT_NAME]
$ perl adpreclone.pl dbTier

With this all the pre clone tasks on the source system have been completed.

Steps to be performed on target system

5)Create OS user accounts

Create a OS user account for your applications

$ useradd -g dba -d [home_directory] -p password username

Similarly create a OS user account for your database

copy the required tops from 

Copy the following application tier directories from the primary node of your source system to the target application tier node, retaining the original directory structure:

– [APPL_TOP]
– [OA_HTML]
– [OA_JAVA]
– [COMMON_TOP/util]
– [COMMON_TOP/clone>
– [806 ORACLE_HOME]
– [iAS ORACLE_HOME]  

6. Copy the required files for merging

Log in as the applications user to each of the secondary source nodes and recursively copy:

directory [COMMON_TOP]/clone/appl
- to -
directory [COMMMON_TOP]/clone/appl on the target system node

Before proceeding with the next steps you must shutdown your oracle applications services and the database on the source system

7. Copy the database tier file system

Log on to the source system as the database user

Copy the database (DBF) files from the source to the target system
Copy the source database ORACLE_HOME to the target system

After this you can now startup the database and applications services on your source system and release it for use.

8. Configure the target system database server

Log on to the target system as the database user and type the following commands to configure and start the database:

$ cd [RDBMS ORACLE_HOME]/appsutil/clone/bin
$ perl adcfgclone.pl dbTier

This will prompt for new ORACLE_SID,ORACLE_HOME,Port Pool,JAVA_TOP and DATA_TOP give the appropriate values matching your target system

Once successful this should start your database and listener

9. Configure the application tier server nodes

The database and its listener should remain up before executing the next set of commands.
Log in to the merged APPL_TOP node as the applications user and execute the following commands:

$ cd [COMMON_TOP]/clone/bin
$ perl adcfgclone.pl appsTier

This will prompt you new port pool for applictaion tier services as well as new APPLTOP,COMMON_TOP,ORACLE_HOME and IAS_TOP

Successful completion of this task will bring up your application tier services on the target or the cloned node.

Post-cloning Tasks

Log in to the target system application tier node as the APPLMGR user.
Run the following tasks in adadmin for all products:
- generate JAR files
- generate message files
- relink executables
- copy files to destination

- Remove the temporary directory [COMMON_TOP]/clone/appl to reduce disk space usage.

Note: Rapid clone takes care of the site level profile options all other profile options need to be manually updated.

Unified APPL_TOP in R12

Unified APPL_TOP is different from separate APPL_TOP in Oracle Applications 11i. Unified APPL_TOP make more sense if you are using multi node Oracle Applications R12.

In Oracle Applications 11i , in multi node installation each APPL_TOP have its different name and files in each APPL_TOP will depend on type of Node (i.e. Forms Node will have fmx or forms related files where as APPL_TOP belonging to CM only node will have rdf or files required to run CM node)

Starting from R12, it will use Unified APPL_TOP which means all files required for all middle tier services are included in all nodes of Multi Node installation (irrespective of services running on that node)

What are few changes because of Unified APPL_TOP?

During Cloning of multi node Oracle Application install, only one copy (any one appl_top) of Application Node files need to copied to target instance.

During Cloning of Multi Node to Single Node you don’t have to merge APPL_TOP’s in R12 as required in 11i .

Services start/stop If you are using adstrtall.sh to start services on a node then it will start services which were configured to start during install on that node (You can still start a specific service even though it was not suppose to configure/start by calling startup script of that specific services).
To explain this better, lets assume you installed multi node R12 instance with Node 1 as Forms & Web Server where as Node 2 was installed with Concurrent Manager. Now on Node1 when you use adstrtall.sh script to start services as expected it will start Forms and Web services but unlike 11i, You can still start Concurrent Manager on Node1 by running adcmctl.sh
–FND_NODES From R12 all nodes will have Y against all services (For multi-node) in FND_NODES table.

Adutilities in Oracle Apps

QWhat are the Ad utilities?
Ad utilities stand for application DBA utilities. These are set of tools used for installing, upgrading, maintaining and patching Oracle application. There are around 15 Ad utilities.

QWhat is maintenance mode and how it enable/disable the same?
Maintenance mode is a feature which is introduced in 11.5.10. For doing any maintenance activity in the application system like applying patch the maintenance mode need to be enable in the application system.It can be disable and enable using adadmin utility .

QWhat is the log file location for Adutilities?
$APPL_TOP/admin/$TWO_TASK/log. format <adutility name>.log

QWhat is adadmin why it is used?

Adadmin is used to perform a number of administrative tasks to maintain the Oracle Applications.Adadmin ensure that Oracle Application run smoothly.
Adadmin perform two types of works -one which is performed at database level and other which is performed at the file system level.
Adadmin can be scheduled to run at a later time in a non-interactive mode.

QHow Adadmin run in a non-interactive mode?

To run Adadmin in a non-interactive mode, you must at first create defaults file. Once the defaults file is created you can run adadmin in non-interactive mode using this file.The defaults file must be located under
APPL_TOP/admin/SID
$adadmin defaultsfile=APPL_TOP/admin/SID/defaults.txt
$adadmin defaultsfile=APPL_TOP/admin/SID/defaults.txt
 logfile-adadamin.log
workers=10
interactive=no
menu_option=DISABLE_MAINT_MODE
In order to choose which task defaults file run you add menu option.

QHow can you regenerate all the forms files using adadmin and what does it do internally?
The generate forms files options of adadmin take  care of generating form files(extension .fmx) from binary forms defination file (extension .fmb). These binary form files are located under AU_TOP and the executable files are stored under each products directory.

QYou have accidentally deleted the environment file how can u recreate it?
You can recreate the environment file by using the options create applications file from adadmin.This options create environment file that your system configuration.

QWhat is the significance of the DUAL table? Who owns this table and how many rows it should have?
The DUAL table is created automatically by Oracle along with the data dictionary. It is located in the schema of SYS. It has one column named DUMMY of type VARCHAR2 and contain one row with the value of 'X'. In case it has more than one row application system may not function properly.

QWhat is ad splice? what does it do?
Oracle often release new products after the base release of Oracle applications. This products are known as off cycle products. Adsplice is a utility which take care of incorporating and off cycle products in to Oracle applications so that it is recognized by the ad utilities as a valid oracle application product.

QWhat are the advantages of distributed AD?
With distributed AD the workers can utilize the additional resources of the remote nodes where they are running apart form the primary node. This capability improves scalability, performance and resource utilization and completion of worker in a faster time.

Distributed AD

Distributed AD

Distributed AD is a parallel processing feature that can reduce downtime by efficiently utilizing all the available resources on a shared application file system. AD Administration and Auto Patch run on the primary node and direct workers running on that node and other nodes in the system. The AD Controller utility controls and monitors the actions of the workers that you specify.
This parallel processing makes use of managers, which direct the actions of worker processes. The manager assigns each worker a processing job and monitors its progress. When a worker completes a job, the manager assigns it another until all jobs are complete. AD Administration and AutoPatch can be directed to distribute processing tasks across multiple remote machines in a multi-node system. This type of parallel processing operation is called Distributed AD. It further reduces the time to complete a maintenance task by utilizing the processing capabilities of all of the nodes in the system.
Distributed AD can only be used on systems that are using a Shared application Tier File System to ensure that files are maintained in a single, centralized location.

Using Distributed AD:

Start AutoPatch or AD Controller with Distributed AD worker options
On one of your shared application tier file system nodes, start your AutoPatch or AD Administration session with the following command line options:

localworkers= workers=For example, to run an AutoPatch session with a total of eight workers (three workers on the local node and five workers on a remote node):

$ adpatch workers=8 localworkers=3 Start AD Controller on remote node(s)
On each of the additional shared application tier file system nodes, start an AD Controller session with the additional distributed command line option:

$ adctrl distributed=yAfter providing basic information, AD Controller will prompt for the worker number(s) to be started. For example, to start workers 4 through 8 on a second node, enter “4 5 6 7 8″ or “4-8″.

The following is an example of starting a three-node session with a total of five workers:

Node 1:
$ adpatch localworkers=3 workers=5Node 2:
$ adctrl distributed=y
Enter the worker range: 4Node 3:
$ adctrl distributed=y
Enter the worker range: 5

During execution of AutoPatch or AD Administration, you can start a normal AD Controller session (without distributed=y) from any of the nodes in the shared application tier file system environment to perform any of the standard AD Controller operations. All of the standard AD Controller options have the same effect on both local and non-local workers, with the following exception:

option 6: Tell manager to start a worker that has shutdown this option will always result in a worker being started on the same node that the AutoPatch or AD Administration utility is running on. This means that if an AutoPatch worker exited on a distributed node, choosing this option will start the worker on the node that is currently running AutoPatch, rather than the node that was originally running the worker.

AD Controller Log Files

The log file created by AD Controller is created on whichever node the AD Controller session is started. This is to prevent file locking issues on certain platforms. It is therefore recommended that the AD Controller log file include the name of the node from which the AD Controller session is being invoked.
Managing Distributed Workers and Nodes

If a worker has exited on a distributed node 
Make sure that the worker is not running. 
Start AD Controller on the same distributed node using: 
$ adctrl distributed=y force_restart=yEnter the worker number(s) to start when prompted. AD Controller will start those workers and then exit. 

AdConfig in Oracle Apps

Autoconfig is an 11i configuration tool (set of java classes run from perl/shell scripts) introduced in 11.5.4, which eliminates the need to manually maintain these configuration files and profile options. Autoconfig maintains the environment specific values for the 11i environment in an xml file (stored in $APPL_TOP/admin). When autoconfig run it overwrites the existing configuration files with new ones that it creates by merging the templates and the context file, it also updates the database by running the script it instantiates (merging a template with the values in the context file is termed as instantiating the template in Oracle parlance) from the script templates.

That’s essentially what autoconfig does i.e. Instantiating configuration files from templates and updating database with values from the xml file. It does not move directories, check for availability of ports, update DNS entries and so on. These have to be done manually.

How Auto config works..

Each configuration file has one corresponding template file (provided by autoconfig patches, stored in $PROD_TOP/admin/template directory). Profile options and other instance specific information in the database is maintained by many sql scripts, called from shell/perl scripts. These scripts also have corresponding template files (also provided by autoconfig patches, in $PROD_TOP/admin/templates).In these template files all the environment specific values are replaced by placeholders.

Configuration files can be easily created from the template file by just replacing all the placeholders with the corresponding values from the context file. This process is termed as “instantiating the template” in Oracle documentation. 

Driver files (stored in $PROD_TOP/admin/driver) store information about what to do with each template (e.g. instantiate it and replace the existing configuration file with it, instantiate it and run it). These files are named as <PROD_TOP>tmpl.drv (e.g. adtmpl.drv, fndtmpl.drv etc.) They contain one line for each template they manage.

When autoconfig (adconfig.pl or adconfig.sh) runs it just processes the driver file for each product, line by line doing what the driver file instructs it to do. The order of execution of each line is not sequential as we shall see later

How to run AutoConfig:

Step 1 : Stop all services 

$ $COMMON_TOP/admin/scripts/adstpall.sh APPS/ APPSPASSWORD

Step 2:Run the autoconfig script, $adautocfg.sh & respond to the prompts. 

Step 3:Check log files for any errors.log file for autoconfig is adconfig.log & is 

located in /admin//log/MMDDhhmm/ 

Step 4: Restart all services : 

$ $COMMON_TOP/admin/scripts/adstrtal.sh APPS/APPSPASSWORD

Roll back autoconfig :

Each execution of AutoConfig creates a roll back script in case we want to revert 

back to the previous configuration settings. For this we perform the following 

steps:

Step 1 :Shut down all services: 

$ cd $COMMON_TOP/admin/scripts/ 

$ adstpall.sh APPS/ 

Step 2:Restore AutoConfig session. 

$ cd $APPL_TOP>/admin//out/MMDDhhmm 

$ restore.sh 

Step 3 : Restart all services: 

$ $COMMON_TOP/admin/scripts//adstrtal.sh APPS/

Restoring the previous configuration 

Log file locations:(11i)

Dbtier

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/MM_DD_YYYY/adconfig.log

Appstier:

$APPL_TOP/admin/$CONTEXT_NAME/log/MMDDYYYY/adconfig.log

R12:
Appstier
$INST_TOP/apps/$CONTEXT_NAME/admin/log/$MMDDHHMM/adconfig.log
Dbtier
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/adconfig.log

Some important Question about Concurrent manager(11i)

Q.In administer concurrent manager form there are two columns target and actual. What are these columns.

  Target columns lists the number of processes that are running for each manager for a particular work shift.

Actual column that is currently how many process are running. If actual column is zero then there are no process are running for that particular manager.

If the Target column is zero then there is no work shift assigned for that manager or the current work-shift does not specify any target processes. If the target column not zero then the manger goes down or manager processes have either failed.

Q.How do i run/schedule a concurrent request from operating system level without logging into the applications.?

A Concurrent request can be scheduled/ru from the operating system using the CONCSUB utility.CONCSUB means concurrent submit.

QWhat are the different ways to check Concurrent manger is running or not?

 By using FNDLIBR fom os level, from the forms it can be check from navigatio>concurrent>Manager>Administer. It can be also checked by using script adcmctl.sh status finally it can be checked from OAM also.

QWaht is the default location of concurrent manager?

If environment variable set
 $APPLCSF/$APPLLOG
$APPL_TOP/admin/SID/log(11i)
$/maildata_test/mcgm/R12/apptier/apps/rdb_mailserver2/logs/appl/conc/log   (R12)
If environment variable not set
$FND_TOP/$APPLLOG

Q i have submitted a request and its showing the status inactive no manager

If the concurrent manager is up and running and the request goes to inactive no manager  for some time it means the cache cycle is less. Cache size set on the concurrent >manager> Define form..
Basically this regulates how many requests a manager will pick up for each sleep cycle.the solution is either increase the sleep cycle or increase the number of manager process. The manager should be either slandered manager or any other manager for which the issue is coming.

Q I have submitted a request  it has gone to pending standby status for a long time whereas other request are getting completed normally with out any issues. what could be the reason.?

 If any particular request is going pending standby status and other request are getting completed it means that either is waiting for other request output or it is conflicting with some other request. If the request is conflicting check the  queue of the conflict resolution manager for troubleshooting.

Q How do i process more concurrent request in parallel?

If you want to process more requests simultaneously there are two way by increasing target process for the manager and other one increasing cache size.


Q When do the tables FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESS need to be purged?

when the tables reached 20000 rows the performance being to diminish. You may want to run purge concurrent request on a regular basis depending on the amount of request being run.

Q What if the internal concurrent manager dies? All the managers also killed immediately after it?

No if the ICM dies the request continue to run normally except queue control requests.

Q Does the ICM run or schedule any request for itself?

No the ICM does not run or schedule any request. It has nothing to do with scheduling requests, or deciding which manager will run a particular request. Its function is only to run queue control requests which are requests to start up or shutdown other manager.It is responsible for start-up and shutdown of the whole concurrent processing facility and it also monitors the other managers periodically and restart them when they goes down. It can also take over the conflict resolution manager  job and resolve incompatibilities. 

Q If the ICM goes down do i need to kill all other managers before starting ICM?

No, If the ICM goes down no need to kill all other managers just simply start ICM

Q Can I delete Concurrent Manager?
Yes you can delete any concurrent manager for deleting , query for the manager in the defined concurrent manager form and delete the row.
Deleting the predefined  manager not recommended and it should never done.

Qwhat is internal monitor?

This manager is used to implement parllel concurrent processing . It monitors whether the ICM is still running and if ICM crashes it will restart it on another node.


QHow do i clean out concurrent manager tables?
For this Oracle provide a script called cmclean.sql.

QI hit the restart button to start the standard manager but it still not started?
Asking a manager to restart sets the status to restart. The ICM will start in the next process monitor session or next time ICM starts. Use activate to start the manager immediately. Also when a manager deactivated manually the ICM will not start it you need to set it to restart or activate it manually.

Q What are the circumstances in which you need to bounce the concurrent manager?

• when you modify the definition of the printer
• when  you modify the environment variables say you have changed the APPLTMP and APPLPTMP variable
• when all the requests are pending or hanging

Q What are the reasons a CM (concurrent manager) hangs?

• Long running jobs
• The Internal manager was activated by some one other than owner of the application system
• The OS file system is full.
• Its not able to create log files
• You've down the Internal Manager but actual has number on it.
• The database hanging may be because of archive logs full
• Pending/standby requests are too many

QThe Different parameter of startmgr utility

Parameters       Description                                                                         Default

sysmgr            sqlplus sername/pwd that owns the foundation tables           Applsys/<pwd>

Mgrname        the name of the manager                                                      Internel Manager

Log file           the logfile of the manager                                                     $FND_TOP/$APPLLOG

                                                                                                                                or

                                                                                                                  $APPLCSF/$APPLLOG

Sleep              the number of seconds the ICM should wait

                       before checking for the new request from the table                         60sec

                       FND_CONCURRENT_REQUESTS                                    

Restart            If the CM goes down abnormally it will automatically
                       restart the manager Y=the  number of minutes the ICM           N=no to restart after abnormal
                       waits before restarting the manager                                         termination

Mailto           MAILTO is a list of users who should receive mail whenever      Current user
                     Manager terminates

Printer           The default printer for sending the output files

Diag               This is used for diagnosis If the CM is started with parameter              N
                      diag=y then full diagnostic output is produced in the log file.

Pmon               The number of sleep cycle ICM will wait before checking                20
                        Failed manager

Quesiz             Number of pmon cycle the ICM waits between times it checks for      1
                        normal changes in concurrent manager operation


Q. What exactly happens when a concurrent request is submitted?
   
     Once a concurrent request is submitted by the user, the table FND_CONCURRENT_REQUEST is automatically updated with the details of the request. The table also updated with the information about schedule of the concurrent whether its immediately scheduled or scheduled at a fixed time. Once the request is scheduled to run the concurrent manager checks the FND_CONCURRENT_REQUEST table to find the request is incompatible with any other request . If the request is incompatible then the Conflict resolution manager take care of the request and find out what are the incompatibilities and resolve them. If there are incompatibilities then it checks for any other special manager assigned to that particular request. If there are any special manager to take care of this request then it goes to the queue of that manager else the standard manager takes care of the same.Once the requset is processed FND_CONCURRENT_REQUEST updated with that status.

Q.In the administer concurrent manager form, what is the significance of the terminate button?

  The terminate button used to terminate any concurrent manager. When you terminate internal manager all the manager automatically ge deactivated and all the running request are terminated. If you want to terminate a particular manager, select that manager and click terminate. The status of the manager changes to deactivate after a few seconds and all the requests are processed by that manager are immediately terminated. Once the manager terminated it does not starts  automatically. You have to manually restart it using the restart button.

Q In administer concurrent manager form what is the significance of the deactivate button and how can you deactivate a manager from there?

For deactivating a particular manager select that manager and press the deactivate button. In case of deactivation, all the requests are processed before shutdown manger does shutdown. If the internal manager goes deactivated all other manager goes deactivated and all requests are processed before manger going down.This is the main difference between terminate and deactivation. In termination all running request are terminated immediately whereas in case of deactivation all the running requests are allowed to complete first.

QIn administer concurrent manager form, what is the significance of the verify button and for which manager its available?

 The verify button enable for only internal manager. One of the function of the internal manager is to monitor the processes of the each concurrent manager. The process of monitoring the other manager process by IM is known as a PMON cycle.When you click the verify button you can force the process monitoring or the PMON activity to occur.The verify button also available for conflict resolution manager.

Q What is parallel processing and what is the significance of the same?
 
   Parallel concurrent processing is the way to distribute concurrent manager across multiple nodes in a cluster, massively parallel or networked environment. It helps in distributing the load across multiple nodes thereby fully utilizing the hardware resource

In case of parallel concurrent processing all the managers are assigned a primary and secondary node. The managers are started on primary node by default. In case of failure or Oracle instance failure all the concurrent manager get into secondary node. If the primary node available all concurrent manger get back to the primary node.During that migration process a manger spread across on both nodes .
In case of parallel concurrent processing it may happen that in a node where parallel co current processing is configured, the Oracle instance may or may not be running. The node which s not running Oracle, the concurrent manager connect via Net8 to a node where is running oracle.

The internal concurrent manager can  run on any node and can active and reactive concurrent managers on all nodes. Since the internal concurrent managers must be active at all times, it needs high fault tolerance. To provide this fault tolerance parallel concuncurent processing uses internal monitor processes. The job of Internal monitor is to monitor constantly the internal manager and start it when it is fails.Only one monitor process can be active on a single node. You have to decide which nodes have an internal monitor process when you configured system. You can also assign each internal monitor process a primary and a secondary node to ensure fail over protection. Internal monitor process like concurrent manager assigned a work shifts and are activated and deactivated by the internal concurrent manager.

Q I have defined  for nodes of the concurrent manager Now do i need to start the concurrent manager from all the nodes?
No, even if you have defined the concurrent manager in four different nodes you need not start the concurrent manager from all the nodes. You just need to start the concurrent on primary node GSM will take care of starting the concurrent manager form all other nodes.

QThe user comes to you saying that the request are taking a lot of time to complete. what is the debugging approach ?

• You can run trace on the request id to find the expensive sqls and then tell the developer to fix the same.
• you can check the program incompatibilities in the concurrent request
• you can check the query which the concurrent program is executing and see if it is creating nay locks in the database 
• many times the users schedule the request to run at a later time.
• You cab check the parameters with which the request is run.

Q Waht are the things take care when you defining a concurrent program.

• selecting an executable file to run the program
• Choosing the execution method for program
• Defining the parameter for the program
• defining the printing information

Oracle Concurrent Manager

The concurrent managers in the Oracle e-Business suite serve several important administrative functions. Foremost, the concurrent managers ensure that the applications are not overwhelmed with requests, and the second areas of functions are the management of batch processing and report generation.

User interaction with oracle application data can be conducted using with HTML application or more traditional form based application. How ever there are also reporting programs and data updating programs that runs periodically or an ad-hoc basis.

These programs will run in background to user continue there work on other tasks. Concurrent processing is an oracle application future that allows these non-interactive and potentially long running functions to be continued. A process that runs on concurrent processing server is call concurrent request.

When you submit a request through the html or form based application a row is inserted into the database table specifying a program to be run. Then concurrent manager reads the request from the table and starts the associate concurrent program. Acting as job scheduling and execution system concurrent manager will run in its own operating system process this operating system is call target process(often called workers) each worker can start one concurrent program at a time.

Whats a Concurrent Program
A program that implements a business functionality and needs to be executed again and again at regular interval or as per business needs is called concurrent program. They can be implement in PL/SQL, Shell Script, C/C++ etc..

Concurrent Manager
Now when a concurrent program is written, it needs to be executed daily at particular time. If we do it manual, there might be chances of delays or it might happen that two different people run the same program at the same time which might lead problems. So we need a manager which can do all this tasks for us. might lead problems. So we need a manager which can do all this tasks for us. Manager, which ensures that each concurrent program can run successfully without any conflicts. They also ensures that the applications are not overwhelmed with requests. They also manages the batch processing and report generation.
The default installation of Oracle Applications comes with a number of pre defined concurrent managers however you can create your custom concurrent managers to spread out the load of your job processing. Apart from taking care of the load of your jobs the concurrent managers can also schedule the jobs periodically. Also we can assign specific priority and specific times to the different programs, so that the concurrent managers can run them in specific workshifts.
Concurrent managers also allows you to tweak the number of concurrent process that it can handle concurrently. If any request exceed this prescribed limit they are automatically put on pending state. The processing of a request takes place based on the time of request submission and priority of the request submitted.

The Oracle e-Business suite has three important master Concurrent Managers:

Internal Concurrent Manager:

The master manager is called the Internal Concurrent Manager (ICM) because it controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers, and reset the other managers after one them has a failure.

Standard Manager
Another important master Concurrent Manager is called the Standard Manager (SM). The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager. Examples of specific concurrent managers include the Inventory Manager, CRP Inquiry Manager, and the Receivables Tax Manager.

Conflict Resolution Manager
The Conflict Resolution Manager (CRM) functions to check concurrent program definitions for incompatibility rules. However, the ICM can be configured to take over the CRM's job to resolve incompatibilities.

When a concurrent program is incompatible with another program, the two programs cannot access or update the same data simultaneously.When you define a concurrent program, you can list those programs you want it to be incompatible with. You can also list the program as incompatible with itself, which means that two instances of the program cannot run simultaneously. You can also make a program incompatible with all other concurrent programs by defining the program to be run-alone.
There are two types of program incompatibilities, "Global" incompatibilities, and "Domain-specific" incompatibilities.
Incompatibility rules is used to govern your request set so that the request set does not run at the same time as other reports or concurrent programs.When you list a program as incompatible with your request set, the program will not run simultaneously within the same conflict domain as the request set or any of the reports within the set.

Transaction Manager
Apart from these three concurrent manages there is another type of concurrent manager known as the Transaction Manager also exists. The transaction manager is responsible for taking the load off the concurrent request table for pooling the request submitted by the user.The transaction manager takes care of these requests and sends it to standard manager directly.In a RACenvironment the Transaction manager is required to be activated on each node of the RAC environment.

From the front end you could view the status of your concurrent manager by logging with the System Administration responsibility and going to the Concurrent Manager administer screen.

Concurrent Manager Processes
The concurrent managers are like other process which run on the oracle applications executable FNDLIBR. The FNDLIBR executable is located at $FND_TOP/bin.

You could also grep the FNDLIBR executable to check if any concurrent manager process are running
$ ps -ef|grep FNDLIBR
The $FND_TOP/sql/afcmstat.sql script gives you a list of concurrent managers and their respective status.

All Concurrent Mangers log files are located in the $APPLCSF/$APPLLOG location.The following bellow options are available to get the latest log files.

Log in R12:/maildata_test/mcgm/R12/apptier/apps/rdb_mailserver2/logs/appl/conc/log

Different Ways to start Concurrent Manager

1)adcmctl.sh--$COMMON_TOP/admin/scripts/ (11i) ,$INST_TOP/admin/scripts(R12)
2)stratmgr.sh --$FND_TOP/bin

Different ways to stop Concurrent Manager

1)adcmctl.sh
2)by using concsub utility, fro the opertating system we can kill the process by checking with FNDLIBR


In Administer concurrent manager form there are two columns labeled as actual and target 

Target:
Target column lists the number of processes that are running for each manager for this particular shift.
Actual:
Actual column list the number processes that are actually running