Cim configuration for ATG 10.2 learnings

Learnings :

* The schema/User created must be given dba privileges.

* The version of jboss should not be higher than 6.1

* All the installations/configurations related to the ATG should be done inside a particular folder.

* If using CRS , the endeca application creation should be performed from <ATG-HOME>/CommerceReferenceStore/Store/Storefront/deploy , not from endeca Tools and framework reference app. This helps evade sample camera store data

ISSUES :

Issue 1 : Not able to install Oracle in local.

Why :

a) linux packages not present .

b) Init run level is set to 1 . As per message while Oracle installation , it should be set 3,5

Solution :

a) downloaded and added the packages. BUT STILL ORACLE did not work

b) Set Init run level to 3 BUT THIS CRASHES THE MACHINE. NEEDED TO REINSTALL UBUNTU

Workaround : Pointed to Oracle VM on 192.168.10.19

Issue 2 : Publishing Schema Data import not working

Why: ClassPath was not setr properly

Triage : Check the cim logs. Some classNotfound error was there for a class file

Solution:

add the j2ee jars (which have those classes in them – Download if needed) to the classpath as followign :

if DEFINED JBOSS_HOME set CLASSPATH=%CLASSPATH%;%JBOSS_HOME%\lib\jboss-javaee.jar;%JBOSS_HOME%\common\lib\jsp-api.jar;%JBOSS_HOME%\common\lib\servlet-api.jar

Workaround: NA

Isssue 3 : Search box and endeca dependent pages not displaying

Why : Data indexing did not happen properly

Solution : Perform a full deployment through BCC and run baseline index to get the pages dependent on endeca

Workaround : NA

Issue 4 : Images on home page not displaying .

Why : The images depend on Slot and Targeters which are dependent on BCC deployment.

Triage : Check the logs of production server which states HomeTheme component not fould

Solution : Perform a full publishing deployment.

Workaround : you can continue working without targetters.

Issue 5 : BCC full delployment failed with general error

Why : Due to java out of memory

Triage : Check in publishing server log which shows that gc ran into memory out of Heap exception

Solution : Increase the java heap in /common/jboss-5.1.0.GA/bin/run.conf as shown below :

if [ "x$JAVA_OPTS" = "x" ]; then

JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"

# CIM - Modifying JBoss JVM Memory Settings

JAVA_OPTS="$JAVA_OPTS -server -XX:MaxNewSize=256m"; export JAVA_OPTS

fi

Issue 6 : Issue while running publishing server for ATG-CRS application. Unable to query table "das_id_generator"

Why : publishing Datasource was not pointing to right schema.

Triage : asked query on OTN : https://community.oracle.com/thread/3638300

Solution : Update atg-ds file @ <jboss-home>/server/<pub-server-name>/atg-ds.xml and point pub schema to ATGPublishing

How to cook Endeca Partial Update with CAS data?

 Today we will learn to cook Endeca Partial update with CAS data. We will pick the raw material and get partial updated data ready to be served.

Ingredients :
Apart from other ingredients , following are the main constituents to cook endeca partial update
* An update Manipulator
* A Record manipulator
* A record spec

How to cook :

Mark the record spec property
Mark a unique property as record spec

Build a pipeline using update manipulator and record manipulator which will look like this :

 

 

Before you put this pipeline on stove ( i.e run partial update from control folder of app) , you should first do following : Update manipulator will look like this :

 





 

Now add some salt to the solution :

 

This salt is most important. Here you add the record spec with source as Endeca.Id and target as your record spec property

 

 

 



Add pass trhoughs to input CAS adapter :

 

 



Following is the code to be added to forge.context.xml :

 

<beans xmlns="http://www.springframework.org/schema/beans"

    xmlns:lang="http://www.springframework.org/schema/lang"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xmlns:util="http://www.springframework.org/schema/util"

    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd

                        http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang.xsd

                        http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.0.xsd">

                                                                                   

 <!-- load external properties, see ./config/script/fcm.properties -->

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">

    <property name="locations">

              <list>

                     <value>classpath:environment.properties</value>

              </list>

    </property>

  </bean>

 

       <!-- base (abstract) record store adapter to avoid config duplication -->

       <bean id="commonRecordStoreAdapter"

              class="com.endeca.esource.springadapter.ConfigurableDelegateAdapter"

              abstract="true">

              <property name="adapter">

                     <bean class="com.endeca.itl.recordstore.forge.RecordStoreSource"/>

              </property>         

              <property name="config">

                     <map>

                            <entry key="HOST" value="${CAS_HOST}" />

                            <entry key="PORT" value="${CAS_PORT}" />

                            <entry key="CLIENT_ID" value="FORGE" />

                     </map>

              </property>

       </bean>

       <!-- Forge record stores -->

 

       <bean id="jobsBaselineRecordStoreAdapter" parent="commonRecordStoreAdapter" lazy-init="true">

              <property name="config">

                     <map merge="true">

                            <entry key="READ_TYPE" value="BASELINE" />

                            <entry key="INSTANCE_NAME" value="jobJdbcDsCmdLine" />

                            <entry key="CLIENT_ID" value="jobsFORGEId" />

                     </map>

              </property>

       </bean> 

       <bean id="jobsPartialRecordStoreAdapter" parent="commonRecordStoreAdapter" lazy-init="true">

              <property name="config">

                     <map merge="true">

                            <entry key="READ_TYPE" value="DELTA" />

                            <entry key="INSTANCE_NAME" value="jobJdbcDsCmdLine" />

                            <entry key="CLIENT_ID" value="jobsFORGEId" />

                     </map>

              </property>

       </bean>

</beans>

 

 

 

Ready to cook good : Run the partial_update script .

 

Ready to Eat. :)

 

Read Data from JDBC connection into endeca CAS

Reading Data using JDBC connection into CAS:

CAS is a component that comes as a separate installer with Oracle Endeca. It stands for Content Acquisition System. As the name suggests, the CAS is used to acquire data from vast data sources and then fetched to the Endeca forge. One such data sources can is JDBC.

To fetch data into CAS, we need to perform couple of operations:

·         Create Data sources

·         Crawl through the data source

 

Creating a Data Source: The data source creation is a simple task which can be primarily accomplished using CAS console in Endeca workbench

Click on Administrative tool dropdown.

 

Here you will see the various administrative options. Click on Data sources option.

 

Here you will see CAS console. You need to click on add data sources option to add new data source.

 

Select JDBC option to select JDBC data source

 Provide all the details asked to connect to the JDBC data source

You will need to provide SQL query and the Key Column. This key column will correspond to record spec option.

But this UI option might not be as useful when trying to automate the process. So to create a data source using command line, do following:

To create the data sources from command line, we need a xml which follows the dtd as needed. Wait wait wait!! Don’t rush on  google for the DTDs and xml formats. We have a way out:

1)      As you have created a data source already using the UI way shown above , you can use following command to get the xml out of the data source created using workbench UI :

cd /endeca/CAS/3.1.2/bin

-bash-3.2$ ./cas-cmd.sh getCrawl --id <existing-job-name> > <some-location>/temp/jobJdbcCrawl.xml

<existing-job-name> is the same one created using the UI.

We will use this xml to get the format of the crawl 

 

2)      Make a copy of this xml file and make changes to the xml copy as per your needs like name of crawl, connections etc. If connection details and stuff you want to use is to be same as used for the data source in CAS console, need not change anything.

For JDBC connection, you will need to add the corresponding value for the ‘password’ key.

 

3)       Run following command to create the crawl :

-bash-3.2$ ./cas-cmd.sh createCrawls -f /endeca/temp/jobJdbcCrawl.xml

 

Crawl through the data source: Crawling is even simpler. For UI it is as simple  as a click of a button. Click on the start button under acquire data header in CAS console against the data sourec you want to crawl

 

Or command line :

cd /endeca/CAS/3.0.2/bin

cas-cmd startCrawl -id CrawlName [-full] [-h HostName]  [-p PortNumber] [-l true|false]

./cas-cmd.sh startCrawl -id jobRecStore -full -h localhost -p 8500 -l false

Voila!! The data has moved from DB to CAS and ready to be ingested to the Pipeline.

 
----------------------------------------------------------------------------------------------------------------------------------- 

Issue you might run into :

You might run into following issue while saving the Data store:

 

Resolve :
The resolve for this issue is to add the drivers ojdbc6.jar (latest version) at <endeca-installation-folder>/CASS/<version-num>/lib/cas-server-plugins/cas-jdbc-datasource and restart the server

 

 

Introduction to ATG-Endeca integration

Oracle acquired “Art technology Group” in early 2011 as a move to feature in the retail services market. But the search capabilities provided by ATG (viz. ATG search) was not up to the standards and most of the ATG customers had been using Endeca for the search capabilities. So as part of its aggressive market strategy, Oracle bought Endeca technologies in late 2011 so as to make use of the Enterprise search software with ATG and eventually replaced the ATG search with Endeca from version 10.1.1 of Oracle ATG.

There is seamless integration that has been envisioned in ATG for the Endeca. A new module has been developed in ATG named ‘Endeca’. This module when included using the application’s MANIFEST.MF file of the application helps the developer to use the Endeca features like baseline updates , product attributes modification, application preview etc. on the fingertips.

There are two aspects of this integration:

Updating MDEX

Accessing MDEX

Both these operations can be easily performed from the ATG side itself, once the integration implementation is done.

Updating MDEX: The changes performed on the web commerce catalog (sku data) can be indexed and transferred to the MDEX from ATG in two ways:

• Manually through /atg/endeca/index/admin/SimpleIndexingAdmin component. This can be done from the dyn-admin (which is a web based ATG application used to make runtime changes to the attributes of ATG components and more. see the image below) Once we manually start the process , various sub-processes run in parallel as threads to run the baseline index.
  

• Automatically by deploying an application on ATG-BCC. BCC is a tool used by Business Users for various business         tasks and each task (say change in catalog) goes as an application deployment in BCC (see the image below).Once       the deployment is finished, the baseline update is executed automatically. Hence this makes sure that whenever the ATG catalog is updated, the records get indexed and go and sit in MDEX

Accessing MDEX:

I already discussed that an ‘Endeca’ module is also provided within latest ATG installations. This module has various APIs– including assembler APIs (helps to manage dynamic content retrieval) and presentation APIs (helps to manage search and navigation result retrieval) – needed to connect to Endeca application and access the MDEX. In addition to this new ATG droplets like invokeAssember have been introduced OOTB that makes the records retrieval and manipulation relatively easier.

The Oracle is coupling ATG more and more with Endeca, the Endeca still can be used with any java and .NET based framework directly through APIs and with any other framework through web service calls.