Starting and stopping endeca components like PlatformServices tools and framework and CAS in linux

Following are the commands that can be used to stop and start Tools and framework , Platform Services and CAS in endeca in Linux. The windows will have corresponding bat files OR can be started/stopped from Windows services

Start ToolsAndFramework
"/ToolsAndFrameworks//server/bin/startup.sh"

Stop ToolsAndFramework
"/ToolsAndFrameworks//server/bin/shutdown.sh"

Start Platform Services
"//PlatformServices/latest/tools/server/bin/startup.sh"

Stop Platform Services
"//PlatformServices/latest/tools/server/bin/shutdown.sh"

Start CAS
"/CAS//bin/cas-service.sh"

Stop CAS
"/CAS//bin/cas-service-shutdown.sh"

ATG-Endeca Integration - NullPointerException: Property value cannot be null (dimval.display_name)

09:24:43,808 ERROR [stderr] (index-/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin) Exception in thread "index-/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin" java.lang.NullPointerException: Property value cannot be null (dimval.display_name)

This can happen for either one of the following 2 reasons :

1) This error is due to a bug in Oracle commerce. There is a patch available with oracle commerce. Consult them and get the patch

2) This apparently indicates that one or more of the item-descriptor tags in customCatalog.xml (in the ATG project) are missing display-name attributes. (Note that some item-descriptor tags – particularly, the OOTB ones – inherit a display-name via the xml-combine="append" attribute value.)

Another symptom of this: In Dyn Admin, RepositoryTypeDimensionExporter component, under "show XML", one of the dimval.display_name entries in the XML shown has an empty PVAL.
(ref : http://blog.jonschneider.com/2013/05/)

Error: Could not find or load main class com.endeca.soleng.eac.toolkit.Controller

This generally happens due to not having jar files in /config/lib/java location of the EAC app. Just make sure all jar file is present.

If jar files are present , make sure the jars are latest and the class is not missing . Compare it with reference application

Convert file format to unix in a vim

use following command to update the fileformat of a file to unix based file format :


:set fileformat=unix


eg -

[atg@endeca control]$ vi initialize_services.sh

:set fileformat=unix

Problem :

Creating crawl TSL1en_en_last_mile_crawl
Error creating crawl TSL1en-last-mile-crawl:
Unable to login to config repository site for user "admin", status code: 401, Response:
com.endeca.itl.cas.cmd.CasCmdException: Failed to create 1 or more crawls
Failure to create crawl.


Solution :
The default username/password set for last-mile-crawl is admin/admin

This configuration is at following location :  /apps/config/cas/last-mile-crawl.xml

edit this file and set the configRepository username password to what you used while installing Tools and framework (one you use to login to workbench)

< outputConfig>
                       
                                com.endeca.cas.output.Mdex
                       
                       
                               
                                        inputDirectory
                                        /data/apps/endeca/apps/Myapp/config/mdex
                               
                               
                                        outputDirectory
                                        /data/apps/endeca/apps/Myapp/data/cas_output
                               
                               
                                        dimensionValueIdManagerInstanceName
                                        Myapp-dimension-value-id-manager
                               
                               
                                        configRepositoryHost
                                        endeca.vm.dsw.com
                               
                               
                                        configRepositoryPort
                                        8006
                               
                               
                                        configRepositoryUserName
                                        admin
                               
                               
                                        configRepositoryPassword
                                        admin
                               
                               
                                        configRepositorySite
                                        Myapp
                               
                       
                </ outputConfig>

Create custom names for the Endeca recordstore

The names of the recordstores for an endeca application can be defined in the initialize_services.bat / initialize_services.sh file at location - /control folder

edit this file and update the following properties to whatever name you like

DATA_RS_NAME=
DIMVALS_RS_NAME=
LAST_MILE_CRAWL_NAME=
DVAL_ID_MGR_NAME=

P.S. - I had used discover-data-cas reference app for deployment. Hence I see (may be) different record stores

On a different note, the last_mile_crawl_name mentioned here should match the crawl name mentioned in

/Myapp/config/cas/last-mile-crawl.xml


                                        dimensionValueIdManagerInstanceName
                                        Myapp_dimension_value_id_manager

                             

Deploy an Endeca CAS based application - version 11.0

[atg@endeca bin]$ ./deploy.sh --app /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml

 ------------------------------------------------------------------------------
   Found a supported version of Platform Services installed in directory
   /data/apps/endeca/PlatformServices/11.0.0. If the location is incorrect, type
   'Q' to quit and adjust your ENDECA_ROOT environment variable. Press enter to
   continue with these settings.

   Continue?


 ------------------------------------------------------------------------------
     Deployment Template installation script.

     This script creates the directory structure for your
     deployment and installs configuration files and scripts
     into the directory structure.

06/22/2017 15:14:11 [deploy.pl] INFO:  Starting deployment template
   installation.

 ------------------------------------------------------------------------------
   The following app modules were specified on the command line argument:


   /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml


   Modules may be deployed standalone or on top of a base deployment type. Do
   you want to install a base deployment with the specified module(s)?

   Install base deployment? [Y/N]:
n
06/22/2017 15:14:15 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml.

 ------------------------------------------------------------------------------
   Enter a short name for your application.

   Note: The name must conform to this regular expression: ^[a-zA-Z0-9]+$
    [default: Discover]

   Choosing a different application name may require additional configuration on
   your web application.

   Application name:
TSLen

 ------------------------------------------------------------------------------
   Specify the path into which the application will be deployed. The specified
   directory must exist and cannot contain spaces.

   For example, to deploy into /localdisk/apps/TSLen, specify the path as
   /localdisk/apps.

   Deployment directory:
/data/apps/endeca/apps

 ------------------------------------------------------------------------------
   Specify the port on which the Endeca Application Controller is running. This
   is configured in the server.xml file in the workspace of the Endeca software
   install and should be the same for all applications deployed in this
   environment. Ports must be in the range 1024-65535 [default: 8888].

   EAC port:

06/22/2017 15:14:38 [deploy.pl] INFO:  Deploying application into
   /data/apps/endeca/apps/TSLen

 ------------------------------------------------------------------------------
   Enter the port where workbench is running. [Default: 8006]


 ------------------------------------------------------------------------------
   Enter the port to use for Dgraph1. [Default: 15000]


 ------------------------------------------------------------------------------
   Enter the port to use for AuthoringDgraph. [Default: 15002]


 ------------------------------------------------------------------------------
   Enter the port to use for LogServer. [Default: 15010]


 ------------------------------------------------------------------------------
   Enter the CAS install directory, including the version number (e.g.
   C:\Endeca\CAS\3.1.2)
/data/apps/endeca/CAS/11.0.0

 ------------------------------------------------------------------------------
   Enter the port where CAS is running. [Default: 8500]


 ------------------------------------------------------------------------------
   Specify the path for the Oracle Wallet jps-config.xml file. [Default:
   /data/apps/endeca/ToolsAndFrameworks/11.0.0/deployment_template/lib/../../server/workspace/credential_store/jps-config.xml]


 ------------------------------------------------------------------------------
   Which directory will application configuration archives be exported to.
   [Default:
   /data/apps/endeca/ToolsAndFrameworks/11.0.0/deployment_template/lib/../../server/workspace/state/repository]


 ------------------------------------------------------------------------------
   Please enter the language code to use (or hit 'enter' to use the default).
   [Default: en]
en
06/22/2017 15:15:05 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml.
06/22/2017 15:15:05 [deploy.pl] INFO:  Processing install with id
   'DefaultTemplateManagerApp'
06/22/2017 15:15:05 [deploy.pl] WARN:  Directory /data/apps/endeca/apps/TSLen
   already exists.
06/22/2017 15:15:05 [FileHelper] INFO:  Renamed /data/apps/endeca/apps/TSLen to
   /data/apps/endeca/apps/TSLen.2017_06_22-15_15_05.bak.
06/22/2017 15:15:05 [deploy.pl] INFO:  Archived existing directory
   /data/apps/endeca/apps/TSLen
06/22/2017 15:15:05 [deploy.pl] INFO:  Application successfully deployed.

Error creating crawl last-mile-crawl: Unable to login to config repository site for user "admin", status code: 401

After deploying cas-based-application , when trying to run initialize services , getting following error -

[06.21.17 20:48:56] INFO: Importing mappings into dimension value id manager 'TSLen-dimension-value-id-manager'.
Finished updating EAC.
Creating crawl TSLen-last-mile-crawl
Error creating crawl TSLen-last-mile-crawl:
Unable to login to config repository site for user "admin", status code: 401, Response:
com.endeca.itl.cas.cmd.CasCmdException: Failed to create 1 or more crawls
Failure to create crawl.

Solution -


The default username/password set for last-mile-crawl is admin/admin

This configuration is at following location :  /apps/config/cas/last-mile-crawl.xml

edit this file and set the configRepository username password to what you used while installing Tools and framework (one you use to login to workbench)




                       
                                com.endeca.cas.output.Mdex
                       

                       
                               
                                        inputDirectory
                                        /data/apps/endeca/apps/Myapp/config/mdex
                               
                               
                                        outputDirectory
                                        /data/apps/endeca/apps/Myapp/data/cas_output
                               
                               
                                        dimensionValueIdManagerInstanceName
                                        Myapp-dimension-value-id-manager
                               
                               
                                        configRepositoryHost
                                        endeca.vm.dsw.com
                               
                               
                                        configRepositoryPort
                                        8006
                               
                               
                                        configRepositoryUserName
                                        guest
                               
                               
                                        configRepositoryPassword
                                        guest123
                               
                               
                                        configRepositorySite
                                        Myapp
                               
                       
               

FATAL: Multiple root installs found. Only one installer config file may have the update attribute unspecified or set to false.

Problem


[atg@endeca bin]$ ./deploy.sh --app /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml

 ------------------------------------------------------------------------------
   Found a supported version of Platform Services installed in directory
   /data/apps/endeca/PlatformServices/11.0.0. If the location is incorrect, type
   'Q' to quit and adjust your ENDECA_ROOT environment variable. Press enter to
   continue with these settings.

   Continue?


 ------------------------------------------------------------------------------
     Deployment Template installation script.

     This script creates the directory structure for your
     deployment and installs configuration files and scripts
     into the directory structure.

06/21/2017 20:38:59 [deploy.pl] INFO:  Starting deployment template
   installation.

 ------------------------------------------------------------------------------
   The following app modules were specified on the command line argument:


   /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml


   Modules may be deployed standalone or on top of a base deployment type. Do
   you want to install a base deployment with the specified module(s)?

   Install base deployment? [Y/N]:
Y
06/21/2017 20:39:01 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/apps/endeca/ToolsAndFrameworks/latest/reference/discover-data-cas/deploy.xml.
06/21/2017 20:39:01 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/apps/endeca/ToolsAndFrameworks/11.0.0/deployment_template/lib/../app-templates/base_descriptor.xml.

 ------------------------------------------------------------------------------
   Enter a short name for your application.

   Note: The name must conform to this regular expression: ^[a-zA-Z0-9]+$
    [default: Discover]

   Choosing a different application name may require additional configuration on
   your web application.

   Application name:
TSLen

 ------------------------------------------------------------------------------
   Specify the path into which the application will be deployed. The specified
   directory must exist and cannot contain spaces.

   For example, to deploy into /localdisk/apps/TSLen, specify the path as
   /localdisk/apps.

   Deployment directory:
/data/apps/endeca/apps

 ------------------------------------------------------------------------------
   Specify the port on which the Endeca Application Controller is running. This
   is configured in the server.xml file in the workspace of the Endeca software
   install and should be the same for all applications deployed in this
   environment. Ports must be in the range 1024-65535 [default: 8888].

   EAC port:

06/21/2017 20:39:17 [deploy.pl] INFO:  Deploying application into
   /data/apps/endeca/apps/TSLen
06/21/2017 20:39:17 [deploy.pl] FATAL: Multiple root installs found. Only one
   installer config file may have the update attribute unspecified or set to
   false.

Solution ;

In the step asking :

 Install base deployment? [Y/N]:

Instead of entering Y , enter N

/bin/sh^M: bad interpreter: No such file or directory

[atg@endeca control]$ ./initialize_services.sh
-bash: ./initialize_services.sh: /bin/sh^M: bad interpreter: No such file or directory

The ^M is a carriage return character. Linux uses the line feed character to mark the end of a line, similar to windows that uses CR LF. So , currently we got windows line endings, which linux is not able to understand is confusing Linux.

To fix this, we will need to remove that confusing characters using command shown below :

[atg@endeca control]$ sed -i -e 's/\r$//' initialize_services.sh.

Now run initialize services :

[atg@endeca control]$ ./initialize_services.sh

Tuning Endeca workbench JVM settings

We can increase the heap size for workbench to make it faster and more efficient by following below steps

1) Go to $ENDECA_TOOLS_ROOT/server/bin
2) Edit workbench .sh using following command
     vi workbench.sh
3) Check for following lines of commands :

# Prepend WB_CATALINA_OPTS to standard Workbench CATALINA_OPTS
CATALINA_OPTS="${WB_CATALINA_OPTS} -Xms256m"
CATALINA_OPTS="${CATALINA_OPTS} -Xmx1024m"
CATALINA_OPTS="${CATALINA_OPTS} -XX:MaxPermSize=512m"
CATALINA_OPTS="${CATALINA_OPTS} -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true"
CATALINA_OPTS="${CATALINA_OPTS} -Dcom.endeca.manager.endecaroot=${ENDECA_TOOLS_CONF}"
export CATALINA_OPTS

Replace CATALINA_OPTS="${CATALINA_OPTS} -Xmx1024m" with CATALINA_OPTS="${CATALINA_OPTS} -Xmx2048m"

Find RSS resident set size for MDEX

Resident set size (RSS) for MDEX is the amount of physical memory currently allocated and used by the MDEX Engine process

Check for MDEX and platformservice processes :

ps -ef | grep MDEX

mayank      15632 48215  0 13:02 ?        00:00:40 /tmp/endeca/MDEX/11.0.0/bin/dgraph --threads 14  --cmem 12288 --spl --spl_sthresh 150 --dym --dym_hthresh 5 --dym_nsug 1 --stat-abins --lang en --port 15000 --pidfile tmp/endeca/apps/EndecaApp/dgraph.pid --log tmp/endeca/apps/EndecaApp/logs/dgraphs/DgraphA1/DgraphB1.reqlog --out tmp/endeca/apps/EndecaApp/logs/dgraphs/DgraphA1/DgraphB1.log --spellpath tmp/endeca/apps/EndecaApp/data/dgraphs/DgraphA1/dgraph_input --updatedir tmp/endeca/apps/EndecaApp/data/dgraphs/DgraphA1/dgraph_input/updates --updatelog tmp/endeca/apps/EndecaApp/logs/dgraphs/DgraphA1/DgraphB1.updatelog tmp/endeca/apps/EndecaApp/data/dgraphs/DgraphA1/dgraph_input/EndecaApp
mbatra 19505 17549  0 14:11 pts/0    00:00:00 grep MDEX


This gives us two process ids viz - 15632


Next run following command :

ps -eo pid,ucomm,rss | grep 15632

15632 dgraph          1245698

This tells Resident set size for dgraph is 1.24 GB

Now check what is total infra memory you have :

free -m

                  total         used       free     shared    buffers     cached
Mem:        64586       5659     61256          1        629       2864
-/+ buffers/cache:       4521     62456
Swap:         4582          0         4582

This tells there is total of 61.2 gigs of memory is free and 5.6 gigs is being used (out of which 1.24 is being used by the dgraph)

Now you can decide if you want to add another dgraph to same server or need another server.

Unable to connect on workbench publishing port localhost/127.0.0.1:8007. Please check your assembler.properties file matches your webstudio.properties file for this port setting.

 SEVERE [com.endeca.infront.publish.AuthoringContentClient] (http-/127.0.0.1:8080-4) Exception while requesting a node from the server.: java.io.IOException: Unable to connect on workbench publishing port localhost/127.0.0.1:8007. Please check your assembler.properties file matches your webstudio.properties file for this port setting.

Check the Tools and framework service is running :

Windows : Go to windows "services" . If tools and frameworks is not started , start it

Linux : Run following  : ps -ef | grep java . Check for java process for ToolsAndFramework. If it is not running , go to /11.0.0/server/bin -> Execute ./startup.sh