Please, don't kill me!

Не убивайте свои системы EPM 11.1.2.0, они сделают это сами :) ):

Бага оказалась небанальной:

Симптомы
Недоступность системы для работы пользователей.
Во время работы пропал доступ к сервису Reporting & Analysis Framework, ответственного за формирование отчетов Financial Reporting и Web Analysis. Cервис Reporting & Analysis Framework в свою очередь связан с модулем Workspace – основной точкой входа во все приложения CIBRIS (Hyperion Planning, Hyperion Financial Consolidation). Риска потери данных в системе не было.

Диагностика
1. Неработоспособность Reporting & Analysis Framework привела к неработоспособности Workspace. Кроме того, была обнаружена неявная ошибка: компонент Reporting & Analysis Framework не мог записывать данные в свой лог-файл (баг вендора), из-за чего также не работал корректно.
2. SmartView не работал, т.к. использует протокол http через workspace.
3. Анализ лога WebLogic Server (WLS) указал на множество накопленных «мертвых», неиспользуемых объектов системы, в частности Reporting & Analysis Framework, из-за чего запросы выполнялись дольше установленного таймаута, что не позволяло возвращать результаты по запросу в установленное время.
4. Антивирус на сервере не настроен в соответствии с рекомендациями вендора Recommendations Regarding Antivirus Settings for Enterprise Performance Management (EPM) System Products [ID 1315173.1].

Лечение
Никогда не оставляйте включенным утюг опцию Change Ownership Event в этой версии здесь:

Workspace - Navigate - Administer - Reporting & Analysis - Event Tracking

и беспощадно убейте этот лог-файл:

Middleware_Home\user_projects\epmsystem1\diagnostics\logs\ReportingAnalysis\rafservletsUTBackup.log

Все подробности в документе Raframework Web Application Crashes Due to Usage Tracking. [ID 1334441.1].

Оптимизация БД EPMA для EPM 11.1.x

Со временем сильно увеличивается БД EPM Architect, что негативно сказывается на скорости работы этого компонента. А в некоторых случаях, веб-сервер не успевает получить ответ в установленный таймаут, и возникает проблема, когда приложение (наример, Planning) может остаться в подвешенном состоянии.

Ниже представлено решение этой проблемы для EPM 11.1.3+

1. Остановить службу EPMA Process Manager
2. Сделать полный бэкап БД EPM Architect
3. Запустить службу EPMA Process Manager
4. Запустить утилиту database_cleanup_utility.exe, которая находится
   либо здесь для 11.1.1.x: \Hyperion\products\Foundation\BPMA\AppServer\DimensionServer\ServerEngine\bin\
   либо здесь для 11.1.2.x: \Oracle\Middleware\EPMSystem11R1\products\Foundation\BPMA\AppServer\DimensionServer\ServerEngine\bin
5. Использовать администраторский логин/пароль EPM
6. Запустить очистку для записей старше 45-60 дней
7. Либо почистить БД EPM Architect скриптом (пример для Oracle DB 10g+):
   

   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 1" FROM DUAL;
   
   CREATE TABLE tmp_DS_MaxTransactions NOLOGGING AS
   SELECT a.i_application_id, MAX(i_transaction_id) AS i_transaction_id
     FROM DS_Transaction_History a
     JOIN DS_Application b
     ON a.i_library_id = b.i_library_id
     AND a.i_application_id = b.i_application_id
     JOIN DS_Library c
     ON a.i_library_id = c.i_library_id
     WHERE c.b_master = 1
     AND a.i_application_id IS NOT NULL
     AND (b.i_first_deploy_transaction_id IS NOT NULL OR b.b_master = 1)
     AND i_transaction_id >= COALESCE(b.i_last_deployed_transaction_id, b.i_first_deploy_transaction_id, 0)
     AND e_transaction_action = 'OrderChildren'
     GROUP BY a.i_application_id;
   COMMIT;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 2" FROM DUAL;
   
   -- Keep newest OrderChildren row for each application.  Logging resumes for this table
   -- after it is created and populated by this statement.
   CREATE TABLE tmp_DS_Transaction_History NOLOGGING AS
     SELECT a.* FROM DS_Transaction_History a
       JOIN tmp_DS_MaxTransactions b
       ON a.i_transaction_id = b.i_transaction_id;
   COMMIT;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 3" FROM DUAL;
   
   DROP TABLE tmp_DS_MaxTransactions;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 4" FROM DUAL;
   
   ALTER TABLE tmp_DS_Transaction_History NOLOGGING;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 5" FROM DUAL;
   
   -- Add back all non-OrderChildren rows.
   INSERT INTO tmp_DS_Transaction_History
     SELECT * FROM DS_Transaction_History
     WHERE e_transaction_action != 'OrderChildren';
   COMMIT;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 6" FROM DUAL;
   
   DROP TABLE DS_Transaction_History;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 7" FROM DUAL;
   
   RENAME tmp_DS_Transaction_History TO DS_Transaction_History;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 8" FROM DUAL;
   
   ALTER TABLE DS_Transaction_History
     ADD CONSTRAINT PK_TranHist PRIMARY KEY (i_transaction_id, i_detail_id);
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 9" FROM DUAL;
   
   CREATE INDEX IdxTranHist01 ON DS_Transaction_History(i_library_id, i_dimension_id, i_member_id);
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 10" FROM DUAL;
   
   CREATE INDEX IdxTranHist02 ON DS_Transaction_History(i_library_id, d_created);
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 11" FROM DUAL;
   
   ALTER TABLE DS_Transaction_History LOGGING;
   
   SELECT TO_CHAR(SYSDATE,'Dy DD-Mon-YYYY HH24:MI:SS') AS "Current Time 12" FROM DUAL;

EPM 11.1.2.x: Tips and Tricks from EPM System Infrastructure Development: Issues 73 and Higher

Здесь (pdf)

Utilities for Troubleshooting Oracle Hyperion Enterprise Performance Management

(1) A new troubleshooting tool introduced in Oracle EPM 11.1.2.1 is \Oracle\Middleware\user_projects\epmsystem1\bin\ziplogs.bat|.sh file. The output of this is generated in \Oracle\Middleware\user_projects\epmsystem1\diagnostics\ziplogs\EPM_logs__11.1.2.1.0.zip

(2) Oracle Configuration Manager (OCM) is an advanced tool which collects and evaluates settings from configuration files (but not Hyperion Registry) and stores them in a central database administered by Oracle. It was first introduced into EPM in version 11.1.2.0

(3) Remote Desktop Assistant (RDA) collects and zips up a large amount of information concerning the software and hardware environments of Oracle products. If Perl 5.1 or later is installed on the computer and accessible from the command line (test: perl -version), then run: \Oracle\Middleware\ohs\rda\rda.cmd from the command line to access the RDA tool. Alternately, precompiled RDA executables are available for several platforms.

(4) Configuration settings have been moved from configuration files to a database repository associated with Shared Services. These may be extracted to an HTML formatted file by running \Oracle\Middleware\user_projects\epmsystem1\bin\epmsys_registry.bat|.sh without parameters from the command line. The output is generated to file \Oracle\Middleware\user_projects\epmsystem1\diagnostics\reports\registry.html The username/login/jdbc URL for the Hyperion/HSS registry are kept in \Oracle\Middleware\user_projects\epmsystem1\config\foundation\11.1.2.0\reg.properties file.

(5) A much enhanced validation tool is available in \Oracle\Middleware\user_projects\epmsystem1\bin\validate.bat. In EPM 11.1.2.1 the output files are of the form \Oracle\Middleware\user_projects\epmsystem1\diagnostics\reports\instance_report_20YYMMDD_######.html (where YY is the year, MM is the month, DD is the day, and ###### is a unique integer). It also runs the tool under (1) and gives the output of that tool.

(6) Individual log files may be found in subdirectories under (those 'starred' are not collected by ziplogs):
\Oracle\Middleware\user_projects\domains\EPMSystem\servers\\logs *
\Oracle\Middleware\logs *
\Oracle\Middleware\wlserver_10.3 *
\Oracle\Middleware\user_projects\epmsystem1\tmp *
\Oracle\Middleware\ohs\cfgtoollogs\opatch *
\Oracle\Middleware\EPMSystem11R1\diagnostics\logs
\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\
\Program Files\Oracle\Inventory\logs
C:\Users\\Oracle\oradiag_\diag\
C:\Users\\.oracle\logs\

(7) Some debug tools (*) can be activated if under http://:19000/workspace/index.jsp > Navigate > Administration > Workspace Server Settings > Client Debug Enabled: is set to Yes (then click 'OK' button, close all browser windows, and restart browser).
* http://hostname:19000/workspace/debug/configInfo.jsp
* http://hostname:19000/workspace/debug/userInfo.jsp
* http://hostname:19000/workspace/debug/userGroupQuery.jsp


Other tools are available at these URLs (largely derived from validate.bat output). A non-Error response indicates that the web service of each particular application is active.:
http://hostname:19000/workspace/status
http://hostname:19000/workspace/BPMContext
http://hostname:8600/mappingtool/faces/info.jspx
http://hostname:10080/easconsole/console.html
http://hostname:16000/WebAnalysis
http://hostname:6373/oracle-epm-fm-webservices/ApplicationService
http://hostname:8300/HyperionPlanning/
http://hostname:13080/aps/APS
http://hostname:8200/hr/status.jsp
http://hostname:19091/awb/conf/AwbConfig.xml
http://hostname:8500/calcmgr/index.htm

Internet Explorer 7 (IE7) and Internet Explorer 8 (IE8) Recommended Settings for Oracle Hyperion Products

1. Configure browser to check for new version every time user visits a page

•Open Internet Explorer
•Go to Tools > Internet options > General

•In 'Browsing history' section select 'Every time I visit the webpage'

•Click Ok, then Apply.
2. Disable default pop-up blocking

•Open Internet Explorer
•Go to Tools > Internet options > Privacy tab

•Uncheck 'Turn on Pop-up Blocker'
3. Add workspace URL to trusted sites

•Open Internet Explorer
•Go to Tools > Internet options > Security tab

•Select Trusted sites from Select a zone to view or change security settings.

•Type your workspace URL in form http://workspaceserver:portnumber in 'Add this website to the zone'
•Uncheck 'Require server verification (https:) for all sites in this zone'
•Click Add, then Close.
•Click Ok and Apply.
4. Minimize security setting for trusted sites

•Open Internet Explorer
•Go to Tools > Internet options > Security tab
•Select Trusted Sites from select a zone to view or change security settings

•Select custom level

•From Reset to list select Low
•Click Ok
•Click Apply, then Ok
5. Customize security settings

•Open Internet Explorer
•Go to Tools > Internet options > Security Tab > Custom Level
•In 'Miscellaneous' section enable the option "Allow script-initiated windows without size or position constraints"
•In 'ActiveX controls and plug-ins' section enable “Run ActiveX controls and plug-ins” and “Script ActiveX controls marked safe for scripting.”
6. Enable option "Always allow session cookies".

•Open Internet Explorer
•Go to Tools > Internet options > Privacy Tab > Advanced > Check the "Override automatic cookie handling" and accept the first and third party cookies and check the "Always allow session cookies" option.
7. Disable option "Enable Native XMLHTTP".

•Open Internet Explorer
•Go to Tools > Internet options > Advanced Tab > Uncheck the option "Enable Native XMLHTTP".