Примерно раз в месяц АТС перезагружается (S8300/G450)

[kitsa]

Добрый день. Примерно раз в месяц АТС перезагружается

Initialized 4 (RELOAD) no Active 10/11 3:55
Initialized 4 (RELOAD) no Active 11/10 12:50
Initialized 4 (RELOAD) no Active 11/30 21:29
Initialized 4 (RELOAD) no Active 12/04 15:06
Initialized 4 (RELOAD) no Active 01/09 8:06

Изначально грешил на питание, поставили второй БП. Теперь вроде как есть резерв. Шлюз заземлен.

Снял следующие скрины

1. list configuration software-versions

SOFTWARE VERSIONS

SOFTWARE VERSION
Memory Resident: R015x.02.0.947.3
Disk Resident: R015x.02.0.947.3

TRANSLATION DATE
Memory Resident: 10:00 pm MON MAR 2, 2015
Disk Resident: 10:00 pm MON MAR 2, 2015
Disk Second Copy: good

2. list configuration all

Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa

001V1 ICC MM S8300C HW02 FW004
001V2 DS1 MM MM710AP HW05 FW021 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 u
001V3 DS1 MM MM710BP HW11 FW047 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
25 26 27 28 29 30 u u
001V4 DCP MM MM712AP HW07 FW015 01 02 03 04 05 06 07 08
001V5 ANA MM MM716AP HW06 FW098 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 u 23 24
001V6 ANA MM MM716AP HW06 FW098 01 02 03 04 05 06 07 08
09 10 11 u u u 15 16
17 u 19 20 21 22 23 24
001V7 ANA MM MM716AP HW06 FW098 01 02 03 04 05 06 07 08
09 10 11 12 u 14 15 16
17 u 19 20 21 u u u
001V8 DS1 MM MM710BP HW11 FW047 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
25 26 27 28 29 30 u u
57 58 59 60 61 62 63 64
002V2 ANA MM MM716AP HW06 FW098 01 02 03 04 05 06 07 08
u u u u u u u u
u u u u u u u u

3. G450-001(super)# show module
SLOT TYPE CODE SUFFIX HW VINTAGE FW VINTAGE
---- -------- ---------- ------ ---------- -----------
v1 S8300 ICC C 2 4
v2 E1T1 MM710 A 5 21
v3 E1T1 MM710 B 11 47
v4 DCP MM712 A 7 9
v5 Analog MM716 A 6 94
v6 Analog MM716 A 6 94
v7 Analog MM716 A 6 94
v8 E1T1 MM710 B 11 47
v10 Mainboard G450 A 1 29.22.3(B)

4. G450-001(super)# show faults

CURRENTLY ACTIVE FAULTS
--------------------------------------------------------------------------
No Fault Messages

Current Alarm Indications, ALM LED is off
--------------------------------------------------------------------------
None

Done!
G450-001(super)#



5. G450-001(super)# show sync t
SYNCHRONIZATION CONTROL: --- Local ---

SOURCE MM or VoIP STATUS FAILURE
--------- ------------------- ----------------------- ---------------
Primary v2 Active None
Secondary Not Configured
Local v0 Standby None

Active Source: v2 Sync Source Switching: Enabled

Done!
G450-001(super)#


6. G450-001(super)# show temp
Ambient
-------
Temperature : 28C
High Warning: 42C
Low Warning : -5C
G450-001(super)#

G450-001(super)# show mg list_config
SLOT TYPE CODE SUFFIX HW VINTAGE FW VINTAGE
---- -------- ---------- ------ ---------- -----------
v1 S8300 ICC C 2 4
v2 E1T1 MM710 A 5 21
v3 E1T1 MM710 B 11 47
v4 DCP MM712 A 7 9
v5 Analog MM716 A 6 94
v6 Analog MM716 A 6 94
v7 Analog MM716 A 6 94
v8 E1T1 MM710 B 11 47
v10 Mainboard G450 A 1 29.22.3(B)
G450-001(super)#

7. G450-001(super)# show volta
PSU#1 PSU#2
-------- -------- --------
Voltages 5v, 48v 5v, 48v
PRESENT YES YES
AC IN YES YES
STATUS OK OK



И самое главное где хоть что то видно


8. G450-001(super)# show logging file content

01/09/2016,08:05:38:170ms, VOICE-Error: ICC reset completed, ID=7000086864

01/09/2016,08:03:27:610ms, VOICE-Error: ICC carries out HARD RESET, ID=7000086864

01/09/2016,07:59:59:195ms, VOICE-Error: ICC carries out SOFT RESET, ID=7000086864

01/09/2016,07:59:19:000ms, VOICE-Error: keepAliveFailed() - Close H248 socket, ID=7000086864

12/04/2015,15:06:01:685ms, VOICE-Error: ICC reset completed, ID=7000086864

12/04/2015,15:03:57:735ms, VOICE-Error: ICC carries out HARD RESET, ID=7000086864

12/04/2015,14:59:55:225ms, VOICE-Error: ICC carries out SOFT RESET, ID=7000086864

12/04/2015,14:59:17:295ms, VOICE-Error: keepAliveFailed() - Close H248 socket, ID=7000086864


И еще данные из System log - Watchdog logs (если честно я отсюда вообще ничего не понял)



20160109:080539120:437:WATCHD-P:HIGH:
20160109:080539170:438:WATCHD-P:HIGH:CWATCHDApp::getConfigData: Watchdog is initializing ...
20160109:080539182:439:WATCHD-P:HIGH:WARNING: Config parm NumOccupancySnapshots= not in file or had no value. Using default(4)
20160109:080539193:440:WATCHD-P:HIGH:CWATCHDApp::makeThreadRecovery: Using CommunicationManager ThreadRecovery type
20160109:080539227:441:WATCHD-P:HIGH:reqsvr thread running, tid 3461
20160109:080540230:442:WATCHD-P:HIGH:Watchdog started. Config File = /etc/opt/ecs/watchd.conf
20160109:080540230:443:WATCHD-P:HIGH:ENTERING MAIN LOOP ...
20160109:080540230:444:WATCHD-P:HIGH:Heartbeats started for application Watchdog (3430)
20160109:080538540WATCHD:HIGH:Watchd version: 3.2.6-466
20160109:080538540WATCHD:HIGH:Build date: Sat Apr 4 09:56:27 IST 2009
20160109:080538540WATCHD:HIGH:Build machine: puscmbldlin06
20160109:080539015WATCHD:HIGH:Running parent (1) and child watchd (3179)
20160109:080539017WATCHD:HIGH:AdoptMgr::Open: open failed for /proc/adopt, errno 2
20160109:080539104WATCHD:HIGH:
20160109:080539120WATCHD:HIGH:CWATCHDApp::getConfigData: Watchdog is initializing ...
20160109:080539161WATCHD:HIGH:WARNING: Config parm NumOccupancySnapshots= not in file or had no value. Using default(4)
20160109:080539170WATCHD:HIGH:CWATCHDApp::getConfigData: Set SvcMon Search Interval to 4 seconds
20160109:080539171WATCHD:HIGH:CWATCHDApp::getConfigData: Set SvcMon Poll Interval to 120 seconds
20160109:080539181WATCHD:HIGH:CWATCHDApp::getConfigData: Set SvcMon SetPrio Interval to 180 seconds
20160109:080539183WATCHD:HIGH:CWATCHDApp::makeThreadRecovery: Using CommunicationManager ThreadRecovery type
20160109:080539240WATCHD:HIGH:tSanityThr STARTING
20160109:080539242WATCHD:HIGH:Gateway sanity enabled
20160109:080539242WATCHD:HIGH:INFO: open of gateway watchdog device
20160109:080539242WATCHD:HIGH:S8300C using MgBeatDevice
20160109:080539243WATCHD:HIGH:INFO: WatchdDevice::Strobe() passed initial Strobe, strobe rate is 2500 mSec
20160109:080539243WATCHD:HIGH:CWATCHDApp::makeThreadRecovery: Using CommunicationManager SystemRecovery type
20160109:080539255WATCHD:HIGH:reqsvr thread running, tid 3464
20160109:080540258WATCHD:HIGH:ProcessOccupancy::ProcessOccupancy: Using high-resolution CPU timing
20160109:080540258WATCHD:HIGH:Watchdog started. Config File = /etc/opt/ecs/watchd.conf
20160109:080540260WATCHD:HIGH:ERROR: stat() failed for "/mom/bin/start_mom". errno = 2 (No such file or directory).
20160109:080540260WATCHD:HIGH:ERROR: Executable "/mom/bin/start_mom" does not exist or is not executable for Application Audix
20160109:080540260WATCHD:HIGH:Application Audix not started, parm file errors
20160109:080540261WATCHD:HIGH:Application ImLogger not started due to OffBoot, OffBoot is set to 1
20160109:080540261WATCHD:HIGH:Application SipServer not started due to OffBoot, OffBoot is set to 1
20160109:080540261WATCHD:HIGH:ENTERING MAIN LOOP ...
20160109:080540262WATCHD:HIGH:ProcStateTrans::_adjustPriority: set priority for dbgserv, prio 62, exe /opt/ecs/sys/dbgserv, pid 2735
20160109:080540262WATCHD:HIGH:ProcStateTrans::_logRunning: Service dbgserv is running with exe /opt/ecs/sys/dbgserv pid 2735
20160109:080540264WATCHD:HIGH:ProcStateTrans::_logRunning: Service atd is running with exe /usr/sbin/atd pid 2864
20160109:080540265WATCHD:HIGH:ProcStateTrans::_logRunning: Service crond is running with exe /usr/sbin/crond pid 2855
20160109:080540266WATCHD:HIGH:postgresql Service not started due to OffBoot being set
20160109:080540267WATCHD:HIGH:hp-sshd Service not started due to OffBoot being set
20160109:080540268WATCHD:HIGH:ProcStateTrans::_logRunning: Service httpd is running with exe /usr/sbin/httpd pid 2814
20160109:080540269WATCHD:HIGH:ProcStateTrans::_logRunning: Service ntpd is running with exe /usr/sbin/ntpd pid 2725
20160109:080540270WATCHD:HIGH:ProcStateTrans::_logRunning: Service prune is running with exe /opt/ecs/sys/prune pid 2845
20160109:080540272WATCHD:HIGH:ProcStateTrans::_lockInMemory: locked syslog into memory, pid 3132, exe /sbin/syslogd, sig 1
20160109:080540272WATCHD:HIGH:ProcStateTrans::_lockInMemory: locked syslog into memory, miscellaneous pid 3142, exe /sbin/klogd, sig 10
20160109:080540273WATCHD:HIGH:ProcStateTrans::_lockInMemory: locked syslog into memory, miscellaneous pid 3137, exe /sbin/nbsyslogd, sig 1
20160109:080540273WATCHD:HIGH:ProcStateTrans::_logRunning: Service syslog is running with exe /sbin/syslogd pid 3132, exe /sbin/klogd pid 3142, exe /sbin/nbsyslogd pid 3137
20160109:080540274WATCHD:HIGH:ProcStateTrans::_adjustPriority: set priority for tomcat4, prio 0, exe /usr/sbin/tomcat4, pid 2786
20160109:080540274WATCHD:HIGH:ProcStateTrans::_logRunning: Service tomcat4 is running with exe /usr/sbin/tomcat4 pid 2786
20160109:080540275WATCHD:HIGH:ProcStateTrans::_logRunning: Service twrcinit is running with exe /etc/tripwire/cmds/twqueue pid 2882
20160109:080540276WATCHD:HIGH:ProcStateTrans::_logRunning: Service xinetd is running with exe /usr/sbin/xinetd pid 2707
20160109:080540277WATCHD:HIGH:activemq Service not started due to OffBoot being set
20160109:080540279WATCHD:HIGH:sds Service not started due to OffBoot being set
20160109:080540280WATCHD:HIGH:monitor Service not started due to OffBoot being set
20160109:080540281WATCHD:HIGH:ppmhttpd Service not started due to OffBoot being set
20160109:080540284WATCHD:HIGH:Application ModemMtty started (/opt/ecs/sys/start_mdmtty, 3467)
20160109:080540286WATCHD:HIGH:Application TraceLogger started (/opt/ecs/sbin/logmanager, 3468)
20160109:080540286WATCHD:HIGH:StartupWait started sync to 5 for application TraceLogger
20160109:080540539WATCHD:HIGH:Heartbeats started for application TraceLogger (3468)
20160109:080540545WATCHD:HIGH:Application slotmon started (/opt/ecs/sbin/slotmon, 3511)
20160109:080540586WATCHD:HIGH:Application LicenseServer started (/opt/ecs/sbin/licsvr, 3512)
20160109:080540586WATCHD:HIGH:StartupWait started sync to 5 for application LicenseServer
20160109:080540976WATCHD:HIGH:Heartbeats started for application slotmon (3511)
20160109:080541014WATCHD:HIGH:Heartbeats started for application LicenseServer (3512)
20160109:080541738WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 0
20160109:080542019WATCHD:HIGH:Application SME started (/opt/ecs/sbin/sme, 3594)
20160109:080542264WATCHD:HIGH:Heartbeats started for application SME (3594)
20160109:080544242WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 1
20160109:080546746WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 2
20160109:080547027WATCHD:HIGH:Application MasterAgent started (/opt/ecs/sbin/snmpdm, 3641)
20160109:080549250WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 3
20160109:080551754WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 4
20160109:080552035WATCHD:HIGH:Application MIB2Agent started (/opt/ecs/sbin/mib2agt, 3681)
20160109:080554258WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 5
20160109:080556762WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 6
20160109:080557047WATCHD:HIGH:Application MVSubAgent started (/opt/ecs/sbin/mvmgt, 3721)
20160109:080559262WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 7
20160109:080601766WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 8
20160109:080602056WATCHD:HIGH:Application LoadAgent started (/opt/ecs/sbin/loadagt, 3764)
20160109:080604271WATCHD:HIGH:INFO: WatchdogDevWakeupCall::Wakeup: Invoked WatchdDevice::Strobe(), numHbeats 9, last message logged
20160109:080607068WATCHD:HIGH:Application FPAgent started (/opt/ecs/sbin/fpagt, 3804)
20160109:080612076WATCHD:HIGH:Application INADSAlarmAgent started (/opt/ecs/sbin/almindsagt, 3845)
20160109:080617096WATCHD:HIGH:Application GMM started (/opt/ecs/sbin/gmm, 3885)
20160109:080617360WATCHD:HIGH:Heartbeats started for application GMM (3885)
20160109:080620263WATCHD:HIGH:INFO: lo-monitor thread is now heartbeating, counter 7
20160109:080622340WATCHD:HIGH:Application SNMPManager started (/opt/ecs/sbin/trapmgr, 3928)
20160109:080622342WATCHD:HIGH:Application filesyncd started (/opt/ecs/sbin/filesyncd, 3929)
20160109:080622344WATCHD:HIGH:Application fsy_rcv started (/opt/ecs/sbin/fsy_rcv, 3930)
20160109:080623358WATCHD:HIGH:Application MCD started (/opt/ecs/sbin/mcd, 3976)
20160109:080623539WATCHD:HIGH:Heartbeats started for application filesyncd (3929)
20160109:080624165WATCHD:HIGH:Heartbeats started for application MCD (3976)
20160109:080628364WATCHD:HIGH:Application CommunicaMgr started (/bin/bash, 4102)
20160109:080628367WATCHD:HIGH:Application lock3rd started (/opt/ecs/sbin/lock3rd, 4103)
20160109:080628367WATCHD:HIGH:StartupGlobalMode::NextAction: ALL PROCESSES HAVE BEEN STARTED
20160109:080628393WATCHD:HIGH:Heartbeats started for application CommunicaMgr (4102)
20160109:080628705WATCHD:HIGH:INFO: process 4103 exited with exit code=0 (0x0)
20160109:080628705WATCHD:HIGH:Application lock3rd (4103) terminated
20160109:080840320WATCHD:HIGH:Watchdog received delwatch request:
20160109:080840320WATCHD:HIGH: -- delwatch ModemMtty
20160109:080840320WATCHD:HIGH:CWATCHDApp::UnregProc: Unregistered Application ModemMtty (3467)
20160109:080841327WATCHD:HIGH:INFO: process 3467 exited with exit code=0 (0x0)
20160109:080841327WATCHD:HIGH:WARNING: process (pid 3467) terminated for not-active application ModemMtty

[gosha]

Поднимаемся до 5.2.1, включая шлюз и модули, и смотрим, что происходит
Да, и покажите sh platf main со шлюза - свербит у меня в голове некий PSN с похожими симптомами...

[kitsa]

G450-001(super)# sh platform main
MAINBOARD BOARD
------------------
Type : G450 mainboard
Description : G450 field replaceable mainboard
Serial Number : 08IS10173220
HW Vintage : 1
HW Suffix : A
FW Version : 29.24.4
Faults : No Fault Messages

MEMORY SOCKET #1
------------------
Type : 256MB DDR SDRAM memory module with ECC support
Serial Number : 009baf76
Manufacture Part Num : SG572328AVA32CD1
Faults : No Fault Messages

MEMORY SOCKET #2
------------------
Type : UNKNOWN
Serial Number : UNKNOWN
Manufacture Part Num : UNKNOWN
Faults : NOT PRESENT

COMPACT FLASH MEMORY
--------------------
Type : No CompactFlash card is installed
Serial Number : N/A
Model Number : N/A
Faults : N/A

MEDIA SOCKET #1
------------------
Type : MP20 VoIP DSP Module
Description : VoIP DSP resource with 20 channels
Serial Number : 08IS09167780
HW Vintage : 0
HW Suffix : C
Faults : No Fault Messages

MEDIA SOCKET #2
------------------
Type : UNKNOWN
Description : UNKNOWN
Serial Number : UNKNOWN
HW Vintage : UNKNOWN
HW Suffix : UNKNOWN
Faults : NOT PRESENT

MEDIA SOCKET #3
------------------
Type : UNKNOWN
Description : UNKNOWN
Serial Number : UNKNOWN
HW Vintage : UNKNOWN
HW Suffix : UNKNOWN
Faults : NOT PRESENT

MEDIA SOCKET #4
------------------
Type : UNKNOWN
Description : UNKNOWN
Serial Number : UNKNOWN
HW Vintage : UNKNOWN
HW Suffix : UNKNOWN
Faults : NOT PRESENT

[kitsa]

Обновление до 5.2.1 платное?
Я имею ввиду не стоимость работ а саму прошивку

[Pterodahtil]

Обновление до 5.2.1 платное?
Я имею ввиду не стоимость работ а саму прошивку

Скажем так, никаких лицензий покупать не надо, но не имея на руках сервисного контракта, ничего скачать с support.avaya.com - нельзя. Так что, если сам софт есть, ничего не мешает обновиться.

Вроде версия шлюза не подпадает под PSN с ребутом.

[kitsa]

Про доступ к прошивки я в курсе.
Меня все таки интересует вопрос кроме обновления прошивки можно что то еще сделать. А если скажем и это не поможет.
И самое интересное кто нибудь может обьяснить как так. Станция стоит, работает несколько лет, от всех внешних связей отключена, обновлений ПО и прочее не проводилось.
С какого фига начались глюки исправить котореы может обновление ПО, ладно я понимаю кондеры бы или еще что по физике, но логика!!!
Такое ощущение что где то закладки стоят

[gosha]

Софт станции - не статичная конструкция (на то, что вы так думаете, наводит использованное вами слово "прошивка"), меняются конфиги, пмшутся логи, допускают ошибки программисты. Со временем это может превысить некий порог, и будет эффект, похожий на ваш.