Alison Shiue Reporter
	Description • 9 years ago

crash report: https://crash-stats.mozilla.com/report/index/8a84b3a1-f464-4a21-b1f1-e37202150417

STR: 
Running MTBF for 34 more hours.

Version info:
Build ID               20150414162502
Gaia Revision          16e948bfaaa15dbc0200135d52f16257b4eab193
Gaia Date              2015-04-14 21:08:25
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/0eec28e78eb1
Gecko Version          37.0
Device Name            hammerhead
Firmware(Release)      5.1
Firmware(Incremental)  eng.cltbld.20150414.201650
Firmware Date          Tue Apr 14 20:17:04 EDT 2015
Bootloader             HHZ12f

	Alison Shiue Reporter
	Comment 1 • 9 years ago

Hit this issue again: https://crash-stats.mozilla.com/report/index/764e3d82-a19e-4aef-9d05-302872150422

Build ID               20150420162501
Gaia Revision          828dd03a0e3b140d74b2e49355197df4d91d9227
Gaia Date              2015-04-20 18:28:39
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/36f72a3efb9b
Gecko Version          37.0
Device Name            hammerhead
Firmware(Release)      5.1
Firmware(Incremental)  eng.cltbld.20150420.195827
Firmware Date          Mon Apr 20 19:58:43 EDT 2015
Bootloader             HHZ12f

	Alastor Wu [:alwu]
	Comment 2 • 9 years ago

It seems that Android media server died due to unknown reasons. 
After media server died, the Gecko would trigger the RecoverTask to get new service.
At that time, the original audio flinger should have been reset to null, then we create a new one.

But, I don't know why we can't get the new audio flinger.
One potential reason is that the gAudioFlinger.clear() does't work, so that the AudioSystem return the non-existent address.

Need to see more logs if you can provide it.
Thanks.

	Alison Shiue Reporter
	Comment 3 • 9 years ago

I think this logcat recorded the crashed moment. Hope this would give some help.

	Bobby Chien
	Comment 4 • 9 years ago

Do we have idea to narrow down reproduce steps and test case?

	Alastor Wu [:alwu]
	Comment 5 • 9 years ago

Hi, Alison,
I found that the log is different with the crash report, is this logcat correct?
Because the log shows that the crash point is caused by some error in the android::Camera3Device, instead of the android::AudioSystem::get_audio_flinger.

Thasnks :)

	Alison Shiue Reporter
	Comment 6 • 9 years ago

I gripped "AudioSystem" on all logcat files, and found following info on logcat127: 

04-21 23:41:40.473  2057  2070 W AudioSystem: AudioPolicyService server died!
04-21 23:41:40.477  2057 30574 W AudioSystem: AudioFlinger server died!

Is this related to crash android::AudioSystem::get_audio_flinger?

	Alastor Wu [:alwu]
	Comment 7 • 9 years ago

Yes, I know that. 
This may related with the crash on AudioSystem::get_audio_flinger. 
But, because of the difference between the log and crash report, I am wondering whether there is possible that the crash of the camera resulted in this crash.

Therefore, I want to know why the log shows another different crash result?  
Thanks!

	Alison Shiue Reporter
	Comment 8 • 9 years ago

It's really weird. And according to jenkins logs, these two crashes seem happened after running camera case. (test_mtbf_camera_multiple_shots.py)

	Alison Shiue Reporter
	Comment 9 • 9 years ago

Here are logs for this two run tests which hit the crashed issue: https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-lNDWe2ofktWbFdlTUQ2c0xxR0xQNXBXU0loSXFsWlRNc1dfLVNrZ0xpLUJEak1tQVU&usp=sharing

	Alastor Wu [:alwu]
	Comment 10 • 9 years ago

From the log on comment3 & comment9, I think the problem might not be related with the audio channel.

I still saw the crash info about the camera (the same as the attachment of the comment5).
It seems that some unknown crash of camera resulted in the dead of the Android media server.

Although the crash report points that the get_audio_flinger() is the crash point, I can't see any possible clue about it. Because the AudioFliger would be re-created after the media server is re-started.

---

Steven, do you know who can help this issue?
Thanks!

	Alastor Wu [:alwu]
	Comment 12 • 9 years ago

free this, unless this issue can be identified as the audio channel problem.

	Bobby Chien
	Comment 13 • 9 years ago

(In reply to Alison Shiue from comment #8)
> It's really weird. And according to jenkins logs, these two crashes seem
> happened after running camera case. (test_mtbf_camera_multiple_shots.py)

Alison, is it possible to run camera-only mtbf tests? if we could narrow down to single test/scenario, we could provide more info for follower. what do you think?

	Alison Shiue Reporter
	Comment 14 • 9 years ago

Yes, I just get this crash again when I only run camera test in mtbf at my local environment.
And the crash report still points to get_audio_flinger(): https://crash-stats.mozilla.com/report/index/01816550-0566-4320-b663-627712150424

	Bobby Chien
	Comment 15 • 9 years ago

per comment 14, blocking as reproducible.

	Bobby Chien
	Comment 16 • 9 years ago

becker, could you help investigate this?

	becker hsieh{:behsieh}
	Comment 17 • 9 years ago

not sure whether this was the root cause of crash
but looks like there is a error event send from camera driver.

04-21 23:41:37.491   195 19182 E Camera3-Device: Camera 0: clearStreamingRequest: Device has encountered a serious error
04-21 23:41:38.142   195 19183 E Camera3-Device: Camera 0: clearStreamingRequest: Device has encountered a serious error
04-21 23:41:38.783   195   731 E Camera3-Device: Camera 0: clearStreamingRequest: Device has encountered a serious error
04-21 23:41:39.497   195   731 E Camera3-Device: Camera 0: disconnect: Shutting down in an error state

	StevenLee[:slee]
	Comment 18 • 9 years ago

Hi Alison,

Can you provide the full log of comment 14? I checked  the log of output_149. The some logcat seems to be washed out. I cannot find the crash ID from b2gps or b2ginfo. Can you apply the patch and try again? Thanks.

	Alison Shiue Reporter
	Comment 19 • 9 years ago

(In reply to StevenLee[:slee] from comment #18)
> Created attachment 8598467 [details] [diff] [review]
> log.patch
> 
> Hi Alison,
> 
> Can you provide the full log of comment 14? I checked  the log of
> output_149. The some logcat seems to be washed out. I cannot find the crash
> ID from b2gps or b2ginfo. Can you apply the patch and try again? Thanks.

I cannot provide more log info of comment 14 because the test was running at my local environment and I only got logcat.   

I would update test result after I tried with your patch. Thanks.

	Alison Shiue Reporter
	Comment 20 • 9 years ago

Add another test result which hit this issue 9 times.  (this build not apply Steven's patch)

Please refer thees log info: https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-lNDWe2oflBjNUNVN1ZLaUl2M0JlbTFTdG11OW9IUWdpMG9naVFpMDkzMld0VEJhbXc&usp=sharing

Crash report: 
https://crash-stats.mozilla.com/report/index/54b962b0-c5a6-4a0d-9007-f51ce2150427
https://crash-stats.mozilla.com/report/index/7cba7e11-6397-4e4f-ad3e-a72302150424
https://crash-stats.mozilla.com/report/index/7f8ce306-717c-48eb-8de5-5c2542150424
https://crash-stats.mozilla.com/report/index/8c3ed878-7964-485d-ad9f-dba642150427
https://crash-stats.mozilla.com/report/index/91640862-dc55-49c1-a0cd-83d5c2150427
https://crash-stats.mozilla.com/report/index/99f52438-5337-4544-ac9f-dc7172150427
https://crash-stats.mozilla.com/report/index/aa8690f5-850f-4ee0-a274-94e092150427
https://crash-stats.mozilla.com/report/index/c26e276c-7f23-4c45-9b42-003b92150427
https://crash-stats.mozilla.com/report/index/ed401bb4-c24a-48bf-bc4c-724d42150427


Build info:
Serial: 0ae0abef0337dcbf (State: device)
Build ID               20150422162503
Gaia Revision          41a85c5f9db291d4f7c0e94c8416b5115b4ee407
Gaia Date              2015-04-21 17:23:41
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/367b3e608cd8
Gecko Version          37.0
Device Name            hammerhead
Firmware(Release)      5.1
Firmware(Incremental)  eng.cltbld.20150422.195718
Firmware Date          Wed Apr 22 19:57:35 EDT 2015
Bootloader             HHZ12f

	becker hsieh{:behsieh}
	Comment 21 • 9 years ago

Hi alison:
per discussion, attached zip is another set of camera library.
please follow below step
1. unzip backup_camera_lib_hammerhead-lmy47d.zip
2. go to backup_camera_lib_hammerhead-lmy47d folder
3. sh push_lib.sh
4. reboot device.

	StevenLee[:slee]
	Comment 22 • 9 years ago

(In reply to Alison Shiue from comment #20)
> Add another test result which hit this issue 9 times.  (this build not apply
> Steven's patch)
> 
> Please refer thees log info:
> https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-
> lNDWe2oflBjNUNVN1ZLaUl2M0JlbTFTdG11OW9IUWdpMG9naVFpMDkzMld0VEJhbXc&usp=sharin
> g
> 
> Crash report: 
> https://crash-stats.mozilla.com/report/index/54b962b0-c5a6-4a0d-9007-
> f51ce2150427
> https://crash-stats.mozilla.com/report/index/7cba7e11-6397-4e4f-ad3e-
> a72302150424
> https://crash-stats.mozilla.com/report/index/7f8ce306-717c-48eb-8de5-
> 5c2542150424
> https://crash-stats.mozilla.com/report/index/8c3ed878-7964-485d-ad9f-
> dba642150427
> https://crash-stats.mozilla.com/report/index/91640862-dc55-49c1-a0cd-
> 83d5c2150427
> https://crash-stats.mozilla.com/report/index/99f52438-5337-4544-ac9f-
> dc7172150427
> https://crash-stats.mozilla.com/report/index/aa8690f5-850f-4ee0-a274-
> 94e092150427
> https://crash-stats.mozilla.com/report/index/c26e276c-7f23-4c45-9b42-
> 003b92150427
> https://crash-stats.mozilla.com/report/index/ed401bb4-c24a-48bf-bc4c-
> 724d42150427
Hi Alison,
Is there any way to know the log files for a certain crash report?

	Alison Shiue Reporter
	Comment 23 • 9 years ago

According to the final logcat (logcat766), it looks like only happened 4 crashes at : "04-23 21:59:12", "04-24 14:47:53", "04-27 18:35:04", and "04-28 01:11:24". 
(I am not sure which is correct since crash dump number cannot match the crash number at logcat....)

Because the jenkins logs were not real time log, I am afraid those did not record these crash moments.
So please refer the realtime log(logcat_163_0ae, this logcat start from 04-27 17:52:45), I think it would include the last 2 crash moments.

	becker hsieh{:behsieh}
	Comment 24 • 9 years ago

from logcat128
start camera in here
04-23 21:23:52.459   195  1785 D QCamera3HWI: int qcamera::QCamera3HardwareInterface::configureStreams(camera3_stream_configuration_t*): newStream type = 0, stream format = 34 stream size : 1600 x 1200
04-23 21:23:52.459   195  1785 D QCamera3HWI: int qcamera::QCamera3HardwareInterface::configureStreams(camera3_stream_configuration_t*): newStream type = 0, stream format = 33 stream size : 3264 x 2448
04-23 21:23:52.673   195 14461 D QCamera3HWI: int qcamera::QCamera3HardwareInterface::processCaptureRequest(camera3_capture_request_t*): Start META Channel
04-23 21:23:52.695   195 14461 D QCamera3HWI: int qcamera::QCamera3HardwareInterface::processCaptureRequest(camera3_capture_request_t*): Start Regular Channel mask=2

and then take picture
04-23 21:23:55.426   195 14461 D QCamera3HWI: bool qcamera::QCamera3HardwareInterface::needReprocess(): need do reprocess for ZSL WNR or min PP reprocess
04-23 21:23:55.433   195 14461 D QCamera3HWI: bool qcamera::QCamera3HardwareInterface::needRotationReprocess(): need do reprocess for rotation
04-23 21:23:55.728   195 14727 D QCamera3HWI: bool qcamera::QCamera3HardwareInterface::needReprocess(): need do reprocess for ZSL WNR or min PP reprocess
04-23 21:23:55.771   195 14707 D QCamera3HWI: bool qcamera::QCamera3HardwareInterface::needRotationReprocess(): need do reprocess for rotation

and there are keeping have AF stats process error
04-23 21:23:54.783   211 14710 E mm-camera: af_process_parse_stats: Invalid FV data!
04-23 21:23:54.783   211 14710 E mm-camera: af_process: Error processing AF stats!
04-23 21:23:54.884   211 14710 E mm-camera: af_process_parse_stats: Invalid FV data!
04-23 21:23:54.884   211 14710 E mm-camera: af_process: Error processing AF stats!
04-23 21:24:01.941   211 14710 E mm-camera: af_process_parse_stats: Invalid FV data!
04-23 21:24:01.941   211 14710 E mm-camera: af_process: Error processing AF stats!

but there is no further camera action , and camera crashed in logcat135

04-23 21:59:12.784   191   191 I DEBUG   : backtrace:
04-23 21:59:12.784   191   191 I DEBUG   :     #00 pc 0000403c  /system/lib/libcutils.so (android_atomic_add+16)
04-23 21:59:12.784   191   191 I DEBUG   :     #01 pc 0000ec37  /system/lib/libutils.so (android::RefBase::decStrong(void const*) const+10)
04-23 21:59:12.784   191   191 I DEBUG   :     #02 pc 00047153  /system/lib/libcameraservice.so (android::sp<android::IBinder>::~sp()+18)
04-23 21:59:12.784   191   191 I DEBUG   :     #03 pc 00069351  /system/lib/libcameraservice.so (android::Vector<android::sp<android::camera3::Camera3StreamInterface> >::do_destroy(void*, unsigned int) const+12)
04-23 21:59:12.784   191   191 I DEBUG   :     #04 pc 00010ffb  /system/lib/libutils.so
04-23 21:59:12.785   191   191 I DEBUG   :     #05 pc 0001101b  /system/lib/libutils.so (android::VectorImpl::release_storage()+28)
04-23 21:59:12.785   191   191 I DEBUG   :     #06 pc 00011089  /system/lib/libutils.so (android::VectorImpl::finish_vector()+4)
04-23 21:59:12.785   191   191 I DEBUG   :     #07 pc 00066737  /system/lib/libcameraservice.so
04-23 21:59:12.785   191   191 I DEBUG   :     #08 pc 00068c6b  /system/lib/libcameraservice.so (android::Camera3Device::~Camera3Device()+126)
04-23 21:59:12.785   191   191 I DEBUG   :     #09 pc 00068cf1  /system/lib/libcameraservice.so (android::Camera3Device::~Camera3Device()+4)
04-23 21:59:12.785   191   191 I DEBUG   :     #10 pc 0000ec55  /system/lib/libutils.so (android::RefBase::decStrong(void const*) const+40)
04-23 21:59:12.785   191   191 I DEBUG   :     #11 pc 0005299d  /system/lib/libcameraservice.so (android::Camera2Client::disconnect()+332)
04-23 21:59:12.785   191   191 I DEBUG   :     #12 pc 000233c3  /system/lib/libcamera_client.so (android::BnCamera::onTransact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+86)
04-23 21:59:12.785   191   191 I DEBUG   :     #13 pc 0001a6cd  /system/lib/libbinder.so (android::BBinder::transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+60)
04-23 21:59:12.785   191   191 I DEBUG   :     #14 pc 0001f77b  /system/lib/libbinder.so (android::IPCThreadState::executeCommand(int)+582)
04-23 21:59:12.785   191   191 I DEBUG   :     #15 pc 0001f89f  /system/lib/libbinder.so (android::IPCThreadState::getAndExecuteCommand()+38)
04-23 21:59:12.785   191   191 I DEBUG   :     #16 pc 0001f8e1  /system/lib/libbinder.so (android::IPCThreadState::joinThreadPool(bool)+48)
04-23 21:59:12.785   191   191 I DEBUG   :     #17 pc 00023a5b  /system/lib/libbinder.so
04-23 21:59:12.785   191   191 I DEBUG   :     #18 pc 000104d5  /system/lib/libutils.so (android::Thread::_threadLoop(void*)+112)
04-23 21:59:12.785   191   191 I DEBUG   :     #19 pc 00010045  /system/lib/libutils.so
04-23 21:59:12.785   191   191 I DEBUG   :     #20 pc 00016baf  /system/lib/libc.so (__pthread_start(void*)+30)
04-23 21:59:12.785   191   191 I DEBUG   :     #21 pc 00014af3  /system/lib/libc.so (__start_thread+6)

	Alison Shiue Reporter
	Comment 25 • 9 years ago

Hit this issue again at about "04-30 21:05:59", here are logs from device with comment 21 library: https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-lNDWe2ofmRBYWVZMC1zMDkxaXJkbDg1bV9WSFBkQnBuUW1kQjFhdnNha1VyQ3QtWDA&usp=sharing

	becker hsieh{:behsieh}
	Comment 26 • 9 years ago

Hi alison:
attached file is another set of camera library.
please follow below step to setup your test environment.
1. unzip zip file
2. run "sh push_lib.sh"
3. reboot device

	Ken Chang[:kenkai]
	Comment 27 • 9 years ago

Hi Steven, could you take this bug?

	StevenLee[:slee]
	Comment 28 • 9 years ago

(In reply to Ken Chang[:ken] from comment #27)
> Hi Steven, could you take this bug?
Hi Ken,

I think becker is looking this problem. And from comment 25, I don't find log related audioflinger crash. So that I suggest becker can take this problem first.

Hi becker,
Could you please take this bug first? Thanks.

	Alison Shiue Reporter
	Comment 29 • 9 years ago

Hit this issue again at about "05-05 00:08:05", here are logs from device with comment 26 library:
https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-lNDWe2ofmFQT2dGbURkYVNqZGR0N2pXYy1jYzVjZ2xEOW9wbHJVeHMyRTc0bC1aTlk&usp=sharing

	becker hsieh{:behsieh}
	Comment 30 • 9 years ago

from log
there is a error event send from camera proprietary library in shutting down
05-05 00:08:05.006 I/Camera3-Device(  189): disconnect: mStatus:0 E
05-05 00:08:05.006 E/Camera3-Device(  189): Shutting down in an error state
05-05 00:08:05.006 I/Camera3-Device(  189): ~Camera3Device: Tearing down for camera id 0
05-05 00:08:05.006 I/Camera3-Device(  189): disconnect: mStatus:0 E
05-05 00:08:05.006 E/Camera3-Device(  189): Shutting down in an error state
and crash happen
05-05 00:08:05.087 I/DEBUG   (  186): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-05 00:08:05.087 I/DEBUG   (  186): Build fingerprint: 'Android/full_hammerhead/hammerhead:5.1/LMY47D/cltbld05031959:eng/test-keys'
05-05 00:08:05.087 I/DEBUG   (  186): Revision: '11'
05-05 00:08:05.087 I/DEBUG   (  186): ABI: 'arm'
05-05 00:08:05.087 I/DEBUG   (  186): pid: 189, tid: 727, name: Binder_2  >>> /system/bin/mediaserver <<<
05-05 00:08:05.087 I/DEBUG   (  186): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
05-05 00:08:05.124 I/DEBUG   (  186):     r0 ffffffff  r1 00000000  r2 b6e09a2c  r3 b4449520
05-05 00:08:05.124 I/DEBUG   (  186):     r4 00000000  r5 ffffffff  r6 b4456370  r7 b37f22a4
05-05 00:08:05.124 I/DEBUG   (  186):     r8 00000000  r9 00000000  sl 000003f5  fp 000000bd
05-05 00:08:05.124 I/DEBUG   (  186):     ip b6d7ed7c  sp b43ffb40  lr b6d7703c  pc b6d7703c  cpsr 200f0010
05-05 00:08:05.139 I/DEBUG   (  186): 
05-05 00:08:05.139 I/DEBUG   (  186): backtrace:
05-05 00:08:05.139 I/DEBUG   (  186):     #00 pc 0000403c  /system/lib/libcutils.so (android_atomic_add+16)
05-05 00:08:05.139 I/DEBUG   (  186):     #01 pc 0000ec37  /system/lib/libutils.so (android::RefBase::decStrong(void const*) const+10)
05-05 00:08:05.139 I/DEBUG   (  186):     #02 pc 00047153  /system/lib/libcameraservice.so (android::sp<android::IBinder>::~sp()+18)
05-05 00:08:05.139 I/DEBUG   (  186):     #03 pc 000693bd  /system/lib/libcameraservice.so (android::Vector<android::sp<android::camera3::Camera3StreamInterface> >::do_destroy(void*, unsigned int) const+12)
05-05 00:08:05.139 I/DEBUG   (  186):     #04 pc 00010ffb  /system/lib/libutils.so
05-05 00:08:05.139 I/DEBUG   (  186):     #05 pc 0001101b  /system/lib/libutils.so (android::VectorImpl::release_storage()+28)
05-05 00:08:05.139 I/DEBUG   (  186):     #06 pc 00011089  /system/lib/libutils.so (android::VectorImpl::finish_vector()+4)
05-05 00:08:05.139 I/DEBUG   (  186):     #07 pc 00066737  /system/lib/libcameraservice.so
05-05 00:08:05.139 I/DEBUG   (  186):     #08 pc 00068cd3  /system/lib/libcameraservice.so (android::Camera3Device::~Camera3Device()+150)
05-05 00:08:05.139 I/DEBUG   (  186):     #09 pc 00068d5d  /system/lib/libcameraservice.so (android::Camera3Device::~Camera3Device()+4)
05-05 00:08:05.139 I/DEBUG   (  186):     #10 pc 0000ec55  /system/lib/libutils.so (android::RefBase::decStrong(void const*) const+40)
05-05 00:08:05.139 I/DEBUG   (  186):     #11 pc 0005299d  /system/lib/libcameraservice.so (android::Camera2Client::disconnect()+332)
05-05 00:08:05.139 I/DEBUG   (  186):     #12 pc 000233c3  /system/lib/libcamera_client.so (android::BnCamera::onTransact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+86)
05-05 00:08:05.139 I/DEBUG   (  186):     #13 pc 0001a6cd  /system/lib/libbinder.so (android::BBinder::transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+60)
05-05 00:08:05.139 I/DEBUG   (  186):     #14 pc 0001f77b  /system/lib/libbinder.so (android::IPCThreadState::executeCommand(int)+582)
05-05 00:08:05.139 I/DEBUG   (  186):     #15 pc 0001f89f  /system/lib/libbinder.so (android::IPCThreadState::getAndExecuteCommand()+38)
05-05 00:08:05.139 I/DEBUG   (  186):     #16 pc 0001f8e1  /system/lib/libbinder.so (android::IPCThreadState::joinThreadPool(bool)+48)
05-05 00:08:05.139 I/DEBUG   (  186):     #17 pc 00023a5b  /system/lib/libbinder.so
05-05 00:08:05.139 I/DEBUG   (  186):     #18 pc 000104d5  /system/lib/libutils.so (android::Thread::_threadLoop(void*)+112)
05-05 00:08:05.139 I/DEBUG   (  186):     #19 pc 00010045  /system/lib/libutils.so
05-05 00:08:05.140 I/DEBUG   (  186):     #20 pc 00016baf  /system/lib/libc.so (__pthread_start(void*)+30)
05-05 00:08:05.140 I/DEBUG   (  186):     #21 pc 00014af3  /system/lib/libc.so (__start_thread+6)

	becker hsieh{:behsieh}
	Comment 31 • 9 years ago

Hi alison:
attached file will show us why camera send error event
please follow below step to push library
1. adb remount
2. adb push libcameraservice.so /system/lib
3. reboot device

	Alison Shiue Reporter
	Comment 32 • 9 years ago

Hit this issue again at about "05-08 06:32:07", here are logs from device with comment 31 library:
https://drive.google.com/a/mozilla.com/folderview?id=0B_lQ-lNDWe2oflB6TUl6dmRVYWdiQlR0YXNSMnVPV0xEZjRtZXJBMFlVamxOZFRlNWp1cHM&usp=sharing

	becker hsieh{:behsieh}
	Comment 33 • 9 years ago

looks from driver side capture request process time out
05-08 06:30:17.211 E/QCamera3HWI(  176): int qcamera::QCamera3HardwareInterface::processCaptureRequest(camera3_capture_request_t*): Unblocked on timeout!!!!
05-08 06:30:17.211 E/Camera3-Device(  176): becker Camera 0: threadLoop: RequestThread: Unable to submit capture request 2249 to HAL device: No such device (-19)
05-08 06:30:17.211 E/Camera2ClientBase(  176): Error condition 1 reported by HAL, requestId -1

	becker hsieh{:behsieh}
	Comment 34 • 9 years ago

looks from driver side capture request process time out
05-08 06:30:17.211 E/QCamera3HWI(  176): int qcamera::QCamera3HardwareInterface::processCaptureRequest(camera3_capture_request_t*): Unblocked on timeout!!!!
05-08 06:30:17.211 E/Camera3-Device(  176): becker Camera 0: threadLoop: RequestThread: Unable to submit capture request 2249 to HAL device: No such device (-19)
05-08 06:30:17.211 E/Camera2ClientBase(  176): Error condition 1 reported by HAL, requestId -1

	becker hsieh{:behsieh}
	Comment 35 • 9 years ago

Hi alison:
attached file is possible solution.
here is step
1. adb remount
2. adb push camera.hammerhead.so /system/lib/hw
3. adb push libcameraservice.so /system/lib
4. adb reboot.

	Alison Shiue Reporter
	Comment 37 • 9 years ago

Executing MTBF test with the possible solution about 1 week, until now, we do not hit the camera crash issue.

	Josh Cheng [:josh]
	Comment 38 • 9 years ago

Hi Becker,
The patch seems work fine. Can you help to raise review request or anyone else can help?
Thanks!

	StevenLee[:slee]
	Comment 39 • 9 years ago

Hi Josh,

Becker has left Mozilla. :(

	Bobby Chien
	Comment 40 • 9 years ago

Hayden, please help attach patch for tracking, it could help further Flame and z3 troubleshooting. Please also help to update binary into nexus 5 image. 

According to above solution, remove v2.2+.

	becker hsieh{:behsieh}
	Comment 41 • 9 years ago

I will give my patch to hayden and discuss with best solution.

	Hayden Huang [:haydenh] Assignee
	Comment 42 • 9 years ago

submit Becker's patch

	Alison Shiue Reporter
	Comment 43 • 9 years ago

Hi Hayden,
Could you please find someone review this patch and then land code in v2.2? Thank you.

	Hayden Huang [:haydenh] Assignee
	Comment 44 • 9 years ago

Comment on attachment 8606957 [details] [review]
pull request

>https://github.com/mozilla-b2g/device-hammerhead/pull/28

	Kai-Zhen Li [:seinlin][:kli]
	Comment 45 • 9 years ago

Comment on attachment 8606957 [details] [review]
pull request

Rubber-stamping.

	Kai-Zhen Li [:seinlin][:kli]
	Comment 46 • 9 years ago

It's more likely a vendor issue. See also https://code.google.com/p/android/issues/detail?id=160609

	Kai-Zhen Li [:seinlin][:kli]
	Comment 47 • 9 years ago

merged: https://github.com/mozilla-b2g/device-hammerhead/commit/74919c3e2742b3237b2f30a8cefe182d0caf1e31

Alison, could you resolve this bug once it is verified fixed? Thanks!

	Alison Shiue Reporter
	Comment 48 • 9 years ago

Verified on v2.2:

Build ID 20150524162504
Gaia Revision de5942bd99b801d4d04b676f1e70a77e93115d95
Gaia Date 2015-05-22 19:38:55
Gecko Revision https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/1bd024f4e171
Gecko Version 37.0
Device Name hammerhead
Firmware(Release) 5.1
Firmware(Incremental)  eng.cltbld.20150524.201333
Firmware Date Sun May 24 20:13:50 EDT 2015
Bootloader HHZ12f