EXC_BAD_ACCESS crashes on iOS due to ANE's?

timo

Hi, we are experiencing EXC_BAD_ACCESS crashes on iOS that we're having hard time debugging since it's very difficult to reproduce locally (I've only managed to crash twice over a long period).

We are getting our bug reports into Bugsnag, and the description of the crash is "Attempted to dereference garbage pointer 0x38". The pointer address varies. It's happening only in about 1% of our sessions, but still, thousands of crashes and several disappointed users in a month.

I'm suspecting the problems are related to ANE-Google-Analytics (github.com/mparrett/ANE-Google-Analytics) and ANE-Bugsnag (github.com/DigitalStrawberry/ANE-Bugsnag) (ironically) as the only clues I get from the stacktraces and thread dumps lead to "trackSocial" (a method of ANE-Google-Analytics) and libcom.bugsnag.ANEBugsnag.

I'm now thinking about replacing the ANE-Google-Analytics with corresponding MMG ANE and seeing if this would at least get rid of the other half of the crashes. However, I believe there's no replacement for the ANE-Bugsnag, so I'm kinda stuck with it unless I change the whole Bugsnag service into something else.

If anyone's got ideas, any help would be really appreciated, thanks! 🙂

henke37

A low pointer value like 0x38 sounds like an offset applied to a null pointer. Look for those?

timo

Right, we've got low and high pointers values.

How would you suggest I look for null pointers? I've not done much outside AS3... 🙁

timo

Hi Guys,

I'm still stuck with a native iOS crash which doesn't seem to bother the users (it happens after the app gets suspended), but does bother me. Is there anybody who could shed any light on how to debug native crashes?

This is my stack trace I get into Bugsnag (crash reporting tool):

EXC_BAD_ACCESS: Attempted to dereference garbage pointer 0x18.

0 thisissand /var/folders/h7/b3r397290c75zjcqnb1kbdnm0000gn/T/d20422bf-b5b3-43b1-a9d6-12a263a3fa57/libcom.bugsnag.ANEBugsnag.a(ANEBugsnag.o) (thisissand)
1 thisissand main (thisissand)
2 thisissand main (thisissand)
3 thisissand main (thisissand)
4 thisissand main (thisissand)
5 thisissand main (thisissand)
6 thisissand main (thisissand)
7 thisissand /var/folders/h7/b3r397290c75zjcqnb1kbdnm0000gn/T/d20422bf-b5b3-43b1-a9d6-12a263a3fa57/libcom.bugsnag.ANEBugsnag.a(ANEBugsnag.o) (thisissand)
8 thisissand /var/folders/h7/b3r397290c75zjcqnb1kbdnm0000gn/T/d20422bf-b5b3-43b1-a9d6-12a263a3fa57/libcom.bugsnag.ANEBugsnag.a(ANEBugsnag.o) (thisissand)
9 thisissand /var/folders/h7/b3r397290c75zjcqnb1kbdnm0000gn/T/d20422bf-b5b3-43b1-a9d6-12a263a3fa57/libcom.bugsnag.ANEBugsnag.a(ANEBugsnag.o) (thisissand)
10 UIKit -[UIApplication _terminateWithStatus:]
11 UIKit ___102-[UIApplication _handleApplicationDeactivationWithScene:shouldForceExit:transitionContext:completion:]_block_invoke.2098
12 UIKit __runAfterCACommitDeferredBlocks
13 UIKit __cleanUpAfterCAFlushAndRunDeferredBlocks
14 UIKit __afterCACommitHandler
15 CoreFoundation ___CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__
16 CoreFoundation ___CFRunLoopDoObservers
17 CoreFoundation ___CFRunLoopRun
18 CoreFoundation CFRunLoopRunSpecific
19 GraphicsServices GSEventRunModal
20 UIKit -[UIApplication _run]
21 UIKit UIApplicationMain
22 thisissand main (thisissand)
23 libdyld.dylib start

Does this indicate that the crash happened in libcom.bugsnag.ANEBugnsag.a (thisissand is my app name). Or could it be that the crash happens in main (thisissand), and ANEBugsnag is just reporting the crash...? If it happens in main (thisissand), then why is this a native crash and not an uncaught as3 error? Also, is there any way to know what's happening on lines 1-6...?

Thanks for any tips! 🙂

marpies

Hey,

We see crash reports like this in our app as well, and we see mentions of different ANEs in the stack trace:

EXC_BAD_ACCESS Attempted to dereference garbage pointer 0x18.

0	AppName AppName main
1	AppName AppName /var/folders/0s/8gb3zk456y5g8qx1gnwz0tjr0000gp/T/5fea6bca-c13e-4b0a-87c2-c5030658b114/libcom.distriqt.Share.a(DTSHSocialPost.o)
2	QuartzCore CA::Display::DisplayLinkItem::dispatch(unsigned long long)
3	QuartzCore CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long)
4	IOKit IODispatchCalloutFromCFMessage
5	CoreFoundation ___CFMachPortPerform
6	CoreFoundation ___CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__
7	CoreFoundation ___CFRunLoopDoSource1
8	CoreFoundation ___CFRunLoopRun
9	CoreFoundation CFRunLoopRunSpecific
10	GraphicsServices GSEventRunModal
11	UIKit -[UIApplication _run]
12	UIKit UIApplicationMain
13	AppName AppName	main
14	libdyld.dylib start

We're not really sure how to debug or reproduce this. We often see the stack trace ending in main so we wonder if it's got to do something with AIR doing some clean up of objects allocated by ANEs that are passed to AS3.

Did have any user reports of this happening in runtime or only when the app is suspended?

timo

There are also reports that have happened while the app has been in foreground, like the one below. There are many that are identical with the one that I posted before (app just suspended), and then many slightly different kinds like the one below (app in foreground). Some of the latter kinds have the ANEBugsnag involved, some not. I haven't lately seen other ANEs than the ANEBugsnag mentioned in the reports.

I'm also unable to reproduce myself.

EXC_BAD_ACCESS: Attempted to dereference garbage pointer 0x8.

0 thisissand main (thisissand)
1 thisissand 0x1459c (0x17e8 + 77236) (thisissand)
2 QuartzCore CA::Display::DisplayLinkItem::dispatch()
3 QuartzCore CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long)
4 IOMobileFramebuffer <redacted>
5 IOKit IODispatchCalloutFromCFMessage
6 CoreFoundation <redacted>
7 CoreFoundation <redacted>
8 CoreFoundation <redacted>
9 CoreFoundation <redacted>
10 CoreFoundation CFRunLoopRunSpecific
11 CoreFoundation CFRunLoopRunInMode
12 GraphicsServices GSEventRunModal
13 UIKit UIApplicationMain
14 thisissand main (thisissand)

timo

I've made another build without the ANEBugsnag for our test user whose iPhone is spitting out these crash logs. The crashes still keep occurring, but now as I don't have the Bugsnag enabled, I'm not able to symbolicate the crashes.. 🙁

The stacktrace otherwise did look identical, so I'm pretty convinced that the problem is not in ANEBugsnag, but possibly something like @marpies was guessing.

I was trying to manually symbolicate with the symbolicatecrash tool, but without luck. As usual, another issue popped up which I assume is preventing the manual symbolication: https://tracker.adobe.com/#/view/AIR-4073680.

My dSYM does not have the arm7 build included, and although the crash happens on arm64, the symbolicatecrash is not able to detect the dSYM and does not symbolicate.

marpies

Hi timo,

Any luck in finding the cause for these reports? We're still seeing a lot of these in the Bugsnag dashboard.

timo

Nope, unfortunately not. 🙁

We're also still seeing these, but I'm suspecting most of them are in the background and not noticeable for users since we haven't gotten user reports of crashes.

Still, would be nice to get the issue sorted!

ma

This is probably caused by some shared data between flash and the native code.

Most common issue I've seen like this has been when passing a bitmap into an ANE and flash disposing of it before the ANE operation has completed.

You often can get around this by storing a reference to the bitmap data you are passing in until the ANE operation has completed, for example, referencing as a class variable.

We sometimes copy the bitmap data internally however this can cause memory issues when duplicating large bitmaps, so we restrict this to long operations where its common for the bitmap to be disposed.

timo

We are loading images from camera roll to the app, but on iOS we use the standard CameraRoll, not an ANE. We also save PNG's to the camera roll for which we use

https://github.com/katopz/ane-photosalbum

(since it's still not possible to save PNG's on iOS with CameraRoll...). And looking at the code, we're actually clearing the PNG bytearray as soon as we have passed it to the ANE. Maybe this is something we should take a look at...

Thanks for the tips anyways ma! 🙂