MMICrashReporter

Martin Redington

martin@mildmanneredindustries.com

Contents

Introduction

Collecting user feedback is a necessary part of the software development life-cycle. In an ideal world, this feedback would consist solely of praise, gratitude, and PayPal donations.

Sadly, much, if not most, user feedback consists of problem or bug reports of one kind of another. No matter how thorough your testing process, some problems will only appear once your software in the world in the hands of real users.

In the Mac world, developers usually provide an email address or a web form for users to submit initial problem reports.

Having received the initial report, the developer can perform an initial diagnosis of the underlying issue, and whether it is already known, or not.

In many cases however, further information is required to accurately diagnose the problem. The initial report may not contain basic information (such as the application version number or operating system version), or slightly more esoteric information may be needed for specific diagnosis, such as the users' network configuration (as shown by the output of running /sbin/ifconfig -a in the Terminal).

Once the initial problem report is received, further correspondence is usually maintained via email, and several emails back and forth may be required to diagnose a single report, with additional requests for information, and responses on each cycle.

The flow diagram below shows a characterisation of this process.

This process can relatively intensive, both for the developer, and the end-user. Explaining what information is required, and gathering that information, especially where the user is relatively technically unsophisticated, can be especially time-consuming, and in many cases, users, or developers, can simply lose interest in pursuing the issue, or simply get distracted, especially where the application is not critical for the user.

Streamlining Problem Reporting

How could we improve the problem reporting and management process?

Anything we can do to increase the quality and information content of the initial report sent by the user, and to reduce the overhead of asking for (from the support perspective) and providing (from the end-user perspective) further diagnostic information would reduce the burden on, and cost to, both parties.

Interestingly, Apple does provide some automated problem reporting facilities as part of the Mac OS X operating system.

A process called crashreporterd watches the signals that are sent to running applications, and when a signal indicates an application crash, launches the CrashReporter. This collects a crash log, which contains stack traces and other technical information, as well as a profile of the system, and a user provided description of the circumstances leading to the crash, for delivery back to Apple.

This has a few disadvantages:

  1. Reports are sent to Apple (who generally does not pass them on), rather than to the application developer.

  2. Reports are only generated when a crash occurs. While all crashes are bugs, not all bugs lead to crashes.

  3. The information provided is fixed, and relatively limited.

  4. Users never receive any replies to these reports, and even if users send them initially, they will tend to stop sending them after a while.

A number of people have written their own CrashReporters to address the first of these issues.

The most well-known example is Insanity's Smart Crash Reports (see ZonicBugReporter or ILCrashReporter for other examples).

Although these alternative do at least send the report to the application developer, they are generally reimplementations of Apple's CrashReporter with a different delivery address, and still suffer from its remaining shortcomings.

What if we had a CrashReporter that could be easily configured to collect a wider range of information, according to the application developer's specific requirements, in response both to crashes, and user-initiated bug reports, and that also maintained a channel back to the user who originated the report, that could be used to follow up the problem?

MMICrashReporter aims to fill this gap.

MMICrashReporter

MMICrashReporter is a second (or third) generation crash reporter, with the following features:

An Example

Consider an example application, called MMIExampleApp.

We would like to receive both crash reports and normal bug reports for MMIExampleApp.

The default configuration of MMICrashReporter allows us to collect crash reports when MMIExampleApp crashes, and to connect an action in MMIExampleApp's user interface (e.g. a Report Bug item in the Help menu) with bug report collection. In either case, MMICrashReporter will be launched to present the report, and to collect any additional user input.

The default MMICrashReporter crash report looks like this:

This is almost identical to a traditional crash report, except that the user's name and email address are present (these were pre-filled from the user's account setting and Address Book entry). This allows the developer to follow up the problem with the user, and ask for more further diagnostic information if required.

The default MMICrashReporter bug report looks like this:

This is identical to the crash report, except that no crash report field is present.

Already, we have some advantages over current CrashReporters, but in this case of MMIExampleApp, we would also like to collect some additional information.

When MMIExampleApp crashes, we would like to obtain any relevant console log entries.

We simply add a new report item to MMICrashReporter's XML property list configuration, like so: 

<dict>
     <key>name</key>
     <string>ConsoleLogEntries</string>
     <key>label</key>
     <string>Console Log Entries:</string>
     <key>type</key>
     <string>ConsoleLog</string>
     <key>description</key>
     <string>Console Log Entries for MMIExampleApp.</string>
     <key>pattern</key>
     <string>MMIExampleApp</string>
     <key>fixedHeight</key>
     <true/>
 </dict>







The screenshot below shows the resulting crash report window:











In the case of bug reports, we would like to provide a summary field,
where the user will hopefully give us a quick idea of the nature of
the problem, and a severity field, so that we can prioritise our
response to the report (presumably user's will expect, or at least
prefer, a faster response to more serious bugs, and will not mind
waiting a bit longer for a response to a minor or trivial issue).





Note that these are just examples. You can configure MMICrashReporter
however you like, using either the built-in ReportItems, or if they
are insufficient, you can extend MMICrashReporter by writing your own
ReportItem plug-ins.





The XML for the new fields is shown below:









<dict>
    <key>name</key>
    <string>Summary</string>
    <key>label</key>
    <string>Summary:</string>
    <key>type</key>
    <string>TextField</string>
    <key>description</key>
    <string>Please enter a one line summary of your problem here.</string>
</dict>

<dict>
    <key>name</key>
    <string>Severity</string>
    <key>label</key>
    <string>Severity:</string>
    <key>type</key>
    <string>Select</string>
    <key>description</key>
    <string>Please indicate the severity of the bug. Major indicates a crash, data loss or serious problem. Minor indicates something that doesn't interfere with main functionality, but is still a bug. Trivial indicates a minor annoyance.</string>
    <key>options</key>
    <array>
        <string>Unclassified</string>
        <string>Major</string>
        <string>Minor</string>
        <string>Trivial</string>
    </array>
    <key>selected</key>
    <string>Unclassified</string>
</dict>








The resulting bug report window is shown here:











We have now tailored the information in our initial crash or bug
reports to our requirements. Imagine that a problem report has been
received, but that further information is required from the user, in
order to diagnose the issue.





One possibility is for us to email the user:








From: support@mildmanneredindustries.com
To: user@somewhere.com
Subject: Your problem

Dear user,

    in order to diagnose your problem, I need some more information. Could you do the following for me:

In the Terminal, run

/sbin/ifconfig -a 

and cut and paste the output into a mail

Then run

defaults read com.mildmanneredindustries.MMIExampleApp

and send me the output.

Finally, could you send me any MMIExample app entries from the console.log

if you run 

grep MMIEXampleApp /Library/Logs/Console/`id -u`/console.log

in the Terminal, and send me the output

Thanks very much for your help

    MildMannered Industries Support







Many users will fall at this hurdle. Alternatively, using MMICrashReporter, we could send them a slightly different email.








From: support@mildmanneredindustries.com
To: user@somewhere.com
Subject: Your problem

Dear user,

    in order to diagnose your problem, I need some more information. Could you double click on the attached file.

A customised bug report will open, and automatically collect the information I need. Just click send when its done.

Thanks very much for your help

    MildMannered Industries Support

Attachment: CustomReport.crashreporterconfig







Here is the custom report:











As well as the information mentioned in the first email, this report
also contains a bug id. This could be a reference to an identifier in
the developer's bug tracker, allowing the response to be easily (or
ideally, automatically) added to the bug tracker case.





When the user hits send on any of the reports, MMICrashReporter will
attempt to send the report via a delivery agent, to the address
specified in the configuration (you can specify different addresses
for each mode, if you wish). Mail and http delivery agents are
built-in.





The actual report output format is also XML property-list based.  Here
is a sample output file. Although this
format is not especially person friendly, it is very suitable for
automated processing.





Possibilities include using XSLT to transform the output into a
suitable format for viewing in a browser, or to inject it directly
into an issue tracker, or a specialised Cocoa browser. It is likely
that support for some or all of the options will appear in future
distributions of MMICrashReporter.





The current XML output also contains the configuration parameters of
each report item. The intention is that these can be used to help a
report viewer construct a user interface without access to the
original report configuration. The exact output format is subject to
change.




Next Steps




The source and binary for the sample application MMIExampleApp is
included in the distribution. Further detail is available here





A number of sample configuration files are also provided - you can
launch MMICrashReporter with a new bug report by double clicking on
these (if you hold down the option key at the same time, you will get
a crash report instead).





Or you could check out the documentation on integration
and configuration.