Minor Support Ticket Incident (RESOLVED)

Today around 13:40pm Finnish time (UTC+2) a very small number of our users’ Zendesk support tickets were accidentally merged together due to a human error.

Due to privacy reasons, we take these issues very seriously. Not only due to EU GDPR legislation, which the whole Sniffie ecosystem & processes are fully compatible by the way.

Everyone who was affected was personally contacted by Sniffie team by phone (if they were reachable and we happened to know their phone), email and / or support chat.

Read more to see what happened, how the situation was handled and what was done to prevent this from happening again.

Description of what happened

Instead of solving a number of support tickets this afternoon, a Sniffie support person assigned to these tickets accidentally merged tickets together. This was done as a manually triggered bulk operation in Zendesk.

All users that had sent these merged tickets received an email notifying them for this merge. The notification email also contained a chat transcript of one user. This chat transcript contained a short chat between this user and a Sniffie employee. The transcript contained a description of one bug in an extraction and Sniffie support person’s response to it. Other merged users’ emails were a part of the CCs in the newly merged ticket at Zendesk.

The support personnel noticed the issue after one affected user replied to the ticket. Sniffie personnel immediately started fixing the issue and contacting the people affected.

Immediate response by Sniffie team

  1. All CCs within the ticket were removed from the Zendesk ticketing portal.
  2. The merged ticket was deleted from the ticketing system to prevent accidental email chains.
  3. Ticket merging at Zendesk was made impossible for Sniffie support personnel.
  4. Sniffie team started to find out about the people affected. Initially the number of people affected was estimated to be 3. However, a detailed log analysis indicated that a total 5+1 people were affected by this incident, representing less than 1% of people that have been in contact with Sniffie support staff. That is, 5 people erroneously received a support chat log of one user as part of an accidental merge notification.
  5. Sniffie team started to contact the affected people. If the user’s phone number was known, the user was contacted via phone at least 3 times until they answered. If we could not reach them, a personal email was sent to the user describing the issue and containing contact information of Sniffie team should they feel like calling. All affected people were first contacted within the first 60 minutes of the incident occurring. The situation was explained in detail and we apologized to everyone personally either in the phone or in the email.
  6. An additional contact was made later in the afternoon should the user want to talk with Sniffie personnel on the phone again regarding the incident.

Response by Sniffie team after immediate response

  1. The incident was documented.
  2. Support personnel training material was updated to not do bulk operations in Zendesk dashboards without four-eyes principle.
  3. Support personnel training material was updated to include a description why ticket merging is no longer possible at Sniffie support platform.
  4. Support personnel training material was updated to include a specific check for ticket requester and possible CCs every time an update to a support ticket is made. This check is now performed before and after updating any ticket.
  5. An additional training session for all Sniffie support personnel was conducted by me (Niko) after normal working hours to account for the above changes.
  6. This blog post was written as a public notice of the occurred incident, how it was resolved and what actions were taken to make sure such an incident will not happen again.

The whole Sniffie team deeply apologizes for the incident. This will not happen again.

Best regards,
Niko & the Sniffie team

By | 2018-05-24T13:18:42+00:00 April 24th, 2018|General, Sniffie support|

About the Author:

An engineer with an interest in automation, healthcare and mathematics. A strong believer in the idea that any data can be useful as long as it reaches the right person.