tag:status.kindly.ai,2005:/historyKindly Status - Incident History2024-03-28T13:51:07+01:00Kindlytag:status.kindly.ai,2005:Incident/190622382023-11-09T22:28:24+01:002023-11-09T22:28:24+01:00Kindly GPT slow or not responding<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:28</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>21:24</var> CET</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>21:24</var> CET</small><br><strong>Monitoring</strong> - Bots using Kindly GPT will experience a degraded performance linked to issues on OpenAI servers.</p>tag:status.kindly.ai,2005:Incident/190576552023-11-08T16:50:51+01:002023-11-08T16:50:51+01:00Kindly GPT slow or not responding<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:50</var> CET</small><br><strong>Resolved</strong> - OpenAI is back up and bots using Kindly GPT is back in full operation.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>15:07</var> CET</small><br><strong>Identified</strong> - Bots using Kindly GPT will experience a degraded performance linked to issues on OpenAI servers.</p>tag:status.kindly.ai,2005:Incident/188116062023-10-30T23:21:07+01:002023-10-30T23:21:07+01:00Upgrade of task queue backend<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>23:21</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>23:16</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>23:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>11:20</var> CEST</small><br><strong>Scheduled</strong> - We will be upgrading a backend service used for asynchronous task execution. During this time the Kindly platform will be unavailable to bot builders and users of the inbox.<br /><br />Bot messaging will not be affected, end users will still be able to chat with bots during this maintenance period.</p>tag:status.kindly.ai,2005:Incident/172044632023-05-10T13:58:23+02:002023-05-10T14:12:52+02:00Issues with DNS provider<p><small>May <var data-var='date'>10</var>, <var data-var='time'>13:58</var> CEST</small><br><strong>Resolved</strong> - All services are globally available as normal again.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>12:48</var> CEST</small><br><strong>Monitoring</strong> - DNS updates are still being propagated. As this progresses, our services are becoming available again for users who were not able to connect previously.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>11:47</var> CEST</small><br><strong>Update</strong> - DNS servers are progressively coming back up. <br /><br />The outage currently seem to affect about 50% of users.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>10:36</var> CEST</small><br><strong>Identified</strong> - Currently experiencing issues with our DNS provider. Our services might be unreachable for some users.</p>tag:status.kindly.ai,2005:Incident/161766682023-02-17T14:32:04+01:002023-02-17T14:32:04+01:00Platform not responding<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>14:32</var> CET</small><br><strong>Resolved</strong> - Problem should be fixed and resolved.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>14:29</var> CET</small><br><strong>Identified</strong> - Issue is identified. Trying out potential fixes to the problem.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>14:26</var> CET</small><br><strong>Investigating</strong> - We are currently having issues with loading content in the platform. We are working on solving the issue. <br />All chatbots are running as normal.</p>tag:status.kindly.ai,2005:Incident/159826962023-01-31T10:32:02+01:002023-01-31T10:32:02+01:00Bots not responding with Greeting<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:32</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>09:41</var> CET</small><br><strong>Identified</strong> - We are experiencing issues with Greetings and performance. <br />We have a solution to the problem and are working on getting it out.</p>tag:status.kindly.ai,2005:Incident/135535312022-11-19T14:00:00+01:002022-11-21T12:36:04+01:00Issues with custom handover functionality<p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>14:00</var> CET</small><br><strong>Resolved</strong> - We experienced problems relating to the custom handover functionality. The cause was identified as an issue related to message event listeners in Kindly Chat and a fix was rolled out.</p>tag:status.kindly.ai,2005:Incident/127447302022-11-07T21:14:33+01:002022-11-07T21:14:33+01:00Database maintenance<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>21:14</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>21:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>17:25</var> CET</small><br><strong>Scheduled</strong> - app.kindly.ai will be unavailable for short period while we perform a routine database migration. Bot messaging is unaffected.</p>tag:status.kindly.ai,2005:Incident/107583742022-08-04T14:26:54+02:002022-08-04T14:26:54+02:00Analytics availability<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>14:26</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>11:13</var> CEST</small><br><strong>Identified</strong> - The analytics component is having database related issues. Analytics views in the platform are currently unavailable or slow to load. We are working on fixing the performance issue.</p>tag:status.kindly.ai,2005:Incident/95157972022-03-11T13:04:12+01:002022-03-11T13:04:12+01:00Bot is not updated when making changes in Build<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>13:04</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:31</var> CET</small><br><strong>Monitoring</strong> - The queue looks healthy now and updating seems to work, continuing to monitor queue for a while</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:18</var> CET</small><br><strong>Identified</strong> - We have a working theory on why the update queue is struggling and are adjusting some infrastructure to mitigate the pressure</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>10:50</var> CET</small><br><strong>Investigating</strong> - We have received reports that bots aren't registering new dialogues that are added in Build</p>tag:status.kindly.ai,2005:Incident/92356892022-02-10T10:23:00+01:002022-02-10T10:23:00+01:00Facebook app deactivated<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>10:23</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>21:08</var> CET</small><br><strong>Monitoring</strong> - The app has been reactivated by Facebook and bot owners need to reconnect their Facebook pages to start receiving messages again.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>10:38</var> CET</small><br><strong>Identified</strong> - The Facebook app used to connect a Facebook page to a Kindly bot or Inbox has been temporarily deactivated. This means messages from users are not forwarded to bots/Inbox at the moment. Once the app has been reactivated by Meta, the messages will start flowing again.<br /><br />We are waiting for the app to be re-reviewed and reactivated by Meta after we have supplied the necessary information required to perform a review.</p>tag:status.kindly.ai,2005:Incident/88560332021-12-15T10:30:00+01:002021-12-15T17:10:24+01:00Plugin: Zendesk ticket<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>10:30</var> CET</small><br><strong>Resolved</strong> - We have identified an issue which affected users of the Zendesk ticket plugin and a fix has been deployed. Please review chats after ~10:30 CET making sure your customers receive the proper response.<br /><br />If investigating webhook responses in the debug console to verify a proper response, look in the _request body_ for `exchange_slug: zendesk_succcess`. Please note that even though the plugin returns HTTP 200 OK, it does not mean the ticket has been successfully submitted.</p>tag:status.kindly.ai,2005:Incident/85972412021-11-16T18:30:00+01:002021-11-19T14:33:35+01:00Google Cloud major outage<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>18:30</var> CET</small><br><strong>Resolved</strong> - Our cloud provider had a major outage on their network for the eu and us regions. The effect of this was that all our bot delivery services and platform interface was unavailable and unresponsive for about 20 minutes.<br /><br />Link to their incident report can be found here: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh</p>tag:status.kindly.ai,2005:Incident/79617742021-09-10T18:10:39+02:002021-09-10T18:10:39+02:00Message queue provider network issue causing spurious errors<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>18:10</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>17:46</var> CEST</small><br><strong>Monitoring</strong> - The network issue with our message queue provider should be fixed and we are monitoring the situation.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>12:05</var> CEST</small><br><strong>Identified</strong> - We are experiencing a network related issue with our message queue provider. This is causing spurious errors on app.kindly.ai when performing actions like creating or updating a dialogue, logging in, etc (these all write to the message queue). We are working on remediating the issue</p>tag:status.kindly.ai,2005:Incident/73867572021-07-02T16:52:38+02:002022-11-28T17:25:35+01:00Analytics service availability<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>16:52</var> CEST</small><br><strong>Resolved</strong> - We have deployed several changes to fix a performance regression after a database upgrade. We are continuing to implement improvements even after the performance of the statistics component have stabilized.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>13:37</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:09</var> CEST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:06</var> CEST</small><br><strong>Investigating</strong> - The analytics component is having database related issues. Analytics in the platform will be unavailable until this is resolved.</p>tag:status.kindly.ai,2005:Incident/70294702021-05-18T20:00:36+02:002021-05-18T20:00:36+02:00Database maintainance<p><small>May <var data-var='date'>18</var>, <var data-var='time'>20:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>19:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>12:42</var> CEST</small><br><strong>Scheduled</strong> - app.kindly.ai will be unavailable for short period while we perform a routine database migration. Bot messaging is unaffected.</p>tag:status.kindly.ai,2005:Incident/69820682021-05-12T20:00:35+02:002021-05-12T20:00:35+02:00Database maintenance<p><small>May <var data-var='date'>12</var>, <var data-var='time'>20:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>19:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>12:49</var> CEST</small><br><strong>Scheduled</strong> - app.kindly.ai will be unavailable for short period while we perform a routine database migration. Bot messaging is unaffected.</p>tag:status.kindly.ai,2005:Incident/61353442021-01-29T18:00:14+01:002021-02-01T13:00:44+01:00Bot not responding in some cases<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>18:00</var> CET</small><br><strong>Resolved</strong> - Immediate issues with bots having Draft & Publish enabled are resolved. We are continuing to improve this features operational reliability going forward.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>17:25</var> CET</small><br><strong>Identified</strong> - Issue has been identified to do with bots that had Draft & Publish enabled. The affected bots are now operational again.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>14:58</var> CET</small><br><strong>Investigating</strong> - Some users are experiencing lack of response from the bot. <br />We are currently investigating the cause of this.</p>tag:status.kindly.ai,2005:Incident/61258082021-01-27T12:53:27+01:002021-01-27T15:51:57+01:00Issue accessing bots<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>12:53</var> CET</small><br><strong>Resolved</strong> - The incident has been resolved. The cause of failed requests was a non-production database migration unintentionally run in the production environment.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>12:23</var> CET</small><br><strong>Monitoring</strong> - The issue has been resolved and app.kindly.ai is accessible again</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>11:53</var> CET</small><br><strong>Identified</strong> - The issue has been identified and we are rolling out a fix. app.kindly.ai is currently set in maintenance mode to avoid data inconsistencies.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>11:42</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.kindly.ai,2005:Incident/53568912020-11-01T18:00:25+01:002020-11-19T15:59:45+01:00Incomplete statistics<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>18:00</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>03:20</var> CEST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>03:19</var> CEST</small><br><strong>Identified</strong> - New events are now being processed and fresh statistics should be available again. We are working on backfilling unprocessed events between ca 2020-10-18 21:00 and 2020-10-19 03:00 UTC+2.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>01:49</var> CEST</small><br><strong>Investigating</strong> - After migrating our services we are experiencing issues with inserting events into our statistics database resulting in incomplete data from around 2020-10-18 21:00 and later. We are working on restoring these events and allowing fresh data to be visible on app.kindly.ai dashboards and directly from sage.kindly.ai.</p>tag:status.kindly.ai,2005:Incident/52606242020-10-19T01:00:29+02:002020-10-19T01:00:29+02:00Google Cloud migration<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>01:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>19:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>16:27</var> CEST</small><br><strong>Scheduled</strong> - We are moving to Google Cloud for our core services to improve scalability, stability and cost to name a few.<br /><br />- Agent inbox and content management of bots from app.kindly.ai will be unavailable.<br />- All bots will be operational.<br />- Analytics dashboards will show stale data for a little while after the migration.</p>tag:status.kindly.ai,2005:Incident/49607422020-08-28T23:57:59+02:002020-08-28T23:57:59+02:00E-mail domain switch<p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>23:57</var> CEST</small><br><strong>Completed</strong> - The domain switch is now complete and you can now reach us at support@kindly.ai (or even at support@convertelligence.no for a while longer)</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>18:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>16:27</var> CEST</small><br><strong>Scheduled</strong> - We are switching our primary domain to kindly.ai, from convertelligence.no. Email sent to either support@kindly.ai or support@convertelligence.no may not be delivered during the maintenance window. At least one of the support email address will be available during the window, so send to both support@kindly.ai and support@convertelligence.no to ensure delivery.</p>tag:status.kindly.ai,2005:Incident/46110912020-07-14T14:36:00+02:002020-07-14T15:23:51+02:00Database outage<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>14:36</var> CEST</small><br><strong>Resolved</strong> - At 14:36 CEST we were alerted that chatbots were unresponsive. We found that one of our critical databases (Heroku Redis on AWS) had become unavailable. While we were still investigating, at 14:52 the service became available again and chatbots resumed working.<br /><br />We are monitoring the corresponding incident at Heroku (https://status.heroku.com/incidents/2084) to determine what happened to the database and how it can be mitigated in the future.</p>tag:status.kindly.ai,2005:Incident/44296592020-06-23T21:58:25+02:002020-06-23T21:58:25+02:00Organization upgrade<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>21:58</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>21:50</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>19:01</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>08:07</var> CEST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>12:31</var> CEST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>12:29</var> CEST</small><br><strong>Scheduled</strong> - We will be releasing an update of the Organization structure in Kindly, including some changes in the app layout. After this it should be much easier to organize your team!</p>tag:status.kindly.ai,2005:Incident/36730892020-03-17T00:00:29+01:002020-03-17T00:00:29+01:00Message store upgrade<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>00:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>22:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>14:00</var> CET</small><br><strong>Scheduled</strong> - We will perform an upgrade of our message store to support future increased message volumes and load. This should not affect bot uptime.</p>