Duration: 30 minutes

Affected Components:

Oct 30, 12:20:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 30, 11:58:31 GMT+0
Identified - Maintenance has completed successfully..

Oct 30, 11:50:01 GMT+0
Identified - Maintenance is now in progress.

Oct 30, 11:45:51 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 6 days, 22 hours and 21 minutes

Affected Components:

Oct 29, 11:01:22 GMT+0
Resolved - This incident has been resolved..

Oct 22, 12:40:04 GMT+0
Identified - We've identified the cause of this issue and are working on a fix..

Duration: 3 hours and 50 minutes

Affected Components:

Oct 22, 12:50:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 22, 11:57:10 GMT+0
Identified - We're continuing to work on this migration..

Oct 22, 10:33:15 GMT+0
Identified - We're continuing to work on this migration. We would like to thank everyone for their patience..

Oct 22, 09:20:45 GMT+0
Identified - Migration beginning.

Oct 22, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 21, 12:52:22 GMT+0
Identified - We're migrating TingleFlow's direct messaging media to a new content delivery service (CDN) as part of our ongoing infrastructure improvements. This migration is intended to improve how media shared through direct and group messages is delivered and accessed. During the migration, some messaging media may be temporarily unavailable or experience interruptions while content is transferred and the new delivery configuration is put in place. We'll provide updates as the migration progresses and confirm once the process is complete..

Duration: 3 hours and 44 minutes

Affected Components:

Oct 21, 12:44:18 GMT+0
Completed - Maintenance has completed successfully..

Oct 21, 12:27:14 GMT+0
Identified - **We expect maintenance to be completed within the next 10-20 minutes.**.

Oct 21, 11:47:37 GMT+0
Identified - We ran into an issue and need to extend the maintenance by 30 minutes after the expected finish time. We're incredibly sorry about this inconvenience..

Oct 21, 11:16:54 GMT+0
Identified - Migration is still underway..

Oct 21, 10:40:25 GMT+0
Identified - We continue to work on the migration of our CDN.

Oct 21, 10:20:59 GMT+0
Identified - The migration is now underway. We're beginning to transfer TingleFlow's content to our new content delivery service. We'll provide further updates as the upgrade progresses..

Oct 21, 09:00:01 GMT+0
Identified - **TingleFlow is currently undergoing scheduled infrastructure maintenance and is temporarily unavailable to all users.** The maintenance period has officially begun as we proceed with migrating TingleFlow's content to a new content delivery service (CDN). This migration is part of our ongoing infrastructure improvements and involves transitioning our content delivery systems to new infrastructure. During this time, access to TingleFlow and its features will be unavailable while the migration is carried out. This includes accessing the platform, watching content, interacting with creators, and using other TingleFlow services that depend on the affected infrastructure. Our team is working to complete the migration as smoothly as possible. Once the transition is complete, we will verify that the relevant systems are operating as expected before restoring access to the platform. We understand that this interruption may be inconvenient, particularly for users who regularly visit TingleFlow to watch content or interact with the community. We apologise for the disruption and appreciate your patience while we carry out this work. **What happens next?** We will continue to publish updates on this page as maintenance progresses. Once the migration has been completed and TingleFlow is available again, we will post a final update confirming that service has been restored. At this stage, please refer to this status page for the latest information regarding the maintenance. We recommend checking back periodically for updates rather than repeatedly attempting to access the platform while the maintenance is ongoing. Thank you for your patience and understanding as we continue improving the infrastructure that powers TingleFlow..

Oct 21, 08:50:03 GMT+0
Identified - Scheduled maintenance is due to begin in approximately 10 minutes. During this time, TingleFlow will be temporarily unavailable to all users while we carry out essential infrastructure improvements. We apologise for any inconvenience this may cause and appreciate your patience while we work to improve TingleFlow. Please continue to monitor this status page for further updates..

Oct 21, 01:00:11 GMT+0
Identified - TingleFlow will be temporarily unavailable to all users while we migrate our content to a new content delivery service (CDN). This migration is part of our ongoing infrastructure improvements and is intended to improve how content is delivered across the platform. During the migration, TingleFlow and most of its features will be inaccessible until the process is complete. We will post updates on this page as the migration progresses and confirm once all services have been restored. _We encourage all users to learn more about this here:_ <https://help.tingleflow.com/hc/en-us/platform/infrastructure-improvements-at-tingleflow> Thank you for your patience while we carry out this maintenance..

Duration: 13 hours and 18 minutes

Affected Components:

Oct 20, 22:44:24 GMT+0
Resolved - This incident has been resolved..

Oct 20, 11:49:07 GMT+0
Monitoring - A fix has been implemented and we are seeing excellent recovery. Some services may load slower than usual..

Oct 20, 09:26:25 GMT+0
Investigating - Users connecting to TingleFlow through Google Cloud’s London region (europe-west2) may be unable to connect. Google has acknowledged the issue, and we’ll provide updates as more information becomes available. They have not yet provided any time estimate for recovery. We're keeping an eye on this and we apologise for the inconvenience this is causing. This is an issue with a third-party platform which TingleFlow has no control of..

Duration: 1 hour and 40 minutes

Affected Components:

Oct 20, 00:59:00 GMT+0
Resolved - This incident has been resolved..

Oct 19, 23:36:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 27 minutes

Affected Components:

Oct 19, 00:28:53 GMT+0
Resolved - This incident has been resolved..

Oct 19, 00:24:40 GMT+0
Monitoring - We have deployed a fix and are seeing significant recovery..

Oct 19, 00:02:22 GMT+0
Investigating - We are currently investigating this incident. Pages may show errors and some videos may display as having no views or likes. We're working on a fix for this issue..

Duration: 11 minutes

Affected Components:

Oct 16, 22:28:05 GMT+0
Resolved - TingleFlow has returned to a healthy state. Incident resolved..

Oct 16, 22:25:16 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Oct 16, 22:16:44 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components:

Oct 12, 10:32:45 GMT+0
Postmortem - This was a deployment issue. We reversed it and it mitigated all errors..

Oct 12, 10:07:23 GMT+0
Resolved - This incident has been resolved..

Oct 12, 10:04:38 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. The site is returning to normal..

Oct 12, 10:03:03 GMT+0
Identified - We have identified the root cause of the issue and are deploying a fix.

Oct 12, 09:57:21 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 54 minutes

Affected Components:

Oct 10, 00:30:00 GMT+0
Resolved - This incident has been resolved..

Oct 9, 23:12:45 GMT+0
Identified - We have identified the issue and are working on a fix..

Oct 9, 22:36:21 GMT+0
Investigating - We are currently investigating this incident. Some aspects of the website are experiencing issues and may not load..

Affected Components:

Oct 5, 06:38:38 GMT+0
Resolved - Help Center is back up. This incident was automatically resolved by Instatus monitoring..

Oct 4, 16:58:00 GMT+0
Investigating - We are continuing to investigate this issue..

Oct 4, 12:24:08 GMT+0
Investigating - Help Center is down at the moment. This incident was automatically created by Instatus monitoring..

Duration: 21 minutes

Affected Components:

Oct 4, 11:53:19 GMT+0
Resolved - Scheduled maintenance has been completed..

Oct 4, 11:40:14 GMT+0
Identified - Scheduled maintenance has begun..

Oct 4, 11:32:05 GMT+0
Identified - This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

Duration: 36 minutes

Affected Components:

Sep 16, 11:17:40 GMT+0
Resolved - This incident has been resolved..

Sep 16, 10:54:53 GMT+0
Monitoring - We implemented a fix and the website appears to be operational again. We will continue to monitor..

Sep 16, 10:51:34 GMT+0
Identified - We have identified the issue and are working on a fix..

Sep 16, 10:41:33 GMT+0
Investigating - We are investigating an issue where users may see errors while connecting to Tingleflow, Tingleflow creator hub, Tingleflow ad manager, and help center. Stay tuned..

Duration: 1 hour and 20 minutes

Affected Components:

Sep 13, 14:29:32 GMT+0
Completed - Maintenance has completed successfully..

Sep 13, 13:10:01 GMT+0
Identified - Maintenance is now in progress.

Sep 13, 12:53:14 GMT+0
Identified - This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

Duration: 2 hours and 23 minutes

Affected Components:

Aug 20, 10:21:05 GMT+0
Resolved - API is fixed..

Aug 20, 08:30:52 GMT+0
Monitoring - ### All regions except for places with server delays are now back online. ### North America Central America South America (Excludes all servers in Rio de Janeiro (BR) and Bogotá (CO) due to server delay, expected return is 20 minutes, users may be routed through servers in São Paulo (BR) or Buenos Aires (AR) or any other smaller servers that can handle extra traffic. Europe Africa & Middle East Asia (excluding Southeast Asia, expected return 20 minutes) Oceania (excluding servers in Melbourne (AU) and Sydney (AU) due to server delay, expected return is 10 minutes, sers may be routed through servers in Auckland (NZ) or any other smaller servers that can handle extra traffic..

Aug 20, 08:23:02 GMT+0
Monitoring - The following regions are now back online: North America Central America South America (Excludes all servers in Rio de Janeiro (BR) and Bogotá (CO) due to server delay, expected return is 20 minutes, users may be routed through servers in São Paulo (BR) or Buenos Aires (AR) or any other smaller servers that can handle extra traffic. More regions will be returning shortly..

Aug 20, 08:11:42 GMT+0
Identified - We have identified the issue and are working on a fix. Our ETA is currently quite broad as propagation can take up to 48 hours, we do not believe it will take this long but we will keep you updated as locations come back online..

Aug 20, 07:58:13 GMT+0
Investigating - We are currently investigating this incident. Users may be unable to connect at this time..

Duration: 15 minutes

Affected Components:

Aug 15, 07:39:30 GMT+0
Completed - Maintenance has completed successfully..

Aug 15, 07:20:01 GMT+0
Identified - Maintenance is now in progress.

Aug 15, 07:15:53 GMT+0
Identified - We're occasionally undergo maintenance to make sure TingleFlow is constantly up to date and secure..

Duration: 3 hours and 1 minute

Affected Components:

Aug 15, 05:33:19 GMT+0
Resolved - This incident has been resolved..

Aug 15, 04:47:45 GMT+0
Monitoring - System appears to be operating as expected..

Aug 15, 02:32:26 GMT+0
Investigating - We are currently investigating this incident..

Duration: 22 minutes

Affected Components:

Aug 14, 12:20:25 GMT+0
Resolved - This incident has been resolved..

Aug 14, 12:15:07 GMT+0
Monitoring - Traffic is now returning to normal..

Aug 14, 12:12:23 GMT+0
Monitoring - Help center and contacts are now back up. Website should return within the next 15 minutes..

Aug 14, 12:10:02 GMT+0
Identified - We are seeing the system stabilizing and will begin letting traffic in incrementally shortly. Our customer service desk is also returning, if you have opened a ticket, please re-send it to us when this page states everything is operational..

Aug 14, 11:58:48 GMT+0
Investigating - Users may experience issues connecting to TingleFlow. Access is currently limited due to a system bottleneck. We are investigating the issue and working to resolve it..

Duration: 28 minutes

Affected Components:

Aug 14, 01:30:00 GMT+0
Resolved - This incident has been resolved..

Aug 14, 01:25:27 GMT+0
Monitoring - We reroute traffic and see improvement. some user may have access again.

Aug 14, 01:17:25 GMT+0
Identified - We identify the root cause as third party issue and we reach out to them to get updates.

Aug 14, 01:01:47 GMT+0
Investigating - Multiple systems are impacted. stay tuned..

Duration: 11 minutes

Affected Components:

Aug 3, 13:00:02 GMT+0
Resolved - This incident has been resolved..

Aug 3, 12:56:47 GMT+0
Monitoring - Users are now being routed through AU-823 again. We will continue monitoring the system closely to ensure everything remains stable and is operating as expected..

Aug 3, 12:48:47 GMT+0
Identified - We are performing an emergency maintenance on AU server 823, located in the Sydney, Australia. Users routed through there could be temporarily inaccessible or might load slower than usual. Our team is currently working to restore normal performance levels. We apologize for any inconvenience caused..

Duration: 2 days, 22 hours and 48 minutes

Affected Components:

Jul 24, 10:15:22 GMT+0
Resolved - All users now have access to TingleFlow, and our systems remain stable. We sincerely apologize for the length of this outage. It was a highly unusual incident due to both its complexity and the nature of the underlying issue. Over the coming days, we will conduct a thorough review to understand exactly what went wrong, identify areas for improvement, and help prevent similar incidents from happening again..

Jul 24, 05:06:15 GMT+0
Monitoring - We have begun allowing traffic back into TingleFlow incrementally. Some users will regain access before others as we carefully monitor system stability and performance. Users may regain access in the following priority order: 1. TingleFlow+ Subscribers (Creators) 2. TingleFlow+ Subscribers (Viewers) 3. Creators (users with two or more uploaded videos) 4. Viewers To further reduce system load, access will also be rolled out gradually by geographic region. This means users within the same category may regain access at different times. Regional rollout order: 1. North America 2. South America 3. Africa 4. Europe 5. Asia 6. Oceania This phased rollout helps us closely monitor system performance and ensure the platform remains stable as users regain access. We appreciate everyone's patience while we complete the recovery..

Jul 23, 23:25:23 GMT+0
Monitoring - It appears that the fix has been successful, and our systems are returning to normal. We expect to begin allowing users back onto TingleFlow incrementally within the next few hours. To help ensure a stable recovery and avoid overloading our infrastructure, we will gradually restore access using DNS routing. We will share more information on which users will gain access first when we're ready to start letting users access TingleFlow again..

Jul 23, 11:20:31 GMT+0
Monitoring - We believe that we have deployed an appropriate fix. We will ensure it is working as expected, then we will be able to let traffic in incrementally if all goes well. .

Jul 23, 03:59:43 GMT+0
Identified - We have identified the exact root cause of the issue and have began working on a solution. Our team is actively working to restore all services, and we'll continue to provide updates throughout the day..

Jul 23, 00:47:11 GMT+0
Identified - We are continuing to work on a fix for this incident. .

Jul 22, 12:28:42 GMT+0
Identified - We **believe** we have identified the internal issue that contributed to the TingleFlow outage. We are currently carrying out the required fixes, testing, and maintenance work to safely restore services. Thank you for your continued patience and understanding while we work to bring TingleFlow back online..

Jul 22, 11:10:30 GMT+0
Investigating - ## TingleFlow has released a statement regarding this outage. You can read it here: <https://blog.tingleflow.com/tingleflow-addresses-unplanned-service-outage/>.

Jul 22, 09:12:25 GMT+0
Investigating - We believe we have narrowed down the issue and are very close to identifying it..

Jul 22, 07:25:06 GMT+0
Investigating - We are continuing to investigate the root cause of this outage. We will provide another update when available. We believe we are close to identifying the root cause..

Jul 22, 05:15:52 GMT+0
Investigating - We are continuing to investigate the root cause of this outage. We will provide another update when available..

Jul 22, 04:04:01 GMT+0
Investigating - The engineering team has dug deeper into debug logs and operating system-level metrics to better understand the issue. This data showed that Consul KV writes were getting blocked for long periods of time, resulting in what is known as “contention.” The cause of the contention is not currently obvious, though one theory is that the shift from 64 to 128 CPU Core servers early in the outage may have made the problem worse. After reviewing the HTOP data and performance debugging data shown in the screenshots below, we have concluded that it is ideal to return to 64 Core servers, similar to those used before the outage. We have begun preparing the hardware for this transition. Consul is installed, operating system configurations have been triple checked, and the machines have been readied for service in as detailed a manner as possible. We will then transition the Consul cluster back to 64 CPU Core servers. This will be our fourth attempt at diagnosing the root cause of the incident..

Jul 22, 02:17:14 GMT+0
Investigating - We are continuing to look for the root cause of this issue. We deeply apologise for the inconvenience caused during this incident..

Jul 22, 00:51:55 GMT+0
Investigating - At this point, it is now clear to us that overall Consul usage is not the only contributing factor to the performance degradation. Given this realization, we will again pivot. Instead of looking at Consul from the perspective of the TingleFlow services that depend on it, we will start looking at Consul internals for clues..

Jul 21, 23:47:31 GMT+0
Investigating - We are currently looking into whether hardware failure may be a part of the issue. However, faster hardware hasn't helped and, as we now know, likely hurt stability. Resetting Consul’s internal state hasn't helped either. There is no user traffic coming in, yet Consul is still slow. We have again leveraged iptables to let traffic back into the cluster slowly. We are looking into whether the cluster simply getting pushed back into an unhealthy state by the sheer volume of thousands of containers trying to reconnect has caused this. This was our third attempt at diagnosing the root cause of the incident**, no cause has yet been identified.** ### **We have ruled out:** * DDoS/Cyber attack * Frontend issues * Authentication issues Our next step is to reduce Consul usage and then systematically reintroduce it. **_TingleFlow also currently reports 0 users online, for the first time._**.

Jul 21, 22:50:02 GMT+0
Investigating - The reset went smoothly, and initially, the metrics looked good. When we removed the iptables block, the service discovery and health check load from the internal services returned as expected. However, Consul performance began to degrade again, and now we are back to where we started: 50th percentile on KV write operations was back at 4 seconds. Services and systems that depended on Consul are now starting to mark themselves “unhealthy.” The system has fell back into the problematic state. There is clearly something about our load on Consul that is causing problems, and over 12 hours into the incident, we still are unsure as to what it is..

Jul 21, 21:34:56 GMT+0
Investigating - We expected that restoring from a snapshot taken when the system was healthy would bring the cluster into a healthy state, but we had one additional concern. Even though TingleFlow did not have any user-generated traffic flowing through the system at this point, internal TingleFlow services and systems were still live and reaching out to Consul to learn the location of their dependencies and to update their health information. These reads and writes were generating a significant load on the cluster. We were worried that this load might immediately push the cluster back into an unhealthy state even if the cluster reset was successful. To address this concern, we configured iptables on the cluster to block access. This would allow us to bring the cluster back up in a controlled way and help us understand if the load we were putting on Consul independent of user traffic was part of the problem..

Jul 21, 12:20:41 GMT+0
Investigating - Based on the severity of this incident, we have changed the status to Critical. We cannot yet provide an ETA but we encourage users to stay informed here..

Jul 21, 11:27:05 GMT+0
Investigating - We are currently investigating this incident. We will provide updates when they become available..

Duration: 1 hour and 3 minutes

Affected Components:

Jul 20, 00:27:49 GMT+0
Resolved - This incident has been resolved..

Jul 20, 00:05:56 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 19, 23:59:54 GMT+0
Identified - We believe we have identified the cause of this issue and we will begin working on a fix..

Jul 19, 23:24:41 GMT+0
Investigating - We are currently investigating this incident..

Duration: 6 hours and 25 minutes

Affected Components:

Jun 1, 07:12:00 GMT+0
Resolved - This incident has been resolved..

Jun 1, 01:49:50 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jun 1, 01:38:03 GMT+0
Investigating - We are currently investigating this incident..

Duration: 8 minutes

Affected Components:

May 30, 11:22:39 GMT+0
Completed - Maintenance has completed successfully..

May 30, 11:15:01 GMT+0
Identified - Maintenance is now in progress.

May 30, 11:15:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Some services may be unavailable..

Duration: 12 hours and 21 minutes

Affected Components:

Apr 4, 00:43:10 GMT+0
Resolved - This incident has been resolved..

Apr 3, 12:22:38 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 hours

Affected Components:

Mar 26, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 26, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 26, 11:00:00 GMT+0
Identified - This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

Duration: 7 hours and 14 minutes

Affected Components:

Mar 23, 21:40:51 GMT+0
Resolved - This incident has been resolved..

Mar 23, 14:26:21 GMT+0
Investigating - We are currently investigating this incident..

Duration: 26 minutes

Affected Components:

Mar 23, 11:52:44 GMT+0
Resolved - This incident has been resolved..

Mar 23, 11:41:13 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 23, 11:31:57 GMT+0
Identified - Identified as data streaming issue..

Mar 23, 11:27:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 30 minutes

Affected Components:

Mar 21, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 21, 13:30:01 GMT+0
Identified - Maintenance is now in progress.

Mar 21, 13:30:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time..

Duration: 10 hours and 53 minutes

Affected Components:

Mar 18, 23:28:19 GMT+0
Resolved - This incident has been resolved..

Mar 18, 13:59:26 GMT+0
Monitoring - The issues have resided and systems are operating nominally or in clear recovery. We will continue to monitor. We are letting users back in incrementally..

Mar 18, 12:35:48 GMT+0
Investigating - We are aware of the issues impacting our services and are working on mitigations..

Duration: 5 hours and 52 minutes

Affected Components:

Mar 14, 07:31:41 GMT+0
Resolved - This incident has been resolved..

Mar 14, 01:39:18 GMT+0
Investigating - We are currently investigating this incident. **A small number of videos may not play.**.

Duration: 13 days, 22 hours and 8 minutes

Affected Components:

Mar 25, 09:14:37 GMT+0
Resolved - This incident has been resolved..

Mar 11, 11:06:31 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours

Affected Components:

Mar 9, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 9, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 9, 01:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Access to TingleFlow may be slow or disrupted during this time. This will affect users that connect to TingleFlow via the Singapore region..

Duration: 3 days, 13 hours and 10 minutes

Affected Components:

Mar 8, 01:37:50 GMT+0
Resolved - This incident has been resolved..

Mar 5, 08:54:25 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 4, 12:27:22 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components:

Dec 30, 23:24:56 GMT+0
Completed - Maintenance has completed successfully..

Dec 30, 23:15:01 GMT+0
Identified - Maintenance is now in progress.

Dec 30, 23:15:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 26 days, 2 hours and 48 minutes

Affected Components:

Oct 29, 21:10:00 GMT+0
Completed - Maintenance has completed successfully..

Oct 29, 21:00:00 GMT+0
Identified - You may experience intermittent issues while maintenance is taking place..