Current network status and maintenance

Updates

• Sep 7, 2026 9:01 AM SAST

  Our ongoing efforts over the weekend have resulted in improvements to email services.
  Many affected customers should now experience improved performance. However, some customers may continue to experience intermittent issues.

  If you are completely unable to access your email, please report this to us using the following form so that our team can investigate:

  https://email-report.xneelo.co.za

  Our engineers continue to monitor the environment closely and are working to restore normal service for all affected customers.

  We will provide our next update at 14h00 SAST today.

  Identified
• Sep 6, 2026 1:05 PM SAST (19 hours and 55 minutes earlier)

  We are on day six of the email service disruption affecting some of our Web Hosting customers on shared servers in Johannesburg.

  We know how serious and disruptive this has been for our customers. This is an unprecedented and complex incident in our environment and continues to have the full focus of our engineering and leadership teams.

  Below is a clearer breakdown of how we got here, what we have done so far, and where things stand today.

  How we got here:
  Earlier this year, we started the migration of our shared web hosting platform from bare-metal servers to our virtualised cloud environment.

  Our new virtualised environment consists of two different storage pools optimised for high-intensity web workloads and standard email workloads. We prioritised the web workloads on premium storage.

  This was designed to improve performance and scalability and has been operating successfully.

  In August, we rolled out a major operating-system (Debian Linux) upgrade across our entire managed hosting platform (approximately 4,000 servers).

  This upgrade followed our tried and tested process:
  - Extensive development and testing on test servers.
  - Production rollout to a small number of customer hosting servers.
  - Monitoring the environment and customer feedback for any observed issues.
  - Proceeding with increasing batch sizes, while continuing to monitor for issues between batches.

  At the time of initiating the upgrade on our cloud-based shared hosting servers, we had completed the upgrades of approximately 3,400 dedicated and shared servers.
  - No issues were observed or reported during these upgrades.
  - During the weekend beginning 28 August, we continued with upgrades on our cloud-based shared hosting servers, with no observed or reported issues on Monday 31 August.

  On Tuesday,1 September, it became clear that something was wrong.

  The underlining issue:
  Our storage infrastructure includes redundancy and failover for hardware failures. This incident is different: it is caused by software-level behaviour introduced by the operating-system upgrade and was present across the affected environment. Moving workloads between redundant hardware therefore could not bypass the underlying bottleneck.

  An ongoing investigation has identified that the recent operating system upgrade introduced a fundamental change in how servers save data to and read data from our Cloud storage drives.

  Why we didn’t enact a roll-back:
  We did not roll back the operating system because the security risks and potential impact of running on end-of-life software were too high.

  What we have done:
  As soon as the scale and impact of the degradation became apparent, we initiated our incident response process.

  Because there was no single, obvious failure, such as a hardware fault, and the issue only emerged under full production load, identifying the cause was complex.

  Our engineers had to work systematically through a number of interrelated possibilities, ruling them out one by one while protecting the integrity of customer email and data.

  This troubleshooting phase took significant time. Several interventions addressed individual symptoms or reduced pressure on the environment, but did not fully resolve the underlying storage bottleneck.

  Once we had isolated storage performance as the core constraint and established that moving servers could safely relieve pressure on the affected environment, we began migrating affected servers to a premium storage tier.

  We also introduced new controls to restrict resource-intensive background processes and moved some logging and mailbox-management workloads onto the premium storage tier.

  Moving a server requires very large volumes of live email data to be transferred safely. These migrations themselves place additional demand on the storage infrastructure, so they have had to be completed carefully and in a controlled sequence to ensure data integrity and avoid any risk of email loss. This careful migration process has accounted for much of the recovery time.

  We have now successfully migrated 50% of the affected servers to more premium storage. Because the impact was widespread across the affected environment, we did not prioritise specific servers for migration.

  Our premium storage tier has finite capacity. We are therefore not able to migrate any more servers to this premium tier with the capacity currently available.

  These migrations have removed a significant amount of demand from the affected storage environment. We expected the reduction in load to improve email performance for customers whose servers remain on the standard storage tier as well.

  What remains
  Customers hosted on servers that have been migrated to premium storage have had their email services restored.

  Approximately 50% of affected servers remain on the standard storage tier and cannot currently be migrated to premium storage.

  We are continuing to tune the operating system and storage environment while monitoring email delivery, connection performance and the processing of queued mail.

  Some customers may therefore continue to experience delayed email delivery, connection timeouts or unreliable mailbox access while this work continues.

  We have no evidence that customer email data has been lost. Delayed email remains securely queued and is processed as storage performance allows.

  When will email service be fully restored?
  We know this is the question our affected customers most need us to answer.

  We recognise that after six days without reliable email, another update without a restoration time is extremely difficult. At this stage, we cannot provide an ETA that we can confidently stand behind.

  The work completed so far has reduced pressure on the affected storage infrastructure. We now need to confirm that this translates into sustained, reliable email performance for customers whose servers remain on the standard storage tier as email activity increases.

  Testing continues today using simulating busy workloads to assess whether our recovery measures can support the demands of a busy business day.

  Our teams remain fully focused on restoring reliable email service. We are deeply sorry for the duration of this incident and the impact it has had on you and your business.

  We will provide our next update by 09h00 SAST on Monday, 7 September 2026.

  Identified
• Sep 5, 2026 5:42 PM SAST (19 hours and 23 minutes earlier)

  Our engineering team is making progress while continuing to troubleshoot and closely monitor performance of those servers not yet migrated. We have processed a significant volume of queued email, although delivery backlogs remain.

  Temporary suspensions of email access may still occur on specific servers while this work continues. Incoming email will remain securely queued for delivery, with no loss of email, and website functionality will not be affected.

  Our next update will be Sunday, 15h00 SAST.

  Identified
• Sep 5, 2026 11:58 AM SAST (5 hours and 43 minutes earlier)

  Our engineering team continues to progress with the remaining server migrations and closely monitor performance.

  Temporary suspensions of email access may still occur on specific servers while migrations continue. Incoming email will remain securely queued for delivery, with no loss of email, and website functionality will not be affected.

  We will no longer provide hourly updates. The next update will be provided at 17:00 SAST.

  Identified
• Sep 5, 2026 9:02 AM SAST (2 hours and 56 minutes earlier)

  Our engineering team made significant progress overnight and continues to work on and closely monitor our servers.

  To complete the remaining migrations safely and efficiently, we will temporarily suspend email access on specific servers at intervals throughout the day.

  During these periods, email clients and Webmail will be unable to connect. Incoming email will remain securely queued and will be delivered once each migration is complete, with no loss of email.

  Website functionality will not be affected.

  We will provide the next update at 12:00 SAST.

  Identified
• Sep 4, 2026 4:46 PM SAST (16 hours and 16 minutes earlier)

  Our engineering teams will work throughout the weekend to migrate the remaining servers to our high-performance storage tier, with the goal of restoring stable email services before Monday morning.

  To complete these migrations safely and efficiently, we may need to temporarily suspend email access on specific servers. During this time, email clients and Webmail will be unable to connect, but incoming email will remain securely queued and will be delivered once the migration is complete.

  We will only take this step if absolutely necessary.

  Identified
• Sep 4, 2026 2:34 PM SAST (2 hours and 11 minutes earlier)

  Migrations and the targeted platform adjustment rollout continue.

  Customers on servers awaiting migration may continue to experience email delays, slow performance and connection timeouts.

  Identified
• Sep 4, 2026 1:30 PM SAST (1 hour and 3 minutes earlier)

  Migrations are continuing as planned and the targeted platform adjustment rollout remains underway. Impacted customers on servers awaiting migration may still experience temporary email delays or timeout errors.

  Identified
• Sep 4, 2026 12:17 PM SAST (1 hour and 12 minutes earlier)

  In addition to the ongoing migrations, we are in the process of rolling out a targeted platform adjustment by moving the background system responsible for cataloguing and organising mailboxes to our high-performance storage tier.

  Ongoing Impact
  By processing these constant "directory lookups" on our faster drives, we expect this to noticeably reduce the burden on the underlying high-capacity storage. We are closely monitoring the platform right now to see if this provides the anticipated relief.

  Customers on servers awaiting migration may continue to experience email delays, slow performance and connection timeouts.

  Some customers may also receive a “failed to save to Sent folder” error after sending an email. Despite this error, the email will still have been sent successfully, although a copy may not appear in the Sent folder.

  Identified
• Sep 4, 2026 11:04 AM SAST (1 hour and 13 minutes earlier)

  Our engineering team is steadily progressing with the careful migration of server data to the high-performance storage tier.

  Identified
• Sep 4, 2026 10:08 AM SAST (56 minutes earlier)

  Our engineering team continues to carefully migrate server data to the high-performance storage tier. The process is progressing slowly but steadily.

  Identified
• Sep 4, 2026 9:11 AM SAST (56 minutes earlier)

  While we have successfully migrated some servers to faster storage, we recognise that customers on servers still awaiting migration are experiencing the same level of disruption as yesterday. We are very sorry for the continued impact on your business.

  Ongoing Migrations and Temporary Impact
  Our engineers are continuing to migrate additional servers from the high-capacity storage tier to faster, high-performance storage. Safely moving large volumes of email data unfortunately takes time.

  The data transfers required for these migrations temporarily add load to the existing infrastructure. During this process, you may experience increased delays, sluggish performance or timeouts.

  This temporary impact is necessary to achieve a stable recovery. As each migration is completed, performance should improve rapidly for customers on that server, while also reducing the overall load affecting the remaining servers.

  Identified
• Sep 4, 2026 8:13 AM SAST (57 minutes earlier)

  We are monitoring peak business-hours load to confirm that the ongoing migration of some servers to faster storage tiers has sufficiently reduced the overall burden on the high-capacity storage layer. The changes implemented thus far have reduced the load, and we are monitoring performance closely to confirm that this is sufficient to stabilise performance during peak times.

  Email remains securely queued and we have no evidence that customer messages or data have been lost.

  A detailed incident report will be shared with affected customers once service is fully restored and stable. We will continue to post updates here on our Network Status page. Our next update will be provided at 09:00 SAST.

  Identified
• Sep 3, 2026 3:15 PM SAST (16 hours and 57 minutes earlier)

  We have successfully moved some background logging processes to our high-performance storage tier, significantly reducing the pressure on our high-capacity email storage infrastructure.

  We are continuing to migrate additional workloads away from this infrastructure to further improve email performance and reliability.

  Investigating
• Sep 3, 2026 12:55 PM SAST (2 hours and 20 minutes earlier)

  Following our recent major operating system upgrade, our email storage infrastructure is experiencing an unexpected performance bottleneck.

  Because email hosting requires significant storage capacity, it is housed on our high-capacity storage tier. This infrastructure has historically been stable and reliable. However, the new operating system changed how data is queued and written to these drives.

  The Technical Cause
  This is a complex software-related issue rather than a hardware failure. Despite our usual testing process prior to OS updates, the new operating system has changed how certain background services process and record data. As a result, routine tasks and system logging are generating a significantly higher volume of small disk writes than expected, consuming resources needed to process email efficiently.

  Our engineers are actively implementing changes to reduce this additional load and restore normal email performance as quickly as possible.

  We apologise for the continued disruption and appreciate your patience while we work towards a stable resolution.

  Investigating
• Sep 3, 2026 10:07 AM SAST (2 hours and 48 minutes earlier)

  Following our recent operating system upgrades, we identified an unexpected resource-balancing issue affecting some of our Shared Servers in Johannesburg. This has resulted in reduced performance across our email storage systems.

  As a result, customers may experience delays when sending and receiving email, as well as intermittent timeouts or slower connections when using Webmail, Outlook and other email applications.

  All email remains securely queued, and no messages or data have been lost.
  Our engineers have identified the root cause and are actively rolling out a configuration change across the affected servers. We expect email performance to improve progressively as the fix is implemented.

  Thank you for your patience while we work to fully restore services.

  Investigating
• Sep 3, 2026 8:30 AM SAST (1 hour and 36 minutes earlier)

  Our engineers have rolled out mitigation steps to address the intermittent email connectivity affecting Servers in Johannesburg. These measures are currently taking effect, and some intermittent connectivity may still be experienced across webmail and email applications.

  As services stabilise, customers may also experience delays in sending and receiving email while the backlog that developed during the incident is processed.

  We continue to monitor the environment closely and will provide further updates as services return to normal.

  Investigating
• Sep 2, 2026 3:47 PM SAST (16 hours and 42 minutes earlier)

  Our engineers are continuing their efforts to restore services on the affected servers. For now, connection remains intermittent across both webmail and email applications.

  Investigating
• Sep 2, 2026 11:46 AM SAST (4 hours and 49 seconds earlier)

  Our engineers are actively working to restore email services on the affected servers.

  We apologise for the disruption and appreciate your continued patience.

  Investigating
• Sep 1, 2026 4:23 PM SAST (19 hours and 23 minutes earlier)

  Our engineers are still working to identify the cause of the email delays and connectivity issues experienced on the affected servers.

  We appreciate your patience and understanding while we work to restore services to normal.

  Investigating
• Sep 1, 2026 8:47 AM SAST (7 hours and 35 minutes earlier)

  We are currently investigating email sending delays and connectivity issues affecting some of our servers. Our team is actively working to resolve the issue, and we will share updates as soon as more information becomes available. We appreciate your patience.

  Identified