Customer Portal

Due to an emergency maintenance that was performed at 9:00 PM EDT 7/12/2017, a portion of users may notice a disruption of service with their Hosted Email mailbox. 

Some of the symptoms that are being reported would include:

1. Latency when navigating the Webmail interface
2. Slowness upon logging in to the Webmail interface
3. Delayed email delivery
4. POP Error 'Your incoming (POP3) email server has reported an internal error. If you continue to receive this message, contact your server administrator or Internet service provider (ISP). The server responded: 
5. ERR [SYS/TEMP] Internal error occurred. Refer to server log for more information.
6. Users unable to delete folders in Webmail "The selected folder does not exist...."
7. Messages previously marked as read are now unread
8. "Lost Folders" appear in webmail
9. Delayed email delivery

Our teams are actively working to reduce load and return to normal operation. In the meantime, users affected by this may also experience the following:

1. Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
2. Latency in accessing mail or mail folders
3. Delivery delays (approximately 22 minutes)

Another update will be posted at 1:00 PM CDT.

No additional updates or ETA available at this time. Our teams continue to work on resolving the issue. Users may continue to experience the symptoms below.

1. Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
2. Latency in accessing mail or mail folders
3. Delivery delays (approximately 1.3 hours)
4. Intermittent access to POP/IMAP

Next update will be provided by 7:00 PM EDT.

Message delivery times remain at normal levels.

System performance has stabilized to normal levels. We will continue to monitor the environment as system load increases throughout the day.

As system load increases throughout the day users may experience: 

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
- Latency in accessing mail or mail folders.
- POP/IMAP access issues.

The next update will be provided by 8:00 AM EDT / 12:00 PM GMT.

At this time, message delivery times and webmail responsiveness have remained stable but we will continue to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
- Latency in accessing mail or mail folders.
- POP/IMAP access issues. 

There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. While some of these symptoms may resolve themselves over time, if you are experiencing the below symptoms over a prolonged period of time , please contact Support for further assistance. Symptoms include but are not limited to:

• Unread Items Issue: A large number (or entire folder) of messages has changed from "Read" to "Unread" 
• Missing Subfolders Issue: Missing Subfolders that seemingly disappeared without intervention. (These folders are hidden and not actually missing)
•  “Lost” Folders Issue: User will report a group of folders that have "lost" in the folder name followed by a string of characters "Ex: lost-asfkjl3q5kjafls" 
• POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined). 
• Folder Deletion Issue: Users unable to delete folders in Webmail. Error received will be "The selected folder does not exist...." 
• Webmail Access Issue Users unable to access their email through webmail or Outlook and receive a hard hat or "an error occurred ..." message. 

We are continuing to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
- Latency in accessing mail or mail folders.
- POP/IMAP access issues.

There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. While some of these symptoms may resolve themselves over time, if you are experiencing the below symptoms over a prolonged period of time , please contact Support for further assistance. 

The next update will be provided by 12:00 PM EDT / 4:00 PM GMT.

We continue to monitor the environment as more users access the system throughout the day and are doing everything we can to drive system load down and improve responsiveness. Users may experience: 

- Latency in receiving and accessing mail or mail folders, 
- POP/IMAP access issues, and/or 
- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information." 

In addition, there are persistent issues with a small portion of mailboxes due to an emergency maintenance performed on 12 July 2017 at 21:00 EDT. Symptoms include, but are not limited to, the following: 

- Unread Items Issue: A large number (or entire folder) of messages changed from "Read" to "Unread" 
- Missing Subfolders Issue: Missing Subfolders removed without intervention. (These folders are hidden and not actually missing) 
- "Lost" Folders Issue: a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls" 
- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined). 
- Folder Deletion Issue: Users unable to delete folders in Webmail. Error Message: "The selected folder does not exist...." 
- Webmail Access Issue: Users unable to access their email through webmail or Outlook, could receive error message web hard hat or "an error occurred ...", depending on the operation. 
- Message duplication issue: Users may experience duplication of recently sent messages. 

The next update will be provided by 13:30 EDT / 17:30 PM GMT.

We are continuing to monitor the environment and are making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience the aforementioned symptoms. 

We encourage those users who are experiencing any of the following specific persistent issues to contact their support team for resolution: 

- Unread Items Issue: A large number (or entire folder) of messages changed from "Read" to "Unread" 
- Missing Subfolders Issue: Missing Subfolders removed without intervention. (These folders are hidden and not actually missing) 
- "Lost" Folders Issue: a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls" 
- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in Webmail (error code undefined). 
- Folder Deletion Issue: Users unable to delete folders in Webmail. Error Message: "The selected folder does not exist...." 
- Webmail Access Issue: Users unable to access their email through Webmail or Outlook, could receive error message web hard hat or "an error occurred ...", depending on the operation. 
- Message duplication issue: Users may experience duplication of recently sent messages. 

We will continue to provide updates as they become available. 

Progress is being made in making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience the aforementioned symptoms. 

We will continue to provide updates as they become available.

System load is progressing towards normal levels and most users should find the email system operational. Until the system is fully restored, users who experience occasional timeouts and connection errors should refresh their connection. 

- Folder or Message Visibility Issue: Affected users have partial or no access to their email and/or folders are able to log in, but do not see all expected folders or messages. Engineers are scanning system logs to detect these issues and compiling a list of affected users. We have prepared an automated fix to be applied following restoration of systems and a complete database cleanup. 

- Folder Deletion Issue: Affected users are unable to delete folders in Webmail, receiving the error Message: "The selected folder does not exist...." This issue is related to the Visibility Issue noted above and will be remediated in parallel. 

- "Lost" Folders Issue: Affected users note a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls". This issue will be addressed for many users during the weekend. If the issue persists following the weekend, please contact a member of your support team. 

- POP/IMAP Setup & Webmail Last Page Issues: Affected users receive an error when trying to configure a mail client or navigate to last page in Inbox in Webmail (error code undefined). These issues are related to the timeouts and connection errors noted above. Currently, most issues are resolved. Any remaining issues should be resolved once load returns to normal levels. 

- Message duplication issue: Affected users may experience duplication of recently sent messages. This issue is a side effect of the system issue. Users can safely delete the duplicates. If manual deletion of a large number of duplicate messages is impractical, please contact a member of your support team, as we can assist with de-duplication once the system is fully restored and our maintenance is complete. 

- Unread Items Issue: For affected users, a number of messages or entire folder changed from "Read" to "Unread". Affected users may safely mark the messages as Read. We urge those users who find it impractical to mark a large number of messages as Read to contact a member of your support team, as we can assist with this task once the system is fully restored and our maintenance is complete. 

We will continue to provide updates as they become available

We have made progress in repairing the impacted mailboxes. Some users may experience sporadic delays today while the repair continues.

We will continue to provide updates as they become available.

Over the weekend we have worked through a system-wide cleanup to address the after-effects of the high system load experienced last week. The load was the result of an unknown bug that impacted an effort to proactively double system capacity. Please see the system status update posted 06:49 PM EDT on 07/15/17 for a brief, preliminary review of the causes of the system disruption.

Our priority today is to maintain good access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. In an abundance of caution, we are limiting the number of simultaneous connections to a single mailbox to 2 POP connections and 4 IMAP connections. Mail programs that exceed the limit will usually retry and users are unlikely to see errors due to connection limits.

We are also rate limiting email delivery for mailboxes that receive hundreds or thousands of email messages per minute; that email will safely queue for delivery over a longer period of time. If a user normally receives less email than this, they should see normal delivery speeds.

After peak load hours today, usually finished before 2pm Eastern, we will consider a bulk mailbox level cleanup to address several issues we have identified proactively from system logs. These issues include:

- some users have not yet been able to successfully access their mailbox
- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with “lost-“

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

We will continue to provide updates as they become available.

We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load.
 
We have begun running a fix to address the issues for users that have not yet been able to successfully access their mailbox.  During this time, some users may see some slowness within Webmail.
 
Once completed, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:
 
- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with “lost-“
 
Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

We will continue to work on these mailboxes through the evening.  We will provide our next update during the morning of Tuesday, July 18th.

Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues.

Once complete, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"

As we continue to progress through our mailbox cleanup efforts, some users may see slowness within Webmail.

Some mailboxes may require additional attention to resolve some of the issues experienced, and our team is actively working through those.

We will provide updates as we progress through these cleanup efforts.