Tuesday 20 September 2016

RESOLVED: Hapara Dashboard Service Disruption: 2016-09-19

UPDATE (Resolved): 16:15 pm PDT: Google has confirmed that the root cause of the issue is due to an API update on the Google side.

We are continuing to work with Google's Engineering team to understand the change, and implement a more robust long term resolution.
---


UPDATE (Resolved): 12:30 pm PDT: We've completed fixing all impacted domains now, and do not anticipate this problem recurring again today.

If you continue to experience issues with access to Hapara Dashboard, please submit a request to our Support Team, with examples of affected teacher emails, so we may investigate further.

We are investigating the root cause of the problem with Google and will update this post with more, once this is determined.
---

UPDATE: 12:00 pm PDT: We're working on the tail end of these fixes now. The majority of impacted domains have been resolved.
---

UPDATE: 11:00 am PDT: We have a fix running which should bring classes back into visibility, and allow teachers to login successfully.

Hapara's little hamster wheels are currently running at max capacity to get these fixes completed, and we'll provide another update in an hour. Please try refreshing Hapara Dashboard to check if your domain has been resolved.
---

UPDATE: 10:00 am PDT: We're working to identify all domains affected by this outage, and are beginning a fix process. This will take approx hour or two to complete in full, but you will likely see resolution sooner than this.
---

UPDATE: 09:00 am PDT: We have a critical ticket logged with the Google team to help investigate the root cause of the issue.

Our support team has a workaround fix we are implementing in the Hapara backend now. More updates to come.
---

UPDATE: 08:15 am PDT: We have found a potential cause and are working toward a resolution. In the interim, teachers can try these troubleshooting steps to resolve.

We'll update this post again as soon as we have more.
---

07:30 am PDT: We have had elevated reports of Hapara Dashboard being inaccessible for some users. We are investigating this at the highest priority and will post more updates as we learn more.

You may receive a login or authorization error, which usually should be resolved by logging out of all Google Accounts. We will update this article again shortly as we learn more.

Best regards,
Hapara Support Team

Saturday 17 September 2016

RESOLVED: Highlights Service Disruption: 2016-09-16


RESOLVED: 10:30 am PDT

This morning at approx 09:00 am PDT we noticed elevated error rates for the Highlights product. 

Teachers may have seen errors saying "Student device is offline" or student's may have had no activities/tabs despite the student being online and active.

Our Engineering team implemented a fix for this issue by 10:00 am PDT, and we are confident this has resolved the root cause of the problem.

If you continue to experience issues with Highlights, please submit a request to our Support Team.

Best regards,
Hapara Support

Friday 9 September 2016

RESOLVED: Google Service Disruption: 2016-09-08

RESOLVED 11:30am PDT: We have had confirmation from Google that the disruption today was caused by a configuration update in Google's Identity, and has since been rolled back.

We are marking this Service Disruption as resolved, and if you experience any further login issues please try these resolution steps in the first instance.

---

UPDATE 10:10am PDT: We are receiving reports that the problem is self resolving. Our team has had some success accessing Hapara Dashboard, Workspace, Highlight, and the Hapara Admin Console after refreshing a browser session and recommend you try this in the first instance:

1.  Log out of all Google Apps accounts
2. Clear Chrome Browser's Cache and Cookies
3. Try log back into your school account and re-access Hapara Dashboard

Our Engineering team is still investigating the root cause of this issue, and we will update this post again when we know more.

Best regards,
Hapara Support

---

10:00am PDT: We are experiencing a global service disruption for Hapara Dashboard, Workspace, Highlight, and the Hapara Admin Console.

This issue looks to be caused by a failure in Google's authentication engine. Our team is currently investigating the source of the problem and reaching out to Google directly.

We will post another update to this issue when we have more information

Best regards,
Hapara Support

Friday 2 September 2016

RESOLVED: Data Loader Service Disruption - Manual Spreadsheet Loads

RESOLVED 2030hrs PDT. We have implemented a fix that handles most of the spreadsheet errors.  We have tested this extensively, and it is handling the majority of errors seen.  If you continue to experience issues with loading data, please submit a request to our Support Team.

Best regards,
Hapara Support

---

UPDATE 1412hrs PDT: We've made progress diagnosing the issue at hand, and have engaged Google to investigate.

We will provide another update as soon as we hear back from the Google Team.

While this is under investigation, you can successfully complete a manual load via CSV using the workaround below.

Best regards,
Hapara Support

---

1308hrs PDT: We are currently investigating an intermittent service disruption for Manual Data Loads from Google Sheets. Some loads of Google Sheets are returning errors and failing to load.

 The issue is not system wide, and to confirm if you're affected by this problem, you might see this or a similar error in the Data Loader Status page:

Unexpected error encountered while loading worksheet <Sheet name>. Please try to load the file again. If this problem persists please contact Hapara Support.

While our development team investigates and works to resolve this issue, you can successfully complete a manual load via CSV using the following workaround:
  1. In the Google Sheet, click File > Download as > Comma-separated values
  2. On your Hapara Admin console, go to the Manual Configuration menu.
  3. Click Start Configuration, then choose CSV file and click Next through the menu options until "Save".
  4. Go to the Manual Load menu, click Next, then choose the file that you downloaded in Step 1.
  5. Click Start Load to process the file.
If you have any questions about the above process, or continue to experience issues with loading data, please submit a request to our Support Team.

We will post another update to this issue when we have any additional information.

Best regards,
Hapara Support