tag:status.vanillasoft.com,2005:/historyVanillaSoft Status - Incident History2024-03-19T04:48:24-04:00VanillaSofttag:status.vanillasoft.com,2005:Incident/194840772023-12-19T18:20:52-05:002023-12-19T18:20:52-05:00VanillaSoft Servers Timing Out<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>18:20</var> EST</small><br><strong>Resolved</strong> - The fix has been deployed and is holding. All services are operational.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>18:09</var> EST</small><br><strong>Identified</strong> - We have identified an issue with the VanillaSoft servers that is causing long load times and preventing some users from using the platform. A fix is being worked on and will be deployed shortly.</p>tag:status.vanillasoft.com,2005:Incident/194128482023-12-12T08:33:53-05:002023-12-12T08:33:53-05:00VanillaSoft VoIP Inbound and Outbound Call Issue<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>08:33</var> EST</small><br><strong>Resolved</strong> - Our VoIP services are now operational.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>08:05</var> EST</small><br><strong>Monitoring</strong> - Our engineers have identified the issue and have implemented a workaround. Our engineers are currently monitoring the system and will provide an update upon resolution. Again, we thank you for your patience and understanding.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>07:56</var> EST</small><br><strong>Investigating</strong> - Our engineers are currently investigating a major outage affecting all inbound, outbound, and SMS traffic. This outage is also affecting call center call routing. We will provide an update as soon as the team is able to identify the root cause. We apologize for any inconvenience, and we thank you for your patience.</p>tag:status.vanillasoft.com,2005:Incident/192442992023-11-29T14:13:59-05:002023-11-29T14:13:59-05:00No Audio on VS Connect (Inbound/Outbound Calls)<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>14:13</var> EST</small><br><strong>Resolved</strong> - Our VoIP audio for inbound and outbound has been restored. Thank you for your patience, and we apologize for any inconveniences that this may have caused.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>14:00</var> EST</small><br><strong>Investigating</strong> - Our VoIP services are currently experiencing intermittent audio issues affecting inbound and outbound calls for some clients. We are actively investigating and hope to have this resolved</p>tag:status.vanillasoft.com,2005:Incident/189679142023-10-30T10:14:41-04:002023-10-30T10:14:42-04:00VoIP Services experiencing SIP 408 Errors<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>10:14</var> EDT</small><br><strong>Resolved</strong> - Our VanillaSoft VoIP services are now operational. Thank you for your patience and understanding.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>09:30</var> EDT</small><br><strong>Monitoring</strong> - Our engineers have identified and addressed the issues causing the SIP errors. Dialing and receiving calls have been re-established. The team will continue to monitor.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>09:21</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of a possible service outage affecting some of our VoIP services. Our engineers are currently working on addressing and resolving these issues.</p>tag:status.vanillasoft.com,2005:Incident/188299452023-10-17T14:19:40-04:002023-10-17T14:19:42-04:00VanillaSoft VoIP Performance Degradatoin<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:19</var> EDT</small><br><strong>Resolved</strong> - Our engineers were able to identify the root cause of the degradation and systems are now operational.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:11</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of performance degradation for some of our VoIP service which is only impacting some of our clients. Our engineers are currently working on addressing and resolving these issues.</p>tag:status.vanillasoft.com,2005:Incident/185385222023-09-18T16:27:55-04:002023-09-18T16:27:55-04:00VanillaSoft VoIP Intermittent Call Issue<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>16:27</var> EDT</small><br><strong>Resolved</strong> - The VoIP call issues that have been reported by some of our clients have now been resolved. All Dialing and Call Center functionalities have been restored.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:58</var> EDT</small><br><strong>Monitoring</strong> - The VanillaSoft Support team is aware of the call issues that are happening to some of our customers. We have isolated the problem and are working on a resolution.<br /><br />Affected Components<br />-Inbound Dialing<br />-Outbound Dialing<br />-Call Center</p>tag:status.vanillasoft.com,2005:Incident/182934922023-08-28T09:58:26-04:002023-08-28T09:58:26-04:00Voicemail drops, HUD and SMS are currently experiencing rolling outages impacting some customers<p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>09:58</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>09:22</var> EDT</small><br><strong>Investigating</strong> - Voicemail drops, HUD and SMS are currently experiencing rolling outages impacting some customers. We are currently investigating and working to resolve the issue as soon as possible.</p>tag:status.vanillasoft.com,2005:Incident/179129302023-07-20T11:45:38-04:002023-07-20T11:45:38-04:00No ringing on outbound calls<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>11:45</var> EDT</small><br><strong>Resolved</strong> - The VanillaSoft team were monitoring an intermittent call connection issues that occurred between 10:00am and 10:45am EDT this morning. The issue was related to a file access problem that has been recovered. The problem was sporadic and not happening system wide. Outbound call and audio issues have now been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>11:05</var> EDT</small><br><strong>Monitoring</strong> - We are currently investigating "no ringing on outbound calls" with our VanillaSoft VoIP Systems. Calls are still connecting to the intended party. There may be some clients that are currently experiencing no audio afterwards. This issue is only affecting a subset of client and may be intermittent. We will continue to monitor and provide an update once a resolution has been established.</p>tag:status.vanillasoft.com,2005:Incident/176050212023-06-17T21:31:49-04:002023-06-17T21:32:24-04:00Intermittent VoIP Issues<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>21:31</var> EDT</small><br><strong>Resolved</strong> - Service remains stable. We continue to monitor but consider this issue resolved.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>17:18</var> EDT</small><br><strong>Monitoring</strong> - Service has been restored, but we are still monitoring the situation.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>16:51</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating intermittent VoIP connectivity problems that are impacting some customers.</p>tag:status.vanillasoft.com,2005:Incident/162567782023-02-23T21:19:15-05:002023-02-23T21:19:15-05:00VoIP outage<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>21:19</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>16:49</var> EST</small><br><strong>Investigating</strong> - Our VoIP partner is currently having an outage. The issue is under investigation and looking to resolve it as soon as possible.</p>tag:status.vanillasoft.com,2005:Incident/102690192022-06-13T15:44:38-04:002022-06-13T15:44:38-04:00VanillaSoft VoIP degraded service<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:44</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:57</var> EDT</small><br><strong>Investigating</strong> - Our VoIP partner is currently experiencing intermittent service issues that are impacting inbound and outbound calls. We are actively investigating and hope to have this resolved shortly.</p>tag:status.vanillasoft.com,2005:Incident/98796632022-04-29T16:36:25-04:002022-04-29T16:36:25-04:00Networking Issue Causing Outage<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:36</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>10:33</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>10:00</var> EDT</small><br><strong>Investigating</strong> - Our Google Cloud hosting center is currently experiencing some network issues that are being investigated and that are impacting performance. We will update as soon as possible.</p>tag:status.vanillasoft.com,2005:Incident/85528742021-11-16T15:50:47-05:002021-11-16T15:50:47-05:00VanillaSoft brief Outage due to Google Cloud outage<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:50</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:41</var> EST</small><br><strong>Monitoring</strong> - At around 12:50pm EST, there was an outage with VanillaSoft due to our cloud hosting provider, Google Cloud, experiencing networking issues with their global load balancer. Google has identified the issue, and all appears to be back to normal - the outage lasted about 15 minutes, We will continue to monitor the situation.</p>tag:status.vanillasoft.com,2005:Incident/63522302021-02-21T19:25:24-05:002021-02-21T19:25:25-05:00VoIP is currently down for all customers<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>19:25</var> EST</small><br><strong>Resolved</strong> - All systems are officially operational. This brief outage is resolved. We thank you for your patience.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:57</var> EST</small><br><strong>Monitoring</strong> - Phone lines have been restored. We continue to investigate the root cause of the incident, while also testing that all features are fully functional.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:48</var> EST</small><br><strong>Investigating</strong> - VanillaSoft VoIP is currently down for all customers. The team is aware of the issue with working diligently to restore service. We will update with timelines and updates as they become available.</p>tag:status.vanillasoft.com,2005:Incident/53497542020-10-17T16:56:05-04:002020-10-17T16:56:05-04:00VS Connect softphone login service disruption<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>16:56</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:46</var> EDT</small><br><strong>Monitoring</strong> - Our softphone provider (Counterpath) has had a data center outage this morning causing our VS Connect softphone login service to fail. They believe they have identified the issue and are currently in the process of restoring the service. VoIP services were not impacted, so service would be uninterrupted for users who were logged in before the issue began. If you would like to read details on the outage, please go to https://support.counterpath.com/hc/en-us/articles/360057898473-Data-Center-Outage-Oct-17</p>tag:status.vanillasoft.com,2005:Incident/52210242020-10-03T13:40:31-04:002020-10-03T13:40:31-04:00VoIP outage<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>13:40</var> EDT</small><br><strong>Resolved</strong> - Our VS Connect (VoIP) services were restored to the production servers yesterday (Friday October 2nd). All services, including voicemail drops and the creation of new lines, are now working normally.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>16:42</var> EDT</small><br><strong>Monitoring</strong> - All VanillaSoft VoIP services were restored last night to a backup server, with the exception of some limitations outlined below. We are working diligently with our partners to have the full service resolved as soon as possible, and anticipate reverting back to our regular servers tomorrow - Friday, October 2nd.<br /><br />VM Drop: Your VM (voicemail) Drop recordings are currently unavailable. In order to use VM Drop before we switch back to our regular servers, you will need to create new temporary recordings. Once we do switch back to our regular servers, all of your original recordings will once again be available, but any new temporary recordings you create will be lost.<br /><br />Call Recordings: You will still be able to record calls and they will be available once we move back to the regular server. However, while we are on the backup server you may not be able to access your previous call recordings until the system is fully restored.<br /><br />New Lines/Features: The provisioning of new VoIP/SMS lines is currently disabled. Additionally, any new changes made to VoIP settings during this period while we are working from a backup server, such as call paths/forwarding/other features, will be lost when the production server is restored.<br /><br />We understand the impact this has on your business, and appreciate your patience during this process.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>20:28</var> EDT</small><br><strong>Identified</strong> - All VoIP services have been restored, with the exception of the CMV HUD, which is currently being worked on.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing a VoIP outage that is impacting all of our clients using CMV VoIP. CMV is aware of the issues and are working toward a resolution as soon as possible. We thank you for your patience and understanding.</p>tag:status.vanillasoft.com,2005:Incident/48396602020-08-12T14:42:24-04:002020-08-12T14:42:25-04:00Intermittent email delivery delays for some users<p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>14:42</var> EDT</small><br><strong>Resolved</strong> - The backlog of undelivered emails is now fully processed. Oracle appears to have officially resolved their issues, as per the guidance we previously shared. The email delivery issue is fully functional again. We are marking this incident as resolved. Thank you for your patience.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>13:28</var> EDT</small><br><strong>Monitoring</strong> - The Oracle engineers have just informed us that the backlog causing these delays should now be resolved. However, it may take some time for the delayed emails to correctly reach their destination as the backlog clears.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>13:16</var> EDT</small><br><strong>Identified</strong> - VanillaSoft's mail service provider, Oracle, is currently experiencing intermittent email delivery delays. They are aware of the issue and have engineers working on a resolution. This only impacts users who do not have an SMTP configuration established.</p>tag:status.vanillasoft.com,2005:Incident/31204442019-10-20T07:10:13-04:002019-10-20T07:10:13-04:00VanillaSoft maintenance<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>07:10</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>06:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:15</var> EDT</small><br><strong>Scheduled</strong> - Maintenance work is being done to VanillaSoft on Sunday October 20 starting at 6am EDT. We expect this to take about one hour, during which time VanillaSoft will be inaccessible.</p>tag:status.vanillasoft.com,2005:Incident/31056892019-10-15T14:57:25-04:002019-10-15T14:57:25-04:00Gryphon DNC (Do Not Call)<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>14:57</var> EDT</small><br><strong>Resolved</strong> - All issues with the Gryphon DNC service have been resolved.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>09:59</var> EDT</small><br><strong>Investigating</strong> - We are currently having problems communicating with the Gryphon DNC service. This will cause routing issues for any customers using Gryphon DNC.</p>tag:status.vanillasoft.com,2005:Incident/27109712019-07-15T17:18:09-04:002019-07-15T17:18:10-04:00Update for VanillaSoft VoIP Users<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>17:18</var> EDT</small><br><strong>Resolved</strong> - This issue has been resolved. Please note the instructions (as described in the last update) in case you encounter the problem.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>11:22</var> EDT</small><br><strong>Identified</strong> - This morning we became aware of an issue in which older versions of VS Connect (the VanillaSoft softphone from CounterPath) are unable to connect to the CounterPath server. If you are affected, you will see an error message stating, “the data received from this license server appears to be corrupted”. This issue can be resolved by uninstalling your VS Connect phone, and then installing the latest version which can be found here: https://support.vanillasoft.com/hc/en-us/articles/360001655151-Download-and-Login-to-VS-Connect<br /><br />If your organization does not use the VS Connect softphone this issue does not affect you and no action is required.</p>tag:status.vanillasoft.com,2005:Incident/25747442019-06-20T11:32:34-04:002019-06-20T11:32:35-04:00VanillaSoft VoIP outage<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>11:32</var> EDT</small><br><strong>Resolved</strong> - Our VoIP service is once again fully functional. Any recordings or voicemails made yesterday between 3:40pm and 10:00pm ET are in the process of being restored, and should be available later today. Call reports may temporarily be inaccurate while the call records are updated on the main servers. We would suggest waiting until tomorrow to run reports to ensure their accuracy.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>15:22</var> EDT</small><br><strong>Investigating</strong> - Some users are experiencing an outage with our VoIP service. Our VoIP provider is actively investigating the issue.</p>tag:status.vanillasoft.com,2005:Incident/24943082019-06-07T11:18:21-04:002019-06-07T11:18:21-04:00VanillaSoft is now up and running on secondary servers<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:18</var> EDT</small><br><strong>Resolved</strong> - Our testing has found no further issues.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>16:20</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>16:17</var> EDT</small><br><strong>Update</strong> - We believe that we have identified the root cause of the intermittent latency issues that were experienced by some customers today. We will continue to monitor closely to validate.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>12:09</var> EDT</small><br><strong>Update</strong> - We have seen intermittent, regional access issues that we are looking into. We will continue to monitor, but it does not seem to be impacting all regions.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>14:39</var> EDT</small><br><strong>Monitoring</strong> - VanillaSoft is now up and running on secondary servers, and testing is showing that everything is working well.</p>tag:status.vanillasoft.com,2005:Incident/24935052019-06-04T14:30:59-04:002019-06-04T14:30:59-04:00VanillaSoft currently offline<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>14:30</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>14:22</var> EDT</small><br><strong>Monitoring</strong> - We have switched over to our Google backup servers, which are located in a region that is currently not impacted by Google’s outage. VanillaSoft appears to be back to normal, but we will continue to monitor. Thanks for your patience.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>12:13</var> EDT</small><br><strong>Identified</strong> - Our Google hosting platform is experiencing a regional outage. We are in communication with Google’s engineers as they work towards a resolution. We will update as soon as we know more.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>11:38</var> EDT</small><br><strong>Monitoring</strong> - VanillaSoft is back online but we are awaiting confirmation from Google that the issue is fully resolved. In the interim we will continue to monitor.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>11:27</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating a possible outage with Google. We will update as soon as we know more.</p>tag:status.vanillasoft.com,2005:Incident/24839562019-06-02T20:42:58-04:002019-06-02T20:42:59-04:00VanillaSoft Currently Offline due to Possible Network Issue with Google Cloud Platform<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>20:42</var> EDT</small><br><strong>Resolved</strong> - Google has confirmed that the issue has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>18:46</var> EDT</small><br><strong>Monitoring</strong> - VanillaSoft is back up and running, but we are still waiting for an update from Google to confirm that the issue has been permanently resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>17:11</var> EDT</small><br><strong>Update</strong> - Google believes they have identified the root cause of the outage and expect to return to normal service shortly. We will update again when we get the next update from Google.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>15:37</var> EDT</small><br><strong>Update</strong> - Google has confirmed a multi-region issue. We will update our status as we receive updates from Google.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>15:18</var> EDT</small><br><strong>Investigating</strong> - VanillaSoft is currently offline. This is believed to be a result of issues with the Google Cloud Platform, which we use to host our software-as-a-service. We are actively and aggressively pursuing this issue with Google. Updates will follow soon, once we have confirmed the cause and identified the corrective course of action and associated timelines. We apologize for this downtime and appreciate your patience.</p>tag:status.vanillasoft.com,2005:Incident/22567942019-03-12T19:18:13-04:002019-03-12T19:18:14-04:00Performance issues have been reported and are being investigated.<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>19:18</var> EDT</small><br><strong>Resolved</strong> - The performance degradation has been resolved. We are actively continuing to monitor it. If anything changes we will issue another status alert. Sorry for any inconvenience. We sincerely appreciate your support and loyalty.<br /><br />#TeamVanillaSoft</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>18:59</var> EDT</small><br><strong>Monitoring</strong> - The issue has been identified. We have implemented a workaround and are monitoring. More updates to follow.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>18:28</var> EDT</small><br><strong>Investigating</strong> - VanillaSoft is currently experiencing performance degradation. Users may experience slow speeds or have trouble logging in. We are investigating the issue and will provide updates as they become available. <br /><br />All hands are on deck for this one folks. We feel your pain. Hold tight. We want this resolved as soon as possible, just like I’m sure you do, too. We appreciate your patience, and we apologize for any inconvenience.</p>