25 Mar 2017

Outage Report from Friday March 24, 2017

What Happened

We’d like to give you some additional information about the service disruption that occurred in the Montreal (PRD.MNTL) Region on the morning of March 24th.

The Green-Light Security and Abuse team were assisting a customer with a security-related incident and as we were working with our upstream provider to resolve the issue all traffic to the system was interrupted. We engaged with the technical team regarding possible reasons for the traffic loss it became evident that the problem was an administrative oversite with the upstream data centre that resulted in our system being unprovisioned.

The issue compounded with an early created set of technical tickets related to the redundancy configuration of our DNS system that was working towards resolution from the previous week, the issues compounded and took out DNS service on (PRD.VAN) soon after. Without both locations providing DNS all other services were soon unreachable. 


Our team and our business have evolved in the last 10 years and organise around these types of problems require a new way of thinking. Therefore we are going to be outsourcing the management of our hosting platform to the technical team at RepairFactory.ca.

The Repair Factory is an IT company started by Green-Light in 2012, it’s a separate team specifically focused on IT and as such is better suited from an organisational perspective to design and build out our next generation cluster and provide design and management of our future infrastructure. This is going to result in better service delivery for our legacy hosting clients. 

The Changes Coming

Starting in the next few weeks Repair Factory will be provisioning a new high availability cluster with OVH on Green-Light’s behalf, one of the largest data centre companies in the world, this system will operate in a Proxmox Virtual Environment with a clustered xfs+ceph storage backend that will allow us to seamlessly and automatically remove any problem system from service without affecting availably of email, web or DNS. This style of environment is one that The Repair Factory team has been deploying and managing for clients with great success, we believe it will prevent future problems. 

Is Green-Light a Good fit?

The problems we see need solving in the world today are not the problems we saw when we started.

One sentiment I’ve heard loud and clear from yesterday is phone access.

As our role has grown to handle behind the scenes business problems like; automating payroll, building online stories, integrating internal data silos with external portals, and the moon shots like aquaponic food security. It increasingly makes us ill-suited to handle short-term spikes in phone demand.

The point of including this here is that it’s important you know that our ability to staff the phone is something we’re less able to manage, it’s not because we don’t love you, it’s that we’re a small focused team of only 5 people, and working on an outage and staffing the phone during a spike in demand is too much for us to handle all at once.

Friday was a day where we put the phone down so we could fix the problem. 

So while we’re committed to improving the infrastructure and reducing downtime it comes without the promise of free on-demand phone support, as we move forward that will increasingly be the case.

So for those of you require that type of access we’re no longer a good fit for service and it’s better to have us help you move to someone wants to staff that call centre 24/7 than to promise that someday we might have a better call centre. 

Email help@green-light.ca to setup some time to review your needs and help you evaluate a switch if required, or call 8668045359 and if required and leave a voicemail for a callback. I promise it won’t be too long and we’ll find you a provider who’s a great fit with all the phone support you need.

Keith Page


Share this
30 Jan 2017

HyperLoop Pod Competition Held at SpaceX This Weekend

University teams around the world met to test out their designs at the first of two Hyperloop Pod Competitions. SpaceX has spent the last 6 months constructing the world’s second largest low-pressure environment in the form of a one mile test track, second only the famous LHC. 

The Hyperloop track built near SpaceX’s headquarters in Hawthorne, California, since early September allows teams to test vehicles of three main types:  Wheel, Air Bearings and Magnetic levitation. 

Hyperloop Track Spec

SpaceX also constructed a vehicle it calls the Pusher to get pods up to operational speed and outlined basic specifications of a track for students to design their pods around. 

Initial interest in the competition was in the thousands, but by the time of competition only 27  teams actually made it through the rigours of the process. Of those 27 teams only 3 won the right to run their pod in the test track. 

Delft, MIT, and WARR

The competition was one of truly global participation but after a battery of safety and engineering tests conducted Saturday on the 27 pods brought to California only MIT here in America,  WARR of Germany, and DELFT of the Netherlands earned the right to run the track. 

MIT won the award for safety and reliability, while WARR won for fastest pod. DELFT walked away with the best overall score in all categories. 

Eyes to the Future

While it was an exciting weekend for many geeks worldwide, we are already looking forward to this summer when the second competition will take place. Teams will return home with the lessons learned, contacts made and a drive to see how they can improve their PODs for the second competition on a yet to be announced date this summer.

We wish them luck and look forward to further innovation in this exciting transport system. 


Share this
20 Dec 2012
26 Jun 2012

Sociocracy: Good Governance

The following is an excerpt from the book, We the people: Consenting to a deeper democracy by John Buck and Sharon Villines

Creating More Perfect Organizations

One of the struggles in building effective organizations whether they are associations, businesses, or governments is finding an effective decision-making method. In democratic organizations, majority vote is the accepted standard. Majority rule, however, automatically creates a minority. This encourages factions and divisiveness rather than harmony. Majority rule encourages people to build strategic alliances and to trade favors rather than to think in terms of the best direction for the organization.

In business, decisions are generally made autocratically by the owner or manager or by a Board on behalf of investors. This can lead to poor decisions because those who execute them may not be free to express their views and critical information is thus not available in the decision-making process. As in majority vote, those who are not included in the decision making may also feel less comitted and thus will not enthusiastically support the organization. Autocratic decision-making also does not encourage leadership.

Sociocracy was developed to correct the deficiencies in both these methods. Sociocratic businesses and organizations set policy by consent and use a governance structure in which each person in the organization is appropriately engaged in making and evaluating the policies that affect their domain of responsibility. Working in self-organizing, semi-autonomous circles, they decide how they will meet the aims of their organization most effectively. This creates more effective, productive, and harmonious organizations — both businesses and associations.

1. Consent governs policy decision-making. Consent means there are no argued and paramount objections to a proposed decision.

2. Circles are the primary governance unit. Circles are semi-autonomous and self-organizing. Within their domain, they make policy decisions; set aims; delegate the functions of leading, doing, and measuring to their own members; and maintain their own memory system and program of ongoing development.

3. Circles are connected by a double-link consisting of the functional leader elected by the next higher circle, and two or more representatives elected by the circle, all of whom participate fully in both circles.

4. People are elected to functions and tasks by consent after open discussion.

Not all decisions in a sociocratic organization must be made by consent. The group can decide by consent to use majority vote for some decisions (when to hold the next meeting) or autocratic decisions for others (letting the shop supervisor assign daily tasks). But everyone must consent before another decision-making method is used and everyone must consent to the policies that determine the parameters for such decisions (who must be included in meetings and how daily tasks are defined).

Self-Organization and Governance

“To govern” means to steer. Everyone wants an organization with lots of energy but an energetic organization needs good steering so its energy is directed and not dissipated. The analogy used to describe how to design and manage a sociocratic organization is to “steer chaos.” To steer, not stifle, energy.Most of us think of chaos as a negative state, but chaos is a very powerful and energetic condition, not to be confused with random or purposeless activity. In chaos, each element is full of energy and freely pursuing its aim without restraint. Free, uninhibited energy creates good conditions for self-organization. In a sociocratic organization, each person and each cluster of persons is encouraged, even required, to self organize and to steer their energy toward their shared aims as energetically as possible. By establishing shared aims, and steering everyone toward them, the sociocratic structure uses all the available energy to move forward quickly and efficiently.

Share this
19 Jun 2012
12 Jun 2012

Beauitful from First Principles

What impresses us the most about some developers is the combination of power and beauty some can bring to our kitchen table. The Apple Design awards announced yesterday highlighted some great examples of this. Check out the our staff favorites. These are truly a labour of love and sweat, and the best part is they’re free… okay two are free one is ten bucks.

read more
Share this

© 2018 Green-Light. All rights reserved.