25 Mar 2017

Outage Report from Friday March 24, 2017

What Happened

We’d like to give you some additional information about the service disruption that occurred in the Montreal (PRD.MNTL) Region on the morning of March 24th.

The Green-Light Security and Abuse team were assisting a customer with a security-related incident and as we were working with our upstream provider to resolve the issue all traffic to the system was interrupted. We engaged with the technical team regarding possible reasons for the traffic loss it became evident that the problem was an administrative oversite with the upstream data centre that resulted in our system being unprovisioned.

The issue compounded with an early created set of technical tickets related to the redundancy configuration of our DNS system that was working towards resolution from the previous week, the issues compounded and took out DNS service on (PRD.VAN) soon after. Without both locations providing DNS all other services were soon unreachable. 


Our team and our business have evolved in the last 10 years and organise around these types of problems require a new way of thinking. Therefore we are going to be outsourcing the management of our hosting platform to the technical team at RepairFactory.ca.

The Repair Factory is an IT company started by Green-Light in 2012, it’s a separate team specifically focused on IT and as such is better suited from an organisational perspective to design and build out our next generation cluster and provide design and management of our future infrastructure. This is going to result in better service delivery for our legacy hosting clients. 

The Changes Coming

Starting in the next few weeks Repair Factory will be provisioning a new high availability cluster with OVH on Green-Light’s behalf, one of the largest data centre companies in the world, this system will operate in a Proxmox Virtual Environment with a clustered xfs+ceph storage backend that will allow us to seamlessly and automatically remove any problem system from service without affecting availably of email, web or DNS. This style of environment is one that The Repair Factory team has been deploying and managing for clients with great success, we believe it will prevent future problems. 

Is Green-Light a Good fit?

The problems we see need solving in the world today are not the problems we saw when we started.

One sentiment I’ve heard loud and clear from yesterday is phone access.

As our role has grown to handle behind the scenes business problems like; automating payroll, building online stories, integrating internal data silos with external portals, and the moon shots like aquaponic food security. It increasingly makes us ill-suited to handle short-term spikes in phone demand.

The point of including this here is that it’s important you know that our ability to staff the phone is something we’re less able to manage, it’s not because we don’t love you, it’s that we’re a small focused team of only 5 people, and working on an outage and staffing the phone during a spike in demand is too much for us to handle all at once.

Friday was a day where we put the phone down so we could fix the problem. 

So while we’re committed to improving the infrastructure and reducing downtime it comes without the promise of free on-demand phone support, as we move forward that will increasingly be the case.

So for those of you require that type of access we’re no longer a good fit for service and it’s better to have us help you move to someone wants to staff that call centre 24/7 than to promise that someday we might have a better call centre. 

Email help@green-light.ca to setup some time to review your needs and help you evaluate a switch if required, or call 8668045359 and if required and leave a voicemail for a callback. I promise it won’t be too long and we’ll find you a provider who’s a great fit with all the phone support you need.

Keith Page


Share this
26 Jun 2012

Sociocracy: Good Governance

The following is an excerpt from the book, We the people: Consenting to a deeper democracy by John Buck and Sharon Villines

Creating More Perfect Organizations

One of the struggles in building effective organizations whether they are associations, businesses, or governments is finding an effective decision-making method. In democratic organizations, majority vote is the accepted standard. Majority rule, however, automatically creates a minority. This encourages factions and divisiveness rather than harmony. Majority rule encourages people to build strategic alliances and to trade favors rather than to think in terms of the best direction for the organization.

In business, decisions are generally made autocratically by the owner or manager or by a Board on behalf of investors. This can lead to poor decisions because those who execute them may not be free to express their views and critical information is thus not available in the decision-making process. As in majority vote, those who are not included in the decision making may also feel less comitted and thus will not enthusiastically support the organization. Autocratic decision-making also does not encourage leadership.

Sociocracy was developed to correct the deficiencies in both these methods. Sociocratic businesses and organizations set policy by consent and use a governance structure in which each person in the organization is appropriately engaged in making and evaluating the policies that affect their domain of responsibility. Working in self-organizing, semi-autonomous circles, they decide how they will meet the aims of their organization most effectively. This creates more effective, productive, and harmonious organizations — both businesses and associations.

1. Consent governs policy decision-making. Consent means there are no argued and paramount objections to a proposed decision.

2. Circles are the primary governance unit. Circles are semi-autonomous and self-organizing. Within their domain, they make policy decisions; set aims; delegate the functions of leading, doing, and measuring to their own members; and maintain their own memory system and program of ongoing development.

3. Circles are connected by a double-link consisting of the functional leader elected by the next higher circle, and two or more representatives elected by the circle, all of whom participate fully in both circles.

4. People are elected to functions and tasks by consent after open discussion.

Not all decisions in a sociocratic organization must be made by consent. The group can decide by consent to use majority vote for some decisions (when to hold the next meeting) or autocratic decisions for others (letting the shop supervisor assign daily tasks). But everyone must consent before another decision-making method is used and everyone must consent to the policies that determine the parameters for such decisions (who must be included in meetings and how daily tasks are defined).

Self-Organization and Governance

“To govern” means to steer. Everyone wants an organization with lots of energy but an energetic organization needs good steering so its energy is directed and not dissipated. The analogy used to describe how to design and manage a sociocratic organization is to “steer chaos.” To steer, not stifle, energy.Most of us think of chaos as a negative state, but chaos is a very powerful and energetic condition, not to be confused with random or purposeless activity. In chaos, each element is full of energy and freely pursuing its aim without restraint. Free, uninhibited energy creates good conditions for self-organization. In a sociocratic organization, each person and each cluster of persons is encouraged, even required, to self organize and to steer their energy toward their shared aims as energetically as possible. By establishing shared aims, and steering everyone toward them, the sociocratic structure uses all the available energy to move forward quickly and efficiently.

Share this
19 Jun 2012
13 Jun 2012
29 Jan 2010

The Folly of CIRA WHOIS Privacy Changes

We are currently in the midst of trying to re-obtain an expired .ca domain for one of our customers. This particular individual decided many years after a re-branding they no longer needed their old domain name and let it expire. Letting your domain expire is typical, getting it back is hard, CIRA is now making it close to impossible.

Most times a domain squatter will pick up one of these domains and advertise something benign something totally harmless like backpacks, school supplies or baby products. Although they can also host not so benign material. The particular domain squatter that grabbed this domain decided on some rather obscene content instead. The former holder being a family oriented business was shocked when customers started calling and asking why some web searches were taking them to this material.

In hurried panic we were brought in, our first step was to establish trademark rights. Did the customer own the trademark to the name itself? A little bit of digging and the whirl of our fax machine shortly produced a collection of legal documents showing the customer in fact owned the trademark completely and outright. The problem now seems like an easy win. Contact the domain holder, present the documents showing trademark, and pay their modest ransom fee to re-obtain the domain. As we found out, not so easy.

CIRA has decided recently to offer domain privacy as an option. On it’s face this is great, it stops you from being spammed, it stops other registries from sending the less tech savvy domain holders misleading renewal bills. However these protection must be balanced with the legitimate need of the Public to know who is behind a particular domain. For instance, the if domain operator has registered a domain in bad faith and is damaging the trademark holders name and reputation with obscene content one needs this information in order try and resolve the problem in  any method short of a full blown lawsuit.

Going to the Registry of the domain obviously get’s you sent back to CIRA, which does offer a privacy bypass contact form on their site. Using this form you may contact the domain holder, without any of their information being revealed. The form even says that if the domain holder does not respond to contact CIRA after two weeks.This seems like a very fair and balance solution, it protects the majority but seems to indicate that one can still obtain the information if required through a two week waiting process and by going through CIRA directly.

After waiting those two weeks and getting no response we just found out that this is in fact a farce. We received this response from CIRA today.

Individual registrant information is always private and will not be given out. You can try again to contact the registrant and hopefully they will reply. You message will be again forwarded to the Registrant’s Administrative Contact but CIRA cannot guarantee that messages will be read, and/or responded to.”

It appears that CIRA was not all that enlightened in doing it’s public duty. I’m not quite sure why they bother to even invite us to contact them after two weeks. It seems obvious that while a domain holder has a right to some privacy they also have an obligation to respond to these contacts, and if they fail to do so then they waive that right to privacy. Just as one can do a search in the public registry for who has registered a trademark, or corporation name, so too should one be able to search for who owns a domain.

Allowing domain holders to hide under a complete veil of secrecy invites all sorts of problems. As I stated earlier barriers to this information are fine and very much correct, but complete obstruction undermines the fact that this is ultimately public information. .CA is a domain property owned by Canadians, CIRA is a government body in charge or registration, and most importantly domains are publicly accessible portals.

We have a right to be able to know who is running these portals, and we shouldn’t have to resort lawsuits just to find out who is damaging a trademark. Why are we protecting the needs of domain squatters over the needs of legitimate business? Balance CIRA, balance!

Share this

© 2018 Green-Light. All rights reserved.