We’d like to give you some additional information about the service disruption that occurred in the Montreal (PRD.MNTL) Region on the morning of March 24th.
The Green-Light Security and Abuse team were assisting a customer with a security-related incident and as we were working with our upstream provider to resolve the issue all traffic to the system was interrupted. We engaged with the technical team regarding possible reasons for the traffic loss it became evident that the problem was an administrative oversite with the upstream data centre that resulted in our system being unprovisioned.
The issue compounded with an early created set of technical tickets related to the redundancy configuration of our DNS system that was working towards resolution from the previous week, the issues compounded and took out DNS service on (PRD.VAN) soon after. Without both locations providing DNS all other services were soon unreachable.
Our team and our business have evolved in the last 10 years and organise around these types of problems require a new way of thinking. Therefore we are going to be outsourcing the management of our hosting platform to the technical team at RepairFactory.ca.
The Repair Factory is an IT company started by Green-Light in 2012, it’s a separate team specifically focused on IT and as such is better suited from an organisational perspective to design and build out our next generation cluster and provide design and management of our future infrastructure. This is going to result in better service delivery for our legacy hosting clients.
The Changes Coming
Starting in the next few weeks Repair Factory will be provisioning a new high availability cluster with OVH on Green-Light’s behalf, one of the largest data centre companies in the world, this system will operate in a Proxmox Virtual Environment with a clustered xfs+ceph storage backend that will allow us to seamlessly and automatically remove any problem system from service without affecting availably of email, web or DNS. This style of environment is one that The Repair Factory team has been deploying and managing for clients with great success, we believe it will prevent future problems.
Is Green-Light a Good fit?
The problems we see need solving in the world today are not the problems we saw when we started.
One sentiment I’ve heard loud and clear from yesterday is phone access.
As our role has grown to handle behind the scenes business problems like; automating payroll, building online stories, integrating internal data silos with external portals, and the moon shots like aquaponic food security. It increasingly makes us ill-suited to handle short-term spikes in phone demand.
The point of including this here is that it’s important you know that our ability to staff the phone is something we’re less able to manage, it’s not because we don’t love you, it’s that we’re a small focused team of only 5 people, and working on an outage and staffing the phone during a spike in demand is too much for us to handle all at once.
Friday was a day where we put the phone down so we could fix the problem.
So while we’re committed to improving the infrastructure and reducing downtime it comes without the promise of free on-demand phone support, as we move forward that will increasingly be the case.
So for those of you require that type of access we’re no longer a good fit for service and it’s better to have us help you move to someone wants to staff that call centre 24/7 than to promise that someday we might have a better call centre.
Email email@example.com to setup some time to review your needs and help you evaluate a switch if required, or call 8668045359 and if required and leave a voicemail for a callback. I promise it won’t be too long and we’ll find you a provider who’s a great fit with all the phone support you need.
Whether your politics fall to the right or left of centre there’s one thing we’re all sure of. We’re fighting blood sweat and tears to wrestle the sacred truth from the other so that our world might be restored to the path of peace and prosperity. Centre stage is the singular struggle that has defined life for my generation. The reality of climate change; or not.
Sock Puppets Cheering on the Destruction of Discourse
Somewhere in-between the punches and the scratches and the underhanded snatches the fight has become less about truth and more about winning, winning no matter the cost. From Stephen Colbert to Bill O’riley the narrative that we’re losing the battle for values and characters is told by both sides and every player. In the ultimate twist of irony, winning the fight for the truth has become a reason to lie, cheat, and deceive. Our online world allows paid actors and provocative trolls an ever increasing ability to form and define our shared reality by providing every ill-conceived question a chorus of sock puppets cheering on the destruction of discourse.
A Steady Hand
It is in such times that one looks to the steady hands of the village elders and their traditional boring institutions for stoic and measured responses. A steady hand to hold back the passions of youthful folly that would lead us down false paths.
A village elder such as the esteemed members of the U.S. House of Representatives Committee on Science, Space, and Technology or the venerable NOAA.
So what exactly is going on between NOAA and U.S. House of Representatives Committee on Science, Space, and Technology (CSPTech)?
In response to an article by the Daily Mail, the Committee released this press statement “Former NOAA Scientist Confirms Colleagues Manipulated Climate Records”. Which consists of committee member quotes praising the source of the article, Dr John Bates, for his brave stand against the data manipulation efforts of the head of NOAA’s Centers for Environmental Information (NCEI) Dr. Thomas R. Karl.
Garbage in, Garbage out…
The centre of Dr. Bate’s complaint is that NOAA made compromises in the archival and verification process of its climate data sets, rejecting a craftsman approach of polishing every data set before release. NOAA opted instead to follow an iterative approach of processing the data incrementally as its software modeling improved and verifying its approach by matching it against measurements taken with newer higher quality instruments.
The complaint centres around doing things perfect at the expense of running out of time with nothing, or doing what you can in the time you have.
Within the software and data development fields the concept of applying 20% of your initial effort to produce the broad strokes that produce 80% of your product is a well known approach to getting actionable results sooner that you further refine with time.
In the case of NOAA we are talking about actual software built to process the data sets, over time NOAA will continue to debug and tune the software model to gain greater accuracy and resolution of the data. However in our current environment the Daily Mail and Dr. Bates use the argument that we have to throw all babies out with all bath waters, that the trade off made by Dr. Thomas R. Karl amount to an outright lie. A position that the House committee members now repeat based on nothing more than a reading of the Mail’s article.
Timing is Everything?
Dr. Bates clarified his remarks to EEnews today by saying, “The issue here is not an issue of tampering with data, but rather really of timing of a release of a paper that had not properly disclosed everything it was.”
His complaint, he said, was that NOAA did not extensively archive data set versions through intermediary steps of the process leading to the report; steps that were deemed extraneous by management because the new results were independently verifiable through external measurements, but that the omission of these steps did not in fact compromise the data or the report.
Basically a data wonk disagreed with management how extensively to archive the changed set of a massive data set.
This small inner conflict over process between egg heads is now seized upon by the very committee tasked to bring clarity and guidance to the Committee on Science, Space, and Technology (CSPTech) and instead Chairman Lamar Smith uses the tabloid’s report to push a manufactured controversy that further erodes the honest discourse of our scientific leaders. In a word it betrays its mission, the American people, and by extension the world.
University teams around the world met to test out their designs at the first of two Hyperloop Pod Competitions. SpaceX has spent the last 6 months constructing the world’s second largest low-pressure environment in the form of a one mile test track, second only the famous LHC.
The Hyperloop track built near SpaceX’s headquarters in Hawthorne, California, since early September allows teams to test vehicles of three main types: Wheel, Air Bearings and Magnetic levitation.
SpaceX also constructed a vehicle it calls the Pusher to get pods up to operational speed and outlined basic specifications of a track for students to design their pods around.
Initial interest in the competition was in the thousands, but by the time of competition only 27 teams actually made it through the rigours of the process. Of those 27 teams only 3 won the right to run their pod in the test track.
Delft, MIT, and WARR
The competition was one of truly global participation but after a battery of safety and engineering tests conducted Saturday on the 27 pods brought to California only MIT here in America, WARR of Germany, and DELFT of the Netherlands earned the right to run the track.
MIT won the award for safety and reliability, while WARR won for fastest pod. DELFT walked away with the best overall score in all categories.
Eyes to the Future
While it was an exciting weekend for many geeks worldwide, we are already looking forward to this summer when the second competition will take place. Teams will return home with the lessons learned, contacts made and a drive to see how they can improve their PODs for the second competition on a yet to be announced date this summer.
We wish them luck and look forward to further innovation in this exciting transport system.
PopCorn time is a great little app that took the internet by storm over the last week. But no sooner had the application started to go viral then the creators removed it citing legal concerns for themslves as it’s makers.
But not all good things have to come to an end. The project was always meant to be open sourced and so it’s been reborn on github.
Feel the chilly wind in your beard as you fall to earth collecting candy canes and presents for the good boys and girls. This fun new game holiday game uses HTML5 and CSS3 to achieve most of it’s 3D effects. We think it’s pretty darn cool.
A Canadian team has won and elusive prize for $250,000 setup in 1980 for the first human powered helicopter. The technology aside this thing is simply a thing of beauty.
Some clients are getting Mail Delivery Problems to Yahoo. The root of this problem has been fixed and we expect the issue to dissipate completely within 48 hours.
The issues was caused by an exposed script from an outdated version of software being used to send spam to Yahoo and a few other providers. It only takes a few hours of this type of activity to temporarily cause reputation problems with providers like Yahoo that can last many days. It highlighted an oversight in our monitoring of this particular system that prevented us from being notified until the reputation of host3 had already begun to sink. We’re sorry for that and have made changes to prevent it in the future.
One of the direct benefits of our CodeWatch program is that it would have prevented this issues regardless of our oversight. We’re systemizing much of the work done by traditional web developers to deliver a faster more secure platform.With CodeWatch your website code is reviewed on a regular basis for outdated components and updated by qualified professionals. Outdated software can be a very real security problem in the online world, used to amplify and distribute the messages of malware authors, identity thieves and their ilk to unsuspecting individuals world wide.
It’s therefor increasingly important to keep any application, online or off, updated and secure with CodeWatch