Default Single Page
Developing a flexible, scalable and secure cloud hosting solution
The Education & Training Foundation (ETF) provides CPD for staff across the Further Education and training sector. Part of this support is provided through Foundation Online Learning, the ETF’s Learner Management System, LMS, for online staff training. In 2016 the Department for Education asked the ETF to additionally commission the delivery of online resources for learners to support the government’s Prevent strategy to counter radicalisation and extremism. To host these new resources, the ETF commissioned DCL to provide a cloud hosted solution that would host the new learner LMS and also the existing Foundation Online Learning systems.
Peter Munday, Strategic Change Adviser at ETF said: “We needed our hosting solution to be a flexible, scalable service, able to cope with significant fluctuations in demand. This project, because of its very nature, is high profile and very sensitive. It’s important our client base is able to easily and securely review materials as it has a reputational risk to us when they can’t. The reputational risk and trust in always being able to deliver were important factors for us. We feel confident from the outset we can always rely on Digital Craftsmen expertise and professionalism.”
One of the key initiatives for 2017 was to support the provision of these learner Prevent resources via https://www.etflearners.org.uk/, a site which provides learners in the FE sector with valuable and important access to training materials and information to help them stay safe online.
A hosting tender to consolidate the hosting into a secure environment was issued and companies were asked to pitch. Digital Craftsmen was awarded the contract after the tender response demonstrated the best services solution, system hosting and value for money. ETF knew Digital Craftsmen had the ability to deliver having worked with them previously and were confident any issues with migrating the different hosting solutions into one central place would be handled quickly and efficiently.
Bernadette Bourgoin, IT Services Manager at ETF also commented: “We needed the element of security, with the reassurance of and understanding security protocols are rigorously followed. Working with the Digital Craftsmen team, we can see they are following all the required IT controls, management and procedures. There’s a clear methodology in their work which is understood by all.”
The Digital Craftsmen Solution:
The DCL team proposed that the Moodle platforms were consolidated into an existing ETF hosting Environment – this allowed the ETF team to take advantage of economies of scale and the existing High-Availability design and infrastructure which had already been operating successfully for several years.
The DCL team then split the project into several component parts.
- Testing system design, automation and auto-scaling
- Migration of existing Moodle Websites into the existing ETF Hosting environment
- Development and Launch of new Moodle website for Prevent initiative
Migration
After the DCL team had successfully tested running the websites within a development environment, they
were confident there would be no issues when migrating the websites to the ETF Hosting environment. Virtual Machines (VMs) use the AWS Elastic Load Balancer (ELB) service to monitor system health and deploy new VMs when required. This trigger also notifies DCL Lambda functions to complete the deployment of a new VM (Notifications to the DCL team and adding the VM to monitoring automatically.
The migration of the websites was coordinated by Ben Sanders from the DCL team with the involvement of all stakeholders. Good communication was key, combined with a well-documented plan and rollback options. On the day of the migration, the websites were transferred and the existing ETF Moodle websites were deployed into the ETF hosting environment. This allowed the development team to focus solely on the next phase: Developing the new Moodle Prevent website.
Development
To facilitate testing for the migration and continued development of the Prevent website, the DCL team
deployed ETF a temporary Development Environment entirely isolated from live systems. This was built and ready for use within the hour thanks to the DCL Automator tool which leverages the excellent AWS
CloudFormation technology.
The DCL Automator provided a VPC with dual NAT, a VPN Server, Configuration Management server (Puppet) and allowed the DCL team to quickly deploy ELB’s, Webservers and Database components that closely matched the setup of live systems.
Key features included with the DCL Automator tool include the ability to automate snapshots of instances on a schedule, power instances on\off for business hours (Saving up to 66% of AWS compute costs) and access to easily build new VMs using Puppet (Foreman).
All combined, the temporary ETF Development environment enabled ETF and their developers to get started very quickly, reduce expenditure on AWS costs in a secure and resilient environment. Once the ETF developers had finished and the newly developed website was launched, all stakeholders agreed the development environment would not be needed in the short term and it was entirely deleted to further reduce costs (Safe in the knowledge that DCL can build another Development Environment on-demand, using CloudFormation).
An Example Development Environment built using DCL Automator
Auto-scaling
With the existing Moodle websites, peak usage is predictable and occurs at several points each year. The
newly developed Prevent website was a complete unknown in terms of usage patterns, so the ETF team was
keen that it should also respond to demand and auto-scale as necessary.
DCL Developed an auto-scaling system using the following technologies:
- CloudWatch to Monitor System Health with alarms set to send SNS notifications
- ELB setup to scale up\down when receiving SNS notifications – new instances using official machine images from Linux distributors.
- Additional SNS notification by email and Slack for the DCL team
- Additional SNS notification to add\remove instances from monitoring
- New instances receive configuration from Puppet and configure VMs (ready in 8 minutes)
- ELB sends traffic to newly built VMs when they pass health checks.
This has allowed the new hosting Environment to dynamically adjust website capacity based on usage and provides the most efficient and cost effective way of utilising AWS resources.
Tuning & Continual Improvement
DCL understand that complex systems evolve over time based on business and security requirements. Business Critical Systems such as the ETF environment need a dedicated change control and access control policies to manage change and enable only authorised members of the team to effect change.
Since the Moodle Websites have migrated to the ETF Hosting environment, we have helped to improve these processes for the system and deploy a number of changes which have improved performance and capacity as usage has grown.
Database Session Management -To enable multiple web servers to work together, a method of sharing user sessions is essential. The initial approach was to use database session management and although this worked well for months after launch, it soon became clear that database session management would not scale as demand on the websites peaked. DCL investigated the available options and then submitted requests for a phased implementation of Memcached across all Moodle websites for session management. This improvement has noticeably sped up page response times during peak usage and has improved the visitor experience.
Caching Improvements – Moodle makes extensive use of caching and when scaled up to 10+ webservers the impact on disk usage is noticeable. When the DCL team found some IO wait within the system we advised the ETF team to upgrade the HA shared file system with more resources to prevent any bottlenecks, we also submitted and implemented requests for change to move some parts of caching locally to the webservers. Both changes have had a positive impact on the performance of the system and reduced IO usage on the shared file system.
Scaling of resources as demand has grown (RDS Database and HA Shared Storage) – has been achieved by Requests for Change including all stakeholders in the process and seeking approval from system owners to manage capacity; DCL have upgraded several components in the system to match visitor demand.
Return of the Development Environment – Continual improvement without testing comes with many risks. As 2017 comes to a conclusion all stakeholders on the project have agreed the need for a new development system to test upgrades, changes and capacity for 2018. DCL will once again leverage the DCL Automator tool to quickly and efficiently replicate the Line Hosting Environment for these purposes.
Cost Savings
With the new components of the ETF Hosting Environment in place and operating successfully – DCL
continually monitor capacity and advise the ETF team through regular meetings and reports. Where
opportunities arise for change or cost savings the DCL team submit requests for change via our agreed change control process.
An example of this is the DCL recommendation to take advantage of AWS Reserved instances; DCL have
compiled all the details and potential cost savings available over the course of 1 and 3 years and submitted that for consideration by the ETF team. When agreed, DCL will implement the change and save ETF up to 46% of AWS Compute costs dependant on which combination of options are chosen.
Peter Munday concludes:
“The most impressive thing about Digital Craftsmen is their level of service which is second to none. At all
times, they are quick to respond and any issues dealt with, it’s a personal service built on trust they will get things done, and fast.”
If you’re planning to move your websites or looking to manage your existing cloud services more efficiently and cost effectively, then speak to one of the Digital Craftsmen team and see how they can help you.
