Tagged: home

Moodle with AWS opensearch

following is a step-by-step guide to setup moodle global search with AWS OpenSearch service. This provides enhanced searching in rich content.

Moodle

A powerful open-source LMS. 

AWS opensearch

This is a forked version of the popular elastic search which is provided by AWS. This is extended by AWS and available as a service under AWS.

Search 

As we start everything from google because of its searching capability, the same we want at many places, especially those which are rich in content.

If the content is not searchable, then it is not visible to user.

Moodle comes with a default DB based search but that is an extra burden over the existing database and not powerful as well.
Along with DB search, moodle is shipped with SOLR, which is also an open-source search engine.

Why AWS openserach instead of SOLR ?

SOLR

A very powerful search engine and integration code comes with moodle core, however, 

1- you will require PHP Solr extension to use this. That might not be directly available on the machine and would not be directly installed. It may require an extension building step then only it can work.

2- you need to set up solr server on a machine. However, there are pretty instructions written, still, monitoring and administration will be a concern over the long run.

Why AWS opensearch ?

[our system is also on AWS, then it is the first choice]

1- Easy to set up and administrator and monitoring.

2- no need for any system based dependency. You just need to install two moodle plugins from moodle admin interface and that’s done.

How to setup AWS open search ?

From the AWS console,  just visit OpenSearch service,

  1. Create a domain
  2. you can start with dev mode, and t3.small.search with 1-az only for cost perspective
  3. Under the network, enable public access
  4. Under fine-grained access,  create a master user and password
  5. Set access policy to “Only Use Fine-grained access control”

And that’s done. In a few minutes, your cluster will be ready and you will get a URL as well.

 you can also login on opensearch console by using same username and password to explore more.

How to setup moodle for this.

1- download this plugin from here:

this is a search plugin, compatible with moodle version 3.2+ and it is extended from the existing plugin to support AWS OpenSearch.

2- this plugin requires another plugin for AWS libraries  : https://moodle.org/plugins/local_aws

3- once installed, just do configure the search as

  • enable global search
  •  setup default as elastic search
  •  setup url as : https://<username>:<password>@<url you copy from aws cluster>
  •  port 443
  •  index your data and this is done.

For more about moodle global search

https://docs.moodle.org/400/en/Elasticsearch

https://docs.moodle.org/401/en/Global_search

AWS Devops Cheat Sheet

There is a plethora of services provided by AWS, however, the following is oriented in the aspect of AWS DEVOPS professional certification and AWS professional solution architect certification.

General

We can view the AWS service in the following way

  • By Cost
  • By Resiliency
  • By Security
  • By Governance
  • By Monitoring
  • By Services

Following is just the generalization as per my view.

By Cost

  • Cost is calculated where the cloud subscriber
    • use storage
    • use compute
    • use network for outbound data transfer

eg: AWS Storage family has the cost on basis of storage used, and what kind of storage used [HDD, SDD, IOPS provisioned]

Compute Family on basis of computing resources are running along with lambda

Network, for outbound data, to the internet, to the region, and to AZ as well.

  • There are three modal of cost
    • Always Free under the limit
    • 12 Months free under the limit
    • Trial
  • There is a pay-per-use, a monthly bill is generated on the 3rd of every month
  • In general, many of the serverless services have an always free tier limit, such as SNS, SQS, SES, Lambda, API Gateway, Cognito, etc, while provisioned resources are paid
  • Cost is roughly per minute or per hour round of depend upon service
  • You can create an estimation at: https://calculator.aws/#/

By Resiliency

  • AWS services are focused on resiliency whether zonal failure, regional failure
  • AWS serverless services are by default regional in nature, providing a regional availability and those are highly available
  • generally, Provisioned resources are in AZ, which provides zonal level availability, To maintain regional level availability, there are options, by which you can apply different types of DR policies according to your RTO and RPO need as well you can maintain HA, like for ec2, there are AMI for backup restore, there is ASG to maintain availability and can set up to use the instance in different AZ, one can deploy Multi-AZ RDS, etc.
  • To achieve the regional level DR and HA for the provisioned resources, there is a cost, this is why those are optional.
  • One can also apply a multi-region modal for DR policies to avoid regional-level failure downtime.
  • AWS provides SLA, different for different services, for availability and durability.
  • AWS provides services that can be used altogether to provide operation-level resiliency as
    • ec2 can be set up to reboot or auto recover using cloud watch.
    • You can define redemtion option under aws config to monitor the change and recover
    • You can use route 53 to apply failover rerouting,
    • you can also apply health checks to [route53, elb, asg, target groups] monitor, and perform the failover action.

By Security

  • Runs on a Shared security modal, physical security are with AWS, while authN/authZ and data security are managed by a cloud subscriber
  • Authentication is managed by IAM
  • Authorization is managed by Policies and roles
  • Everything is governed by AuthN/AuthZ, whether single account, interaction among services, multi-account or cross-account, whether it is data at S3 or branch at code commit.
  • In general, there are 2 types of credentials. username/password and programmatic credential [accesskey/secret key]. there are 2 others as well, for special services like code commit and Cassandra
  • For data [In-rest], all the services provide an option to store the data as encrypted by using AWS KMS [key management service] where the key can be generated by the client and stored at AWS, encryption using keys also has a cost impact as encryption-decryption utilize key
  • For -Intransit, all the provision resources are under VPC which is internal to cloud subscribers and secured by AWS, all the outbound should use HTTPS, however, this is not a compulsion. All the AWS server-less services are accessible through HTTPS only. and more into this, you can also create a VPC endpoint for serverless services so that, your resources are available within your VPC.
  • All resources are private by default, if you need to make them public, you can do it explicitly, like AMI, volumes, snapshot, s3 bucket etc

By Governance

  • There are AWS services for this, like,
    • AWS Config to maintain the complaints
    • SSM [System manager] for operations like, run command, patching, etc
    • Cloud shell for the in-browser command line tool
    • Cloudformation for IAC [infra as code], Elastic beanstalk and Opsworks
    • AWS control tower, AWS Trusted advisor
  • AWS Organization, for a multi-organization model
  • You can interact with or manage the resources by
    • Console [browser based interaction]
    • SDK [Software Development Kit in multiple lanaguage]
    • CLI [Command Line Interface]
    • CDK [cloud development Kit]
    • Cloud Control API [Unified Rest API]
    • Cloudformation/ SAM templates [json/yaml based templte, internally used API]

By Monitoring & Logging

  • Cloudwatch is the service for this, which is a unified service for aws resources as well as you can use that on-premise resources
  • You can use this service for monitoring, for alarming, for communicating with other AWS services
  • You can use Cloudwatch Logging for the logging perspective
  • AWS cloud trail is the service to log and insights all the communication with your cloud account whether on Management Plane [managing infra] or Data plane [s3 bucket data], which is governed by AWS
  • You can automate the operation using Cloudwatch Events / Lambda, and can use SNS for notification

By Services

  • Services are categorized by their usage type

Notes

  • EFA  Elastic Fabric Adapter: EFA brings the scalability, flexibility, and elasticity of the cloud to tightly-coupled HPC applications
  • SR-IOV (Single Root I/O Virtualization). SR-IOV is a method of device virtualization that provides higher I/O performance and lower CPU utilization compared to traditional implementations
  • Yes, Amazon CloudWatch stores metrics for terminated Amazon EC2 instances or deleted Elastic Load Balancers for 2 weeks.
  • AWS CDK: cloud development kit: use to launch resource, an abstraction top of cloud formation
  • AWS SDK: use to interact with those resources.
  • Cloudformation Vs Elastic Benastalk : We can use cloud formation to make resources, where we need to write the code.EB [Elastic Beanstalk], do this thing for use from front end by option and configuration. By EB: one can monitor and configuration can be part of the version management system by having .ebextension file. Elastic beanstalk is more towards application, CF is more towards aws service, as aws service rollback on failure on update the configuration
  • AWS copilot is the CLI based automation tool for ECS, where we create a manifest file
  • ECS as well as EB, can be fully automated with cloud front.
  • Swap url feature [Elastic Beanstalk] can be suitable for two identical env , basically blue/green deployment
  • Codedeploy and Opsworks and many other services which utilize agent support on-premise servers, like CW, inspector, etc.

More