Category: Moodle

Setup Solr with Moodle for search inside files content

Following is the step by step guide to setup Solr with Moodle for search inside files content. This will need setting up of SOLR, setting up php-solr extension and solr core configuration.

Moodle has an option for global search, which provide an option to search among course, activities and other areas. This is a unified search, which extrac the data from database , basis on user serach query and context.

Context means : Search inside only enrolled courses or All the courses.

We will divide this topic into following part,

What is the need for Solr

Out of many benefits like,

  • fast searching
  • ranking
  • search stats

the most benefit is, searching inside the files.

Majorly, files are uploaded in the system, as pdf, doc, txt, ppt etc whether as an activity or assignment submission etc.

these files are not much relevan if we are not able to search within the file content as an 100 pages pdf will be irrelevant if not comes with a saearch query,

Here, SOLR do its job to provide the result by searching inside the file content at a speed.

There are alternatives to SOLR as elasticsearch, but, SOLR is avialable and integrated directly in moodle.

more about solr, https://solr.apache.org/features.html

What is solr

the next section is about, what is solr ?

SOLR is an open-source search engine that provides a REST API Interface to ingest the content and query against that content and the output is in JSON.

Solr is a standalone enterprise search server with a REST-like API. You put documents in it (called “indexing”) via JSON, XML, CSV or binary over HTTP. You query it via HTTP GET and receive JSON, XML, CSV or binary results.

more about solr , https://solr.apache.org/

How to setup solr ?

SOLR can be setup on any linux or windows or mac server.

It will run independently and can be run on the same server at which the moodle runs or on any other server.

there are many tutorial to install solr on specific platforms

https://www.vultr.com/docs/install-apache-solr-on-ubuntu-20-04/

Remember this,

  • During setup on Linux, you will create a solr user and try to run solr by that user only, you can switch the user from sudo su command and then do the solr-related operation for sake of permission issues in Linux.
  • Once you have set up the solr and able to access the solr home page from the web, you need to create a collection. The collection is like a logical area, where ingested data and operations will perform
su - solr -c "/opt/solr/bin/solr create -c moodle 
  • Apart from the solar, you will require the php solr extension at the moodle server. that is not available directly for installation. You can install it through pear or pecl. and depend upon the version , you may require to build the extension from zip.
  • following command can help you in centos based system
yum install php-pear php-pecl  php-devel curl-devel zlib-devel pcre-devel gcc ibxml2-devel
pecl install solr

How to setup it with Moodle

Now, you have

  • setup the solr
  • create a basic collection name as moodle
  • setup the php solr extension

it comes to the integration part,

  • login through site admin in moodle
  • enable global serach under Site administration > advance features
  • Go on Administration > Plugins > Search > Manage global search
  • Select search engine as Solr
  • Configure the Solr under Administration > Plugins > Search > Solr
  • Enable File Indexing option, and setup upper size, if 0 then unlimited.

if dependencies are met, it will show the screen like this

  • Once the first three are yes, Click on Index data. this will ingest the data into solr and will show a screen like this.
  • You can enable the global search block and can try with query.

If you are not getting the result, then try to query the in solr directly, like below

Still, the file content would not be searchable.

Setting up file content serachable

still, the file content is not searchable if you have created a core from the default configuration. To make it searchable, we need to apply some changes at solr side.

Add following at line number 70 around

<lib dir="${solr.install.dir:../../../..}/contrib/extraction/lib" regex=".*\.jar" />
  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-cell-\d.*\.jar" />

  <lib dir="${solr.install.dir:../../../..}/contrib/langid/lib/" regex=".*\.jar" />
  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-langid-\d.*\.jar" />

  <lib dir="${solr.install.dir:../../../..}/contrib/velocity/lib" regex=".*\.jar" />
  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-velocity-\d.*\.jar" />

Add following at line number 850 around

            <requestHandler name="/update/extract"
                  startup="lazy"
                  class="solr.extraction.ExtractingRequestHandler" >
    <lst name="defaults">
      <str name="lowernames">true</str>
      <str name="uprefix">ignored_</str>

      <!-- capture link hrefs but ignore div attributes -->
      <str name="captureAttr">true</str>
      <str name="fmap.a">links</str>
      <str name="fmap.div">ignored_</str>
    </lst>
  </requestHandler>
  • restart the solr service
  • re-index the data from moodle form Index Data Page
  • Now you can query from solr or from global search , and if all goes well, it will display the result

Moodle Search result

Solr Search Result

Customizing core features in Moodle through Events and Callbacks

At some point of time, we always need to tweak or customize the core functionality or flow of the application.

If you are aware of WordPress, then there are hooks and filters for this.

In moodle, We can extend the core features and even tweak them through events and callback functions, even without making any change in the core files.

What is moodle EVENT ?

Moodle events, are the hooks, on which you can register your functions to be called. there are many-many core events available .

you need to create a plugin , under db folder , create an observer.php file.

put following code.

// create a class name as core_event_sample_observer and write a function observe_all, function would be called as soon as user is created in system.

$observers = array(
    array(
          'eventname'   => '\core\event\user_created',
        'callback'    => 'core_event_sample_observer::observe_user',        'includefile' => null,
        'internal'    => true,
        'priority'    => 9999,
    ),

);

You can include your class by putting file path . and you can get the user detail , by defining parameters in a function.

class core_event_sample_observer {

public static function observe_user($event){
    $userid = $event->objectid; // created user id
    print_R($event);
// write your code here and do the needful operation
   }
}

An administrator can view a list of all types of events from Site administration > Reports > Events list.

more information can be grabbed form here,

https://docs.moodle.org/dev/Events_API#Events_API

You can execute the code in events callback but think like this as a background process, as it will not interact with page. Simply, you can not use redirect or you can not put any thing that you want to be a part of the rendereded page. For such cases, there are callbacks.

Extending through Callbacks

To solve the above challenges, we use the callback.

Callbacks are the function, in which moodles try to call during execution. These are hooks as in wordpress.

  • You write the function in lib.php of your plugin in a standard format.
  • The function will be called during execution. Moodle will try to check the function by including files from all the plugins.

following is the sample.

// assuming , xyz is a local plugin and callback will be done while code is being executed from standard_footer_html.

// this will include this js at footer of the moodle page.

function local_xyz_standard_footer_html() {
global $CFG, $DB, $USER;
   
    $output = <<<STR
    <script>
   
    window.onload = function(){
        alert("Hello");
    }();
    </script>
STR;
    return $output;
}
  • as you are writing this, so you do not need to customize and moodle file to inject your code into the page .
  • There are many callbacks already available for core functionalities like page rendering or user login or user registration etc .

List of one-to-many callbacks

CallbackPlugin typeVersionComments
Navigation, see Navigation API
extend_navigation_coursePutting up a menu in the course administration menu*before Moodle 3.0 was only supported by report and tool plugin types
extend_settings_navigationPutting up a menu in the Site Navigation menulocal, booktoolNote, modules have a one-to-one callback with the same name, see below
extend_navigationlocalNote, modules have a one-to-one callback with the same name, see below
extend_navigation_user_settingsPutting up a menu in the User Setting menu*3.0+before Moodle 3.0 was only supported by tool plugin types
extend_navigation_category_settingsPutting up a menu in the Category Administration menu*3.0+
extend_navigation_frontpage*3.1+
extend_navigation_user*3.1+
My profile, see My profile API
myprofile_navigation*2.9+
Before-actions hooks
course_module_background_deletion_recommended*3.2+see upgrade.txt
pre_block_delete*3.1+
pre_course_category_delete*3.1+
pre_course_delete*3.1+
pre_course_module_delete*3.1+
pre_user_delete*3.1+
Login_callbacks
after_config*3.8+Triggered as soon as practical on every moodle bootstrap after config has been loaded. The $USER object is available at this point too.
after_require_login*3.7+eg Add permissions logic across a site or course
check_password_policy*3.6+
print_password_policy*3.8+ (WIP)
pre_signup_requests*3.5eg to do actions before sign up such as acceptance of policies, validations, etc
extend_change_password_form*3.8+ (WIP)eg Injecting form elements into the change password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
extend_set_password_form*3.8+ (WIP)eg Injecting form elements into the set password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
extend_forgot_password_form*3.8+ (WIP)eg Injecting form elements into the forgot password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
extend_signup_form*3.8+ (WIP)eg Injecting form elements into the signup form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
validate_extend_change_password_form*3.8+ (WIP)eg Adding additional validation to the change password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
validate_extend_set_password_form*3.8+ (WIP)eg Adding additional validation to the set password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
validate_extend_forgot_password_form*3.8+ (WIP)eg Adding additional validation to the forgot password form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
validate_extend_signup_form*3.8+ (WIP)eg Adding additional validation to the signup form. Note: This hook is not compatible with the Webservices API or the Moodle mobile app. See MDL-66173 and MOBILE-3181 for more info.
post_change_password_requests*3.8+ (WIP)eg Fire additional actions after a user changes password.
post_set_password_requests*3.8+ (WIP)eg Fire additional actions after a user resets password.
post_forgot_password_requests*3.8+ (WIP)eg Fire additional actions after a user submits a password reset request.
post_signup_requests*3.8+ (WIP)eg Fire additional actions after a user creates an account.
Page rendering, see Output_callbacks
add_htmlattributes*3.3+eg Add open graph xml namespace attributes
before_footer*3.3+eg injecting JS across the site, like analytics
before_http_headers*3.3+Setting http headers across the site like CSP
before_standard_html_head*3.3+A better API alternative to appending to $CFG->additionalhtmlhead
before_standard_top_of_body_html*3.3+
render_navbar_output*3.2+
Course module edit form
coursemodule_edit_post_actions*3.1+
coursemodule_standard_elements*3.1+Takes parameters $formwrapper and $mform, allows manipulation of editing form, e.g. additional fields. See https://github.com/marcusgreen/moodle-local_callbacks
coursemodule_definition_after_data*3.11+
coursemodule_validation*3.1+
Modules
check_updates_sincemod3.2+See NEWMODULE_Documentation
dndupload_registermod
Admin
bulk_user_actions*3.9+Any plugin typically an admin tool can add new bulk user actions
Other
get_fontawesome_icon_map*3.3+https://docs.moodle.org/dev/Moodle_icons#Font_awesome_icons
get_question_bank_search_conditionslocal
oauth2_system_scopes*3.3+
rss_get_feed*
supports_logstorereport
get_shortcutsltisource3.1+
before_launchltisource2.8+
control_view_profile*3.6+
store_profiling_data*3.6+
override_webservice_execution*3.6+
Deprecated
cron*See Task API
delete_coursemod,report,coursereport,formatReplace with observer to course_contents_deleted event
print_overviewmodup to 3.2New dashboard uses Calendar API to populate events on the timeline
report_extend_navigationcoursereportPlugin type coursereport is deprecated, plugin type report should be used instead

List of one-to-one callbacks

CallbackPlugin typeVersionComments
Comments support, see Comment API
comment_add*
comment_display*
comment_permissions*
comment_template*
comment_url*
comment_validate*
Gradebook
export_%_settings_definitiongradeexport% is a plugin name without prefix
import_%_settings_definitiongradeimport% is a plugin name without prefix
report_%_profilereportgradereport% is a plugin name without prefix
report_%_settings_definitiongradereport% is a plugin name without prefix
Groups support, see Groups API
allow_group_member_remove*
restore_group_member*For plugins that create their own groups
Installation and upgrade
xmldb_%_install*Located in db/install.php , % refer to the full plugin name, in case of modules without prefix
xmldb_%_install_recovery*Located in db/install.php , % refer to the full plugin name, in case of modules without prefix
xmldb_%_uninstall*Located in db/uninstall.php , % refer to the full plugin name, in case of modules without prefix
xmldb_%_upgrade*Located in db/upgrade.php , % refer to the full plugin name, in case of modules without prefix
LTI source
add_instance_hookltisource
messagetypeltisourceCallback name is the type of the message
Question bank support, see Question API
question_pluginfile*
question_preview_pluginfile*
questions_in_use*since 3.7.5, 3.8.2 (previously only “mod”)
Ratings support, see Rating API
rating_can_see_item_ratings*2.9.2+
rating_can_see_item_ratings*2.9.2+
rating_get_item_fields*
rating_permissions*
rating_validate*
Themes
$THEME->csspostprocesstheme
$THEME->extralesscallbacktheme
$THEME->lessvariablescallbacktheme
page_inittheme
Modules: Calendar and dashboard
core_calendar_event_action_shows_item_countmod3.3+
core_calendar_is_event_visiblemod3.3+
core_calendar_provide_event_actionmod3.3+
Modules: Course cache
cm_info_dynamicmod
cm_info_viewmod
get_coursemodule_infomod
Modules: Course reset
reset_course_form_defaultsmod
reset_course_form_definitionmod
reset_userdatamod
Modules
delete_instancemod
dndupload_handlemod
export_contentsmodUsed in WS get_course_contents
extend_settings_navigationmod,booktool
extends_navigationmod
get_completion_active_rule_descriptionsmod3.3+Must be present for modules implementing custom completion rules
get_completion_statemod
get_extra_capabilitiesmod
get_file_areasmod
get_recent_mod_activitymod
get_shortcutsmod3.1+
grade_item_updatemod
grading_areas_listmod
print_recent_activitymod
print_recent_mod_activitymod
refresh_eventsmod
rescale_activity_gradesmod3.1+
scale_used_anywheremod
update_gradesmod
user_completemod
user_outlinemod
Glossary formats
glossary_print_entry_%for glossary formatsGlossary formats is not a plugin type yet, however the list of formats is not hardcoded in the mod_glossary, instead the methods similar to callbacks are used for each subfolder in mod/glossar/format. % refer to the subfolder name, the functions are expected in lib.php
glossary_show_entry_%for glossary formats– ” –
glossary_show_entry_%for glossary formats– ” –
glossary_show_entry_%for glossary formats– ” –
Other
global_db_replaceblock
inplace_editable*See Inplace editable
output_fragment_**3.1+see Fragment
page_type_list*Used when displaying page types in block configuration
params_for_jsatto
pluginfile*Callback checking permissions and preparing the file for serving plugin files, see File API. Note, in case of block plugins the list of arguments is slightly different
restore_role_assignment*For plugins that create their own role assignments
strings_for_jsatto
supports*Required for activity modules
h5p\canedit::can_edit_content*4.0+Plugins can implement this class&method to define, if required, any custom behaviour for deciding whether an H5P content used inside the plugin can be edited or not
Deprecated
ajax_section_moveformatup to 2.3See Course formats
ajax_supportformatup to 2.3See Course formats
delete_coursemodup to 3.1use event observer
display_contentformatup to 2.3See Course formats
get_post_actionsmod,booktoolOnly called if legacy log is used on the site
get_section_nameformatup to 2.3See Course formats
get_section_urlformatup to 2.3See Course formats
get_typesmod,ltisourceup to 3.0Replaced with get_shortcuts in 3.1
get_view_actionsmod,booktoolOnly called if legacy log is used on the site
load_contentformatup to 2.3See Course formats
scale_usedmodup to 3.0
uses_sectionsformatup to 2.3See Course formats
question_list_instancesmodup to 3.8Use “questions_in_use” instead