ElasticSearch

v7.3+ Populate using a queue

Introduction

In order to speed up the populating process and avoid exhausting available memory one can populate using a queue. Read more

Instalation

• Make sure you include the YML that configures enqueue in your project:

git remote add skeleton git@git.xalok.com:HML/skeleton-2021.git
git fetch skeleton
git cherry-pick 4eccf0bd356b46b033751f5b2f7167e840842517

• Make sure wfcms/standard is at v7.3.0-alpha.6 or newer:

composer update wfcms/standard

Usage

app/admin/console fos:elastica:queue-populate --fast --memory-limit=512 --nb-subprocess=4

This command has a lot of helpful options described below (you can also see these options by using --help (or -h) after the command):

--fast                           It calls fos:elastica:fast_populate command.
--index[=INDEX]                  The index to repopulate
--no-reset                       Do not reset index before populating.
--no-delete                      Do not delete index after populate.
--sleep=SLEEP                    Sleep time between persisting iterations (microseconds). [default: 0]
--ignore-errors                  Do not stop on errors.
--first-page=FIRST-PAGE          The pager's page to start population from. Including the given page. [default: 1]
--last-page=LAST-PAGE            The pager's page to end population on. Including the given page.
--max-per-page=MAX-PER-PAGE      The pager's page size. [default: 100]

These options go to Consume command (app/admin/console enqueue:consume) that is run automatically inside the populate command:

--nb-subprocess[=NB-SUBPROCESS]  The number of subprocesses for the "consume command". [default: 2]
--message-limit[=MESSAGE-LIMIT]  Consume n messages and exit.
--time-limit[=TIME-LIMIT]        Consume messages during this time.
--memory-limit[=MEMORY-LIMIT]    Consume messages until process reaches this memory limit in MB.
--niceness[=NICENESS]            Set process niceness.

Read more in php-enqueue's documentation.