Listings

Intro

On the public part, every published page ID is stored in a few redis sorted sets (one per category, subcategory, one for each tag, author, etc). When displaying some lists on public, instead of querying MySQL (join page_category join category where category.id = X) - which is very slow when there are many articles - the CMS takes the IDs from the correct redis set and from MySQL it queries by page.id - which is blazingly fast (almost) no matter how many articles there are 😉

The things that make this work:

vendor/wfcms/cms-base-bundle/Wf/Bundle/CmsBaseBundle/Publish/Manager/BaseManager.php:getPageLists - when a page is published, this method is called and populates an array of lists names (strings) where the page "belongs".

vendor/wfcms/cms-base-bundle/Wf/Bundle/CmsBaseBundle/PageRenderer/ListingPageRenderer.php - one example where this is queried:

$qb = $articleRepository->getBaseQB();
$qb->byList($listName);

One very important limitation about displaying results: the redis list is used only when the results are paginated, so don't call $qb->execute() or anything to get the results, instead, put the QB in a paginator. You'll see in the ListingPageRenderer that right below the $qb->byList($listName) that QB is fed into a paginator.

Custom lists

One can create a custom list to be stored in Redis by overwriting the wf_cms.publish.manager:getPageLists method (class Wf\Bundle\CmsBaseBundle\Publish\Manager\BaseManager) in the project:

Example: Adding a new Redis list based on whether the article has a setting checked:

<?php

namespace App\Bundle\CmsBundle\Publish\Manager;

use App\Bundle\CmsBundle\Entity\PageArticle;

class RedisManager extends \Wf\Bundle\CmsBaseBundle\Publish\Manager\RedisManager
{
    public function getPageLists($page)
    {
        $pageLists = parent::getPageLists($page); 

        $settings = $this->getObjectField($page, 'settings');
        if (!empty($settings[PageArticle::SETTING_SPECIAL_LIST])) {
            $pageLists[] = PageArticle::LISTING_SPECIAL;
        }

        return $pageLists;
    }
}

Note: The above code uses a constant PageArticle::LISTING_SPECIAL for the list name, this allows reusing the same constant when one wants to use this list, and PageArticle::SETTING_SPECIAL_LIST that can be reused when adding the checkbox to the article's form.

Note: The same approach can be used to exclude an article from listings: return an empty array if the article fullfills the desired criteria.

Gotchas

Repopulate when changing lists

wf_cms.publish.manager:getPageLists is called when an article is being published, either from the UI or from a script/Symfony command. Thus, making changes to what lists an article belongs to only applies to articles created when that code is in place - it does not apply retroactively to existing articles. To apply it to existing articles one would need to run make publish-populate.

getObjectField

Note how the code above uses $this->getObjectField($page, 'settings') instead of accessing the entity's getters directly ($page->getSettings(), or $page->getSetting(PageArticle::SETTING_SPECIAL_LIST)). This is because $page is an entity when publishing normally, from the admin UI or from a script, but when running make publish-populate $page is an array containing the raw DB result. This is because repopulating the lists has to go through all articles in the database, and hydrating entities from the DB data is a resource intensive task that can significantly slow down the repopulation process.

Automated listing module

Intro

In wf-html one can use the automated listing module:

<div wf-role="listing"></div>

Note: If not using AJAX pagination and each page will have its own URL (e.g. /politica/2, /politica/3, etc.), the board that renders this module must be rendered using the queryPassThrough option

This module allows configuring a list of articles to show.

Behind the scenes the automated listing module uses (for most of the criteria that can be defined through its configuration UI) Redis lists to get the list of articles that it needs to show.

Using a custom list

Configuration UI

One needs to add the option to allow editors to select this new listing type:

<?php

namespace App\Bundle\CmsAdminBundle\Form\Extension;

use App\Bundle\CmsBundle\Entity\PageArticle;
use Symfony\Component\Form\AbstractTypeExtension;
use Symfony\Component\Form\Extension\Core\Type\ChoiceType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wf\Bundle\CmsBaseAdminBundle\Form\Helper\ListingSettingsHelper;
use Wf\Bundle\CmsBaseAdminBundle\Form\Type\ListingSettingsFormType;

class ListingSettingsFormTypeExtension extends AbstractTypeExtension
{
    protected ListingSettingsHelper $helper;

    protected TranslatorInterface $translator;

    public function getExtendedTypes()
    {
        return [ListingSettingsFormType::class];
    }

    /**
     * @required
     */
    public function setHelper(ListingSettingsHelper $helper)
    {
        $this->helper = $helper;
    }

    /**
     * @required
     */
    public function setTranslator(TranslatorInterface $translator)
    {
        $this->translator = $translator;
    }

    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add('content', ChoiceType::class, [
            'choices' => $this->getContentChoices(),
            'label' => 'form.listing.settings.content',
            'translation_domain' => 'WfCmsAdmin',
        ]);
    }

    protected function getContentChoices()
    {
        $choices = $this->helper->getContentChoices();
        $choices[$this->translator->trans('form.listing.settings.content.special_list', [], 'WfCmsAdmin')] = PageArticle::SETTING_SPECIAL_LIST;

        return $choices;
    }
}

Note: The ListingSettingsHelper service was added in v7.2

Note: The above code uses the @required annotation, used by Symfony's autowiring to inject the dependencies - the ListingSettingsFormTypeExtension does not need to be defined in YML/XML.

Other configuration fields

Expanding on the above example, if one needs to add custom configuration fields for this new option (e.g. when selecting "by category" option, an input is being shown to allow selecting the category), one can use for-field wf-form:

<?php
namespace App\Bundle\CmsAdminBundle\Form\Extension;

// ...

class ListingSettingsFormTypeExtension extends AbstractTypeExtension
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        // ...
        $builder->add('inputForSubcriteria', TextType::class, [
            'row_attr' => [
                'data-wf-form-type' => 'for-field',
                'data-wf-for-field' => 'content:'.PageArticle::SETTING_SPECIAL_LIST,
            ],
        ]);
    }
}

Note: One needs to show this input in the @AppCmsAdminBundle/Listing/form.html.twig:

{% extends "@WfCmsBaseAdmin/Listing/base_form.html.twig" %}

{% block wf_listing_group_content %}
    {{ parent() }}
    {{ form_row(listForm.settings.inputForSubcriteria) }}
{% endblock %}

Using the custom list

To use the custom list based on the configuration of the automated listing module, one would have to overwrite wf_cms.page_renderer.listing:getListName (class Wf\Bundle\CmsBaseBundle\PageRenderer\ListingPageRenderer):

<?php

namespace App\Bundle\CmsBundle\PageRenderer;

use App\Bundle\CmsBundle\Entity\PageArticle;

class ListingPageRenderer extends \Wf\Bundle\CmsBaseBundle\PageRenderer\ListingPageRenderer
{
    protected function getListName($listing)
    {
        $settings = $listing->getSettings();
        if ($settings['content'] == PageArticle::SETTING_SPECIAL_LIST) {
            return PageArticle::LISTING_SPECIAL;
        }

        return parent::getListName($listing);
    }
}

v7+ Multiple criteria

Intro

XalokNext:v7 adds a new type of automated listings based on ElasticSearch that allows filtering by multiple criteria.

Setup

To enable this functionality within your project, you need to add the following parameter:

wf_cms_admin.listing.multiple_criteria_enabled: true

Configuration

Upon enabling the feature, a new field labeled multipleCriteria will be presented in the dropdown menu. Selecting this option enables you to filter listings by multiple authors, categories, article types, and tags simultaneously.

How does it work

This enhanced listing functionality utilizes the same search mechanism as the advanced search feature. For detailed insights into its workings, refer to the provider Wf\Bundle\CmsBaseBundle\Listing\Provider\ListingMultipleSearchProvider. Additionally, this feature incorporates the same tag naming conventions as the Redis-based listing, tailored to the selected content types.

Listing utils

Intro

XalokNext offers some utilities to help working with listings outside using them for rendering HTML (e.g. for rendering the articles in a listing as JSON).

• wf_cms.listing.finder (alias Wf\Bundle\CmsBaseBundle\Listing\ListingFinder) that helps you find listings in boards.
• wf_cms.listing.pagination_helper (alias Wf\Bundle\CmsBaseBundle\Listing\PaginationHelper) useful to work with listings' paginations (e.g. can be used to render links to all pages, or to tell whether the listing has a next or previous page).
• vv7.2-next One can use wf_cms.page_renderer.listing (alias Wf\Bundle\CmsBaseBundle\PageRenderer\ListingPageRenderer) to reuse the logic of tagging the response for (Varnish) cache.

Usage

The following examples uses a mix of all the utils listed above. It assumes there's a board with slug category_secondary, it finds the listing defined inside this board for a given category, and renders the articles in this listing as JSON. Additionally, it uses the wf_cms.listing.pagination_helper to add to the JSON response information whether the current results also have next/previous pages.

// in src/App/Bundle/CmsBundle/Controller/CategoryController.php
<?php

namespace App\Bundle\CmsBundle\Controller;

use App\Bundle\CmsBundle\Entity\Category;
use App\Bundle\CmsBundle\PageRenderer\ListingJsonPageRenderer;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\ParamConverter;
use Symfony\Component\Routing\Annotation\Route;
use Wf\Bundle\CmsBaseBundle\Listing\ListingFinder;

class CategoryController extends \Wf\Bundle\CmsBaseBundle\Controller\CategoryController
{
    protected ListingFinder $listingFinder;
    protected ListingJsonPageRenderer $listingJsonPageRenderer;

    /**
     * @required
     */
    public function setListingFinder(ListingFinder $listingFinder): void
    {
        $this->listingFinder = $listingFinder;
    }

    /**
     * @required
     */
    public function setListingJsonPageRenderer(ListingJsonPageRenderer $listingJsonPageRenderer): void
    {
        $this->listingJsonPageRenderer = $listingJsonPageRenderer;
    }

    /**
     * @ParamConverter("category", class="AppCmsBundle:Category")
     *
     * @Route("/listing/{slug}/{page}", name="category_listing", defaults={"page": 1})
     */
    public function listingAction(Category $category, $page)
    {
        $listing = $this->listingFinder->findListingInEntityBoard(
            'category-secondary', 
            $category
        );

        if (!$listing) {
            throw $this->createNotFoundException('Listing not found');
        }

        $this->listingJsonPageRenderer->setPage($page);

        $response = $this->listingJsonPageRenderer->render($listing);

        return $response;
    }
}

// in src/App/Bundle/CmsBundle/PageRenderer/ListingJsonPageRenderer.php
<?php

namespace App\Bundle\CmsBundle\PageRenderer;

use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Serializer\SerializerInterface;
use Wf\Bundle\CmsBaseBundle\Entity\Page;
use Wf\Bundle\CmsBaseBundle\Listing\PaginationHelper;
use Wf\Bundle\CmsBaseBundle\PageRenderer\ListingPageRenderer;
use Wf\Bundle\CommonBundle\Serializer\SerializerUtil;

class ListingJsonPageRenderer
{
    protected ListingPageRenderer $listingPageRenderer;
    protected PaginationHelper $paginationHelper;
    protected SerializerInterface $serializer;
    protected $page;

    /**
     * @required
     */
    public function setListingPageRenderer(ListingPageRenderer $listingPageRenderer): void
    {
        $this->listingPageRenderer = $listingPageRenderer;
    }

    /**
     * @required
     */
    public function setPaginationHelper(PaginationHelper $paginationHelper): void
    {
        $this->paginationHelper = $paginationHelper;
    }

    /**
     * @required
     */
    public function setSerializer(SerializerInterface $serializer): void
    {
        $this->serializer = $serializer;
    }

    public function setPage($page)
    {
        $this->page = $page;
        $this->listingPageRenderer->setPage($page);
    }

    public function render(Page $listing)
    {
        $data = $this->listingPageRenderer->getTemplateData($listing);

        $response = new JsonResponse([
            'pages' => $this->serializer->normalize(
                $data['pages'],
                'json',
                SerializerUtil::createContextForGroups('list')
            ),
            'hasNextPage' => $this->paginationHelper->hasNextPage(
                $listing,
                $this->page
            ),
            'hasPreviousPage' => $this->paginationHelper->hasPreviousPage(
                $listing,
                $this->page
            ),
        ]);

        return $this->listingPageRenderer->processResponse(
            $response,
            $listing
        );
    }
}

Note: The code above relies on using Symfony's autowiring (see the @required annotations) to inject the dependencies - no YML/XML configuration is needed.

Note: The methods from wf_cms.listing.pagination_helper in the example above are passing the $listing directly. Optionally, they also support a method($category, 'category-secondary', $currentPage) signature - similarly to the wf_cms.listing.finder:findListingInEntityBoard method, this would find the listing defined in the board with slug category-secondary corresponding to the $category entity.

Note: The controller action uses the @ParamConverter annotation to automatically fetch the Category entity from the URL parameter slug. You are encouraged to use this, as it has the additional benefit that it throws a 404 exception if there's no such category, a check that sometimes the developers forget, leading to 500 exceptions down the line when the not found $listing is being used.

Commands

Publish populate

See the full article

Debug article lists

XalokNext comes with wf:cms:publish:debug command that helps debugging the Redis lists for a given article.

It compares the lists where the article appears in Redis with the lists that would be populated if the article was republished at the moment. These might be out of sync if, for example, the article was published before a new list was added to the system. Or if the article was manually edited in the database.

_Example_: 

```shell
> ./app/admin/console wf:cms:publish:debug 114
Page details
ID: 114
Title: test redis
Template: default
Main category: 67 (Noticia)
Categories:
 61 (Categoría 1ra)
 67 (Noticia)
Authors:
 12 (admin)
Currently in 19 lists
Repopulating would result in 19 lists
  dske:es:latest:category:67 FOUND total:7
  dske:es:latest:category:67:full FOUND total:7
  dske:es:latest:category:61:full FOUND total:11
  dske:es:latest FOUND total:106
  dske:es:latest:secondary:category:67:full FOUND total:7
  dske:es:latest:secondary:category:67 FOUND total:7
  dske:es:latest:secondary:category:61 FOUND total:11
  dske:es:latest:secondary:category:61:full FOUND total:11
  dske:es:latest:secondary:category:67 FOUND total:7
  dske:es:latest:secondary:category:61 FOUND total:11
  dske:es:latest:author:12 FOUND total:20
  dske:es:latest:author:12:67 FOUND total:7
  dske:es:latest:author:12:61 FOUND total:11
  dske:es:latest:template:default FOUND total:106
  dske:es:latest:template:default:67 FOUND total:7
  dske:es:latest:template:default:61 FOUND total:11
  dske:es:latest:image FOUND total:18
  dske:es:latest:image:67 FOUND total:2
  dske:es:latest:image:61 FOUND total:3
Doesn't need repopulating