Inhalt abgleichen
Drupal.org - aggregated feeds in category Planet Drupal
Aktualisiert: vor 2 Minuten 21 Sekunden

2bits: When Memcached Slows Your Drupal Site's Performance

12. Februar 2018 - 18:09

For all of the sites we consult on, and manage, we use the excellent memcache module, which replaces the core's database caching. Database caching works for low traffic simple sites, but cannot scale for heavy traffic or complex site.

Recently we were asked to consult on the slow performance of a site with an all authenticated audience. The site is indeed complex, with over 235 enabled modules, 130 enabled rules, and 110 views.

The site was moved from a dedicated server to an Amazon AWS cluster, with the site on one EC2 instance, the database on an RDS instance, and memcache on a third instance. This move was in the hope that Amazon's AWS will improve performance.

However, to their dismay, performance after the move went from bad (~ 5 to 7 seconds to load a page) to worse (~ 15 seconds).

We recommended to the client that we perform a Drupal performance assessment server on the site. We got a copy of the site and recreated the site in our lab, and proceeded with the investigation.

After some investigation we found that by having memcached on the same server that runs PHP made a significant performance improvement. This was logical, because with a site with all users logged in, there are plenty of calls to cache_get() and cache_set(). Each of these calls have to do a round trip over the network to the other server and back, even if it returns nothing. The same goes for database queries.

Instead of 29.0, 15.8, 15.9, and 15.5 seconds for different pages on the live site, the page loads in our lab on a single medium server were: 3.6, 5.5, 1.4, 1.5 and 0.6 seconds.

However, this victory was short lived. Once we put load on the web site, other bottlenecks were encountered.

We started with 200 concurrent logged in users on our lab server, and kept investigating and tweaking, running a performance test after each tweak to assess its impact.

The initial figures were: an average response time of 13.93 seconds, and only 6,200 page load attempts for 200 users (with 436 time outs).

So, what did we find? We found that the culprit was memcache! Yes, the very thing that helps site be scaleable was severely impeding scalability!

Why is this so? Because of the way it was configured for locking and stampede protection.

The settings.php for the site had these two lines:

$conf['lock_inc'] = 'sites/all/modules/memcache/memcache-lock.inc';
$conf['memcache_stampede_protection'] = TRUE;

Look at the memcache.inc, lines 180 to 201, in the valid() function:

if (!$cache) {
  if (variable_get('memcache_stampede_protection', FALSE) ... ) {
    static $lock_count = 0;
    $lock_count++;
    if ($lock_count <= variable_get('memcache_stampede_wait_limit', 3)) {
      lock_wait(..., variable_get('memcache_stampede_wait_time', 5));
      $cache = ...;
    }
  }
}

The above is for version 7.x of the module, and the same logic is in the Drupal 8.x branch as well.

If memcache_stampede_protection is set to TRUE, then there will be up to three attempts, with a 5 second delay each. The total then can be as high as 15 seconds when the site is busy, which is exactly what we were seeing. Most of the PHP processes will be waiting for the lock, and no free PHP processes will be available to serve requests from other site visitors.

One possible solution is to lower the number of attempts to 2 (memcache_stampede_wait_limit = 2), and the wait time for each attempt to 1 second (memcache_stampede_wait_time = 1), but that is still 2 seconds of wait!

We did exactly that, and re-ran our tests.

The figures were much better: for 200 concurrent logged in users, the the average response time was 2.89 seconds, and a total of 10,042 page loads, with 100% success (i.e. no time outs).

But a response time of ~ 3 seconds is still slow, and there is still the possibility of a pile up condition when all PHP processes are waiting.

So, we decided that the best course of action is not to use memcache's locking at all, nor its stampede protection, and hence deleted the two lines from settings.php:

//$conf['lock_inc'] = 'sites/all/modules/memcache/memcache-lock.inc';
//$conf['memcache_stampede_protection'] = TRUE;

The results were much better: for 200 concurrent logged in users, the average response time was 1.09 seconds, and a total of 11,196 pages with 100% success rate (no timeouts).

At this point, the server's CPU utilization was 45-55%, meaning that it can handle more users.

But wait! We forgot something: the last test was run with xhprof profiler left enabled by mistake from profiling the web site! That causes lots of CPU time being used up, as well as heavy writes to the disk as well.

So we disabled xhprof and ran another test: and the results were fantastic: for 200 concurrent logged in users, the average response time was just 0.20 seconds, and a total of 11,892 pages with 100% success rate (no timeouts).

Eureka!

Note that for the above tests, we disabled all the rules, disabled a couple of modules that have slow queries, and commented out the history table update query in core's node.module:node_tag_new(). So, these figures are idealized somewhat.

Also, this is a server that is not particularly new (made in 2013), and uses regular spinning disks (not SSDs).

For now, the main bottleneck has been uncovered, and overcome. The site is now only limited by other factors, such as available CPU, speed of its disks, complexity of modules, rules and views ...etc.

So, check your settings.php to see if you have memcache_stampede_protection enabled, and disable it if it is.

Contents: Tags: 

DrupalEasy: DrupalEasy Podcast 205 - Fatima Khalid - First time sprinter's guide

12. Februar 2018 - 13:47

Direct .mp3 file download.

Fatima Khalid (sugaroverflow), web developer with Digital Echidna, and DrupalCon Nashville track chair and sprint mentor joins Mike Anello to talk about how to be a first-time sprinter at a local Drupal event or a DrupalCon and how she came for the community and stayed for the code. Along the way, we talk about Canadian Cheerios, the importance of issue queue triage, and (alleged) creepy monkey guy.

Interview DrupalEasy News Upcoming events Sponsors Follow us on Twitter Subscribe

Subscribe to our podcast on iTunes, Google Play or Miro. Listen to our podcast on Stitcher.

If you'd like to leave us a voicemail, call 321-396-2340. Please keep in mind that we might play your voicemail during one of our future podcasts. Feel free to call in with suggestions, rants, questions, or corrections. If you'd rather just send us an email, please use our contact page.

Lucius Digital: Drupal 8 development: always redirect all logged out visitors to the login page

12. Februar 2018 - 13:11
Login or get out!

Currently we are busy constructing the production of a realtime messaging platform in Drupal and NodeJS, look at it as a ‘WhatsApp for Business’. This Drupal system works like a web app; logging in is mandatory. How do you make sure that logged out visitors must log in to Drupal 8 before they are allowed to continue?

Drupal has many out-of-the-box functionalities, as well as a powerful API, but because it has so many functions many tracks are standardly available for anonymous visitors. We’d want to make all paths unreachable, until you log in.

That means that visitors always will be redirected to the login screen as long as they aren’t logged in. You wouldn’t want an anonymous user reaching internal news on the homepage.

Redirect URL in Drupal 8

Basically, we want all url’s / paths be made unavailable for non-logged in visitors, except explicitly specified pages like:

  • Login (/user)
  • Forgot password (/user/password)
  • Login link (user/reset/login)

in Drupal 7 you could use the module Logintoboggan for that purpose. You could also easily work around it in hook_init() or hook_boot() in a custom Drupal 7 module.

Quest

This was quite a puzzle, and we soon found some examples as well as exceptions. Everytime it didn’t work how we wanted it to. This example was the most useful.

Implementation in Drupal 8

Eventually, we got it working with the help of following code in a custom Drupal 8 module:

services.yml

put this file in your module root, and format yourmodulename.services.yml:

https://medium.com/media/20c294c1890ad778074f8276d5febad1/hrefRedirectAnonymousSubscriber.php

Put the file RedirectAnonymousSubscriber.php in folder /src/EventSubscriber/ and do your custom thing:

https://medium.com/media/1723313a8d58061c5f36a77f32dac0e9/href

This code builds on symfony’s EventSubscriber, the framework on which Drupal8 has been built.

Wrap up

Alright, that’s it. I hope the information as described will help you to always redirect visitors to the login page. Questions or feedback? Let me know!

Drupal 8 development: always redirect all logged out visitors to the login page was originally published in Lucius Digital | Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

OSTraining: Should I Re-use Existing Drupal Fields?

12. Februar 2018 - 7:14

Sometimes we're able to give really clear advice: "Do this!" or "Don't do that!"

This is not going to be one of those blog posts.

Drupal gives you the ability to re-use fields. If you have an "Image" field, you could choose to use that same field on every content type on your site.

However, it's not always clear whether re-using fields is a good idea. Sometimes it is, sometimes it isn't.

Here's an overview of the advantages and disadvantages to consider before re-using Drupal fields.

ADCI Solutions: Top Design Trends for 2017-2018

11. Februar 2018 - 4:39

Development and design always go hand in hand, and our Drupal development team pays a lot of attention to design trends so that we can visualize our solutions in the most beneficial way.

Our lead web designer has come up with a cool review on design trends for 2018. Read and implement!

 

Check design trends for 2018

 

DrupalCon News: You should stay at a DrupalCon partner hotel. Here’s why.

10. Februar 2018 - 1:12

The hotels we chose are a perfect hub connecting you to a rewarding DrupalCon experience. It’s also a great way to show up for the community.

Valuebound: How to debug Drupal 8 website performance using Web Profiler

9. Februar 2018 - 16:31

Over the period of time, website performance degrades significantly affecting the business in terms of traffic, sales, marketing etc. And there can be n-number of reasons for the poor performance of the website. In this scenario, performance enhancement is one of the solutions to keep your website as well as the business running efficiently.

Well, there are various tools to debug the performance of web applications. In case your website is running on Drupal 8, then you need not worry as there are various contributed modules that help to identify the reasons and debug the site. Here we will explore how Web profiler helps in enhancing Drupal 8 site performance.…

Amazee Labs: A brand new website for Zürich Tourismus

9. Februar 2018 - 14:30
A brand new website for Zürich Tourismus

Zürich Tourismus is responsible for the local and international marketing of the city of Zurich and the surrounding region. They aim to showcase the region as both a fascinating tourist destination and the perfect place to hold meetings and congresses.

Nicole Blum Fri, 02/09/2018 - 13:30 The Challenge

The goal of the website was to increase conversion rates and user retention. This required a new structure on the individual pages and the integration of user-generated content.
The website's authenticity is reinforced by offering users the option to show off the best sides of the city on the social channels. Lastly, our offer also included a quick and easy-to-use search function with a map and the option to search within a certain radius.

Our Idea

The new CI/CD would be integrated into all of the content and design elements on the website. The website should have a lot of empty white space which draws the user's focus to the images and videos. The left-justified design and uniform teaser layout create a feeling of order on each individual page. A specially designed company font and the predefined colour palette were used for the website's design. A cluster of topics and the colours used as an additional branding element in the navigation would also be implemented.

The Result

With an immersive video as the point of entry, supplemented by user-generated content, the website not only inspires trust in the user but also builds an emotional connection with them. Interactive options such as the option to send enquiries to service providers (hotels, restaurants, etc.) or share content allow for and inspire additional user interaction, eliminate obstacles which could discourage users from booking services, and let the user know that they are fully informed, have chosen the best activities for their stay in Zurich, and have not missed anything.

With the new branding, which moves from playful to minimalist, the new website is timeless and modern with a touch of "Swissness" thanks to the references to the heyday of Zurich-based design in the 1960s. The website's appearance is rounded out by the specially designed company font - based on Helvetica - and the expanded colour palette, which is inspired by the tram lines that run through Zurich.

Lullabot: The Layout Initiative

9. Februar 2018 - 5:06
Mike and Matt talk to Acquia's Tim Plunkett and Emilie Nouveau about the Layout Initiative, which aims to build layout building functionality into Drupal core similar to Panels / Display Suite. Note that the Layout Initiative did officially make it into Drupal 8.5!

Agiledrop.com Blog: AGILEDROP: How you can try Commerce 2 for Drupal 8

9. Februar 2018 - 4:13
I always felt that any website should be a tool for driving sales. And there is no better way than enabling customers to make a purchase right there on your website. Anf now you can do it with Drupal 8. TLDR: Acro Media created a demo Drupal 8 demo site with Drupal Commerce 2 and made it available on Github. Building commerce websites is not easy. The only way to make it easier is to use a service like Shopify, where there are presets available that will cover requirements for 80% of entrepreneurs doing their first online commerce venture.  Sooner or later things get complicated ...… READ MORE

MidCamp - Midwest Drupal Camp: Call for Volunteers

8. Februar 2018 - 4:33
Call for Volunteers We need you!

Want to give back to the Drupal Community without writing a line of code? Volunteer to help out at MidCamp 2018.  We’re looking for people to help with all kinds of tasks including: 

Setup/Teardown
  • For setup, we need help making sure registration is ready to roll, getting T-shirts ready to move, and getting the rooms prepped for our amazing sessions.

  • For teardown, we need to undo all the setup including packing up all the rooms, the registration desk, cleaning signage, and making it look like we were never there.

Registration and Ticketing
  • We need ticket scanners, program dispersers, and people to answer questions.

Room Monitors
  • Pick your sessions and count heads, make sure the speakers have what they need to survive, and help with the in-room A/V.

If you’re interested in volunteering or would like to find out more, please contact us.

Sign up to Volunteer!

MidCamp - Midwest Drupal Camp: Call for Volunteers

8. Februar 2018 - 4:33
Call for Volunteers We need you!

Want to give back to the Drupal Community without writing a line of code? Volunteer to help out at MidCamp 2018.  We’re looking for people to help with all kinds of tasks including: 

Setup/Teardown
  • For setup, we need help making sure registration is ready to roll, getting T-shirts ready to move, and getting the rooms prepped for our amazing sessions.

  • For teardown, we need to undo all the setup including packing up all the rooms, the registration desk, cleaning signage, and making it look like we were never there.

Registration and Ticketing
  • We need ticket scanners, program dispersers, and people to answer questions.

Room Monitors
  • Pick your sessions and count heads, make sure the speakers have what they need to survive, and help with the in-room A/V.

If you’re interested in volunteering or would like to find out more, please contact us.

Sign up to Volunteer!

Tandem's Drupal Blog: Migrating a Drupal 7 Body Field to a Drupal 8 Paragraph

8. Februar 2018 - 2:00
February 08, 2018 Migrating A Drupal 7 long text field to a Drupal 8 nested text field in a paragraph is a two step process. It is quick and easy once you get the basic grasp of the mechanisms involved. Why are we doing this? We were hired on as to help consult with one of our University clients migration from Drupal 7 to Drupal 8. One of the ...

Roy Scholten: An experiment in removing friction

8. Februar 2018 - 0:28
07 Feb 2018 /sites/default/files/styles/large/public/form-display.png?itok=sKIMMJvv An experiment in removing friction

An experiment triggered by Dries’s blog post To PESOS or to POSSE? and my comment there about the friction in the UI’s for the POSSE approach. The Drupal content creation UI is not as optimized as the ones in native, mobile apps for Twitter, Instagram and the like.

I was curious how far I could take things just by stripping down the default Drupal content creation UI, optimizing it for posting an image + caption as an entry to this here blog.

What I did:
  1. Create a custom, stripped down content creation form
  2. Use evil CSS to hide superfluous items
Custom form mode

The first step was to remove most of the fields that are not required to create a post. The Form Mode Manager module lets you do just that. I set up a “Minimal piece” form mode which hides the fields for inputs like a separate summary, tags, file attachments, url alias, author, date, sticky and the “promoted” checkbox. These are optional (not required) fields that can be left empty or provide sensible defaults (me as the author, use “now” for the publishing date).

This alternative form comes with its own URL I can go to to create that new post.


Evil CSS

The stripped down form still shows bits and pieces I don’t really need to be reminded of every time I go to post something. Nothing a few display: none; CSS staments couldn’t fix. But, I’m using the default Seven admin theme here, which does not provide a body class to uniquely identify a specific page. I couldn’t add my CSS hacks to that theme’s (or sub theme’s) stylesheet because those would then apply to all admin pages.

I use the Firefox browser on my Android phone. Stylish to the rescue. We’re definately in proof-of-concept territory now! Stylish lets you define custom styling for all sites, a site or one or more specific URLs. I made one specifically for the URL where the minimal entry form lives to clean up that screen even more: hiding the header, breadcrumbs, form field descriptions, etc.


And, does it work?

Yes. It’s far from elegant but it’s already a much more focussed and faster experience. I think I already knew but was happily surprised to see that the regular image field does provide access to both the camera (for taking a new picture) and the gallery (for choosing an existing image).


What could be next

One thing that makes the Drupal approach feel old school is the required title field. Twitter, Instagram, Tumblr all let you just type something into what basically amounts to a “body text” field. In Drupal it has to at least be just like email: a subject and a message: a title + a body. I can make the body optional but I want my text to go in there. Hiding the title field is not straightforward, I think because it is tightly coupled with the unique ID for that content item. There’s modules that let you hide the title input field but those still automatically generate a (gibberish) title.

Still, my focus was on smoothing the path for posting an image to my blog from my phone. That “Browse…” button is old school too, but it does provide access to camera and camera roll. Saving a bookmark for this comes with the option to create a shortcut icon for it on the phone home screen for quick access. Not bad for an hour or two of experimenting without coding :)

A couple of years ago swentel created Drupoid, an Android app that let you save an image from your phone as a content item on a Drupal site. It would be an interesting exercise to recreate something like it with React, Drupal’s (future) JavaScript toolkit of choice and figure out how smooth and snappy we can make it there.

Tags Drupal drupalplanet ux process

myDropWizard.com: Drupal 6 security update for FileField Sources!

7. Februar 2018 - 22:23

As you may know, Drupal 6 has reached End-of-Life (EOL) which means the Drupal Security Team is no longer doing Security Advisories or working on security patches for Drupal 6 core or contrib modules - but the Drupal 6 LTS vendors are and we're one of them!

Today, there is a Moderately Critical security release for the FileField Sources module to fix an Access Bypass vulnerability.

This module enables you to upload files to fields via several sources.

The module doesn't sufficiently handle access control on the autocomplete for reference sources.

See the security advisory for Drupal 7 for more information.

Here you can download the Drupal 6 patch.

If you have a Drupal 6 site using the FileField Sources module, we recommend you update immediately! We have already deployed the patch for all of our Drupal 6 Long-Term Support clients. :-)

If you'd like all your Drupal 6 modules to receive security updates and have the fixes deployed the same day they're released, please check out our D6LTS plans.

Note: if you use the myDropWizard module (totally free!), you'll be alerted to these and any future security updates, and will be able to use drush to install them (even though they won't necessarily have a release on Drupal.org).

Drupal Association blog: Updating the DrupalCon Code of Conduct

7. Februar 2018 - 18:50

DrupalCon has long had a Code of Conduct that we all agree to as part of participating in the event. It was originally derived from the Drupal Code of Conduct and added sections specific to in-person events.

We are constantly striving to improve all that we offer to make DrupalCon a great event for all Drupal professionals and it is now time to make a change to our DrupalCon Code of Conduct.

Making a positive change

When I first started at the Drupal Association at the beginning of the year, I was very pleased to discover that the Association staff had already made significant progress on an updated version of the DrupalCon Code of Conduct that included clearer description of what is expected of participants at DrupalCon, where various responsibilities were held for enforcement and what consequences should be expected for falling foul of the code. As the new Community Liaison, though, it was important that I took this work to the Drupal and wider web professional community and gained further insight into producing something that worked for us all.

Consulting the community

Having spent time on the Community Working Group, I was aware that consulting the community at large is hugely important but can be a huge undertaking that can potentially become never ending. Indeed, when the original Drupal Code of Conduct was in development, the process took many months to reach a result. Whilst we wanted to involve people, we also wanted to deliver something better than the current Code in time for DrupalCon Nashville 2018.

So, my methodology was to choose eight people from different parts of our community to comment on the draft Code of Conduct, plus ask them to nominate two extra people who I should also ask to make comment. I requested that they nominate someone who they felt would have a similar outlook to themselves and someone they thought might have a different outlook. This meant we had a focused team of people from the community that could work on this over a very short period, but one that I knew wouldn’t be overly influenced by my personal choices. I was also careful to choose people from several continents.

The vast majority of suggestions and comments they proposed were accepted, either directly or with some adjustment by either Association staff or our lawyers. At the end of the day, the code has to be legal, enforceable and protect both conference participants and the project’s legal entities.

The New Code

The draft has now been reviewed by Drupal Association staff, Drupal Community members and the Drupal Association Lawyers. It is ready to go and will now be applied to DrupalCon Nashville 2018.

I do want to talk about a few of the improvements we have made, though.

Clarity

One of the challenges of the old Code was that, whilst being very positive, it failed to make it clear to all DrupalCon participants exactly what was expected of them. We have taken the time to write about what behaviors are and are not acceptable in the context of DrupalCon. You will also see that we clearly state that the expectations of leaders at DrupalCon are expected to uphold even higher standards.

Our actions have consequences

We have also added sections that detail exactly what is expected of people and how failing those requirements will be dealt with.

One of the significant changes here is we separated the responsibilities for dealing with unacceptable behavior from that of supporting participants in resolving conflicts. Who to contact in the case of each is more clearly displayed, along with exactly who will be initially handling reported incidents.

Best practices

We didn’t just make all this up. We spoke with many organisations and used well-known best practice where it was available. We have detailed where much of the best practice was found at the end of the Code of Conduct.

We are also sharing the new DrupalCon Code of Conduct under a Creative Commons license, as before. If you are organising a Drupal event, you are very welcome to make use of this and add in your own contact information.

Release early, release often

Finally, I want to talk about the future. As mentioned earlier, whilst we consulted members of the Drupal Community, we were not able to incorporate all their comments at this time. Some comments would simply not have met our objectives to protect the Community and the Project but others, whilst beneficial, require further development that would mean we would not be ready for Nashville. There are also other initiatives taking place within the overall Drupal Project, like the refreshing of the Drupal Project’s value statement and a Drupal Community Code of Conduct, that we would want to incorporate back into the DrupalCon Code of Conduct. I’m sure the aforementioned best practices will evolve over time, too, and I want a mechanism where we can take advantage of that.

Well, we already have an established (and, I feel, successful!) process in Drupal Core; which now makes releases on a regular schedule. I propose that we make an annual DrupalCon Code of Conduct release, at this time each year. I will be setting up a project to manage this. So, if you believe there are ways we can still improve our Code of Conduct, there will be a mechanism to achieve it.

Dropsolid: Drupal 8 config management (part 3)

7. Februar 2018 - 17:10
07 Feb Config management in Drupal 8: use cases (part 3/3) Niels A Drupal Drupal 8

In part 2 of this three-part series, we showed you how to set up you config management flow. In this part we’ll be looking at some use cases and common issues related to config management.
 

Back to part 1     Part 2


Starting on a new issue, or finishing one

There is an important must-do before you start developing on a new issue, after you’ve pulled the latest code with Git. Don’t forget to always do an import of the configuration first! It should become second nature to do a “drush config-import -y” or shorthand “drush cim -y” just after a Git pull or checkout.

When you finish one, you should do a “drush config-export -y” or shorthand “drush cex -y” and check the exported files before committing your changes.


Live-specific configuration

To set live-specific configuration, you have two options:

  1. Set the config using the settings.php file directly on that environment
  2. Make split that is active on that environment

The general rule is to export your configuration in the sync folder and override for the environment that differs.

The following example should make this a bit clearer. Imagine you have an API that you need to connect to. Your local, development and staging environment should all connect to the development version of the API, while the live environment should connect to the live API. 

If you have a lot of configuration to do for your API connection, you can choose to use config split to solve your problem. We can state that most environments need to connect to the development API; therefore this config should be stored in the sync folder, so it gets imported onto all environments. We then need to override that config for the live environment by using a config split that is only active on the production environment. 

In practice, you would go about it in the following way:

  1. Make sure you have set up your environment settings files as described above
  2. Configure the dev settings in the configuration form
  3. Use “drush cex” to export all updated configuration
  4. Check Git if your changes have been added to the sync folder
  5. Go back to the config form and configure the live settings as if you were on a live site
  6. Use “drush csex live” to export your updated configuration only to your live split folder based on the live split settings.
  7. Check git if your changes have been added to the live sync folder
  8. Commit all your changes 
  9. Use “drush cim” to import all the config again according to your currently active splits

Important: Make sure your drush version is higher than 8.1.10 or it won’t pickup the splits if you use drush cex or drush cim. Be aware that you still need to use csex and csim if you want to export or import from specific splits.
 

When you are adding configuration that should be ignored or split

If you are just creating new functionality and some of the configuration should be ignored or split. Beware that on the first import you do on other environments the config that should have been ignored or split won’t be.

This is because the newly added config to the Config Ignore or Config split module has not been taken into account yet. Sometimes, this can be to your advantage - but more often, the contrary is true.

The way to get around this, is to manually add the “Config Ignore” or “Config Split” config to the environments where you want to deploy or take a two-step approach at deployment.


Avoid changing your default language

Changing your default language after you’ve already been exporting and importing your config can result in some very strange side effects with translations. 

If you ever find yourself in the situation, a good fix is to manually change the langcode in the yml files in your sync folder to your default language again. Also make sure that the strings you’ve used are in the same language as defined by the langcode.

A good rule to bear in mind: Make sure all your config exports are done in the same language. This sometimes leads to errors, because it uses the language defined by the language negotiation plugins. If you only use Drush to import and export your config, this module can help enforcing a specific language. It will force all your Drush commands to be done in a language you define. The Dropsolid dev team be working together with Bart (Xano on Drupal.org) to see if we can help him get the module to a stable release.


Config translations can be tricky

You can easily translate your configuration using the Config Translation module from core. This sometimes can lead to unexpected side effects if you consider translations to be in the grey zone between config and content.

The issue here is that config translation and interface translation sometimes overlap - take for example the name of your content type. It is stored not only in config translation, but also in string translation

When you do an import, it will by default overwrite the translation you’ve set in interface translation. This is because the locale module has a subscriber that imports those strings on the import event. (See this issue.)
Be aware that this can even happen to custom translated strings that you use in your own code. If there happens to be a same string that is used in config, it will get overridden. In this case you can always assign a context to your translated string so it doesn’t get overridden. 

For those cases where you can’t set a context on the translated string, we’ve developed and contributed the Locale: Config import module. This module will let you change the behaviour of how the translated config gets imported.

The module shows the following options:


Agreeing on best practices for those grey areas for content

As a team of developers, you should agree in advance on a couple of best practices concerning content grey areas. Below I’ve listed a few rules that we’ve been playing with at Dropsolid:

  • If you create a custom block plugin, use translatable markup with context for titles.
  • Initially, treat Roles and permissions as config, until you go live. Next, configure the Config Ignore module to ignore them and update them through update hooks.
  • Always treat webforms as if they are content. Deploy changes through update hooks

 

Working towards a better flow


I hope this three-part post has provided you with some valuable insight!
This article has been a true team effort, so thanks to everyone who contributed (most notably Brent, Marek, Laurens, Kevin, Thomas and Nick). Together, we can make configuration management a little bit clearer for everyone who uses it!

No doubt this article will be changing through the coming months as everyone - including the team here at Dropsolid - is still looking to find that perfect workflow.
 

Back to part 1     Blog overview

Dropsolid: Drupal 8 config management (part 2)

7. Februar 2018 - 17:01
07 Feb Config management in Drupal 8: the set-up (part 2/3) Niels A Drupal 8 Drupal

In the first part of this three-piece blogpost, we explained the existing options for managing your configuration across environments. We also shed some light on grey areas between configuration and content. In this part, I’ll be showing you how to set it all up.
 

Back to part 1     Skip to part 3

 

Setting up configuration management Configuring the splits

Start by installing the config split module, available here.

We like to keep our splits simple and clear. Currently, have four splits:

  • Blacklist: A split that doesn’t have a folder configured, so it doesn’t end up in our Git repository. This split usually has the same configuration as the config ignore module. The reason why will be explained in the part about config ignore.
  • Dev: A split that is used for our development environment-only configuration
  • Staging: A split that is used for our staging environment-only configuration
  • Live: A split that is used for our live environment only-configuration

Our blacklist configuration looks like this:

If you don’t fill in the folder name, the Config Split module will export the config to a table in the database. That way, it doesn’t pollute your Git commits. For our other splits we use a folder outside our docroot for security reasons - e.g. for a dev split: ../config/splits/dev

Another important value here is the weight of the split. This weight will be used when multiple splits are active. The weight defines the order in which conflicting settings from splits get resolved. I will explain this in the part about the settings files, a little further down this post.

For the active checkbox you can choose whatever you want, because we’ll overwrite it later using the settings.php file. 

We’ve only been actively using the complete split configuration because it is pretty straightforward in how it handles your configuration. The additional configuration is handy because you can use wildcards. In the case of our blacklist, for example, we wanted to exclude everything from webform by using “webform.*”

We haven’t come across any use cases where we needed the conditional split, because we have a split active on every environment or the environment just needs the configuration from the sync folder.

If you were to use it, the following use case would be perfect for it. For instance, you have some configuration you need on all environments but you want it to be different on one environment. In our example we want a different email to be set in the reroute e-mail settings on the staging environment. You would change it in the interface and then do a “drush csex staging” to export it to the config split staging folder. This will allow the config file to be present in the sync folder and in the split folder. 


Configuring ignore

 

First, install the config ignore module.

The important thing to know about this module is that it only ignores config on import. This is why we have currently set up a blacklist split. A new module called Config Export Ignore (available here) is also at your disposal, but we haven’t used it in our workflow because it doesn’t have a stable release yet. We will look into this in the future though, because the “blacklist” split feels like a hack in some way.

Our default config ignore settings look like this:

As you can see, you can also add some special rules to your ignore settings. The only downside is that config split does not support these special rules. You’ll have to tweak your blacklist split a little bit and make your peace with the added files in your repo, until we find or create a solution for this. 


The settings files

To get this working across all environments, you need to have a settings file that is aware of its environment. This can be done using an environment variable that you check or use different files that get deployed based on the environment. There you set which config should be active on which environment. Sometimes it happens that you need multiple splits to be active on a single environment. You need to consider these splits as layers of config that you want to manage across environments. 

Our settings.php file on our development environment is identical to our local environment. It contains the following lines:

// Set your config directory outside of your docroot for security $config_directories[CONFIG_SYNC_DIRECTORY] = '../config/sync'; // Configure config split directory $config['config_split.config_split.blacklist']['status'] = TRUE; $config['config_split.config_split.dev']['status'] = TRUE; $config['config_split.config_split.staging']['status'] = TRUE; $config['config_split.config_split.live']['status'] = FALSE;

You might think that setting the staging split to true on the dev/local environment might be a mistake, but this is very intentional. As we are mostly using the complete split settings, this means that a config file can only be in one place. So we leverage the weight of the splits to also have certain staging config on our dev/local environment active. 
For instance, if we only want the reroute e-mail module to be active on local/dev/staging environments, we would add it to the complete split settings of the staging split.

Our settings.php for our staging environment will look as follows:

// Set your config directory outside of your docroot for security $config_directories[CONFIG_SYNC_DIRECTORY] = '../config/sync'; // Configure config split directory $config['config_split.config_split.blacklist']['status'] = TRUE; $config['config_split.config_split.dev']['status'] = FALSE; $config['config_split.config_split.staging']['status'] = TRUE; $config['config_split.config_split.live']['status'] = FALSE;

Consequently, the settings.php for our live environment looks like this:

// Set your config directory outside of your docroot for security $config_directories[CONFIG_SYNC_DIRECTORY] = '../config/sync'; // Configure config split directory $config['config_split.config_split.blacklist']['status'] = TRUE; $config['config_split.config_split.dev']['status'] = FALSE; $config['config_split.config_split.staging']['status'] = FALSE; $config['config_split.config_split.live']['status'] = TRUE;

Our Rocketship install profile (blog post in Dutch) has all these things preconfigured, which enables us to create more value for our clients.


Deploying changes

When you want to deploy changes to your environments, the only thing you need to do is make sure the config files that you’ve exported using ‘drush config-export’ are on the environment you want to deploy. Using a versioning system like Git can greatly help you manage those files across environments. 

When you are ready to deploy your changes, just run the “drush config-import” command on that environment. Settings up drush aliases can save you a lot of time when you want to execute drush commands on remote environments. Read more about them here on this Drupal.org page.

Once you have this in place, you’ll be able to easily manage your configuration across environments. Go check out part 3 if you want to explore a couple of real world use cases and issues we’ve encountered.
 

Revisit part 1     Read part 3

Dropsolid: Drupal 8 config management (part 1)

7. Februar 2018 - 17:00
07 Feb Config management in Drupal 8: tackling common problems and use cases (part 1/3) Niels A Drupal Drupal 8

This is the first of a series of three config management blog posts. In this series, we’ll help you set up a good starting point and provide you with a few solutions for everyday configuration issues. The first part of this multi-part blog post will provide you a bit of context. The second part goes into the nitty gritty of configuration management, and the third part demonstrates some concrete use cases, pitfalls and their solutions!

 

Features and configuration management

Drupal 8 has been around for a while now and at Dropsolid we have substantial experience with large projects that use both the contributed Features module and the core configuration management system.

Essentially, Features leverages the configuration management system to package configuration with your module and overwrites certain rules for importing config from a packaged module after first install.

The alternative is to use configuration management from Drupal 8 core with the contributed Config Split and Config Ignore module.

Config split lets you set up rules for splitting your configuration per environment. Config ignore lets you ignore certain configuration to be imported.

The way you handle configuration fundamentally differs between both solutions. Features lets you whitelist the configuration you want to import. Configuration management from Drupal 8 core with addition of the mentioned contributed modules works more like blacklisting the configuration you don’t want to import. As a developer I have always found it easier to narrow down what I want to have control over, instead of what I don’t want to have control over. 

We’ve come to the conclusion that the latter option actually works faster, which means more value for the client, but only if you have a good configuration to start from. As it turns out, configuration that you don’t want to control, is more often shared between different projects than config that you do want to control.

 

Content and config: blurred lines

One of the most wonderful things about Drupal 8 is config entities, a uniform way of creating and saving configuration throughout your site. It has been leveraged by many contributed modules to give end users a great experience in configuring their website. 

The downside of these configuration entities is that they often cross the line between what is considered content and configuration. We consider content as everything a client needs to be able to change. A good example of this are webforms. Every webform you create is an instance of a configuration entity - whereas the submissions of the webform are instances of a content entity. If you want to know more about the difference between config and content entities, I advise you to read this article by Acquia.

We want clients to have full control over the kind of webforms they create, so they can use them effectively across their site to gather leads.

This brings us to the following issue. As a company we believe that a website is an ever-changing platform that grows over time. To have it grow at the speed that customers require, we need to work on multiple features with multiple teams at once. It needs to be stable and traceable. 

Part of this stability and traceability is having what we developers define as structural configuration being versioned (git) and easily deployable, all without hindering the client to keep on working on their content. 

Thanks to configuration management, config split and config ignore we’ve been able to achieve all this for Drupal 8!

Ready to set up your configuration? Read on in part two of the blog post series!
 

Read part 2

Lullabot: Drupal 8 Release Planning in the Enterprise

7. Februar 2018 - 17:00

Yesterday, your CTO mandated that all new CMS builds would on Drupal 8 and that all existing CMS platforms would be consolidated to Drupal within the next three years. Great! Time to start prioritizing sites, hiring teams, and planning development and launch schedules.

The old methods of release planning don’t work anymore

Most enterprise IT organizations are used to a “development” cycle followed by a “maintenance” cycle, at which point you start over again and rebuild with a new development cycle. This type of planning (which Dave Hansen-Lange called the “boom-bust cycle”) led to poor user experiences with Drupal 7 sites as they became stale with time and is impossible to use for a Drupal 8 site.

Drupal 8 has a different release cycle compared to previous Drupal releases. For example, Drupal 7 was released in January 2011, and ever since has only had bug fixes (with the occasional developer-only feature added). Drupal 5 became unsupported, and Drupal 6 entered it’s last phase of support (until Drupal 8 was released). Product owners of sites big and small found this release process stifling. The web is a very different platform today compared to January 2011—although Drupal 7 is pretty much the same. The slow release cycle also reduced willingness to contribute directly to Drupal after all, why spend time writing a new feature you can’t use until you migrate to Drupal 8?

Part of why Drupal 8’s development took the time it did was to allow for a faster release cycle for features. Now, new features are added in point release (while code compatibility with prior Drupal 8 releases is broadly maintained).

Furthermore, security updates are only reliable for the latest minor release. After a minor release, such as Drupal 8.4, security issues in the prior minor release (such as Drupal 8.3) will only be patched for a few weeks, if at all.  In other words, to keep your sites secure, you have to be on the latest point release.

Upgrading your site every six months is a cost of using Drupal 8.

However, it’s not just Drupal that requires more frequent upgrades. Docker, NodeJS, and PHP itself all have more aggressive support schedules than similar software may have had in the past. Here lies the core of release planning: synchronizing all of these release schedules with your business.

1. Build a schedule of key dates for your business

In the publishing industry, tentpole events such as tournaments, live events, or holiday specials drive production deadlines. In addition to tentpole events, identify conferences, retreats, and other dates where your team may not be available. If they overlap with an EOL of software in your stack, you can try to schedule an upgrade ahead of time.

There are two reasons to identify these early and socialize them with the rest of the business. First, these are times when you don’t want to schedule maintenance or deployments if you can help it. Second, these key dates are times to be prepared and aware of security release windows. For example, Drupal core security releases are usually on the third Wednesday of the month. If this overlaps with a high-traffic event, having a plan to deploy and test a security patch before the patch is issued will ensure your site is kept secure, and that technical teams aren’t stressed out during the process.

2. Build a schedule of your stack and its support windows

Too often, we see organizations mandate a specific tool (like Drupal or Docker) without budgeting time and money for recurring, required upgrades. Maintenance often means “respond to bug reports from our team and handle outages,” instead of “incrementally upgrade software to keep it in supported releases.”

Before development begins, teams should gather a list of all key software used in their expected production stack. Then, go to each software release page, and find out the start and end dates of support for the current and next few releases. Use this to create a Gantt chart of support windows.

Here’s an example of one I created for a client that was deploying Drupal 8, PHP, Nginx, nodejs, Docker, and Ubuntu for their sites. In their case, they weren’t using Ubuntu packages for any of the other software, so they had to track each component directly.

undefined

Having such a chart makes prevents surprises. For example, if a site is using nodejs 6.x, we can see that it’s going to EOL just after Drupal 8.5 is released. It would make sense for a developer to work on upgrading to nodejs 8.x over the December holiday break so that the team isn’t juggling upgrading two core components at the same time.

This type of chart also gives teams the power to know if a given technology stack is maintainable for their organization. If it’s a small team maintaining a large Drupal site, they may not have the bandwidth or expertise to keep up with updates elsewhere in the stack. That may mean they have to stick with LTS releases, or avoid software without an LTS available.

For Drupal 8, that may mean that it is not a fit for your organization. If your budget only allows an initial build and then very minimal maintenance, Drupal 7 or another CMS may be a better fit (at least until Drupal 8 has an LTS). Or, for event and marketing sites, consider using a static site generator like Jekyll, so that when development ends the site can be “serialized” into static HTML and served with a basic stack using LTS software only.

3. Have a top-down process for handling security and feature updates

Many times we see software update requests coming from the development team, and then bubbling up to product owners and managers, meaning management will often prioritize new features with immediate business value before critical updates. Of course, this ignores the fact that a functioning and secure website is the foundation for addressing the needs of the business. And, it places developers in an awkward position where they are responsible for cleaning up any security breaches that occur.

Instead, software updates should be visible to and scheduled by the project managers, ensuring that time is available in the right sprint to test and deploy updates, preventing the development team from being overcommitted or opening your site up to a security breach. For example, let's say a team is using two-week sprints, with a sprint set to start on February 19, 2018. A quick look at the release cycle overview for Drupal 8 shows that 8.5.0-rc1 will be tagged on February 21st. Creating a ticket for that sprint ahead of time will ensure the upgrade won’t be forgotten.

It can be more difficult to handle software—like Drupal contributed modules or npm packages—that doesn't have a defined release schedule. In those cases, having a recurring “update everything” ticket can help. Assuming one team manages an entire deployment, a successful schedule might look like:

  • Sprint 1: Update all Drupal contributed modules
  • Sprint 2: Update all npm packages
  • Sprint 3: Update all system-level packages
  • Sprint 4: Update all Drupal contributed modules
  • … etc.

Remember, just because you schedule software updates doesn’t mean that you have to deploy them to production if QA goes poorly. But, if you do find issues, at least you can try and address them before, and not during a security patch.

4. Promote the new features you deploy to your colleagues and users

After doing all of this work, you should tell your end users and stakeholders about the gains you’ve made in the process. Show editorial users how Drupal 8.4’s Inline Form Errors gives them a better display of form errors on content they write. Tell your stakeholders how your API servers respond 15% faster to mobile clients. High-five your developers when they can remove a patch they’d been manually applying to the site.

For larger teams, newsletters and blogs are a great way to communicate these improvements. They are especially useful for core development teams, who write and maintain software shared throughout an enterprise.

With a solid grasp of how upstream release schedules affect your product, you will be able to reduce the need for unexpected delays and budget requests. If you’re interested in exploring this further with your team, we’d be glad to help.