Twitter löschte 1,2 Millionen Accounts mit terroristischen Inhalten

heise online Newsticker - 9. April 2018 - 9:30
Der Kurzmitteilungsdienst arbeitet weiter daran, terroristische Accounts offline zu nehmen.

Nach c't-Recherchen: Tracking-Smartwatch jetzt ohne Abhörfunktion

heise online Newsticker - 9. April 2018 - 9:30
Besitzer von Vidimensio-GPS-Trackern können aufatmen: Sie können über die Uhren nicht mehr von Fremden aus dem Internet belauscht werden. Wenige Tage nach Veröffentlichung einer investigativen c't-Recherche schaltete der Hersteller die Funktion ab.

Indien geht gegen Kryptowährungen vor – und denkt über eigene nach

heise online Newsticker - 9. April 2018 - 7:30
Ab sofort dürfen in Indien Finanzinstitute keine Geschäfte mit individuellen oder wirtschaftlichen Einrichtungen mehr machen, die mit virtuellen Währungen handeln.

WD erneuert PCIe-SSD aus der Black-Serie

heise online Newsticker - 9. April 2018 - 7:30
Die neue Version der PCIe-SSD WD Black soll bis zu 3,4 GByte/s beim Lesen und 2,8 GByte/s beim Schreiben erreichen – damit könnte sie Samsungs 960 Pro als Leistungsspitzenreiter ablösen.

Digitalisierung stockt – Intelligente Stromzähler kommen später

heise online Newsticker - 9. April 2018 - 7:00
Der Stromzähler soll intelligent werden und die Energiekosten für Verbraucher senken. Doch der Weg dahin ist lang, denn smarte Messgeräte müssen hohe Sicherheitsstandards erfüllen.

Jeff Geerling's Blog: Installing PHP 7 and Composer on Windows 10

Planet Drupal - 9. April 2018 - 5:23

I am working a lot on Composer-based Drupal projects lately (especially gearing up for DrupalCon Nashville and my joint workshop on Drupal and Composer with Matthew Grasmick), and have been trying to come up with the simplest solutions that work across macOS, Linux, and Windows. For macOS and Linux, getting PHP and Composer installed is fairly quick and easy. However, on Windows there seem to crop up little issues here and there.

Since I finally spent a little time getting the official version of PHP for native Windows installed, I figured I'd document the process here. Note that many parts of this process were learned from the concise article Install PHP7 and Composer on Windows 10 from the website KIZU 514.

Install PHP 7 on Windows 10

Love Huria: Cool things you can do with Sass - Part 1

Planet Drupal - 9. April 2018 - 2:00

I have been using Sass for like past two years and now I’m a huge fan. Even though we were doing pretty much alright with writing CSS but it never gave us that kind of flexibility that Sass provides like one of the things could be managing the complexity in stylesheets as our apps get more and more substantial. Anyways, Enough about my experience already as today we have got a bunch of cool things to cover!

What is Sass?

It’s a CSS preprocessor, that’s what you will get if you start googling and its true but hold that...

Wim Leers: API-First Drupal: file uploads — 572 comments summarized

Planet Drupal - 8. April 2018 - 23:11

This blog post summarizes the 572 comments spanning 5 years and 2 months to get REST file upload support in #1927648 committed. Many thanks to everyone who contributed!

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just *look at that commit message!*

Background: API-First Drupal: file uploads!.

  • Little happened between February 2013 (opening of issue) and November 2015 (shipping of Drupal 8).
  • Between February 2013 and April 2014, only half a dozen comments were posted, until moshe weitzman aptly said Still a gaping hole in our REST support. Come on Internets ….
  • The first proof-of-concept patch followed in August 2014 by juampynr, but was still very rough. A fair amount of iteration occurred that month, between juampynr and Arla. It used base64 encoding, which means it needed 33% more bytes on the wire to transfer a file than if it were transmitted in binary rather than base64.
  • Then again a period of silence. Remember that this was around the time when we were trying to get Drupal 8 to a shippable state: the #1 priority was to stabilize, fix critical bugs. Not to add missing features, no matter how important. To the best of my knowledge, the funding for those who originally worked on Drupal 8’s REST API had also dried up.
  • In May 2015, another flurry of activity occurred, this time fueled by marthinal. Comment #100 was posted. Note that all patches up until this point had zero validation logic! Which of course was a massive security risk. marthinal is the first to state that this is really necessary, and does a first iteration of that.
  • A few months of silence, and then again progress in September, around DrupalCon Barcelona 2015. dawehner remarked in a review on the lack of tests for the validation logic.
  • In February 2016 I pointed out that I’m missing integration tests that prove the patch actually works. To which Berdir responded that we’d first need to figure out how to deal with File entity type access control!
  • Meanwhile, marthinal works on the integration test coverage in 2016. And … we reached comment #200.
  • In May 2016, I did a deep review, and found many problems. Quick iterations fix those problems! But then damiankloip pointed out that despite the issue being about the general File (de)serialization problem, it actually only worked for the HAL normalization. We also ended up realizing that the issue so far was about stand-alone File entity creation, even though those entities cannot be viewed stand-alone nor can they be created stand-alone through the existing Drupal UI: they can only be created to be referenced from file fields. And consequently, we have no access control logic for this yet, nor is it clear how access control should work; nor is it how validation should work! Berdir explained this well in comment 232. This lead us to explore moving parts of https://www.drupal.org/project/file_entity into core (which would be a hard blocker). The issue then went quiet again.
  • In July 2016, garphy pointed out that large file uploads still were not yet supported. Some work around that happened. In September, kylebrowning stressed this again, and provided a more detailed rationale.
  • Then … silence. Until damiankloip posted comment #281 on April 3, 2017. Acquia was sponsoring him to work on this issue. Damian is the maintainer of the serialization.module component and therefore of course wanted to see this issue get fixed. My employer Acquia agreed with my proposal to sponsor Damian to work on REST file upload support. Because after 280 comments, some fundamental capabilities are still absent: this was such a complex issue, with so many concerns and needs to balance, that it was nigh impossible to finish it without dedicated time.
    To get this going, I asked Damian to look at the documentation for a bunch of well-known sites to observe how they handle file uploads. I also asked him to read the entire issue. Combined, this should give him a good mental map of how to approach this.
  • #281 was a PoC patch that only barely worked but did support binary (non-base64) uploads. damiankloip articulated the essential things yet to be figured out: validation and access checking. Berdir chimes in with his perspective on that in #291 … in which he basically outlines what ended up in core! Besides Berdir, dagmar, dawehner, garphy, dabito, ibustos all chimed in and influenced the patch. Berdir, damiankloip and I had a meeting about how to deal with validation, and I disagreed with with both of them. And turned out to be very wrong! More feedback is provided by the now familiar names, and the intense progress/activity continues for two months, until comment #376!
  • Damian got stuck on test coverage — and since I’d written most of the REST test coverage in the preceding months, it made sense for me to pick up the baton from Damian. So I did that in July 2017, just making trivial changes that were hard to figure out. Damian then continued again, expanding test coverage and finding a core bug in the process! And so comment #400 was reached!
  • At the beginning of August, the patch was looking pretty good, so I did an architectural review. For the first time, we realized that we first needed to fix the normalization of File entities before this could land. And many more edge cases need to be tested for us to be confident that there were no security vulnerabilities. blainelang did manual testing and posted super helpful feedback based on his experience. Blaine and Damian tag-teamed for a good while, then graphy chimed in again, and we entered September. Then dawehner chimed in once more, followed by tedbow.
  • On September 6 2017, in comment #452 I marked the issue postponed on two other issues, stating that it otherwise looked tantalizingly close to RTBC. aheimlich found a problem nobody else had spotted yet, which Damian fixed.
  • Silence while the other issues get fixed … and December 21 2017 (comment #476), it finally was unblocked! Lots of detailed reviews by tedbow, gabesullice, Berdir and myself followed, as well as rerolls to address them, until I finally RTBC‘d it … in comment #502 on February 1 2018.
  • Due to the pending Drupal 8.5 release, the issue mostly sat waiting in RTBC for about two months … and then got committed on April 3 2018!!!

Damian’s first comment (preceded by many hours of research) was on April 3, 2017. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

  • API
  • Acquia
  • Drupal

Wim Leers: API-First Drupal: file uploads!

Planet Drupal - 8. April 2018 - 23:09

Drupal 8’s REST API has been maturing steadily since the Drupal 8.0.0 was released in November 2015. One of the big missing features has been file upload support. As of April 3 2018, Drupal 8.6 will support it, when it ships in September 2018! See the change record for the practical consequences: https://www.drupal.org/node/2941420.

It doesn’t make sense for me to repeat what is already written in that change record: that already has both a tl;dr and a practical example.

What I’m going to do instead, is give you a high-level overview of what it took to get to this point: why it took so long, which considerations went into it, why this particular approach was chosen. You could read the entire issue (#1927648), but … it’s one of the longest issues in Drupal history, at 572 comments1. You would probably need at least an entire workday to read it all! It’s also one of the longest commit messages ever, thanks to the many, many people who shaped it over the years:

Issue #1927648 by damiankloip, Wim Leers, marthinal, tedbow, Arla, alexpott, juampynr, garphy, bc, ibustos, eiriksm, larowlan, dawehner, gcardinal, vivekvpandya, kylebrowning, Sam152, neclimdul, pnagornyak, drnikki, gaurav.goyal, queenvictoria, kim.pepper, Berdir, clemens.tolboom, blainelang, moshe weitzman, linclark, webchick, Dave Reid, dabito, skyredwang, klausi, dagmar, gabesullice, pwolanin, amateescu, slashrsm, andypost, catch, aheimlich: Allow creation of file entities from binary data via REST requests

Thanks to all of you in that commit message!

I hope it can serve as a reference not just for people interested in Drupal, but also for people outside the Drupal community: there is no One Best Practice Way to handle file uploads for RESTful APIs. There is a surprising spectrum of approaches2. Some even avoid this problem space even entirely, by only allowing to “upload” files by sending a publicly accessible URL to the file. Read on if you’re interested. Otherwise, go and give it a try!

Design rationale

General:

  • Request with Content-Type: application/octet-stream aka “raw binary” as its body, because base64-encoded means 33% more bytes, implying both slower uploads and more memory consumption. Uploading videos (often hundreds of megabytes or even gigabytes) is not really feasible with base64 encoding.
  • Request header Content-Disposition: file; filename="cat.jpg" to name the uploaded file. See the Mozilla docs. This also implies you can only upload one file per request. But of course, a client can issue multiple file upload requests in parallel, to achieve concurrent/batch uploading.
  • The two points above mean we reuse as much as possible from existing HTTP infrastructure.
  • Of course it does not make sense to have a Content-Type: application/octet-stream as the response. Usually, the response is of the same MIME type as the request. File uploads are the sensible exception.
  • This is meant for the raw file upload only; any metadata (for example: source or licensing) cannot be associated in this request: all you can provide is the name and the data for the file. To associate metadata, a second request to “upgrade” the raw file into something richer would be necessary. The performance benefit mentioned above more than makes up for the RTT of a second request in almost all cases.

PHP-specific:

  • php://input because otherwise limited by the PHP memory limit.

Drupal-specific:

  • In the case of Drupal, we know that it always represents files as File entities. They don’t contain metadata (fields), at least not with just Drupal core; it’s the file fields (@FieldType=file or @FieldType=image) that contain the metadata (because the same image may need different captions depending on its use, for example).
  • When a file is uploaded for a field on a bundle on an entity type, a File entity is created with status=false. The response contains the serialized File entity.
  • You then need a second request to make the referencing entity “use” the File entity, which will cause the File entity to get status=true.
  • Validation: Drupal core only has the infrastructure in place to use files in the context of an entity type/bundle’s file field (or derivatives thereof, such as image fields). This is why files can only be uploaded by specifying an entity type ID, bundle ID and field name: that’s the level where we have settings and validation logic in place. While not ideal, it’s pragmatic: first allowing generic file uploads would be a big undertaking and somewhat of a security nightmare.
  • Access control is similar: you need create access for the referencing entity type and field edit access for the file field.
Result

If we combine all these choices, then we end up with a new file_upload @RestResource plugin, which enables clients to upload a file:

  1. by POSTing the file’s contents
  2. to the path /file/upload/{entity_type_id}/{bundle}/{field_name}, which means that we’re uploading a file to be used by the file field of the specified entity type+bundle, and the settings/constraints of that field will be respected.
  3. … don’t forget to include a ?_format URL query argument, this determines what format the response will be in
  4. sending file data as a application/octet-stream binary data stream, that means with a Content-Type: application/octet-stream request header. (This allows uploads of an arbitrary size, including uploads larger than the PHP memory limit.)
  5. and finally, naming the file using the Content-Disposition: file; filename="filename.jpg" header
  6. the five preceding steps result in a successfully uploaded file with status=false — all that remains is to perform a second request to actually start using the file in the referencing entity!
Four years in the making — summarizing 572 comments

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

Want to actually read a summary of those 572 comments? I got you covered!

  1. It currently is the fifth longest Drupal core issue of all time! The first page, with ~300 comments, is >1 MB of HTML. ↩︎

  2. Examples: Contentful, Twitter, Dropbox and others↩︎

  • API
  • Acquia
  • Drupal

Another Drop in the Drupal Sea: Migrating from Drupal 6 to Drupal 8

Planet Drupal - 8. April 2018 - 18:31
Migrating from Drupal 6 to Drupal 8 Marc Sun, 04/08/2018 - 11:31am

Well, I've finally done it! I migrated this blog from Drupal 6 to Drupal 8. I did a test run yesterday with a personal blog of mine and found the process was relatively easy. Both sites are relatively simple.

There are other blog posts about the process as well as documentation on Drupal.org so I won't repeat lots of details.

I'm running Drupal 8.5.1 as of this blog post. I chose to use all of the various migration modules that come with Drupal 8 core, including the two marked as experimental. Once I had them enabled I clicked the link to get to the upgrade form in the UI. One of the sites did have file uploads and all of them were pulled in seamlessly. I had created a backup of the sites/[site-name] directory and placed it in my new Drupal 8 sites directory.

Here are issues I encountered:

  • Administration Menu is (apparently) not properly ported to Drupal 8 and it blew things up on the site yesterday. I had to manually clean things up in the database so that module was not enabled in my Drupal 8 site and clear cache.
  • The taxonomy term reference to the Tags vocabulary needed the field setting updated so that it selected the Tags vocabulary.
  • When I click the user in the Toolbar it does not switch the tray so that it shows View profile, Edit profile and Log out. (That's still an issue as I write this. I haven't investigated it enough to figure out what's going wrong, nor have I filed an issue.)
  • The feed that taxonomy provides for terms has changed slightly. I filed an issue to get Planet Drupal to use the new feed.
  • No views were imported.
  • The pathauto patterns weren't imported.
  • Disqus doesn't handle the migration.
  • Other than those things, I can't say I ran into much of anything else. And, aside from the site blowing up from Administration Menu, there wasn't much that presented a real challenge.

If you are reading this post on Planet Drupal, then you know I'm back up and running!

I'll still need to theme this site again. And I'll need to replace some functionality that was previously provided by Advanced Blog. I haven't yet installed and tested out the Drupal 8 port of Tagadelic.

One more note: I decided to just delete the comments that I had for the Drupal 6 version of this site since I don't want to use the Comment module, preferring to use Disqus instead.

Comments

Projekt in Gera kombiniert eSport und Berufsausbildung

heise online Newsticker - 8. April 2018 - 17:30
Einen neuen Weg in der Professionalisierung von Computersportlern schlägt ein Unternehmen in Gera ein: Gleichberechtigt sollen die eSportler trainieren und sich in einem Beruf ausbilden lassen.

RubyMine 2018.1 erhält neue Core Engine für die statische Analyse

heise online Newsticker - 8. April 2018 - 16:30
Die Entwicklungsumgebung für Ruby erlaubt erstmals auch das partielle Weiterleiten von Code-Abschnitten als Git Commit. Durch Verknüpfung mit einer Changelist lassen diese sich außerdem tracken.

Facebook beschränkt Datenzugriff – und macht Tinder kaputt

heise online Newsticker - 8. April 2018 - 16:00
Facebook hat seine Schnittstellen weiter beschnitten, damit externe Apps weniger Informationen abgreifen können. Davon betroffen ist auch Tinder: Die populäre Dating-App war zeitweise nicht mehr zu gebrauchen.

Keine Killer-Roboter: KI- und Robotik-Forscher drohen südkoreanischer Uni mit Boykott

heise online Newsticker - 8. April 2018 - 16:00
Berichten nach soll an der südkoreanischen Hochschule KAIST künftig auch Militär-Roboter mit künstlicher Intelligenz entwickelt werden. Führende KI- und Robotik-Wissenschaftler drohen mit einem Boykott. Die Universität verneint solche Pläne.

MDM-Plattform Jamf Pro hat Probleme mit iOS 11.3

heise online Newsticker - 8. April 2018 - 14:30
Wer versucht, mit dem populären Adminwerkzeug iPhone und iPad auf die neueste Betriebssystemversion zu aktualisieren, kann diese nachher nicht mehr steuern. Ein Fix ist im Anmarsch.

iPhone-Drosselung: Klagen gegen Apple werden zusammengeführt

heise online Newsticker - 8. April 2018 - 14:30
Angeschlagene Akkus sorgten bei Apple-Smartphones zu einer Leistungsreduktion – was dem Konzern über 60 Rechtsstreitigkeiten allein in den USA einbrachte. Diese werden nun in Kalifornien kombiniert.

Facebook-Datenskandal: EU-Kommission will mit Facebook sprechen

heise online Newsticker - 8. April 2018 - 13:00
Die EU-Kommission hat für die kommenden Tage Gespräche mit dem US-Konzern angekündigt. Die Bundesjustizministerin hat indessen Facebook scharf gerügt.

l+f: Piep, piep, root

heise online Newsticker - 8. April 2018 - 12:30
Wenn der Lautsprecher des Linux-Servers piept, hat sich vielleicht gerade jemand unerlaubt Root-Rechte verschafft.

Apple: App-Store-Bestand schrumpft erstmals

heise online Newsticker - 8. April 2018 - 11:30
iOS verlor 2017 laut einer Analyse rund fünf Prozent seiner Apps. Grund dürfte auch Apples Vorgehen gegen Spammer und Scammer sein – und 32-Bit-Apps.

Daimler-Chef: Mehr Elektroautos nicht so gut für die Bilanz

heise online Newsticker - 8. April 2018 - 10:00
Der Wandel in der Autoindustrie kostet Zeit und vor allem sehr viel Geld. Daimler-Chef Zetsche warnt die Aktionäre schon mal vorsichtig, was das nach den Bestwerten von 2017 für die kommenden Jahre bedeuten könnte.