Latest Plone Posts
Two Jazkarta Projects Competing for Digital Humanities Award
By Sally Kleinfeldt from Planet Plone. Published on Feb 23, 2017.The annual Digital Humanities Awards allow the public to recognize outstanding digital humanities resources on the web. This year not one but two Jazkarta projects are nominated in the “Best Use of Digital Humanities for Public Engagement” category! Pleiades, a gazetteer of ancient places that recently won the Archaeological Institute of America’s 2017 Award for […]
Towards Plone 6
From Planet Plone. Published on Feb 13, 2017.A report from the Alpine City Sprint in Innsbruck
Push coverage report out of an Gitlab CI Runner
From Planet Plone. Published on Feb 06, 2017.For our Plone/Python projects we often generate coverage reports as HTML sites, this posts show how you can push this report out of the Gitlab CI Runner.
No more JS / CSS browser cache refresh pain
Pleiades Wins an Award and Gets an Upgrade
By Sally Kleinfeldt from Planet Plone. Published on Jan 30, 2017.Here at Jazkarta we’ve been working with the Institute for the Study of the Ancient World (ISAW) for the past year on a project funded by the National Endowment for the Humanities to upgrade and improve Pleiades, a gazetteer of ancient places that is free and open to the public. Thus it was very gratifying […]
Simple loop parallelization in Python
By Mikko Ohtamaa from Planet Plone. Published on Jan 30, 2017.Sometimes you are programming a loop to run over tasks that could be easily parallelized. Usual suspects include loads that wait IO like calls to third party API services. Since Python 3.2, there have been easy tool for this kind … Continue reading
Seven Years: A Very Personal History of the Web
By Martijn Faassen from Planet Plone. Published on Jan 25, 2017.
Humans are storytellers. As anyone who knows me can confirm, I definitely enjoy the activity of telling stories. We process and communicate by telling stories. So let me tell you all a story about my life as a web developer the last 7 years. It may just be self-reflection, but hopefully it's also useful for some others. Perhaps we can see my story as a highly personal history of the web.
I always say that what pulls me to software development most is creativity. I am creative; I can't help myself. I enjoy thinking about creativity. So this is also going to be about my creative trajectory over the last 7 years.
Why 7 years? Because in early 2010 I decided to withdraw myself from a software development community I had been involved in for about 12 years previously, and I took new paths after that. It is now early 2017; time to take stock.
Letting go of an intense involvement with a software development project, up to the point where it became part of my identity, was difficult. Perhaps that's a geeky thing to say, but so it is. I needed to process it.
Life after Zope
Zope was a web framework before that concept existed. The modern web framework only materialized sometime in the year 2005, but Zope had been ahead of its time. I was involved with the Zope community from when it was first released, in 1998. I learned a lot from Zope and its community. I made many valued connections with people that last until this day.
Zope helped shape who I am and how I think, especially as a web developer. In 2013 I wrote a retrospective that went into the history of Zope and my involvement with it.
But I did not just process it by writing blog posts. I also processed it creatively.
So I am a web developer. Back in 2010 I saw some people argue that the time of the web framework had passed. Instead, developers should just gather together a collection of building blocks and hook it up to a server using a standard API (WSGI in the case of Python). This would provide more flexibility than a framework ever could.
In a "X considered Y" style post I argued that web frameworks should be considered useful. Not that many people needed convincing, but hey, why not?
The burden of assembling and integrating best of breed components can be shared: that's what the developers of a framework do. And if you base your work on a pre-assembled framework, it's likely to be less work for you to upgrade, because the framework developers will have taken care that the components in the framework work together in newer versions. There is also a larger chance that people will write extensions and documentation for this same assembly, and that is very likely something you may benefit from in the future.
I follow the ReactJS community. The React community definitely is a community that lets you self-assemble a framework out of many parts. This gives that ecosystem flexibility and encourages creativity -- new approaches can be tried and adopted quickly. I like it a lot.
But figuring out how to actually start a React-based project had become a major effort. To get a good development platform, you needed to learn not only about React but also about a host of packaging and build tools: CommonJS and Webpack and npm and Babel and so on. That's quite intimidating and plain work.
So some React developers realized this and created create-react-app which makes it easy to start a working example, with minimal boilerplate code, and with suggestions on how to expand from there. It's a build framework for React that gathers good software in one place and makes it easy to use. It demonstrates how frameworks can make life easier for developers. It even goes a step further and allows you to opt out of the framework once you need more control. Now that's an interesting idea!
Client-side JS as Servant to the Server UI
jQuery was first released in 2006. jQuery provided better APIs over sucky browser APIs, and hid incompatibilities between them. Its core concept is the selector: you select things in the web page so you can implement your dynamic behavior. Selectors fit the server-dominant paradigm very well.
Client-side JS as Master of the UI
By 2010, the notion of single-page web application (SPA) was in the air. SPAs promised more powerful and responsive UIs than server-side development could accomplish. The backend is a HTTP API.
So there we finally get to my idea: to create a client side web framework, by bringing over concepts from server frameworks to the client, and to see what happens to them. Cool things happen! We started with templates and then moved to MVC. We created a notion of components you could compose together. We created a client-side form library based on that. In 2011 we released this as Obviel.
For a little while in 2010, early 2011, I thought I was the only one with this cool idea of a client-side web framework. It turns out that I was not: it was a good idea, and so many people had the same idea at about the same time. Even before we released Obviel, I started to hear about Backbone. Ember and Angular soon followed.
In 2011 and 2012 we built a lot of stuff with Obviel. In the beginning of 2013 those projects were done. Obviel didn't get any traction in the wider community. It was a project I learned a lot from, so I don't regret it. I can claim deep experience in the area of client-side development.
I went to my first JS conference in September of 2013. I had originally submitted a talk about Obviel to it, but it wasn't accepted. Everybody was promoting their shiny new client-side framework by that time.
So was Facebook. Pete Hunt gave a talk about React. This was in fact only the second time React had been introduced to a wider audience. Apparently it went over a lot better than the first time. It certainly made an impression on me: there were some fascinating new ideas in it. The React community became ferment with fascinating new ideas. At the conference I talked to people about another idea I'd had: a client framework that helps coordinate client/server communication; maybe sort of like a database, with transactions that commit UI state to the server? Nobody seemed to know of any at the time. Uh oh. If nobody has the idea at the same time, then it might be a bad one?
Then from the React community came Flux and Redux and Relay and Mobx. I let go of Obviel and started to use React. There is a little irony there: my move client-side frameworks had started with templates, but React actually let go of them.
The Server in Modern Client-side times
I wrote a list of what tasks remain for the server framework:
What remains is still significant, however:
- serving up JSON on URLs with hyperlinks to other URLs
- processing POSTs of JSON content (this may include parts of form validation)
- traversal or routing to content
- integrating with a database (SQLAlchemy in this case)
- authentication - who are you?
- authorization - who can access this URL?
- serve up an initial web page pulling in a whole bunch of static resources (JS, CSS)
I also wrote:
Much of what was useful to a server side web framework is still useful. The main thing that changes is that what goes over the wire from server to client isn't rendered HTML anymore. This is a major change that affects everything, but much does stay the same nonetheless.
I didn't know at the time of writing that I would be working on just such a server web framework very soon.
On the Morepath
In 2013 I put some smaller pieces I had been playing with for a while together and created Morepath, a server Python web framework. I gave an over-long keynote at PyCON DE that year to describe the creative processes that had gone behind it. I gave a more focused talk at EuroPython 2014 that I think works better as an introduction.
For a while now I've been working on Morepath. I thought I'd say a bit about it here.
Morepath is a Python web micro-framework with super powers. It looks much like your average Python micro-framework, but it packs some seriously power beneath the hood.
One of the surprises of Morepath was the discovery that a web framework that tries to be good at being a REST web server actually works very well as a server web framework as well. That does make sense in retrospect: Morepath is good at letting you build REST services, therefore it needs to be good at HTTP, and any HTTP application benefits from that, no matter whether they render their UI on the client or the server. Still, it was only in early 2015 that Morepath gained official support for server templates.
2014 was full with Morepath development. I announced it at EuroPython. It slowed down a little in 2015, then picked up speed again in 2016.
I'm proud that Morepath is micro in implementation, small in its learning surface, but macro in power. The size of Morepath is another surprise: Morepath itself is currently a little over 2000 lines of Python code, but it does a lot, helped by the powerful Reg (<400 lines) and Dectate (<750 lines) libraries. Morepath offers composable, overridable, extensible applications, an extensible configuration system, an extensible view dispatch system, automated link generation, built-in powerful permission rule system, and lots more. Morepath is like Flask, but with a nuclear fusion generator inside. Seriously. Take a look.
Over the last few years Morepath has become a true open source project; we have a small team of core contributors now. And in late 2016 Morepath started to gain a bit more attention in the wider world. I hope that continues. Users that turn into contributors are invaluable for an open source project.
The Django management UI is cool. It makes me want to implement the equivalent for Morepath with PonyORM and React and Mobx. Or something. Want to help?
I've been itching to do something significant on the client-side again. It's been a little while since I got to do React. I enjoyed attending React Europe 2015 and React Europe 2016. I played with React Native for a bit last year. I want to work with that stuff again.
The space where client and server interacts is fertile with creative potential. That's what I've found with Obviel and React on the client, and with Morepath on the server side. While GraphQL replaces the REST paradigm that Morepath is based around (oops!), I'd enjoy working with it too.
Where might the web be going? I like to think that by being creative I sometimes get to briefly peek into its near future. I hope I can continue to be creative for the next 7 years, as I really enjoy it.
I'm a freelancer, so the clients I work for in part shape my creative future. Hint. Let me know if you have something interesting for me to work on.
Fix failing parallel running browser tests with Gitlab & Robotframework
From Planet Plone. Published on Jan 25, 2017.One of our project is organized in sprints where all developer work on the same code at the same time. We use one Gitlab CI server with a simple shell executer and had random failing builds with Robotframwork & Firefox.
Python Meeting Düsseldorf - 2017-01-18
From Planet Plone. Published on Jan 16, 2017.
The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.
Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:
Bereits angemeldete Vorträge
"Kurze Einführung in openpyxl und Pandas"
"Optimierung in Python mit PuLP"
Startzeit und Ort
Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.
Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet
sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.
Über dem Eingang steht ein großes "Schwimm’ in Bilk" Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.
>>> Eingang in Google Street View
Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.
Das Python Meeting Düsseldorf nutzt eine Mischung aus (Lightning) Talks und offener Diskussion.Vorträge können vorher angemeldet werden, oder auch spontan während des Treffens eingebracht werden. Ein Beamer mit XGA Auflösung steht zur Verfügung.
(Lightning) Talk Anmeldung bitte formlos per EMail an email@example.com
Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.
Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.
Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.
Da wir nur für ca. 20 Personen Sitzplätze haben, möchten wir bitten,
sich per EMail anzumelden. Damit wird keine Verpflichtung eingegangen.
Es erleichtert uns allerdings die Planung.
Meeting Anmeldung bitte formlos per EMail an firstname.lastname@example.org
Weitere Informationen finden Sie auf der Webseite des Meetings:
Viel Spaß !
Marc-Andre Lemburg, eGenix.com
How to monitor your Plone servers with Sentry
From Planet Plone. Published on Dec 30, 2016.use the free Sentry error tracking service to stay on top of production errors
Quills blogging add-on for Plone gets some attention
From Planet Plone. Published on Dec 27, 2016.release 1.8.1
How to enable online reading of Taylor & Francis journals from your Plone site
From Planet Plone. Published on Dec 27, 2016.a Plone external method to allow member-only access to reading an online journal
Plone performance: threads or instances?
By hvelarde (email@example.com) from Planet Plone. Published on Dec 23, 2016.Recently we had a discussion on the Plone community forum on how to increase performance of Plone-based sites.
I was arguing in favor of instances, because some time ago I read a blog post by davisagli taking about the impact of the Python GIL on performance of multicore servers. Others, like jensens and djay, were skeptical on this argument and told me not to overestimate that.
So, I decide to test this using the production servers of one of our customers.
The site is currently running on 2 different DigitalOcean servers with 4 processors and 8GB RAM; we are using Cloudflare in front of it, and round-robin, DNS-based load balancing.
Prior to my changes, both servers were running with the same configuration:
- nginx, doing caching of static files and proxy rewrites
- Varnish, doing caching and URL-based load balancing
- 4 Plone instances running on ZEO client mode, with 1 thread and 100.000 objects in cache
Both servers where running also a ZEO server on ZRS configuration, one as a master and the other as a slave, doing blob storage replication.
First, here we have some information from this morning, before I made the changes. Here are some graphics I obtained using New Relic on the master:
Here is the same information from the slave:
As you can see, everything is running smoothly: CPU consumption is low and memory consumption is high and… yes, we have an issue with some PhamtomJS processes left behind.
This is what I did later:
- on the master server, I restarted the 4 instances
- on the slave server, I changed the configuration of instance1 to use 4 threads and restarted it; I stopped the other 3 instances
The servers have been running for a couple of hours now and I can share the results. This is the master server:
And this is now the slave:
The only obvious change here is in memory consumption: wow! the sole instance on the slave server is consuming 1GB less than the 4 instances in the master server!
Let's do a little bit more research now. Here we have some information on database activity on the master server (just one instance for the sake of simplicity):
Now here is some similar information for the slave server:
I can say that I was expecting this: there's a lot more activity and the caching is not very well utilized on the slave server (see, smcmahon, that's the beauty of the URL-based load balancing on Varnish demonstrated).
Let's try a different look, now using the vmstat command:
Not many differences here: the CPU is idle most of the time and the interrupts and context switching are almost the same.
Now let's see how much our instances are being used, with the varnishstat command:
Here you can see why there's not too much difference: in fact Varnish is taking care of nearly 90% of the requests and we have only around 3 requests/second hitting the servers.
Let's make another test to see how quickly we are responding the requests using the varnishhist command:
Again, there's almost no difference here.
Conclusion: for our particular case, using threads seems not to affect too much the performance and has a positive impact on memory consumption.
What I'm going to do now is to change the configuration used in production to have 2 instances and 2 threads… why? because restarting a single instance on a server for maintenance purposes would let us without backends during the process if we were using only one instance.
Share and enjoy!
Plumi now on Debian Jessie, Ubuntu 16.04 and Centos 7
By anna from Planet Plone. Published on Dec 21, 2016.We are very excited to announce that after much effort, Plumi is now available to install on Debian Jessie, Ubuntu 16.04 (latest stable) and Centos 7. The latest code is available here on Github: https://github.com/plumi/plumi.app Documentation on how to install is available here: https://github.com/plumi/plumi.app/blob/master/README.rst Further documentation including an introduction, installation, theming and maintenance guide has […]
Ten years of Plumi and looking ahead to 2017
By anna from Planet Plone. Published on Dec 21, 2016.It’s been ten years now since we released our first public version of Plumi with a vision to provide free democratic access to video distribution, and I’m very proud of EngageMedia’s work to sustain the project through many successes and challenges. I’d like to thank the whole team at EngageMedia, and all our visionaries, programmers, […]